Hennadii Kutomanov PRO

Thank you, you've picked out the main point. Each request has its own difficulty, because it has its own material, and that difficulty links all of its replacements. That is why I use a logistic GLMM with a random intercept for each request: every replacement is a Bernoulli trial, and all replacements of one request share that intercept. This separates the effect of the load from the difficulty of the particular request. The variance σ² of the random intercept is reported in the passport as a model parameter in its own right. It also enters the formula for instruction capacity: IEC_q is the number of rules at which the probability of a correct replacement falls to the level q, and on the logit scale this level corresponds to the threshold η*(q, σ²) ≈ ln(q / (1 − q)) · √(1 + 3σ²/π²).

Hi Dipankar!

Thank you for the interesting distinction between input-side and output-side interference. By the time your message arrived, I had already completed my investigation of the vector similarity hypothesis, so I can now share the results. Most of them concern the similarity between background words and rule triggers, and I come back to the output side near the end.

I added a vector similarity term, E(e), to the Bernoulli GLMM for background words, using the same 4,937,679 GPT-4o mini trials, the same run weights, and the same bins of the orthographic (spelling) measures d and c as in the previous analysis.

Here, e is the minimum cosine distance between a background word and the rule triggers in its run, divided into ten bins. Since GPT-4o mini’s internal vector representations are not accessible, I used embeddings from text-embedding-3-large as an external proxy. All uncertainties reported below are half-widths of 95% confidence intervals.

Vector similarity does have a statistically significant effect on misfires. The misfire rate is 4.33% for the closest 1% of trials, compared with just 0.17% for the most distant 9%. After controlling for orthographic similarity, the odds of a misfire in the closest bin remain 4.3 times those in the most distant bin (95% CI: 2.7–6.7). Across most of the remaining range, however, the relationship is relatively flat.

The effect of vector similarity turned out to be considerably weaker than that of orthographic similarity.

Relative to a model without similarity terms, adding E(e) alone increases the weighted log-likelihood by 1,771, whereas adding the orthographic terms D(d) and C(c) increases it by 10,097.

When the terms are included together, E(e) contributes an additional 772 beyond orthographic similarity, while the orthographic terms contribute an additional 9,098 beyond E(e).

Adding E(e) barely changes the estimates of the orthographic terms: the differences are below 0.1 logits, except for the sparsely populated c ≥ 5 bin.

The Wald χ² statistics minus their degrees of freedom are 311 for C(c), 259 for D(d), and 64 for E(e). All three terms are statistically significant (p < 10⁻¹¹). By these measures, both orthographic terms contribute more than the external vector similarity measure used here. D(d) and C(c) cannot be separated from each other, since the 95% interval of their difference includes zero. These statistics reflect the strength of evidence rather than effect size.

Thus, my hypothesis that vector similarity might be a stronger predictor of misfires was not supported, at least with the chosen representation of words.

Nevertheless, these measures capture distinct properties of words. Across all 2.46 million word pairs in the vocabulary, the absolute correlations among d, c, and e do not exceed 0.27.

At the individual-trial level, the correlations reach 0.47, consistent with the distributions of all three measures shifting as the number of rules increases. However, within each doubling range of the rule count, the correlations remain below 0.3.

Vector and orthographic similarity therefore provide distinct information, even though their effects on errors partly overlap.

As for the main IEC passport, these results support the previous conclusion. The only substantial change concerns the dependence on the number of rules.

The slope per doubling of the rule count is:

• −0.48 ± 0.07 without similarity terms;
• −0.37 ± 0.07 with the vector term alone;
• −0.17 ± 0.08 with the orthographic terms;
• −0.14 ± 0.08 with both types of similarity included.

The run-level random-effect variance changes from 2.27 ± 0.33 to 2.14 ± 0.31. For the other load dependencies, the largest difference among reliably estimated bins is 0.20 logits.

I am therefore keeping the main IEC passport free of similarity terms, so that its characteristics continue to describe model behavior under the actual distribution of words in the experimental tasks.

Of course, using external embeddings does not tell us how similar the results would be with GPT-4o mini’s own internal representations. Nor does it establish at which stage of processing an incorrect substitution occurs.

Your observation about possible competition on the output side adds another dimension to the problem, so I also checked your question by adding the edit distance to the nearest replacement word. It does pick up the far misfires: among words with no trigger nearby, the misfire rate rises from 0.16% at 4 or more edits to 2.86% at one edit. In the full data such misfires make up 19% at m ≥ 2, against 34.5% in your subset. Across all trials, this term adds more on top of spelling than the vector term does (2,974 against 772), and the slope per doubling of the rule count falls from −0.17 ± 0.08 to −0.04 ± 0.08. The passport stays unadjusted for the reason you gave: such a curve describes a word the task almost never contains. Comparing these distances with the internal embeddings of an open-weight model such as gpt-oss could help distinguish between the possible sources of interference.

It is becoming increasingly clear that lexical interference is a research problem in its own right. One particularly interesting question is whether it is specific to rule execution or also appears in other language-model operations.

For example, it could be investigated using an entirely different experimental design: false recognition of absent words that resemble items in a previously presented list, or incorrect attribution of properties to objects with similar names. Such experiments could help determine which of the observed patterns are specific to word substitution and which reflect more general properties of lexical processing.

I see potential for a separate research project here, but for now I need to keep my own work focused on model degradation under load, which remains the central objective of IEC. I therefore plan to present the orthographic and vector interference findings separately, rather than combining them with the load-related characteristics of the model passport.

Thank you for the thoughtful discussion and for the ideas that have helped me develop a much deeper understanding of these errors.

Hi Dipankar!

Thank you for the interesting suggestion. I've considered the possibility of transferring results between vocabularies, but in the case of IEC, its practical value seems rather limited.

For each model provider, I use a single vocabulary constructed for its tokenizer. It consists of monosyllabic nouns at least three letters long, each represented by a single token. Creating a second independent vocabulary of comparable size under the same constraints is practically impossible. And when switching to a model from another provider, not only the vocabulary and tokenizer change, but the model itself does too, so its characteristics must be measured independently anyway.

That said, your findings have revealed an interesting property of the model that deserves separate consideration.

In the Bernoulli GLMM, the spelling-related components D(d) and C(c) have the same mathematical structure as the load components, but they describe a different phenomenon. The dependencies on text length, position in the response, replacement density, and number of rules characterize degradation in instruction execution under load. The spelling-related components characterize the model's sensitivity to orthographic similarity between words.

I'm therefore inclined to retain the load dependencies in the main IEC passport and publish the spelling-related components separately, as additional model characteristics. They may be of independent interest, but they address a different research question.

This also brings up another possibility.

So far, we've examined orthographic similarity, although language models process tokens through vector representations. Similarity between these representations might be an even stronger predictor of misfires on background words than spelling similarity.

Comparing orthographic and vector similarity could be particularly informative. It might help us understand not only the conditions under which errors occur, but also the mechanisms behind them. The structure of incorrect substitutions could potentially reveal a great deal about the stages of model processing at which these failures arise.

What do you think of this hypothesis?

Hi Dipankar!

Thank you for your recommendation on the spelling similarity measure and for your observation about the run-level random-effect variance. You wanted to see what the GLMM would reveal, and I now have the results. In brief, the spelling similarity effect is clearly present in the data, but I have decided not to include the corresponding terms in the model passport. Here is why.

I examined GPT-4o mini’s behavior and degradation on background words using the full experimental dataset, including responses with disrupted line structure. For this analysis, I used a new line- and word-alignment algorithm. The resulting dataset contains 4,937,679 background words whose corresponding positions in the responses could be established, with 32,587 misfires, giving an overall rate of 0.66%.

I fitted two Bernoulli GLMMs to the same trials, using identical run weights.

Without spelling similarity terms:

logit P(no misfire) = V(v) + Q(s/v) + P(p) + M(m) + b

With spelling similarity terms:

logit P(no misfire) = V(v) + Q(s/v) + P(p) + M(m) + D(d) + C(c) + b

Here, v is the text length, s/v is the fraction of the answer generated so far, p = m/D is the replacement density, m is the number of rules, and b ~ N(0, σ²) is the run-level random effect. The variable d denotes the minimum Levenshtein distance between the word and the rule triggers in that run, while c is the maximum common-prefix length between the word and those triggers.

Each term is represented by a set of free coefficients defined over discrete bins. All uncertainties reported below are half-widths of 95% confidence intervals.

To answer your question, I expanded the distance binning rather than optimizing the cut points during fitting. The levels for d are 1, 2, 3, 4, 5, and 6+, while those for c are 0, 1, 2, 3, 4, and 5+. Target words are modeled separately, allowing each word type to have its own spelling similarity terms. This also addresses your earlier question.

The spelling similarity effect is clearly identifiable and follows a regular pattern: the logit changes by approximately one unit per doubling of distance and by about 0.7 units per additional shared prefix character. Your observation is confirmed.

However, including these terms provides little benefit to the model passport, for three reasons.

First, the run-level random-effect variance, which you suggested comparing, changes very little: σ² = 2.27 ± 0.33 without the spelling terms, versus 2.17 ± 0.31 with them.

Second, the dependencies on text length, fraction of the answer generated, and replacement density remain virtually unchanged. The largest difference among bins with reliably estimated coefficients is 0.17 logits.

Third, the only substantial change occurs in the dependence on the number of rules, and here the spelling adjustment conflicts with the intended interpretation of the passport.

Without the adjustment, the logit decreases by approximately 3 units at 114 rules. With the adjustment, the decrease is reduced to about 1 logit, and the curve becomes nearly flat beyond approximately 15 rules. The slope per doubling of the rule count changes from −0.48 ± 0.07 to −0.17 ± 0.08.

Thus, in the GLMM fitted to the full dataset, the dependence on rule count does not disappear, unlike the estimate of −0.01 from your ordinary logistic regression on 1,132 aligned responses.

As you demonstrated, increasing the number of rules reduces the distance between each word and its nearest rule trigger. However, this is a natural consequence of adding rules to a task and therefore part of the load effect being measured.

Once the spelling adjustment is introduced, the resulting dependence describes a word that is orthographically distant from every rule trigger, rather than the actual distribution of words in the task. As a result, the estimated capacity with respect to rule count at the 0.95 level increases by more than eight orders of magnitude.

The limitation of this choice is that background-word capacity estimates are directly comparable only across models evaluated using the same vocabulary.

Because misfires are relatively rare, the standard IEC0.95 and IEC0.5 measures are not particularly informative for background words. I have therefore developed additional measures for them.

Going forward, the passport will characterize model behavior and degradation on both target and background words, as you suggested in your first comment. I will share these new measures when the passport is published.

Thank you again for your contribution. I plan to incorporate the background-word laws into the model passports alongside the other models and discuss the spelling similarity effect in my next research paper.

Hi Dipankar!

Thank you for suggesting the spelling similarity metric. I think it could be useful for further improving my evaluation framework.

Would you mind if I tried incorporating it into IEC? I'll be sure to share the results with you.

I'd also like to acknowledge your contribution in my next research paper. Would you be comfortable with me mentioning your name in the acknowledgments?

Hi Dipankar!

Thank you for the new results. Reducing the observed increase in the error rate from 6.7x to 1.5x after standardizing for spelling distance is a significant finding.

For now, I'd prefer to keep the existing generator unchanged and first try to account for this effect in the IEC statistical model.

IEC uses a Bernoulli GLMM that models the probability of correctly processing a word as a function of text length, word position in the response, replacement density, number of rules, and, for target words, the position of the corresponding rule in the list. The model also includes a random effect to account for the individual difficulty of each run.

I'm considering adding a term to this model that captures the effect of spelling similarity. This would apply not only to background words, which should remain unchanged, but also to target words, where similarity between different rule triggers could potentially affect the selection of the correct replacement.

You've already investigated the relationship between spelling similarity and errors in considerable detail, so I'd appreciate your opinion. What mathematical measure of spelling similarity would you recommend for such a term in the model?

Minimum Levenshtein distance seems like an obvious candidate. However, your finding that 28% of misfires involve a rule other than the closest one by spelling suggests that a measure accounting for the full set of distances to rule triggers might be more appropriate.

I'm interested in finding a reasonably simple measure that works for both background and target words, so I can test its effect on the measurement results before making any changes to the generator.

Hi Dipankar!

The 23x difference is striking, especially since the effect of spelling similarity persists across different rule counts.

However, I see this result from a slightly different perspective. The primary goal of IEC is to measure degradation in instruction following under controlled load. Spelling similarity introduces an additional, unwanted factor whose influence I'd like to minimize.

I don't think eliminating near-spellings from the generator entirely is practical or desirable. They occur naturally in language. So I'm leaning toward a statistical approach instead.

My idea is to account for the spelling distance between each copy word and the active rule triggers, estimate how misfire probability depends on that distance, and use this relationship to separate the effect of spelling similarity from the effect of load.

Rather than making lures a separate research direction within IEC, I'd prefer to treat spelling similarity as a nuisance factor that needs to be controlled for, and where possible adjusted for, in the statistical model. The goal is a cleaner and more comparable measurement of model behavior under load.

Your results provide a strong starting point for this. I'm particularly interested in seeing how much of the apparent dependence of background misfires on rule count remains after accounting for spelling distance.

Hi Dipankar!

It looks like we're exploring the same problem in parallel. After your comment yesterday, I also started examining different types of failures and noticed this phenomenon. But you've already taken it a step further: the relationship between misfires and spelling similarity shown by your analysis is particularly interesting. The fact that positional proximity seems to have relatively little effect is also quite revealing.

I'm becoming increasingly convinced that this needs to become a proper part of the research. In fact, I'm now realizing that my original classification of Type I, II, and III failures was based on a preliminary analysis that didn't fully account for errors in background words. So this may require more than adding another metric — I may need to revisit the analysis methodology and possibly the failure classification itself.

As for the generator, spelling similarity is not explicitly controlled. Words are deliberately selected at random and repeatedly reshuffled for each DMN parameter configuration. The idea was to avoid making the results dependent on any single word arrangement. But your findings suggest that randomization alone may not be sufficient. It would be worth measuring the effect of spelling similarity separately and checking how much of it survives averaging.

I'm particularly interested in whether these errors follow their own patterns and how they relate to the load dimensions already studied. This may reveal another important characteristic of model behavior.

Thank you for continuing to investigate the data in such depth. This is genuinely valuable work, and I'm very glad it's opening up new directions for the research.

Dipankar, thank you so much for taking the time to dig into the raw data! You went well beyond reading the article: you independently verified the data, carefully selected a subset that avoids alignment ambiguities, and discovered something I had overlooked. This is exactly why I made the data public. It's exciting to see that independent analysis is already producing unexpected findings.

Two observations particularly struck me: the 252 answers with perfect trial records that still modified the original text, and the 745 apparent misfires versus just 35 misses with a single rule. The latter is especially intriguing: misfires are clearly not limited to long rule lists. How they depend on the other load dimensions, and whether they reflect a distinct failure mechanism, is an open question.

You've identified an important limitation of the current passport. It captures only one side of model behavior: whether an instruction is followed where it should be. It tells us nothing about what happens where no changes are supposed to occur. And your results show just how significant that missing side may be.

I'm now particularly interested in whether these misfires follow their own patterns, how they depend on different load dimensions, and whether I can characterize them quantitatively. I find your suggestion of adding a precision dimension alongside recall very promising. This may turn out to be another distinct dimension of model behavior.

I plan to investigate this across the full dataset, including answers with broken line structures, which will require additional care with alignment. I'll certainly share what I find, and I'd be very interested in your thoughts.

Thank you again. Your analysis has done more than point out a limitation of the current passport — it has opened up a research direction I hadn't originally considered.