Understanding Neural Network Symbols

Key Takeaways:

• Symbol-like behavior in neural networks spans a spectrum: from genuinely emergent internal structure (grokking, phase codes) to structure that is implanted by design (neuro-symbolic concepts).
• Grokking, where a model generalizes long after overfitting, is now understood as a transition from the “kernel regime” to the limiting behavior of gradient descent, not magic.
• The “emergent abilities” of large language models may partly be a measurement artifact: nonlinear metrics like exact-match accuracy make gradual improvement look like a sudden jump.
• Whether a model has “real” symbolic structure depends on the metric you choose, the task, and whether the structure was learned or hand-coded.

The most consequential debate in modern AI concerns whether the symbol-like structures neural networks appear to use are real, or an artifact of how we measure them. A network trained only to predict the next token will, past a certain scale, start doing arithmetic, following grammatical rules, and composing concepts it was never explicitly taught. Whether that counts as emergent symbolic structure, or just a smooth curve that looks jagged under the wrong ruler, is now a live, contested question in the field.

What Symbolic Structure Actually Means

Classical symbolic AI held that intelligence is built from discrete, composable symbols manipulated by rules. Neural networks rejected that premise: they learn distributed, continuous representations with no obvious unit that maps to “chair” or “left-of.” The two traditions were treated as rivals for decades, a split Marvin Minsky once framed as “neat versus scruffy.”

The last few years have blurred that line. A growing body of work shows that networks trained purely by gradient descent can develop internal structure that looks recognizably symbolic: individual neurons or directions that respond to specific concepts, circuits that implement arithmetic, and oscillatory “phase codes” that encode abstract quantities. The question is no longer whether connectionism and symbolism can coexist, but whether symbolic structure is something the network builds on its own, or something we read into it.

One of the clearest demonstrations of genuine emergence comes from a study of recurrent networks trained on modular arithmetic. Keith Murray’s work on phase codes asked whether oscillatory “phase coding,” where information is stored in the relative phase of neural activity, could arise purely from task optimization, without any architectural bias toward oscillation. It can. But the paper’s more interesting finding is that phase coding is “natural but not inevitable”: the same task admits a “rich diversity” of alternative solutions with qualitatively different dynamics. Emergent structure, in other words, is one solution among many that gradient descent might find.

Grokking: When Structure Arrives Late

The cleanest laboratory for this is grokking, first observed in small transformers trained on modular addition. A model will overfit its training data, memorizing specific examples, and then, long after accuracy on held-out data has plateaued near zero, suddenly generalize. The switch is abrupt, which is why researchers likened it to a phase transition in physics, like water freezing into ice.

A theoretical analysis accepted at ICML 2024 reframed this. The authors show that early in training, the model is in a “kernel regime” where no permutation-equivariant model can generalize on modular addition without seeing a large fraction of all possible inputs. Eventually the model escapes that regime, and two-layer networks that reach zero training loss with bounded norm generalize with far fewer examples. Grokking, in their account, is a transition from kernel-like behavior to the limiting behavior of gradient descent on deep networks, not a mysterious ability that appears out of nowhere.

The practical takeaway is that the symbolic structure, the internal circuit that actually implements modular addition, is being assembled slowly during training even while the model’s test accuracy looks flat. The accuracy curve hides the underlying learning. This is a recurring theme: what looks like a sudden leap in ability is often a continuous process underneath.

The Mirage Debate and Metric Choice

The same tension plays out at the scale of frontier language models. In a 2022 paper, researchers compiling the BIG-bench benchmark reported that some abilities, like three-digit addition, stayed near zero as models grew and then “broke through” at a specific param count. As Quanta Magazine reported, GPT-3 gained the ability to add when trained with 13 billion params, and LaMDA did the same at 68 billion params. The authors called these abilities “emergent.”

A Stanford team led by Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo pushed back. They argued that the sharpness is an artifact of measurement. The BIG-bench studies scored addition by exact-match accuracy: either the model got every digit right or it scored zero. If instead you award partial credit, digit by digit, the “jump” dissolves into a smooth, predictable improvement as params increase. “Strong claims of emergence have as much to do with the way we choose to measure as they do with what models are doing,” Koyejo told Quanta Magazine.

The debate is not settled. Jason Wei, author of the original emergence paper now at OpenAI, has argued that for arithmetic the exact answer really is what matters, so exact-match is the right metric. Alex Tamkin at Anthropic told Quanta that even with continuous metrics, “you still have discontinuities” at scale. Anthropic’s own interpretability work, which decomposes model activations into interpretable features, has found directions corresponding to discrete concepts, suggesting the structure is not purely an artifact. The honest summary is that some of what we call emergence is metric choice, some of it is real, and the field is still building the science to tell them apart.

Two Roads to Symbolic Structure

It helps to separate two very different ways a network ends up with symbolic structure, because they have opposite trade-offs.

The first road is emergence: structure learned from data with no explicit symbolic scaffolding. Grokking and phase codes are examples. The second road is implantation: structure built in by design. The MIT and Stanford researchers behind “Neuro-Symbolic Concepts”, a Communications of the ACM paper, propose representing each concept as a tuple of a symbolic program and a neural embedding. A concept like “orange” is a filter function paired with a learned vector; “left-of” is a relation paired with a cosine-similarity classifier. These concepts compose, so “orange cylinder left of bottle” is built by combining sub-concepts, which buys data efficiency and compositional generalization but requires the structure to be specified up front.

A middle path is the “essence neural network” (ENN) from UT Southwestern, described in a Nature Computational Science paper. ENNs abandon backpropagation entirely and instead assign each neuron a specialized distinction, like “A versus not-A” or “more like A versus more like B,” then arrange neurons hierarchically. The result is intrinsically explainable, can simulate deliberation and symbolic reasoning, and is claimed to be more solid to adversarial attack. The trade-off is that it discards the gradient-descent machinery that made deep learning scale, so it has not showed the same benchmark performance on large, messy datasets.

What This Means for Practitioners

For anyone building or evaluating models, the emergence debate carries a concrete lesson: your metric determines what you think you see. If you score a model on exact-match accuracy, you will observe sudden “breakthroughs” that may not reflect a real change in the model. If you score with a continuous metric, like per-token or per-digit correctness, the same model looks like it was improving all along.

Here is a practical illustration of how metric choice changes the story, using a simplified evaluation loop:

# Example: Comparing exact-match vs. token-level accuracy
def evaluate(model, dataset, metric="exact_match"):
 total = 0
 correct_digits = 0
 exact_matches = 0
 for sample in dataset:
 target = sample["target"]
 prediction = model(sample["input"])
 total += 1
 if metric == "exact_match":
 if prediction == target:
 exact_matches += 1
 elif metric == "digit_accuracy":
 # Calculate per-digit similarity
 min_len = min(len(prediction), len(target))
 matches = sum(1 for p, t in zip(prediction, target) if p == t)
 correct_digits += matches / max(len(target), 1)
 if metric == "exact_match":
 return exact_matches / total
 elif metric == "digit_accuracy":
 return correct_digits / total
 

The same principle applies when you read vendor claims about “emergent” capabilities. A model that “suddenly” mastered a task may simply have crossed an arbitrary all-or-nothing threshold. Ask what metric was used, and whether partial credit tells a different story.

The deeper implication is for interpretability and safety. If the symbolic structure inside a network is real and findable, then the black box is not as opaque as feared, and a data-center compliance or safety team can in principle inspect the concepts the model is actually using. If it is partly an artifact, then much of the interpretability field is chasing a reflection. The truth, as the grokking and phase-code work suggests, sits in between: networks do build real, discoverable structure, but it is one solution among many, dependent on initialization, data, and the metric you use to look for it.

Summary of Approaches

The field is converging on a nuanced position. Symbolic structure in neural networks is neither a pure mirage nor a guaranteed consequence of scale. It emerges in some settings, is implanted in others, and is partly a function of how we choose to measure. The productive question for 2026 is not “do networks have symbols?” but “under what conditions, and how would we know?” The answer will determine whether the next generation of models is something we can inspect and trust, or a black box we simply have to hope behaves.

Related Reading

More in-depth coverage from this blog on closely related topics:

• Top GPU Performance Tips
• How to Highlight Code Based on Importance
• Google Maps Directions
• CPython Support for RISC-V Platforms
• How to Use Tether for Messaging on Linux

Sources and References

Sources cited while researching and writing this article:

• Keith Murray’s work on phase codes
• theoretical analysis accepted at ICML 2024
• How Quickly Do Large Language Models Learn Unexpected Skills?
• [2304.15004] Are Emergent Abilities of Large Language Models a Mirage?
• interpretable features
• Neuro-Symbolic Concepts
• Explainable neural networks that simulate reasoning | Nature Computational Science