In some circles, neural and/or mental representations are controversial. I have recently argued that neural representation clearly occurs, and that the word ‘representation’ is simply a label for a behavior, as opposed to a hypothetical mechanism or entity. I also showed that this pragmatic and causal sense of representation is much older than so-called “cognitivism”, which is the notion that the mind’s operations should be understood solely in terms of symbolic representations modeled on language. In the previous essays I carefully avoided addressing the issue of symbolic representations. But that is not because I think that the human brain doesn’t make use of them. I will now show, using similar logic to the argument in the previous post, that symbolizing is not a hypothetical mechanism, but an observed behavior. The real debate can only be about how the faculty of symbolizing is mediated by the brain (and perhaps by phenomena outside the brain).
Throwing things together: the origins of the term “symbol”
“Symbol” comes from the Greek roots “syn” , which means together, and “bole”, which means a throw or cast, or even a bolt or a beam. It’s the same “bole” root we see in “parabola” and “hyperbole”. So “symbol” was derived from “throwing things together”. The Greek word symbolon came to mean a token or mark that indicated faith in a creed, participation in a contract, or even a ticket or license. The “throwing together” sense comes from the idea of juxtaposing things in order to compare them. So one might “throw together” a person’s religious tokens along with yours, to check if your allegiances… align. 1 In English, this idea gradually broadened to mean “something which stands for something else”. Apparently the first recorded usage of this sort is from 1590, in Edmund Spencer’s Faerie Queene.
The anthropologist David Graeber’s book Debt: The First 5000 Years sheds some additional light on the concept of ‘symbol’. In his chapter on the Middle Ages, he explores the origins of paper money, where the symbolic nature of the value of the monetary tokens becomes impossible to ignore. It’s worth reading an extended excerpt:
When Aristotle argued that coins are merely social conventions, the term he used was symbolon — from which our own word “symbol” is derived. Symbolon was originally the Greek word for “tally”— an object broken in half to mark a contract or agreement, or marked and broken to record a debt. So our word “symbol” traces back originally to objects broken to record debt contracts of one sort or another. This is striking enough. What’s really remarkable, though, is that the contemporary Chinese word for “symbol,” fu, or fu hao, has almost exactly the same origin.
Let’s start with the Greek term “symbolon.” Two friends at dinner might create a symbolon if they took some object — a ring, a knucklebone, a piece of crockery — and broke it in half. Any time in the future when either of them had need of the other’s help, they could bring their halves as reminders of the friendship. Archeologists have found hundreds of little broken friendship tablets of this sort in Athens, often made of clay. Later they became ways of sealing a contract, the object standing in the place of witnesses. The word was also used for tokens of every sort: those given to Athenian jurors entitling them to vote, or tickets for admission to the theater. It could be used to refer to money too, but only if that money had no intrinsic value: bronze coins whose value was fixed only by local convention. Used for written documents, a symbolon could also be passport, contract, commission, or receipt. By extension, it came to mean: omen, portent, symptom, or finally, in the now-familiar sense, symbol.
The path to the latter appears to have been twofold. Aristotle fixed on the fact that a tally could be anything: what the object was didn’t matter; all that mattered was that there was a way to break it in half. It is exactly so with language: words are sounds we use to refer to objects, or to ideas, but the relation is arbitrary: there’s no particular reason, for example, that English-speakers should choose “dog” to refer to an animal and “god” to refer to a deity, rather than the other way around. The only reason is social convention: an agreement between all speakers of a language that this sound shall refer to that thing. In this sense, all words were arbitrary tokens of agreement. So, of course, is money — for Aristotle, not only worthless bronze coins that we agree to treat as if they were worth a certain amount, but all money, even gold, is just a symbolon, a social convention.
All this came to seem almost commonsensical in the thirteenth century of Thomas Aquinas, when rulers could change the value of currency simply by issuing a decree. Still, Medieval theories of symbols derived less from Aristotle than from the Mystery Religions of Antiquity, where “symbolon” came to refer to certain cryptic formulae or talismans that only initiates could understand. It thus came to mean a concrete token, perceptible to the senses, that could only be understood in reference to some hidden reality entirely beyond the domain of sensory experience.
The historical trajectory of ‘symbol’ was a kind of parabola, moving from the concrete to the abstract and then back again, as a chimera of the concrete and the abstract: a physical, tangible token that is a metaphorical ‘fragment’ of some larger — and often less tangible — entity. This notion of ‘symbol’ is close to the usual definition, but left out the notion of “standing for”. For this purpose, the fragment-centric notion of symbol-hood may come in handy. The Oxford English Dictionary provides the following definitions which seem most relevant2:
It’s interesting that definition 2a specifically says “not by exact resemblance but by vague suggestion or by some accidental or conventional relation”. Without this stipulation, “symbol” would become synonymous with “representation”, so the term “non-symbolic representation” would become meaningless.
An alien anthropologist would presumably be able to tell that there is a qualitative difference between how a person uses a map to represent a territory, and how they use a name to refer to an entity. Social convention is clearly required in both cases, but the degree of arbitrariness is greater in names than in maps. The alien researcher watching for long enough, and over a wide enough swathe of the planet, would quickly discover that the sound-patterns that different humans use to refer to the sun and the moon are so diverse and arbitrary that they cannot possibly have any internal structural similarity with the things they symbolize.
I saw the sign (and it opened up my mind)
We should be able to assert confidently that symbolizing is something that humans do, and that it has shades of meaning that do not completely overlap with those of representation; artistic representation, for example, tends to focus on verisimilitude. Unlike beauty, verisimilitude need not be purely in the eye of the beholder. Or rather, it need not be invisible to other beholders with different ‘tastes’.
When talking about the continuum between maps and arbitrary labels, we cannot really do better than Charles Sanders Peirce’s system of signs. Peirce divides signs into three kinds, based on how they relate to objects: icon, index, and symbol.
- An icon, also called a semblance or a likeness, covers our notion of representation as portrayal. So a painting’s relationship with its subject matter is iconic.
- An index relates to its target by some kind of causal connection. So smoke is an index of fire.
- A symbol denotes its object by virtue of interpretation or convention. “Pure” symbols have no resemblance to the things they represent, or any causal connection of the sort that exists between fire and smoke. The word “smoke” is a symbol for smoke, but its form has no necessary causal connection with smoke.
Causality makes this neat trio a little complicated: when we name something, are we not causing some kind of change of plasticity in the brain of the observers witnessing this baptism? Perhaps we should say that an alien anthropologist who only has brief temporal access to a person’s behavior would find it far easier to understand how they relate icons and indices to their targets than how they do so with symbols.
Animals and plants are able to use indices of various sorts: chemical signals are ‘signals’ for this very reason. The ability to remember and recognize associations between stimuli or actions and their consequences is the basis for Pavlovian and instrumental learning. And there is plenty of evidence that this ability is mediated by the brain. Some indices may even be genetically encoded in some animals. For example, mice and rats seem to be born fearing the smell of cat urine.
A more contentious issue is whether non-human animals use icons. Perhaps they can just get by with indices? At any rate, I have never seen a squirrel or a dog navigate by way of a physical map located outside its body. The sophistication of animal navigation has led researchers to use the term “cognitive map” to talk about its neural basis. Place fields in the hippocampus are a famous and relatively recent example of a putative map used for navigation. A lab animal seems to use a place field in ways that are at least loosely analogous to how a human uses a map — it tells us where we are, where potential goals may be, and where the obstacles along the way may be. Since these neural circuits seem to convey information about places and their inter-relationships, they appear to be internalized representations that do a job that humans perform with external maps. Not everyone agrees that the term “map” is appropriate, of course, but one suspects that part of the disagreement comes from unconstrained notions of semblance or “isomorphism”.
But let’s get back to our oracle bone of contention: symbols. The classic notion of symbol is well exemplified by the characters used in a writing system. But the auditory form seems to be far older, given that there are plenty of cultures that never developed writing. The process by which an arbitrary sound comes to be associated with an object remains mysterious. When a parent gestures towards an animal and says “dog” to their child, how exactly does the child know that they are referring to the animal, and not, say, the animal’s activity, or its color, or even something happening in the background? Sorting this issue out seems to require careful study of the non-symbolic representations of object boundaries, and how attention interacts with them. I have written about this elsewhere (and have talked about how Steve Grossberg’s work helps with thinking about this), but I think for the most part this remains a deep mystery. But note: the mystery is how this happens, not that it happens.
Humans do not simply label objects and processes in the world using words and other symbols: we also string symbols together. This falls under the heading of “symbol-manipulation”, since we quite literally concatenate symbols, permuting their order to convey different meanings. Language is the obvious example of this, but mathematics operates like this too.3 It is hard for me to imagine why anyone would claim that symbol manipulation does not take place in the brain — or in the mind for that matter. Granted, sometimes I am capable of uttering a sentence that I did not really anticipate saying. Since I demonstrably remember the words that I am capable of uttering, they must be represented in the brain somehow, even if the process of manipulation is unconscious. But I am also perfectly capable of consciously concocting phrases and sentences in my imagination, unleashing them when appropriate (or even when inappropriate). In this case I am representing the words in my mind.
Hic Sunt Dracones
Let’s recapitulate an idea from the previous post: a neural representation of X is a mediator of X — it makes present some aspect of X, which can then have causal access to other parts of the brain. So when I encounter the word “banana”, some neural pattern (or family of patterns) is kindled in the brain that enables me to draw a banana, or recall banana bread recipes, or conjure up an image of someone slipping on a banana peel.
Central to the power of language is that it allows us to repurpose old words in the construction of new ideas and images. I have never encountered a six-foot tall magenta banana, but now, having encountered the phrase, I can imagine one. In fact, I like to think that the ability to imagine “as many as six impossible things before breakfast” should be viewed as the hallmark of symbolic representation, and of human intelligence more generally. We seem to have the ability to fuse wildly incongruous images together. The dragon is the perfect example and mascot of this phenomenon. The European dragon fuses the serpent with the bird or bat: it is an alchemical marriage of the earthbound and the aerial. We have no idea if non-human animals are capable of imagining chimeras, but we can be reasonably sure that they do not create physical icons of them. It is not impossible that our powers of imagination and our powers of expression co-evolved. Our permutations of words are not unlike our amalgamations of scales and wings: they take on an existence that is altogether distinct from the sum of their parts. We might symbolize (!) this emergent wholeness as the fire that the dragon breathes.
The controversial role of symbol-manipulation in psychology has less to do with dragons than with computers, of course. In the decades following the Second World War, the computer revolution changed how people thought about the mind-body problem. Computers provided a working model of cognition that helped put the final nail in the coffin of dualism — at least in mainstream, secular academia. Alan Turing convincingly showed that a very simple mechanical model of symbol-manipulation — inspired by what human computers were capable of doing without a lot of thinking — could generate all familiar forms of computation. Since computers inspired by this principle became exceptionally successful at certain well-specified tasks, people began to wonder if the mind and/or the brain might be understood as analogous to Turing’s celebrated machine. To cut a very long and fascinating story short, the computer metaphor was the physicalist Trojan horse that brought internal mental states back into psychology after the Behaviorist Interregnum. Along the way, a new field was born: cognitive science. Cognitive science made explicit use of ideas borrowed from what is now called “GOFAI”: Good-Old Fashioned AI. Nowadays, researchers who work with dynamical systems and/or artificial neural networks disparage GOFAI as little more than a set of “if-then” statements. But these strings of conditionals clearly performed some “human-like” tasks. Whether they performed them in the same way as humans remains up for debate.
Symbols as invariants
Instead of diving into this complicated-yet-strangely-boring debate, I would like to move in an orthogonal direction, and explore how the concept of invariance can help us think about symbols in the brain. Invariance is a powerful and widely used framework in physics and mathematics, but it strikes me that its uptake has been limited in neuroscience and psychology. In math and physics, invariance (also known as symmetry) is immunity to a possible change. 4 The fact that invariance relies on a possible change means that the type of change needs to be specified in order for the concept to have meaning. Simply saying that something is “unchanging”, “eternal” or “constant” will not do. More importantly, the presence of some symmetry implies an underlying asymmetry. This will be clear when we look at examples.
Squares are invariant to rotations by multiples of 90 degrees, as well as flips along the diagonals and perpendicular bisectors. That means that they look the same when any of a certain set of transformations are performed. (It is a good exercise to work out which transformations, if any, leave a given shape unchanged.) We would not be able to know that a square was rotated by 90 degrees in the first place, if something other than the square did not look different after the transformation was performed. In the case of rotations, this underlying asymmetry may be provided by a protractor that measures the angles. A measuring device can only tell you about the symmetries revealed by a transformation if it is not immune to the same transformation. So the protractor provides the asymmetry necessarily implied by knowledge of the symmetry of the square.
Human face recognition shows many strong invariance properties. We can recognize faces from great distances, as well as in situations of altered make-up, hairstyle, and facial hair. But there are limits to the invariance properties of our face representations: we are really bad at recognizing upside-down faces, so face recognition is not invariant with respect to 180 degree rotations.
Music provides many fascinating examples of invariant representation. The most basic one is octave similarity: in a musical context, when a frequency is doubled or halved it is heard as the “same” note. So our perception of the note is invariant to octave transforms. Our melody representations show a related invariance: they are symmetrical with respect to shifts in overall pitch. So you can recognize when someone is humming “Happy Birthday to You” even if they are singing it at a pitch that is much higher or lower than you have heard it performed before. There are sophisticated invariances in the time domain too: we can clearly recognized tunes that are slowed down or sped up (up to a point), and many of us can recognized highly syncopated (‘jazzed up’) versions of familiar melodies, which may even contain melodic interpolations and digressions. In all these cases, our musical invariants are not “universals”, since the are not immune to all imaginable transformations, but only to a specific subset of them. And context may well play a role in the degree of invariance displayed: we may struggle to recognize a tune in some locations or during certain activities. Contextual change can be reframed as one of the classes of transformation that a behavior or a neural representation might be invariant with respect to.
We can now ask about the invariance properties of symbols. Music is a nice midway point between the non-symbolic and the symbolic: the domain of spoken language displays many overlapping qualities. We factor out the pitch of a person’s voice when determining the words they are saying, so the ability to recognize words is invariant to pitch-shifts within some range. Similarly, word recognition shows a degree of invariance to speeding up or slowing down the utterance. We should note that the invariance displayed by a cognitive or perceptual faculty does not mean that our entire perceptual system is insensitive to the fact of a transformation: we simultaneously recognize what a person is saying, and perceive how the pitch is modulated. The point is that without invariant representations, we would not be able to extract similarity of structure in the face of so much change. 5
The domain of writing reveals many sophisticated forms of invariance whose underlying transformations may be hard to articulate. In Douglas Hofstadter’s book Metamagical Themas, he shows a picture of the letter ‘A’ in various typefaces (right). It is quite clear that our ability to recognize ‘A’ is invariant to many wild transformations: there is no local feature or combination of features that can be called the essence of the letter ‘A’: there are non-triangular A’s, A’s without a crossbar, and A’s with additional bells and whistles. How is this possible? No one really knows, but that doesn’t take away from the bare description of the phenomenon in terms of invariance and transformation. Whatever model you come up with for human character recognition, it must display invariances of this sort.
There is another way to use invariance to probe the nature of symbolic representation: we transform the user, and see what, if anything, breaks the faculty of symbol-manipulation. When a person uses sentences in conversation, you might claim that they are not that different from Clever Hans, a horse whose feats of cogitation were eventually shown to be simple reactions to his trainer’s body language. Perhaps humans simply “pick up” linguistic information from the environment? Or “couple” with the person they are talking to, like some sentient pendulum? This sort of absurdity is immediately put down like an old horse when we reflect on the invariance properties of human language use. I can climb up to a remote mountaintop, all alone, and construct sentences. I can write them down to prove it to the neighsayers. Similarly, the invariances of a symbol’s references are quite remarkable, if you pause to explore them. You can think about artichokes even if you are nowhere near a farmer’s market or a restaurant. You can follow a recipe for a dish you have never eaten before (granted, with varying degrees of success). You can parse at least some of the streams of gibberish and tragedy on your social media feed even if they have little to do with any familiar context. Human reference can be decontextualized and recontextualized, which is why you can entertain ideas about the potential monetary value of an NFT that depicts a large magenta banana, despite only learning about what NFTs are in the last year. Your ability to understand and create language is not significantly disrupted by loss of limb or even complete paralysis. The ability to process humor — everything ranging from the lowly pun to many-layered equine irony — requires alienating words from their native habitats while simultaneously retaining as many of their associations as might be needed to get the joke (and perhaps a few extra just in case of follow-up jokes).
The faculty of symbol-manipulation is clearly invariant with respect to many transformations of context and bodily configuration. But it is not invariant to removal of certain parts of the brain. Everything is connected to everything else, but the strength of connection is not equal everywhere, which is why we can say that the heart pumps blood without reference to the Crab Nebula. When we say that symbol-manipulation is taking place in the brain, we mean that the observable, impossible-to-doubt phenomenon is not invariant with respect to certain transformations of the brain.6 And of course, researchers who actually study language-related processes in the brain can be more specific: specific regions of the brain have long been known to be crucial for language processing. When Broca’s area is damaged, the ability to produce language (spoken, signed or written) is severely disrupted, but the ability to comprehend language remains more or less intact. Conversely, when Wernicke’s area is damaged, patients cannot comprehend language, but are capable of fluent speech. Damage to other parts of the brain does not produce these particular deficits, so there is little point in insisting that localized processing does not occur. Yes, the brain is an integrated dynamical system, but evidence from lesions, recordings, chemical manipulations, and neurosurgery all confirm that functional specialization cannot be wished away.
Symbols are fragments of… something
I am very sympathetic to the ethical and aesthetic appeal of a more integrated, interaction-centric approach to mind and life. But these powerful motivators of “radical” reform should not blind us to the surface-level obviousness of symbol use, and to the marginally-less-obvious varieties of symbol invariance. I will grant, however, that we do not have particularly good ideas about how the symbol-using faculty emerged in the first place. The “radical” cognitivist solution — symbols all the way down — has no meaningful connection to any neuroscientific data or model. More importantly, it avoids the question of how biological systems implement the rule-like manipulations of ‘natural’ symbols, i.e., the rules that mathematicians, computer scientists and linguists began to make explicit only in the last century. In any case, the meanings of symbols do not completely reside in the computer: if meaning is usage, then surely the meaning of each symbol permuted by a computer is determined in large part by its users, i.e., by us.
From the perspective of the brain and mind sciences, goal-oriented behavior is obvious — but it does not tell us what is distinctive about symbols. Non-symbolic representations, as we have seen, are causal mediators of events in the world and in the rest of the brain/body. Symbolic representations inherit this mediating capacity, and their usefulness is plain to see. But where do the rule-like relationships between them come from? These rules seem far more subtle than the patterns and categories learned by neural networks (whether biological or artificial). All our attempts to create systems that can interact with us using symbols are laughably rudimentary.
What now? Obviously, I am not going to propose a novel theory of symbol-manipulation at the end of a long blog post. I have no idea what researchers should do to emulate human symbol-manipulation. But recently I have been toying with some images that may come together, dragon-like, in the future. So everything that follows is wild speculation. I am just riffing now.
Associative learning is not an especially controversial idea — it is implemented in some form or the other by most neurally plausible models of synaptic plasticity. So the act of arbitrary association, of which naming is a special case, is straightforward to describe mechanistically: it seems to be a variant of supervised learning. Once you have a stable representation of an object (or a category of objects), you can use some variant of Hebbian learning to associate this representation with a (representation of) a name or some other symbol. Steve Grossberg‘s supervised learning model, ARTMAP, has the right structure, since both the name/label and the object being named are put through unsupervised learning networks: this ensures that labels have their own invariance properties (like the letter ‘A’ in the example above) in addition to those of the object.7
Once you have associated an object with a label, you have something new that might as well be called a ‘superobject’. In an ART-style model with top-down excitation, activating a label can cause reactivation of the previously-learned features of the object. And of course, the bottom-up presence of the object, or even part of it, will activate the label. Somehow, humans are able to perform associative learning on all the features as well as their labels, without creating a gigantic kludge memory in which everything is reminiscent of everything else. I have no idea how this happens, but I have no doubt that it happens. The richness of human content-addressable memory provide many great examples.
We can think of a mnemonic superobject as akin to the unbroken symbolon of the ancient Greek world: one part is the label, and the other part is the vast network of features and associations. Among some imaginary protolinguistic ancestors, the mental world might have been composed solely of superobjects interacting in a manner that we might call behavioral free-association: neural superobjects would evoke other superobjects, and the trajectories in neural pattern-space that were useful for survival… survived.
Some sort of partial breaking of symbolons happened eventually, so that the labels could take on a life of their own. Out of the churning neurosemiotic sea, the names/labels floated to the top and became islands — symbols ready for manipulation. But they retained (and still retain) their connections with the fluid matrix of features and associations. We might say that symbols are like discrete lily pads floating on the surface of a pond, offering no clues about the tangle beneath. One is tempted to offend the botanists by imagining that this array of lily pads is somehow a rhizome.
Jumping to another image, it seems to me that symbols are like mountain peaks: easily differentiated, but ultimately part of a single mass of mountain with no sharp boundaries. The mountain is the tectonic mass of associations and features from which symbols thrust upwards. Fusing this metaphor with that of the lily pond, we can imagine symbols as islands that are actually the peaks of underwater mountain ranges. Their apparent discreteness and self-containedness are what usually draw our attention, rather than their underwater linkages.
Rendering this picture dynamic is quite challenging, so perhaps we must speak without the help of coherent analogies now. The relationships among labels are stable and systematic, yet somehow fluid, and are governed in large part by the linkages below the surface 8. It seems like the crisply reliable rule-like nature of symbol-manipulation, particularly in the case of language use, emerges from how the ‘submerged’ portion of each symbolon interacts with its counterparts. In fact, the rule-like quality may only crystallize in the mind of an observer when socially amplified9 attention is drawn to the surface pattern and away from the gyre of features and associations.
In fact, the ability of discrete and seemingly isolated symbols to interact below the surface might give us a little narrative about the emergence of dragons and other “mind-born” creatures. Let us assume for the sake of argument that language was necessary for the invention of dragons. 10 How might this have occurred? Perhaps in a crowd, one person was talking about a snake, while another was talking about a bat. The symbols, juxtaposed in the mind of some listener, might have served as matchmakers for a more intimate engagement: the fusion of incongruent features in that underwater realm. And in this way a new object is born: something stable enough to be given a place above the surface, among the named islands of reference.
Notes
- The ‘lost and found’ trope in Bollywood movies shows that this way of proving allegiances has survived into the modern era, at least notionally. Siblings separated in childhood might rediscover each other years later thanks to a locket, a ring or even a birthmark.
- the OED definition of symbol #1a: “A formal authoritative statement or summary of the religious belief of the Christian church, or of a particular church or sect; a creed or confession of faith, spec. the Apostles’ Creed.”
- In my opinion, mathematical expressions — equations, inequalities, definitions and so on — are a constrained subset of natural language, rather than a separate language. There are no native speakers of pure mathematics. But this does not necessarily mean that mathematical thinking is a subset of “pure” linguistic thinking. In any case, I am not sure that “pure” linguistic thinking is a coherent concept.
- Invariance bears some resemblance to the hoary old concepts of “property” and “universal” in philosophy, but the mathematical mindset has swept away some of the mental cobwebs.
- If these concepts are not yet clear, check out this essay I devoted to the symmetry perspective, and why science itself is a form of symmetry: Science: The Quest for Symmetry. I also discussed invariance in this video on information.
- There are no doubt non-neural transformations that symbol-use is not invariant with respect to, but I suspect that they are better understood as mediators of the emergence of the symbol-faculty in evolutionary, developmental and cultural senses. Clearly language requires a community to impart it. But if, some day, all but one human have departed from this vale of tears, that lone survivor, if they have learned language in the first place, will continue to be able to talk to themselves for quite a while.
- I present an introduction to Grossberg’s Adaptive Resonance Theory in this video.
- Surface of what? Of the water? of consciousness? I don’t know.
- My former thesis advisor wrote an excellent paper on how social interaction mediates language and intelligence: Socializing the Theory of Intellectual Development. For excerpts, see this twitter thread.
- I am aware that there is a simpler theory: that someone saw an eagle carrying a snake in its talons. That doesn’t really explain the sphinx though, does it?
