‘Representing’ means exactly what you think it means

I. The evocative intermediary

In the previous post, I offered a potted history of the debate surrounding neural and/or mental representations, and tried to expand the scope of the typical genealogical narratives, which tend to begin in the post-WWII era, at the birth of cognitive science. I also argued that representations are usually placeholders, especially for experimentalists who have no time to come up with complete theories of how representations arise and become useful in the brain. More importantly, the history of science is littered with placeholders that are gradually turned into fleshed out technical concepts.

But we can go much further in defense of representations. It is not too hard to argue that ‘representing’ is the best term to use when describing certain behaviors routinely observed in human beings. No other word will do. Before we get to the neuroscience and psychology, we have to first establish what we mean by the word ‘represent’. Too often in discussions about representation, the starting point is some technical and/or philosophical notion of representation that is divorced from wider usage. So in the spirit of Wittgenstein’s idea that meaning is use, let’s first look at how ‘representing’ operates in non-academic environments. Dictionaries are not a bad place to start.

The Online Etymology Dictionary tells us that ‘represent’ comes from a Latin word which means “to make present” or “to place before”. The “re” in “represent” does not seem to mean “do again”, even though it does mean this in many other words that start with “re”. It seems to be an intensive prefix in this case.

The Oxford English Dictionary gives us two broad shades of meaning:

At first glance, it is the notion of representation as depiction, portrayal, and “making present” that seems most relevant to descriptions of mind and brain. The other set of meanings has to do with what is expressed in the term ‘House of Representatives’. Political representatives are persons or groups who act on behalf of others, or perform a function in their name. So we have representing as ‘portrayal’ or ‘evocation’, and representing as ‘serving the function of an intermediary’, or as mediation. This second sense of representing will come in handy for causal accounts of how portrayal or evocation works in the brain.

Nowhere in the OED(s) is representation defined as “present twice”, but we can still discern a kind of “two-ness” in representation: we start with some ‘X’ , the entity, process or group being represented, and also have a separate ‘Y’ which is doing the artistic, theatrical or mental representing of X. We can posit a third necessary entity: the observer to whom X is being represented. In CS Peirce’s semiotics, the role of the observer is covered (roughly) by the term “interpretant”. Object, Sign and Interpretant do not necessarily exhaust all the factors one might consider necessary to label an occasion as representation, but the semiotic trio is a good starting point.

How do we determine if something is an artistic representation of something else? We might, for example, come across an odd contrivance in a museum. How do we come to decide that it is a portrayal or depiction of something else? Typically we recognize its form as being similar to, or evocative of, that other thing. ¹ This similarity is clearly in the eye of the beholder. ² So does this mean that representation is useless when talking about the inner workings of the beholder? It is tempting to think that we have arrived at some kind of impasse beyond which lies circular reasoning or infinite regress, but let’s try to keep going.

Let us imagine that we are not interested in whether Y represents X to the context-free viewer from nowhere, but whether Y represents X to a specific observer Zed. In other words, we have specified a beholder other than ourselves. We cannot now rely on our own subjective sense of similarity or evocation. Instead, we must use the observable behavior of Zed to infer something about how Zed thinks X and Y are related. If Zed is an adult human, you can simply ask them whether they think X is similar to Y, and what that similarity consists in. You might ask them what the relationship is between a photograph and a scene that is photographed. If Zed is unaccustomed to the odd questions of mind/brain researchers, they might struggle a bit. You could try more targeted probes though. You could show them X and Y for a bit, and then hide X and ask them to make true claims about X using only Y. If Y is a map and X is some territory, then you can devise more elaborate tests of Zed’s representation ability. You could perhaps ask Zed to hide some object in a room and see if they can use a map Y in a way that helps them find the treasure a day later. You can also expand your test, so that Zed is asked to assemble a physical object, such as a drawing or a clay sculpture, in order to send information to yet another observer, Dubya. You can specifically rule out language, to probe other forms of communicative representation.

It is not too hard to come up with tests to see if Zed is using physical medium Y to represent physical object/process X, either to himself at some later time, or to his confederate, Dubya. You can probe how the specific physical form of the representation medium — pen and paper, clay, knots in a string, grunts, gestures, and finally words and phrases — constrains the types of information that Zed can store or share. In this way, you come up with a pragmatic, constrained, context-specific notion of representing. Once you and your fellow researchers have agreed that your experiments capture something relevant to how Zed represents X using Y, you can look at all the “objective” physical measurables that overlap between X and Y. You might find that there are geometric, topological or textural features that are shared between X and Y. There may prove to be a more complicated relationship involving contextual factors and juxtapositions. Another interpretive challenge involves any sort of conventional code (assigning physical objects to each other in an arbitrary way). But codes can be cracked as long as they continue to be used, and this too can be studied by observing how Zed works with Y to convey X. In this way you could learn, for example, that Y is being used as a name for X, and therefore relies on associative memory to do its work. This is how a code-breaker operates, listening in on the encrypted messages sent between two spies.

Armed with this externalist way of thinking about representation, we can look for it in the wild, i.e., in places other than neuroscience, cognitive science and philosophy. It should not be hard to come up with tests to see if and how a painting or sculpture represents the thing depicted, or how a map represents a city. It isn’t even that difficult to come up with an externalist test for more abstract notions of representation, such as the idea that a work of art or a piece of music represents an emotion. To many listeners, the song ‘I Can See Clearly Now’ makes present the feeling of optimism.

II. Aethereal presences?

All we have done so far is make explicit what we already knew about the various ways the word ‘represent’ is used. Most people learn what words mean by watching them in action, a skill that cannot be fully replaced by a dictionary. This initial step, which can seem redundant to people who feel confident that they know what colloquial usage is, is crucial if we are to use ‘representation’ in the cognitive and neural hall of mirrors. Only if we accept that such tests actually do tell us whether Y represents X to Zed in the colloquial sense can we ask about what representing might mean in the context of the brain. This is what I meant in the previous post when I said that representing is a label for an observable phenomenon, and not a hypothetical mechanism or entity. When X, Y, Zed and Dubya conform to certain stable patterns, we can affirm that representing is by definition under way. In other words, the concept of representation is fully grounded at the level at which it is usually deployed, even if we do not understand anything about what is going on inside the heads of the various beholders. This is how concepts always get grounded in the first place. We were able to use the word “motion” as a description long before physicists understood anything about mass, inertia, or momentum. We could identify a liquid as a liquid long before we knew what its chemical composition was. We did not need to wait around for botanists in order to understand what seeds did. At every stage in the development of science, researchers deploy pragmatic, ostensively defined concepts that are subsequently replaced or refined.

Granted, some concepts are abandoned. But it seems unlikely that ‘representation’ will go the way of phlogiston or luminiferous aether. This is because phlogiston and aether were proposed as inferred, unobserved causes of well-known phenomena (combustion and propagation of light, respectively). ³ Representing is much more like the observable phenomenon itself than any proposed hidden cause. The fact that people can use a map to navigate, or use physical media to communicate, cannot be abandoned like phlogiston, regardless of the terms we use to label such processes.

In any case, the word ‘representation’ accurately conveys the key feature of the phenomenon that no other word in English does: representing involves making things present⁴. And it also involves agents that act on behalf of other agents (as in the ‘functionary’ notion of representation). We can fold these two version of representation into a single notion: Y represents X when it stands for X. In fact the functionary definition might be best: a representation Y mediates X. Just like a political representative, a representation is the intermediary through which something or someone can perform certain functions. So a neural representation can be understood as the intermediary (or quite simply, the medium) for some other phenomenon — including another neural phenomenon. Tying the two notions of representation together, we can say that a neural representation brings some aspect of the phenomenon into the presence of the rest of the brain and body, and in doing so mediates the potential for certain actions. A neat consequence of this is that all forms of memory⁵, from short-term memory and working memory to long-term episodic memory, can be understood in terms of mediating past events, creating the potential for them to be brought into the present. If we can get on board with this notion of ‘representation’, then it will not be hard to recognize that when most neuroscientists use the word, this mediation process is all they mean. ⁶

Memory-guided behavior is a great place to watch representation in action. Let’s say that you threw a party (in that mythical pre-COVID era) and the next day someone calls to tell you they accidentally left their wallet at your place. They send you a picture of it, and that helps you hunt around for it. Using our externalist notion of representing, we can say that the photograph is being used by you as a portrayal of the wallet. It serves as a template, rendering the form of the wallet present and available for comparison with various things in your living room to the wallet (perhaps there are many wallets in your room). Now what if, instead of looking at a photograph, you recall the wallet — perhaps you remember your friend took it out to hand someone a business card. Now you have an internal ‘something’ that does the same job as the external photograph, more or less. It brings into your mental presence the form of the lost wallet, which you then use to enable your search. If some alien neuroscientist were watching this whole scene, equipped with futuristic brain scanning technology, they might note some structural similarities between the real wallet and the remembered image of it.⁷ Functionally, the fact that the neural memory is inside the head, and is presumably less accurate than the photograph, does not make any substantive difference to the nature of the phenomenon. The neural pattern⁸ that corresponds to the subjectively experienced memory is called (by convention, based on the colloquial meaning) a representation of the wallet. To what is the neural pattern relating? To you. In other words, you are the observer of (some of) your neural patterns. If you can observe your own hand, you can in principle observe things inside you, so there is nothing necessarily circular here.

Circularity could arise if we are not careful when fleshing out how a neural representation enables successfully searching for that wallet. What you are, when viewed at the scale of brain dynamics, is a localized neural storm. Recall Leibniz’s mill: when you enter, all you see are whirring cogs and gears (neurons, glia etc.). There is no central hub that corresponds to “you” — only an interpenetrating tapestry of dancing neural patterns. These patterns are ceaselessly “observing” and “comparing” each other in myriad ways.

III. Beware the etymological fallacy

We now need to investigate why it is so tempting to use scare quotes when attributing the power of observation or comparison to components of you. Some philosophers and neuroscientists (including a previous iteration of yours truly⁹) have argued that concepts such as perception and agency only apply to wholes and not to parts. We might call this emergentism by fiat. Failure to adhere to this stricture has been labeled the mereological fallacy. A specific version of it is the double subject fallacy¹⁰, which involves locutions like “my prefrontal cortex made me do it” or “my amygdala made me scared”. I used to think we ought to frown on this kind of category mistake — harrumphing that only the person as a whole can be said to do anything — but I now think that this is simultaneously a severe handicap on analogical thinking and also an inaccurate characterization of the fractious disunity of subjective experience. Sometimes it does feel like the self is multiple partial selves at war with each other — and surely this feeling is in need of mechanistic explanation?

Let’s call strict abstinence from the mereological fallacy the ‘etymological fallacy’. It is the insistence that a concept never be used outside the context in which it was developed. It is a kind of originalism about language. The fact is that treating neural circuits or individual neurons as ‘quasi-agents’ — entities that are a little more complex than cogs and gears, since they have homeostatic ‘goals’ of their own — comes naturally to people who think about the brain. Famous examples of neuro-anthropomorphism include Oliver Selfridge’s Pandemonium model, Marvin Minsky’s Society of Mind, and Daniel Dennett’s idea that consciousness involves fame in the brain. Stephen Grossberg has drawn analogies in the other direction, going from neural networks to social networks. And I myself, in a not-quite-serious spirit, have written about the brain as an economy, using the idea to link some neural phenomena with Goodhart’s law. This way of describing the brain can usually be cashed out in non-agentic terminology, or even in differential equations. We need only be wary of anthropomorphic metaphors when we have no idea how to translate them into toy models.

So let us allow that neurons, ensembles of neurons and neural circuits can meaningfully be described as observing each other. We need not assume that this observation involves conscious experience — it is purely an externalist, functional characterization. Observing is what observing does: in this case that includes receiving, transforming, and transmitting neural signals. If one is still worried about “observers all the way down”, we can just point to the fact that physics, chemistry and molecular biology have no use for anthropomorphic metaphors, so once we cross the level of the single cell on our downward journey, we are in safely physicalist territory. Anthropomorphism eventually “bottoms out”. Of course, this does not solve the problem of how agents arose in the first place. But that doesn’t mean that speaking of agents or representations cannot guide both experimental and theoretical neuroscience.

We may actually be able to use representation-as-mediation to blur the distinction between anthropomorphic observers and causal agents. In other words, the brain is a house of representatives. The reps act on behalf of various parties: the internal organs and circuits of the body, objects and processes in the world, and other neural reps. In doing so, they make aspects of phenomena present and therefore available for potential causal impact on behavior.

IV: Representations mediate flexible control of the body

A motor-centric viewpoint on the brain helps cement the link between representation-as-evocation and representation-as-mediation. ¹¹ Flexible behavior requires many patterns that have a quasi-stable relationship to phenomena in the outside world. For example, imagine a predator chasing after its prey in the jungle. Successfully tracking the prey requires a set of quasi-stable patterns that establish behavioral ‘grip’ on the situation. Since the relative position of the prey can vary, as can the configuration of the predator, sophisticated transformations from eye-centric space to body-centric space to world-centric space to motor command space must be made on the fly. An object moving across the visual field will excite a variety of neurons in regions that are primarily retinotopic. These signals cannot directly excite the motor cortex without further processing: that would be like free jazz played on the muscles. This is likely the reason why so much cortical real estate in primates is dedicated to the visual system: transforming visual signals into stable object representations and movement-relevant coordinates is a complicated business.

Some of these pragmatic patterns maintain the presence of a prey that is temporarily absent from view, such as when it has passed behind a tree or turned around a corner. In the most behaviorally flexible predators, there must also be stable patterns than latch onto the identity and anticipated trajectory of the prey, so that a temporary distraction — a large branch falling nearby, perhaps — does not completely derail the hunt. These are simple examples of invariants that must be constructed by the predator in order to successfully hunt in an environment that is changing in various ways, including as a result of the predator’s own actions. Computational neuroscientists who model active vision often call these patterns “invariant representations“: they bring aspects of the prey into the presence of the brain’s decision-making and navigation-system and mediate the continued presence of these aspects. No word other than “representation” conveys this spectrum of meanings.¹² Even when a computational model neglects to close the perception-action loop — which is the case for most models, since this is difficult to do when there are many degrees of freedom in organism and environment — any invariant representation is implicitly assumed to have a form that renders it “legible” to neural circuits underlying reinforcement learning, attentional modulation, decision-making, and/or motor control. A cognitive map, for example, can be modeled without knowing exactly how it is used by the navigation system or the ‘vicarious trial and error’ system, as long as its structure does not rule out some form of goal-directed reactivation/reconstruction.¹³

Experimental neuroscientists use “representation” in much the same way as the theoreticians and modelers, but with less commitment to specific hypothetical mechanisms. They are sometimes accused of conflating neural patterns that are representations to the researcher with representations to the organism. This may be a rhetorically satisfying flourish, but if you were looking for structures that serve as representations to the organism, what else would you look for other than structures that seem like representations to you? Correlation doesn’t imply causation, but causation usually implies correlation. So correlates are not a bad place to start looking for processes with causal heft. Regardless, correlation is not the only tool at the disposal of the experimentalist. Anatomical tracings, lesion studies, and chemical/electrical manipulations, all performed with cross-species comparisons in mind, are used to triangulate candidate neural representations. Whether they ‘really are’ representations or not is a matter of how theoretical and computational approaches harmonize with data and guide the design of new experiments. Given the way “representation” is actually used by large numbers of neuroscientists, declaring that there are no representations at all comes across as trying to hamstring scientific communication in the name of some kind of metaphysical purity — a purity that many people find very hard to parse, let alone engage with.¹⁴

An exhaustive engagement with anti-representationalism would be… exhausting, since there are many elaborate and seemingly technical arguments against representation, only some of which my discussion above can possibly address.¹⁵ One topic that cannot be ignored, however, is symbolic representation. Symbols are the elephant in the representation room. Some anti-representationalists seem to agree that pragmatic representations of the sort I outline here clearly exist, even if they are given other names in their theories¹⁶. So their real target is symbol-manipulation. I think this is a much richer topic for debate than representation in the sense of mediator of presence. Nevertheless, for human beings who are capable of language, it is obvious that symbol-manipulation is going on, since that is literally what using language is. I aim to pick up this topic in the next post, but the basic idea is very much along the lines I have presented (!) here: just like representation, symbol-manipulation is not a hypothetical mechanism or entity, but an observed phenomenon. The real debate is about the neural basis for symbol-manipulation. Some flavors of “cognitivism” propose that physical symbols exist in the brain/mind, and that their manipulations resemble the rule-governed transformations of symbol-strings that one sees in GOFAI or Chomskyan generative grammar. On this point I lean towards the side of the anti-cognitivists: the neural mechanisms mediating many forms of symbol-manipulation need not look anything like computer code: the rules may instead be emergent from other types of mechanisms. But even here, it is clear that some human beings, such as mathematicians and computer scientists, do represent (in every sense) the rules for symbol manipulation in ways that are in close correspondence with the rules as written explicitly. Another contentious topic is whether symbols exist intrinsically as part of our genetic heritage, or whether they are necessarily social in origin. I think the two sides are stuck in a boring nature-versus-nurture debate, but it is once again obvious to me that the development and meaning-grounding of language and other symbol-systems is observably social and interaction-driven.¹⁷

So just to recap, my point with this post is to argue that neuroscientists who use the term ‘representing’ mean exactly what a random person on the street probably thinks the word means: bringing some phenomenon into the presence of an observer, and in doing so, mediating the potential¹⁸ for some causal interaction. We must just allow that the observer could be a sub-component of the brain.

Notes

1. Some people use the term “isomorphism” here, but that word is best restricted to mathematics, where the word refers to a one-to-one correspondence between the features of two objects. Such a one-to-one mapping may be very tricky to elucidate in everyday contexts such as artistic portrayal. The concept of equivalence classes might help flesh out what exactly is isomorphic between two objects.
2. Where else might it be? In the morphic field? :P
3. It’s also worth noting that these concepts served as crucial stepping stones on the way to better theories, even if we didn’t continue using them afterwards. For example, James Clerk Maxwell’s mechanical modeling of the aether led to the development of his famous equations.
4. Well, maybe a neologism could be derived from David Bohm’s verb ‘relevate’, which comes from the adjective ‘relevant’. But it is hard to imagine neuroscientists reporting their findings and models in terms of neural ‘relevates’ or ‘relevators’. More power to you if you attempt to use this is a publication.
5. Instead of debating whether memory is representation (it is), it is more productive to discuss its specific form. Here I argue that the “inscription metaphor” fails to capture a key feature of human memory: the fact that it is content-addressable: Why human memory is not a bit like a computer’s.
6. This includes the relatively small subset of neuroscientists who explicitly posit symbolic representations.
7. It’s easy to construct more elaborate examples from here. Your friend could describe the wallet to you instead of sending a picture. So you use the phrases to construct an imagined representation of the wallet. The “aboutness” becomes a little quirky to identify in the case of imagination, but as long as we root ourselves in the colloquial notion, we will find that “family resemblances” will guide us when the thing being represented is wholly imaginary, like a dragon.
8. There could be more than one, and they may change over time.
9. See this essay of mine: Persons all the way down: On viewing the scientific conception of the self from the inside out.
10. Another topic on which I have written: Me and My Brain: What the “Double-Subject Fallacy” reveals about contemporary conceptions of the Self.
11. In the previous post I pointed out that many of the earliest uses of the representation concept were sensorimotor in emphasis. In the quotations I dug up from the 19th century, the idea that parts of the body have counterparts in the brain, which receive signals from them, was well known and backed by anatomy. Similarly, the idea that muscles were controlled by descending nerve fibres from the brain was also well established. So both sensory and motor emissaries were a common idea.
12. “Information” might work, but that is, if anything, even more closely tied to the world of discrete symbols than “representation” is. You could also argue that the information that one extracts from a representation can vary depending on the context. But perhaps that is an unnecessary distinction, since nothing rules out representations that are themselves context- and goal-dependent. Suffice it to say that an anti-representationalist who uses the word “information” is going to cause a lot of confusion. Also see note 13.
13. The conclusion of this review on vicarious trial and error by David Redish is a good example of standard use of “representation” in neuroscience, and also why keeping it distinct from “information” is often necessary, if only for reasons of exposition: “In his 1935 critique of Tolman’s VTE hypothesis, Guthrie[152] said that: “So far as the theory is concerned the rat is left buried in thought”. We now know that an internal search through a representation can derive information that is hidden within that representation, and thus that a search through a mental schema of the consequences of one’s actions can derive information that can change decisions. Importantly, computation takes time. The search-and-evaluate process that underlies deliberation therefore requires time to complete. […] the term ‘vicarious trial and error’ makes strong assumptions about cognitive and mental states; it implies that rats are actually searching through possibilities, evaluating those possibilities and making decisions that are based on those evaluations. In other words, it implies that rats really are deliberating. The data that have come in over the past decade are clear: Muenzinger, Gentry and Tolman were completely correct — VTE is a process of vicarious (mental) trial and error (search, evaluate and test). Guthrie was also right: when rats pause at choice points, they are indeed “buried in thought”.”
14. In the comments to the previous post, Gualtiero Piccinini linked to a paper written by him and Eric Thomson: Neural Representations Observed. It is very much in line with the argument I am making here. It features a thorough review of the debate about representation, as well as some relevant neural data. Here is a punchy excerpt: “Neural representations are observable, quantifiable, manipulable, and have received multiple independent lines of empirical support. Therefore, neural representations are real—as real as neurons, action potentials, and other entities routinely observed and manipulated in the laboratory.”
15. Only a subset of these would in any case make much sense outside philosophical circles.
16. Mark Bickhard talks about pragmatic representations in this podcast episode. I have not yet found anything published by him so far that uses this exact phrase.
17. The capacity for these interactions is, of course, ‘intrinsic’.
18. I hope to elaborate on why “potential” is important here in a future post, which may also feature dragons.