expression and content does not cross each other in time and/or space, and they are perceived to be of different natures (Sonesson, in press a). The nature of a picture, as a physical object, and that which it depicts is just very different, especially if you have a developed concept of “picture.”
In terms of knowledge, or expectations, differentiation in the iconic sign relation can thus be many things, which is the reason I have used the term “sufficiently dif-ferentiated” in the definition of pictorial mode in section 1.4. Differentiation can not only be several things, but different for different media. For example in photo-graphs, knowledge of displacement is a central cue for differentiation: the point of view that there is a space and/or time difference between the occurrence of the refer-ent and its photographic counterpart. For constructed pictures the plain knowledge that they are constructed can be something that helps with differentiation. Their visual deviation from reality is another cue, as is the flatness of most pictorial dis-plays, etc. All that one learns about pictures potentially helps with differentiation.
However, one can only learn about pictures, as a category, due to differentiation in the first place. But this initial differentiation might not be enough. Sufficient, or proper, differentiation is a necessity for reference to be possible.
Because differentiation is not an all or none thing, pictures, even when seen as pic-tures, can contain properties of the real. This is for example evident in children who perform “realist errors” (e.g. Thomas et al., 1999; “false photographs”: Zaitchik, 1990), which is when children believe that a referent can continue to affect a picture after creation of the picture. If the referent updates, so does the picture, they reason.
There is enough differentiation to allow a sign function, but not enough to preclude misconceptions about the relation between pictures and their referents.
As adults we are not immune to degrees of differentiation. Attempts have been made to pinpoint what it is that determine, for any given image, whether adults rea-son about the referent of the picture, or the picture as the referent (Schwartz, 1995).
Realism versus schematics has proven to be a promising candidate. This is likely due to retained properties of the real in the latter case.
To experience a retained reality perspective on pictures, despite fully fledged ref-erential competence, try to tear up a photograph of a loved one and note your feel-ings. Or if you are afraid of snakes you probably know already in which books not to browse. A picture can very well have properties of the real when differentiation falters. This suggests that differentiation is partly connected to inhibitory functions.
For some pictures, or content, we are good at inhibiting our reality responses, most likely not even noticing that they are there, while others break the dams, so to speak.
The opposite case is of course also possible. People can turn the very real into non-reality, as when watching a gruesome splatter movie and react with amusement in-stead of disgust. So while parts of our systems treat pictorial stimuli as real, other parts “know” what pictures are and are not. In e.g. Slater et al. (2006) the reality part got the upper hand when subjects reacted with subjective, behavioural and physiological stress when having to administer electric shocks to an animated figure in a virtual reality setting. Subjects that did not have to see the animation but
inter-acted with it through a text based interface did not experience such stress. The for-mer group reacted despite full knowledge of the artificiality of the situation.31
In order to explain why some animals and children act towards pictures as if they were real, even though they can separate expression and content per se, i.e. discrimi-nate pictures from reality, we must invoke a third entity. There seems to be a second necessary differentiation, that between the sign (i.e. expression and content), and that entity in the real or imagined world that appears as content within the sign; i.e.
the referent. The referent is present in the sign, as content, but is not really there. It is the content which is perceived as actually “being there.” If this experience is con-fused with the referent, the sign appreciation has also failed.
At first glance it might seem odd to distinguish between content and referent.
After all, most of the time we do not seem to bother to connect a picture to a spe-cific object in the outside world. However, that there is a need to separate content from referent is perhaps more apparent when we look at words, signs that are grounded on conventionality. “Fox” and “räv” (Swedish) are different expressions that result in the same content. But let us say that “fox” is exchanged for “vixen,”
where Swedish does not have a separate word for a female fox. Then, for the English speaker, content would change with the expression used. However, the English speaker and the Swedish speaker can still talk about the very same vixen/räv (fox), i.e. referent. Gender is a property of the referent, but is only a part of the content, given the expression, in one of the two languages. Similarly for the iconic sign, all drawings of foxes, although very different, captures the content “fox” (otherwise they would for example be dog drawings), but not all drawings of foxes can necessar-ily refer to the same referent fox.32
But there are variations in expression that do indeed affect the content and con-sequently the sign’s possible referent. A fox can be drawn in a way that someone interprets as a specific fox, perhaps a dog, or cannot recognise as a depiction at all.
The properties of the expression which is necessary for designation of a content is called form in the semiotic tradition of Ferdinand de Sausssure, and that which is redundant is named substance. Form and substance exist on both the expression and the content side of the sign. They are separated by what is called the principle of relevance, which defines which properties of the expression that are crucial for a cer-tain content to be expressed, and also which properties of the content that can vary with expression. Putting a horn in the forehead of a horse changes the content to a unicorn. A horn in the forehead is thus form. For example the length, within certain boundaries, of this horn is substance. A unicorn with its head in the bushes can turn back into a horse, given that we did not know anything about the context that hinted that this was in fact a unicorn with its head in the bushes. Such knowledge is on the level of the referent. The principle of relevance is thus dynamic and sensitive to the dictations of reference. The same feature can be form or substance in different contexts.
31 The social pressure stemming from “being bad” in a scientific experiment was arguably equivalent between the two conditions.
32 It is clear from this example that content is closely connected to what is sometimes called catego-ries in cognitive science.
Reference is indeed context. The very same circle (content) made out of lines (ex-pression) on a surface, can be many things (referent), depending on context. It can be a table from above, a ball, a hole, the letter O, etc. Context can take the form of additional elements, such as a plate and fork on the table, or a shadow under the ball. Context can furthermore be the location where a marked surface appears, such as being in a written book, in a picture book, on a road sign, and so forth. Context can also be what we knew about the specific circle from before, for example that it was a table in the picture we saw on the previous page in a picture book, or what people tell us it is through its appearance in communication. It is no wonder that Peirce called the establishing of reference “interpretant,” and that there are several kinds, which can be internal or external to the mind (Deacon, 1997). To conclude, reference can perhaps be seen as the specification of content due to context.
In an experimental setting, which we will see many examples of later in this text, the task at hand potentially set up context differently. This can have crucial impact on the perceived content of a picture. While a picture might be recognisable in e.g. a matching task, it may remain non-identifiable in e.g. a free response task.
Different types of signs designate their referents in different ways. A word, which has an arbitrary relationship between expression and content, relies heavily on con-text that is external to the sign as such. A picture, on the other hand, in virtue of being an iconic sign, can often specify referents with the context in the sign itself.
There is a type of overlap between referent and content in an iconic sign that cannot be found in indexical or conventional signs. Consequently there are interpretive mis-takes that are specific for pictures, such as mistaking picture for referent and act out on it. One seldom acts out on words, as sound waves or ink on paper, and when getting angry at the squirrel that steals food from the bird table one does not throw stones at the paw prints it has left in the snow.
The trinity of expression, content and referent is closely connected to the notions of surface, reality, and pictorial mode of picture processing. One could say that the focus of attention in surface mode is on the expression side of the potential sign, which is the reason recognition fails. In reality mode attention is caught in the con-tent of the picture, but never moves beyond this. In a pictorial mode, atcon-tention is not only on the content, but also on the referent.33