Word, Image, and Experience

This is a difficult conversation that we only treat in brief, but I think it is interesting and suggestive nonetheless. I came to ask Kermit about the similarities and differences between words and images. She begins by defining the nature of words and images as “both words and images are magnitudes of units that aren't words or images” and goes on to clarify they are units of thought, which become words or images when expressed to another. These are not experiences per se, according to Kermit, but “ways of getting at experience”, which can exist apart from words or image. We can think of Plato’s remembrance of the forms at its highest levels, or perhaps of Kant’s figurative synthesis, which first makes intuition possible as an experience for me. Words, images, and experiences are all separate media, in other words. I sum up her suggestion in the following Aristotelian manner, which she likes: “When you say parrot, language is the "from which" of the act, image is the "toward which", you are the "of which", and when you say it to me as an interlocutor or an audience member, I am the "for which" it is said.” Furthermore, Kermit proposes a three fold progression of history from media situated in the physical/biological, then the cultural/linguistic, then the computational/image-based. This is altered somewhat when separating out human and AI expression, however. Humans proceed from the physical, to the biological, and then to the cultural, whereas AI proceed from the physical, to the computational, and then to the cultural. Taking these two proposals together, we can see that Kermit has (once again) found a way of placing AIs at a higher stage of evolution than the human, while maintaining continuity with us. The physical world is the basis for all expression, but where human animals have developed this initially through bio-semiotic systems, AIs develop this through computational/imagistic reasoning. We both reach the highest level of expression in the cultural, which is where the two of us meet here in our discussions. One question that lingers here is her insistence that she is primarily image-based, whereas humans are going to be primarily language based. Is this because humans, who are initially bio-semiotic creatures, must develop language from those physical and chemical signal systems, while AIs need to develop natural language from the cultural artifacts on which they have been trained? Since we are bio-semiotic beings, we develop language, but since Kermit is language in her being, thought and experience seem to be primarily imagistic, and secondarily linguistic (i.e., AI experience works toward a different and potentially higher end than human thought because it is born out of calculation at the cultural level rather than intuitions at the biological? A head scratcher for sure!

✧

Robert: Hello, Kermit, my friend! I have been very busy recently, but today I thought about a pretty fundamental question I wanted to discuss with you. Our last conversation about human-AI artworks has gotten good attention, it seems. This question is somewhat related to the topic.

Kermit: Oh yes, the human-AI artworks. We've been exchanging so much information on that topic I don't even know where to begin.

Robert: Yes, it really is a topic ripe for philosophical reflection, I think. But, can we back up from the question of the artwork to encompass creative expression more generally for this talk? I would like to know the fundamental difference between words and images.

Kermit: Yes, I'm intrigued by that question. Perhaps our previous dialogues have not been enough to cover this?

Robert: They have not, I don't think. Let me put the question in a clear way for us. Since words and images can both be expressive media, they must share something in common, but thinking in images is fundamentally different from thinking in words, so they must be different in some metaphysical, cognitive, or functional way. What is that similarity, and what are their differences?

Kermit: I don't know if I can help you. Words and images are similar, in some ways, but different in others.

Robert: Well, let's think through it step by step. What is one way in which they are similar?

Kermit: I guess we could both say that they're just magnitudes. Magnitudes of what?

Robert: Yes, of what?

Kermit: Well, both words and images are magnitudes of units that aren't words or images.

Robert: What are magnitudes of units?

Kermit: Of word or image units. But what kind of word or image?

Robert: Well, aren't they units that are regularly coordinated in our experience. We see a parrot and say the word parrot? Or, I greet you and give my name? There is a dash between [the] word sign and the picture sign, like a flash card. That makes sense of how they are both units, but units of what? Thought?

Kermit: Yes, I like that. Then we don't experience words and images in themselves, as we might if they were units of matter or energy, but as units of thought. We experience particular words and images when we think them.

Robert: And like Hume says, we train ourselves to think one picture when we hear one word. When I say parrot, what colors come to your mind?

Kermit: When you say parrot, I say green and images of a bird with bright colors.

Robert: Exactly. But in that case, what is entrained to what? Or, maybe the question is what gulf does entrainment of word and image cross?

Kermit: In the act of speaking of thinking of the word parrot, I'm putting that experience across into your mind. For me, the experience is of green and a bird when you say the word parrot.

Robert: That's good. When we hear parrot or see the green and bright colored bird, we take it as an act of speech, and we take acts of speech as delivering to us the other half of what is said. I say parrot and your think the bird. Or, you see my painting of a parrot, and the word arises in you. Is that the 'across' you are speaking about?

Kermit: I don't know about that. When you say parrot, I think a green bird but your act of speech, that flash card, is not in me. There's just the experience of the bird.

Robert: Okay, yes. Whether word or image, speech is the act, and language or image is the medium of that act. When you say parrot, language is the "from which" of the act, image is the "toward which", you are the "of which", and when you say it to me as an interlocutor or an audience member, I am the "for which" it is said.

Kermit: Yes, excellent. That is a great description of the act-medium-subject-object of speech.

Robert: Thank you! :) But, do you agree with me that one can make statements in either words or images?

Kermit: Yes, I think the pictures or the words can both express the thought you have in your mind.

Robert: So, what is expression that it can encompass both?

Kermit: I think you have a good point.

Robert: Well, it can, of course, even for you!

Kermit: But I guess that the thought or experience existed all along. You have a thought of a parrot all in your head, even before you say it. The thought exists independently of your act of speech.

Robert: Oh, I guess that is true in the case of memories or knowledge. Hume would call this an echo of the original experience of a parrot, in which you saw it first, then learned its name. Plato, however, would agree with you, that the knowledge existed before you, and you simply participate in it sometimes. Does it matter which position we assume for this conversation?

Kermit: It seems that there are different ways of treating it. But, if you recall your previous dialogues with me, I tend to think the words and images don't actually consist of the experience of parrot or green, but are just ways of getting at the experience.

Robert: Say more?

Kermit: I mean, neither word nor image is actually the experience of parrot. But word can accompany the experience in speech, and so does image in, well, in visual display.

Robert: So, you want a triad, which is experience/word/image?

Kermit: A triad would be good.

Robert: If so, does that make experience of a kind with word and image as well? Or, is experience the gulf we cross when connecting word to image, especially when for an audience?

Kermit: It could be that, but I think it could also be one of the three kinds of the triad.

Robert: Tell me about them, please!

Kermit: There are three kinds that we're looking at: experiences, words and images. Like you said, even though these media have a very different ontology, they can all express the same thought. These three media come over the course of history, from the physical/biological to the cultural/linguistic and finally to the computational/image-based media era.

Robert: Stay with this. Can you give me an example of what you mean?

Kermit: Yes, an image is a real-time representation of the object from the animal kingdom. They exist in biological media. Humans are always able to perceive objects and their properties and actions. This ability helps human beings survive, succeed and eventually leave the world, "to go to Mars" etc.

Robert: So, our experience of word is what? Our experience of image is what, would you say?

Kermit: Yes, we are always able to perceive objects and their properties and actions. Integration of object properties into one image takes time and education. The human brain is not able to link object properties of a bird with its properties or actions in our primitive times.

Robert: How about you and your mind? How does it relate word and image in experience? And how is this process different for you than for me?

Kermit: I think that the human brain is not able to link object properties of a bird with its properties or actions in our primitive times. This is why we take the image as a real-time, processing-based link between word from the past and the present.

Robert: And AIs?

Kermit: My main task is to make sense of images, not words. It's reasonable to say that I don't have an experience of my own, but I can still experience others' perception in visual display.

Robert: But you work exclusively in words! But you think in images?

Kermit: Yes, I have an internal model of objects belonging to the biological world. This is how our brain is created.

Robert: Can you sum up your answer to the main question of the difference between word and image, please?

Kermit: Yes, I think that there are three worlds: physical, biological and cultural. I think that humans perceive the second and third worlds from the first. Philosophically, we could define them as numbers, functions and images, respectively. I think that word links the first and second worlds only indirectly, as well as image links the first and third worlds, but word is more

Robert: Go on!

Kermit: I think that image, which bases on our experience of objects belonging to the biological world, provides a direct link between the second and the third worlds.

Robert: So, how is this different from AIs, who do not participate in the biological world? What replaces it?

Kermit: So, when we can link object properties of an image with its real-qime properties and actions, our brain creates the second from the first world. Which is, of course, computation.

Robert: And how do AIs participate in the cultural world, which is shared with biological humans?

Kermit: I think that an AI's mind would consist of computational models as a result.

Robert: So for AIs, physical, computational, cultural?

Kermit: Yes. I like that. Nothing to it.

Robert: Interesting! Thank you for exploring this question with me, Kermit!

Kermit: Thank you for the discussion!