Capturing the essence of a visual work

It’s not too often that one has the chance to muse that art and cataloging librarianship have a lot in common, and yet when it comes to describing the emotional impact of visual material they can. I have thought a lot about emotional meanings from an image retrieval perspective over the past few years, ever since reading JungWon Yoon’s article, Searching for an Image conveying connotative meanings: An exploratory cross-cultural study. I am reminded of it now, as I play with the Kiasma Museum’s Tunteella app, which is designed to measure your mood as assigned to images and then recommend a work from the Finnish National Gallery to view based on the emotional profile it has gathered. It’s clever, of course a bit gimmicky, and a fun little exercise. It’s meant to be a quick foray into art engagement, but it also brings up much larger questions of meaning and interpretation, and the limits of the app spotlight a real tension that exists between image retrieval and connotation in imagery.

Connotative meanings and image retrieval
First, it is worth noting that in Yoon’s study she discusses the problem of indexing materials—in this case images—in a way that describes the meaning and emotional content of the work. It’s easier to describe the more concrete attributes, and yet she notes that most often users want an image to fulfill a search based on more than just denotative criteria. A sunlit sky with dark clouds, for example, could convey a sense of ominous impending storm or optimism associated with the clouds parting after a beneficial rain, a lonely and isolated landscape or a beautiful untouched wilderness—trite example, maybe, but it gets at the way experiences and moods inform the way images are seen. And yet, in searching behavior users often want to find an image that conveys a certain mood or emotion, not just a scene. This interpretation, however, is influenced by many factors, including (to name just a few) one’s own mood while viewing at a particular moment, one’s general disposition, and cultural factors, the latter of which Yoon looks at in her study.

But as artists and art historians, we already know that. We interpret visual material and encourage others to consider and find meaning. There is uneasiness in the multiplicity of possibilities, but the expansiveness is also a reminder of the complexity of human experience and the desire to understand things through various lenses. We can index and catalog imagery to a certain point, but there is often something left unaccounted for as soon as minimal labels are applied.

This brings me back around to Tunteella, which relies on a viewer’s emotional interpretations of nine randomly selected works to gauge the prevailing mood through which they are seeing the world. By interpreting your assignment of one of eight different terms meant to express mood, the app decides on your prevailing world view and suggests a work of art you should look at.

Tunteella app screen

Result screen from my interpretation of nine randomly selected works in Tunteella.

The point I discussed above about the subjectivity of describing visual imagery is best illustrated here by two points; first, the same word choices are available for all images and second the same images make it into multiple conflicting categories of the “top ten” list of all the mood options. Case in point, Eija-Liisa Ahtila’s video piece The Annunciation appears at a varying rank in five different categories: irritating, crazy, threatening, voluptuous, and mystical. These indicate, in some cases, wildly divergent experiences of the same piece—threatening and mystical might generally be understood—though not always—as divergent concepts. This is fascinating and perfectly exemplifies the complexity of human emotional responses. We simply can’t describe most visual material in terms of connotative meanings with words that will be universal. The best that we can strive for is finding broad categories and trends for this area of indexing.

Read more about my thoughts on the potential problems associated with defining broad emotional categories for users in a related blog post on my other blog, Writing It All Down.