The modalities of communication – speech, song, writing, images, etc. – constitute general registers of human expression and experience. As such, they lend themselves to different forms of technological mediation, both individually and in combination as multimodal communication.
The various modalities are at once biologically conditioned and culturally shaped. On the one hand, the human senses enable people to identify particular affordances (Gibson 1979) or possibilities for action in their environment. On the other hand, representations of this environment are realized in culturally conventional forms, in embodied as well as technologically mediated communication. Such representations, further, come to constitute affordances in their own right. Research on, for example, computer-mediated communication has examined media as cultural artifacts with specific affordances for social interaction (Norman 1990). Hutchby (2001) identified distinctive affordances of interaction that takes place either face-to-face, via telephone, or in online chat. In a perspective of media and communication theory, the modalities can be considered potentials of the human senses for meaningful discourse in and about the natural and cultural environment – potentials that are extended through information and communication technologies.
While everyday language, like Aristotle, distinguishes five main senses – sight, hearing, smell, taste, and touch – communication research has focused special attention on aspects of sight and hearing in social interaction, even while recognizing, for example, the role of touch in interpersonal communication and experiments integrating smell into cinema (“odorama”). A key issue is how to typologize modalities in relation to sight and hearing: how many modalities are there? With reference to face-to-face interaction, Stivers and Sidnell (2005) employed a main distinction between a vocal–aural and a visuospatial modality, sub-dividing the latter into gesture, gaze, and body posture and the former into lexico-syntactic, vocal, and prosodic channels, because each of these could be said to make a difference in the process and outcome of communication. From a perspective of technologically mediated communication, Kress and Van Leeuwen rather suggested that language as such is a modality or mode, “because it can be realised either as speech or as writing,” adding that writing is itself a semiotic mode that can be realized as “engraving in stone [or] print on glossy paper” (Kress & Van Leeuwen 2001, 6).
In a historical perspective, modalities can be understood as an intermediate category between the material vehicles of communication – human bodies and information technologies – and its discursive vehicles – texts or messages. Technologies – from writing to the Internet – have separated and recombined modalities in specific ways. As brought home by McLuhan (1964), the material and modal forms of messages affect their interpreted contents. In an oral culture, face-to-face communication is, by definition, multimodal, involving several sensory registers at once. Writing and print focus the vision of the individual reader into a communicative gaze on texts, although reading aloud and subvocal reading persist. Radio reproduces a different, secondary orality (Ong 1982), and simultaneously integrates multiple modalities of language, music, and soundscapes. Cinema and television, in public and private contexts of reception, recreate a rich experience of natural and social settings, and of other humans, in sound as well as vision. Digital media, such as online virtual worlds, arguably, take multimodal communication to the next degree, simulating features of face-to-face interaction.
While it may, thus, be argued that communication history has witnessed a trajectory from multimodal, via tendentially monomodal, and back toward multimodal media, the configuration of modalities in any given period has been complex, both within the available technological media – for instance, illustrated books and magazines – and in the multistep flows involving such media and face-to-face communication (Katz & Lazarsfeld 1955). All along, language has been a master modality of sorts, providing both a model for other communications and the means by which all other modalities could be redescribed and explained. As noted by Émile Benveniste (1969/1985, 236), “the signs of society can be interpreted integrally by those of language, but the reverse is not so.” Also in this regard, language can be considered a key component of individual psychological development as well as of self-reflective cultures.
The concept of modalities remains relatively undeveloped in media and communication research. The coming of more explicitly and emphatically multimodal digital media technologies may be an opportune moment to reconsider the definition and operationalization of modality, also in historical studies of analog media. A renewed round of sound media studies has raised questions, for instance, of whether music and speech should be considered media, or modalities that are realized in different media (Cook 1998; Jensen 2006). In retrospect, “new” multimodal media can throw light on “old” media, their aspects and degrees of monoand multimodality.
- Benveniste, É. (1985). The semiology of language. In R. E. Innis (ed.), Semiotics: An introductory anthology. Bloomington, IN: Indiana University Press, pp. 228 –246. (Original work published 1969).
- Cook, N. (1998). Analyzing musical multimedia. Oxford: Oxford University Press.
- Gibson, J. J. (1979). The ecological approach to visual perception. Boston, MA: Houghton Mifflin.
- Hutchby, I. (2001). Conversation and technology: From the telephone to the Internet. Cambridge: Polity.
- Jensen, K. B. (2006). Sounding the media: An interdisciplinary review and a research agenda for digital sound studies. Nordicom Review, 27(2), 7–33.
- Katz, E., & Lazarsfeld, P. F. (1955). Personal influence. Glencoe, IL: Free Press.
- Kress, G., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold.
- McLuhan, M. (1964). Understanding media. New York: McGraw-Hill.
- Norman, D. A. (1990). The design of everyday things. New York: Doubleday.
- Ong, W. (1982). Orality and literacy. London: Methuen.
- Stivers, T., & Sidnell, J. (2005). Introduction: Multimodal interaction. Semiotica, 156(1/4), 1– 20.
Back to Communication Theory and Philosophy.