One of the distinctive characteristics of our species consists in the creation of fictional worlds, parallel to the world of physical reality. Humans seem to have always been dissatisfied with the prosaic nature of the everyday world, inventing fictional realms through the production of images and the creation of stories. Creating and watching images, telling and listening to stories are essential ingredients to account for the “diversity” of Homo sapiens, when compared to other representatives of the animal kingdom. As argued by scholars coming from different fields of research, such as Brian Boyd (2009), Jonathan Gottschall (2012), or Michele Cometa (2018), narrative behavior has given shape to and conditioned the development of the cognitive abilities of Homo sapiens.
Cognitive neuroscience can provide new heuristic tools linking nature and culture through the empirical investigation of the brain-body mechanisms supporting both human creative processes and the reception of their results. By studying the brain-body in relation to cultural artifacts and their reception, we can better understand the building blocks of what makes us humans.
Neuroscientific research in the last thirty years has newly defined the themes of the subject and intersubjectivity, and in more general terms has given us a new conception of the so-called human mind, putting the body and emotions at the center of the debate. One of the most interesting contributions of neuroscience has been the discovery of the cognitive role of the cortical motor system. The teleological dimension of behavior, made up of actions directed toward the accomplishment of goals, is an integral part of the functional properties of the motor system, in close coupling with the brain centers that mediate appetitive and defensive behaviors, reward, and emotions.
An important aspect of the cognitive role of the motor system is its involvement in perception: premotor and parietal areas contain motor neurons that also respond to visual, auditory, and tactile inputs, in the absence of any movement. In this “perceptual” modality, the cortical motor system is partly active, but movement is not produced: it is only simulated. Similarly, the motor system activates during mental motor imagery (for a review, see Gallese 2014; Gallese and Guerra 2020).
This logic can be applied to film reception, as several theories have proposed and suggested since the very beginning of film history. When watching films, what we see is not only the simple “visual” recording in our brain of what stands in front of our eyes—that is, images moving on a screen—but the result of a complex construction whose outcome requires the fundamental contribution of our body with its motor potentialities, our senses and emotions, our imagination, and our memories. Neuroscience has swept away the “oculocentric” concept of “purely visual” vision, empirically grounding a new model of visual perception. Vision is a complex experience, intrinsically synesthetic, that is, made of attributes that largely exceed the mere transposition in visual coordinates of what we experience any time we lay our eyes on something. The expression “laying the eyes on” indeed betrays the haptic quality of vision: our eyes are not just optical instruments but are also “hands” touching and exploring the visible, turning it into something seen by someone (Gallese 2018: 77). In fact, perception cannot be thought of as the mere direct recording of what is in the world. Computational machines do that. Perception in animals, however, is always accompanied by the experience of emotions, and it is from the intermingling of perception and emotions that we can explain most of our behaviors. This has profound consequences for our relationship with cultural artifacts, like the experience of the moving images of film.
Indeed, the relationship between film and neuroscience has been recently addressed (Gallese and Guerra 2020) by focusing on three key aspects. The first one has to do with the contribution that the neuroscientific data and research methodologies can provide to film theory and philosophy: neuroscience can shed new light on neglected theories and approaches, like those related to embodiment and motor cognition already present within the debate on film since the 1910s, or it can update and renew pioneering paradigms of research, like that of French filmology (D'Aloia and Eugeni 2014). In every field, the theoretical approach to neuroscientific data and evidence supports the need of a “critical neuroscience,” non-reductionist and non-reifying, bridging the long perspective of the aesthetic reflection across the arts and within disciplines and the biocultural insights inspired by the study of our brain-body.
It is, for instance, quite interesting to notice that the most relevant and durable conception of cinematic storytelling—namely, continuity editing—was theorized in the 1910s in reference to “the laws of natural movement and action” (Phillips 1914), stressing the idea that the human body and motor cognition play a role not only in film reception, but also in the construction of its style and affordance, very similarly to what Sergei Eisenstein would theorize in his writings and movies (Belodubrovskaya 2018; Hedberg Olenina 2021) from the 1920s to 1940s. A motor approach to film style and perception characterized the scientific approach of a physician like Edouard Toulouse, who observed—in the 1910s and 1920s—that “motor suggestibility” should have been considered the strongest and most essential source of the sense of reality elicited by moving images (Morel 2010), and the same belief has been passed on through the writings of Merleau-Ponty (2011) and Kracauer (1960) on film. A confrontation with the tradition of film studies and theory is the only way to include cognitive neuroscience within a serious debate on film aesthetics, escaping the two main risks implied by a “neuromaniac” tendency: on the one hand, a reductionist and “de-humanizing” approach to humanities and, in our case, film studies; on the other hand, the transformation of “neurofilmology” into another apparatus theory, which would be a defeat after the season of post-theory.
To us, cognitive neuroscience is a way to underpin the human experience of art and to favor a multilayered analysis and study of film reception capable of creating the space for new thinking about basic apprehension and aesthetic appreciation of film (Fingerhut 2020). The brain-body, as investigated by neuroscience, becomes the place where it is possible to re-discuss the viewer as a biological organism that reacts to very specific stimuli, as a body capable of establishing new intersubjective relationships, and eventually as a human being who can live a complete experience in front of a work of art.
The second aspect has to do with the empirical nature of neurocognitive film studies, enabling the investigation of film reception, focusing on the impact of different filmic solutions on spectators’ brain-body, correlating the sub-personal level of description, dealing with neurophysiological mechanisms and bodily responses to the way we experience film. As we have demonstrated in our works (Kaltwasser et al. 2019; Heimann et al. 2019; Calbi et al. 2019; Gallese 2018; Heimann et al. 2017; Heimann et al. 2014), this aspect contributes to complete film analyses also from a stylistic and historical perspective, contrasting the idea that a neurocognitive approach would be ahistorical (Onians 2007). The debate around experiments on film is sometimes quite hot. Despite the delimitation of the field for empirical investigation (camera movements, editing, cinematic shot-scales, affective responses, etc.), a lab is still not a movie theatre, and participants in the experiment are not members of a film audience. The whole experimental context aims at mimicking those conditions in the best possible way, but it remains true that experimental conditions are nothing but an oversimplified and coarse replica of real life situations. And yet, this does not apply just to our experimental approach to cinema, but to cognitive neuroscience at large.
After Uri Hasson's pioneering works, the literature on neurocognitive film experiments has been increasing in the last decade (e.g., see Raz et al. 2014; Lankinen et al. 2018; Schmälzle and Grall 2020; Sun et al. 2022), and a group pursuing research on neurohumanities—consisting of neuroscientists, humanists, and data analysts—are shaping new models to widen and strengthen the common area between humanities and science. We have included some good examples in this issue as well.
The third aspect aims to reflect on the relocation of film and the transformation of experience due to the new digital devices (Casetti 2015). The Lumières’ claim, according to which the most relevant cinematic revolution was bringing “the world within our reach,” is becoming ad litteram a new filmic condition, and the role of our brain-body must be profoundly rethought in relation to the new devices and the new forms of cinematic relocation. An interesting and challenging path of research within neurocognitive film and media studies must cope with the virtual life of film, the new devices of representation and mediation, and the new forms of intersubjectivity they imply (Gallese and Guerra 2018). In particular, a fertile ground for future transdisciplinary research is the role played by the ever-growing presence of portable devices, like smart phones and tablets, in our media-sphere. Indeed, a new proxemics of the audio-visual experience of film is brought along by mobile digital dispositives: images and sounds are perceived exclusively within peripersonal space, being literally at hand. Furthermore, touch-screens introduce a new agent-driven quality into the digital experience of film, bringing along a performative quality of perception: the subject of the filmic experience also becomes the active agent of its delivery. The impact of these new visual practices on film reception is practically unknown and as yet poorly investigated (see Ackerman, Grespi and Pinotti 2020).
What Embodiment?
The debate around embodiment and the arts is still at the heart of such a way of thinking. For a significant part of the twentieth century, classic cognitive science conceived human cognition as the outcome of a representational and computational mind, which manipulates amodal symbols (Fodor 1975, 1983; Pylyshyn 1984). This model has been polemically caricatured as the “sandwich model” (Hurley 1998, 2008), where cognition stands for the juicy filling, with action and perception as the two peripheral and unconnected slices of bread. For many years, and partly also today, neuroscience has endorsed this model, trying to locate in the human brain a variety of cognitive modules supposedly related to different cognitive functions, like language and mindreading. Such an approach suffers from ontological reductionism because it reifies human subjectivity and intersubjectivity within a mass of neurons variously distributed in the brain. This ontological reductionism chooses as its level of description the activation of segregated cerebral areas or, at best, the activation of circuits that connect different areas and regions of the brain. However, the competence of understanding others—real people as well as fictional characters—is uniquely describable at the personal level, and therefore is not entirely reducible to the sub-personal activation of neural networks in the brain that are hypothetically specialized in mindreading, as too many neuroscientists nowadays think. Indeed, neurons are not epistemic agents.
In the final part of the last century, however, empirical research in neuroscience and the cognitive sciences and a series of theoretical proposals radically changed our view about the mind and its relationship to the body: the so-called “embodied cognition paradigm.” Two books greatly contributed in different ways to undermining the theoretical tenets of classic cognitivism: Metaphors We Live By, by George Lakoff and Mark Johnson (1980), and The Embodied Mind, by Francisco Varela, Evan Thompson, and Eleanor Rosch (1991). The first one introduced the notion of conceptual metaphors, holding that humans’ abstract thought and conceptual system rely on the use of metaphors, which map onto the body. The second one, by relying on the contribution of phenomenology, gave great emphasis to the thus far neglected dimension of experience.
Goldman and de Vignemont (2009) provided a very useful taxonomy of the different notions of embodiment: “embodied” means that body parts, bodily actions, or body representations play a crucial role in cognition. Body representations, however, can be interpreted in terms of mental representations, either with a bodily content (representations of the body) or with a bodily format. The theory of embodied simulation chooses the second option, namely that the bodily format of a mental representation constrains what such mental representation can map, because of the bodily constraints posed by the specific configuration of the human body. In other words, neurons and brain circuits function the way they do just because they are connected to the body (Gallese 2016).
Embodiment and Film Reception
Neuroscientific experiments, such as those described in The Empathic Screen (Gallese and Guerra 2020), aim to shed light upon the contribution of the brain-body to our relationship with films. As we wrote in the book, “What appears on the screen, the ways and means, the strategies of appearance in cinema are the result of the direct relationship our perceptual system has with technology and are influenced by production processes and needs that consolidate the social aspects” (Gallese and Guerra 2020, 89). Our experiments provide a preliminary and partial naturalized account of what it means to experience a film. Many more elements are at stake, like music, lighting, the script, actors’ performances, etc. All these elements, however, can be connected to embodiment. Let's focus briefly, as an example, on spectators’ engagement with the fictional characters. A recent fMRI study (Broom et al. 2021) investigated whether identification with fictional characters is associated with increased neural overlap between the self and fictional others. These authors selected fans of a famous HBO TV series and asked them to perform trait evaluations for the self, nine real-world friends, and nine fictional characters during functional neuroimaging. The results showed a larger response for self than for friends and fictional others in the ventro-medial prefrontal cortex (vMPFC), a brain region normally activated when thinking about the self or close others. Furthermore, the participants higher in trait identification, the tendency to inhabit the first-person experiences of fictional characters during narrative engagement, showed greater neural overlap in the vMPFC between self and fictional characters; the magnitude of this correlation was greater for the fictional characters that participants felt closest to/liked the most as compared to those they felt least close to/liked the least. Another interesting result of this study was that midline brain regions belonging to the so-called Default Mode Network (DMN), implicated both in self-referential and social cognition, and a supposedly mindreading-related area, the temporo-parietal junction (TPJ) responded more strongly to the self than to close friends and fictional characters. These results show that self and others—both real and fictional ones—activate overlapping brain networks, showing the reuse rule of embodied simulation, with different intensities, the strongest being related to the self. We hypothesize that the systematic activation of these brain regions during mindreading tasks does not depend upon the fact that they contain mindreading-specific neurons, but because self-other differentiation at the bodily level is a necessary ingredient of any mentalizing activity, when targeting both real persons and fictional characters on film (see Ammaniti and Gallese 2014).
The Issue
The goal of this issue is to offer an overview of the lines of research inspired by cognitive neuroscience and to elicit a wider, informed debate. The three possible paradigms we briefly discussed in this introduction are basically present in the different approaches adopted in the following articles: a historical and theoretical one (Fingerhut and Heimann, Kolesnikov, Tikka), a philosophical interpretation of the relationship between neuroscience and the arts (Smith, Seeley), and a more committed approach in the empirical study of moving images and their effects on the audience (Cabañas et al., Levin et al.).
References
Ackerman, Ada, Barbara Grespi, and Andrea Pinotti, eds. 2020. “Mediatic Handology: Shaping Images, Interacting, Magicking,” special issue, Cinéma & Cie. International Film Studies Journal 20 (35).
Ammaniti, Massimo, and Vittorio Gallese. 2014. The Birth of Intersubjectivity: Psychodynamics, Neurobiology and the Self. New York: W. W. Norton & Company.
Belodubrovskaya, Maria. 2018. “The Cine-Fist: Eisenstein's Attractions, Mirror Neurons, and Contemporary Action Cinema.” Projections 12 (1): 1–18. https://doi.org/10.3167/proj.2018.120102
Boyd, Brian. 2009. On the Origin of Stories: Evolution, Cognition, and Fiction. Cambridge, MA: Belknap Press.
Broom, Timothy W., Robert S. Chavez, and Dylan D. Wagner. 2021. “Becoming the King in the North: Identification with Fictional Characters is Associated with Greater Self–Other Neural Overlap.” Social Cognitive and Affective Neuroscience 16 (6): 541–551. https://doi.org/10.1093/scan/nsab021
Calbi, Marta, Francesca Siri, Katrin Heimann, Daniel Barratt, Vittorio Gallese, Anna Kolesnikov, Maria Alessandra Umiltà. 2019. “How Context Influences the Interpretations of Facial Expressions: A Source Localization High-Density EEG Study on the Kuleshov Effect.” Scientific Reports 9 (1). doi: https://doi.org/10.1038/s41598-018-37786-y.
Casetti, Francesco. 2015. The Lumière Galaxy. New York: Columbia University Press.
Cometa, Michele. 2018. Perché le Storie ci Aiutano a Vivere: La letteratura necessaria. Milano: Raffaello Cortina Editore.
D'Aloia, Adriano, and Ruggero Eugeni, eds. 2014. “Neurofilmology. Audiovisual Studies and the Challenge of Neuroscience,” special issue, Cinéma & Cie. International Film Studies Journal 14 (22-23).
Fingerhut, Joerg. 2020. “Twofoldness in Moving Images: the Philosophy and Neuroscience of Filmic Experience.” Projections 14 (3): 1–20. https://doi.org/10.3167/proj.2020.140302
Fodor, Jerry A. 1983. The Modularity of Mind: An Essay on Faculty Psychology. Cambridge, MA: MIT Press.
Fodor, Jerry A. 1975. The Language of Thought. Cambridge, MA: Harvard University Press.
Gallese, Vittorio. 2014. “Bodily Selves in Relation: Embodied Simulation as Second-Person Perspective on Intersubjectivity.” Philos Trans R Soc Lond B Biol Sci. 369 (1644): 20130177. https://doi.org/10.1098/rstb.2013.0177.
Gallese Vittorio. 2016. “Finding the Body in the Brain: From Simulation Theory to Embodied Simulation.” In Alvin Goldman and His Critics, ed. Hilary Kornblith and Brian P. McLaughlin, 297–317. New York: Blackwell.
Gallese, Vittorio. 2018. “The Power of Images: A View from the Brain-Body.” Phenomenology and Mind (14): 70–79. https://doi.org/10.13128/Phe_Mi-23626.
Gallese, Vittorio, and Michele Guerra. 2018. “L'empathie d'une machine.” In Des Pouvoirs des Écrans, ed. Mauro Carbone, Jacopo G. Bodini, and Anna C. Dalmasso. Paris: Mimesis International.
Gallese, Vittorio, and Michele Guerra. 2020. The Empathic Screen: Cinema and Neuroscience. Trans. Frances Anderson. Oxford: Oxford University Press.
Goldman, Alvin, and Federique de Vignemont. 2009. “Is social cognition embodied?” Trends Cogn Sci. 13 (4): 154–159. https://doi.org/10.1016/j.tics.2009.01.007
Gottschall, Jonathan. 2012. The Storytelling Animal: How Stories Make Us Human. New York: Houghton Mifflin Harcourt Publishing.
Hedberg Olenina, Ana. 2021. “Sergei Eisenstein, Neurocinematics, and Embodied Cogniition: a Reassessment.” Discourse 43 (3): 351–382. https://doi.org/10.13110/discourse.43.3.0351
Heimann, Katrin, Maria Alessandra Umiltà, Michele Guerra, and Vittorio Gallese. 2014. “Moving Mirrors: A High-Density EEG Study Investigating the Effects of Camera Movements on Motor Cortex Activation during Action Observation.” Journal of Cognitive Neuroscience 26 (9): 2087–2102. https://doi.org/10.1162/jocn_a_00602
Heimann, Katrin, Sebo Uithol, Marta Calbi, Maria Alessandra Umiltà, Michele Guerra, and Vittorio Gallese. 2017. “Cuts in Action: A High-Density EEG Study Investigating the Neural Correlates of Different Editing Techniques in Film.” Cognitive Science 41 (6): 1555–1588. https://doi.org/10.1111/cogs.12439
Heimann, Katrin, Sebo Uithol, Marta Calbi, Maria Alessandra Umiltà, Michele Guerra, and Vittorio Gallese. 2019. “Embodying the Camera: An EEG Study on the Effects of Camera Movements on Film Spectators’ Sensorimotor Cortex Activation.” Plos One 14 (3): e0211026. https://doi.org/10.1371/journal.pone.0211026.
Hurley, Susan L. 1998. Consciousness in Action. Cambridge, MA: Harvard University Press.
Hurley, Susan L. 2008. “Understanding Simulation.” Philosophy and Phenomenological Research 77 (3): 755–774. https://doi.org/10.1111/j.1933-1592.2008.00220.x
Kaltwasser, Laura, Nicolas Rost, Martina Ardizzi, Marta Calbi, Luca Settembrino, Jeorg Fingerhut, Michael Pauen, and Vittorio Gallese. 2019. “Sharing the Filmic Experience. The Physiology of Socio-Emotional Processes in the Cinema.” Plos One, 14 (10): e0223259. https://doi.org/10.1371/journal.pone.0223259.
Kracauer, Siegfried. 1960. Theory of Film: The Redemption of Physical Reality. Princeton, NJ: Princeton University Press.
Lakoff, George, and Mark Johnson. 1980. Metaphors We Live By. Chicago: Chicago University Press.
Lankinen, Kaisu, Jukka Saari, Yevehen Hlushchuk, Pia Tikka, Lauri Parkkonen, Piita Hari, and Miika Koskinen. 2018. “Consistency and Similarity of MEG- and fMRI-signal Time Courses during Movie Viewing.” Neuroimage 15 (173): 361–369. https://doi.org/10.1016/j.neuroimage.2018.02.045
Merleau-Ponty, Maurice. 2011. Le monde sensible et le monde de l'expression. Cours au Collège de France. Notes, 1953. Geneva: MetisPresses.
Morel, Jean-Paul. 2010. “Le Docteur Toulouse ou le cinéma vu par un psycho-physiologiste (1912-1928).” 1895 60: 122–155. https://doi.org/10.4000/1895.3874.
Onians, John. 2007. Neuroarthistory. New Haven, CT: Yale University Press.
Phillips, Henry A. 1914. The Photodrama. New York: The Stanhope-Dodge Publishing Company.
Pylyshyn, Zenon W. 1984. Computation and Cognition: Toward a Foundation for Cognitive Science. Cambridge, MA: MIT Press.
Raz, Gal, Yael Jacob, Tal Gonen, Yonatan Winetraub, Tamar Flash, Eyal Soreq, and Talma Hendler. 2014. “Cry for Her or Cry with Her: Context-Dependent Dissociation of Two Modes of Cinematic Empathy Reflected in Network Cohesion Dynamics.” Social Cognitive and Affective Neuroscience 9 (1): 30–38. https://doi.org/10.1093/scan/nst052
Schmälzle, Ralf, and Clare Grall. 2020. “The Coupled Brains of Captivated Audiences: An Investigation of the Collective Brain Dynamics of an Audience Watching a Suspenseful Film.” Journal of Media Psychology: Theories, Methods, and Applications 32 (4): 187–199. http://dx.doi.org/10.1027/1864-1105/a000271
Sun, Yadi, Junji Ma, Miner Huang, Yangyang Yi, Yiheng Wang, Yue Gu, Ying Lin, Liman Man Wai Li, and Zhengjia Dai. 2022. “Functional Connectivity Dynamics as a Function of the Fluctuation of Tension during Film Watching.” Brain Imaging and Behavior. https://doi.org/10.1007/s11682-021-00593-7.
Varela, Francisco J., Evan Thompson, and Eleanor Rosch. 1991. The Embodied Mind: Cognitive Science and Human Experience. Boston: MIT Press.