Strikingly Low Agreement in the Appraisal of Motion Pictures

in Projections
View More View Less
  • 1 New York University pascal.wallisch@nyu.edu
  • 2 Rutgers University—Newark jake.whritner@rutgers.edu
Restricted access

Abstract

Neuroimaging research suggests that watching a movie synchronizes brain activity between observers. This is surprising in light of anecdotal reports that viewers construct their experience radically differently, consistent with contemporary cognitive media theory. This article empirically tests the degree of agreement in the appraisal of commercially produced major motion pictures. Ratings for more than two hundred carefully selected movies were solicited from a diverse pool of more than three thousand study participants. Doing so shows that intersubjective movie appraisal is strikingly low but significantly different from zero. The article also shows that these ratings correlate only weakly with the judgment of professional movie critics. Taken together, this study supports the notion that movies are an extremely rich, highly dimensional narrative stimulus with many degrees of freedom for viewers to construct their subjective experience in a highly idiosyncratic fashion.

Contributor Notes

Pascal Wallisch, PhD, serves as a clinical assistant professor of psychology at New York University. His research interests concern motion perception, generally conceived, for example, its neural basis, its attentional impact, and its appraisal. His work was recognized with a variety of honors, including the Wayne C. Booth Prize and Golden Dozen Award for Excellence in Teaching, as well as the First Eagleman Prize. He also was a recipient of a German National Merit Scholarship.

Jake Alden Whritner is a laboratory/tech assistant at the Cognitive and Data Science (CoDaS) Lab at the Rutgers University-Newark, where he works on a visual perception project that uses augmented reality to study and train the visual system. His research interests include dynamic event cognition and the interaction between visual processing and attention.

Projections

The Journal for Movies and Mind

  • Barrett, Brian. 2016. “Netflix’s Grand, Daring, Maybe Crazy Plan to Conquer the World.” Wired, 27 March. http://www.wired.com/2016/03/netflixs-grand-maybe-crazy-plan-conquer-world.

    • Search Google Scholar
    • Export Citation
  • Boatwright, Peter, Suman Basuroy, and Wagner Kamakura. 2007. “Reviewing the Reviewers: The Impact of Individual Film Critics on Box Office Performance.” Quantitative Marketing and Economics 5 (4): 401425.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Boor, Myron. 1990. “Reliability of Ratings of Movies by Professional Movie Critics.” Psychological Reports 67: 243257.

  • Boor, Myron. 1992. “Relationships Among Ratings of Movie Pictures by Viewers and Six Professional Movie Critics.” Psychological Reports 70: 10111021.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bordwell, David. 2013. “The Viewer’s Share: Models of Mind in Explaining Film.” In Psychocinematics: Exploring Cognition at the Movies, ed. Arthur P. Shimamura, 2953. New York: Oxford University Press.

    • Search Google Scholar
    • Export Citation
  • Bordwell, David, and Kristin Thompson. 2010. Film Art: An Introduction. 9th ed. New York: McGraw-Hill.

  • Bourdieu, Pierre. 1984. Distinction: A Social Critique of the Judgement of Taste. Cambridge, MA: Harvard University Press.

  • Cameron, Sam. “On the Role of Critics in the Cultural Industry.” Journal of Cultural Economics 19 (4): 321331.

  • Child, Ben. 2016. “Is the Biggest Batman v Superman Smackdown between Fans and Critics?Guardian, 28 March. http://www.theguardian.com/film/filmblog/2016/mar/28/is-the-biggest-batman-v-superman-smackdown-between-fans-and-critics.

    • Search Google Scholar
    • Export Citation
  • Dargis, Manohla. 2014. “In Hollywood, It’s a Men’s, Men’s, Men’s World.” New York Times, 24 December. http://www.nytimes.com/2014/12/28/movies/in-hollywood-its-a-mens-mens-mens-world.html.

    • Search Google Scholar
    • Export Citation
  • Debenedetti, Stéphane. 2006. “The Role of Media Critics in the Cultural Industries.” International Journal of Arts Management 8 (3): 3042.

    • Search Google Scholar
    • Export Citation
  • Dellarocas, Chysanthos, Neveen Farag Awad, and Xiaoquan (Michael) Zhang. 2004. “Exploring the Value of Online Reviews to Organizations: Implications for Revenue Forecasting and Planning.” ICIS 2004 Proceedings 30. http://aisel.aisnet.org/icis2004/30.

    • Search Google Scholar
    • Export Citation
  • Doane, Mary Ann. 1987. The Desire to Desire: The Woman’s Film of the 1940s. Bloomington: Indiana University Press.

  • Eliashberg, Jehoshua, and Steven M. Shugan. 1997. “Film Critics: Influencers or Predictors?Journal of Marketing 61: 6878.

  • Felsen, Gidon, and Yang Dan. 2005. “A Natural Approach to Studying Vision.” Nature Neuroscience 8: 16431646. doi:10.1038/nn1608.

  • Geisler, Wilson S. 2008. “Visual Perception and the Statistical Properties of Natural Scenes.” Annual Review of Psychology 59: 167192. doi:10.1146/annurev.psych.58.110405.085632.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Grodal, Torben. 2006. “The PECMA Flow: A General Model of Visual Aesthetics.” Film Studies 8 (1): 111. doi:10.7227/FS.8.3.

  • Grubb, Michael, David J. Heeger, Uri Hasson, and Pascal Wallisch. 2010. “Subjective Preference and Its Effect on the Reliability of Cortical Activity during Movie Viewing.” Poster presented at the 40th Annual Meeting of the Society for Neuroscience, San Diego, CA, 13–17 November.

    • Search Google Scholar
    • Export Citation
  • Hadida, Allègre L. 2008. “Motion Picture Performance: A Review and Research Agenda.” International Journal of Management Reviews 11 (3): 297335. doi:10.1111/j.1468–2370.2008.00240.x.

    • Search Google Scholar
    • Export Citation
  • Hasson, Uri, Ohad Landesman, Barbara Knappmeyer, Ignacio Vallines, Nava Rubin, and David J. Heeger. 2008. “Neurocinematics: The Neuroscience of Film.” Projections: The Journal for Movies and Mind 2 (1): 126. doi:10.3167/proj.2008.020102.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hasson, Uri, Rafael Malach, and David J. Heeger. 2009. “Reliability of Cortical Activity during Natural Stimulation.” Trends in Cognitive Sciences 14 (1): 4048.

    • Search Google Scholar
    • Export Citation
  • Hasson, Uri, Yuval Nir, Ifat Levy, Galit Fuhrmann, and Rafael Malach. 2004. “Intersubject Synchronization of Cortical Activity during Natural Vision.” Science 303 (5664): 16341640. doi:10.1126/science.1089506.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Holbrook, Morris B. 1999. “Popular Appeal versus Expert Judgments of Motion Pictures.” Journal of Consumer Research 26: 144155.

  • Jacobs, Ruud S., Ard Heuvelman, Somaya Ben Allouch, and Oscar Peters. 2015. “Everyone’s a Critic: The Power of Expert and Consumer Reviews to Shape Readers’ Post-Viewing Motion Picture Evaluations.” Poetics 52: 91103. doi:10.1016/j.poetic.2015.07.002.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jazayeri, Mehrdad, Pascal Wallisch, and J. Anthony Movshon. 2012. “Dynamics of Macaque MT Cell Responses to Grating Triplets.” Journal of Neuroscience 32 (24): 82428253. doi:10.1523/JNEUROSCI.5787–11.2012.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kim, Sang Ho, Namkee Park, and Seung Hyun Park. 2013. “Exploring the Effects of Online Word of Mouth and Expert Reviews on Theatrical Movies’ Box Office Success.” Journal of Media Economics 26 (2): 98114. doi:10.1080/08997764.2013.785551.

    • Search Google Scholar
    • Export Citation
  • Kivy, Peter. 2015. De Gustibus: Arguing about Taste and Why We Do It. Oxford: Oxford University Press.

  • Ladhari, Riadh, and Miguel Morales. 2004. “The Influence of Individual Characteristics on Moviegoers’ Satisfaction Ratings.” Paper presented at the Atlantic Schools of Business Conference, Mount Saint Vincent University, Halifax, Nova Scotia, 4–6 November.

    • Search Google Scholar
    • Export Citation
  • Lundy, Duane E. 2010. “A Test of Consensus in Aesthetic Evaluation among Professional Critics of Modern Music.” Empirical Studies of the Arts 28 (2): 243258. doi:10.2190/EM.28.2.h.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lundy, Duane E. 2013. “Critiquing the Critics: Statistical Analysis of Music Critics’ Rating Distributions as a Measure of Individual Refinement.” Empirical Studies of the Arts 31 (1): 5979. doi:10.2190/EM.31.1.d.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mancini, Vince. 2016. “‘Batman v Superman’ and the Myth That Box Office Repudiates Critics.” Uproxx, 28 March. http://uproxx.com/filmdrunk/batman-v-superman-audience-critic-divide.

    • Search Google Scholar
    • Export Citation
  • Mulvey, Laura. 1975. “Visual Pleasure and Narrative Cinema.” Screen 16 (3): 618. doi:10.1093/screen/16.3.6.

  • Pearson, Karl. 1920. “Notes on the History of Correlation.” Biometrika 13 (1): 2545.

  • Plucker, Jonathan A., James C. Kaufman, Jason S. Temple, and Meihua Qian. 2009. “Do Experts and Novices Evaluate Movies the Same Way?Psychology & Marketing 26 (5): 470478. doi:10.1002/mar.20283.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rust, Nicole, C., and J. Anthony Movshon. 2005. “In Praise of Artifice.” Nature Neuroscience 8: 16471650. doi:10.1038/nn1606.

  • Schrage, Scott. 2012. “The Impact of Movie Reviews vs. Word of Mouth on Post-Viewing Evaluations of Films.” MA thesis, Iowa State University.

    • Search Google Scholar
    • Export Citation
  • Seeley, William, and Noë l Carroll. 2014. “Cognitive Theory and the Individual Film: The Case of Rear Window.” In Cognitive Media Theory, ed. Ted Nannicelli and Paul Taberham, 235253. New York: Routledge.

    • Search Google Scholar
    • Export Citation
  • Simonton, Dean Keith. 2007. “Is Bad Art the Opposite of Good Art? Positive Versus Negative Cinematic Assessments of 877 Feature Films.” Empirical Studies of the Arts 25 (2): 143161. doi:10.2190/2447–30T2–6088-7752.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Simonton, Dean Keith. 2011. Great Flicks: Scientific Studies of Cinematic Creativity and Aesthetics. Oxford: Oxford University Press.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Smith, Tim J. 2012. “The Attentional Theory of Cinematic Continuity.” Projections: The Journal for Movies and Mind 6 (1): 127.

  • Smith, Tim J., Daniel Levin, and James E. Cutting. 2012. “A Window on Reality: Perceiving Edited Moving Images.” Current Directions in Psychological Science 21: 101106. doi:1177/0963721412436809.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Spearman, C. 1904. “The Proof and Measurement of Association between Two Things.” The American Journal of Psychology 15 (1): 72101. doi:10.2307/1412159.

    • Search Google Scholar
    • Export Citation
  • Stevens, S.S. 1946. “On the Theory of Scales of Measurement.” Science 103 (2684): 677680.

  • Surowiecki, James. 2004. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. New York: Anchor.

    • Search Google Scholar
    • Export Citation
  • Tetlock, Philip E. 2005. Expert Political Judgment: How Good Is It? How Can We Know? Princeton, NJ: Princeton University Press.

  • Tetlock, Philip E., and Dan Gardner. 2015. Superforecasting: The Art and Science of Prediction. New York: Crown.

  • Vessel, Edward A., G. Gabrielle Starr, and Nava Rubin. 2012. “The Brain on Art: Intense Aesthetic Experience Activates the Default Mode Network.” Frontiers in Human Neuroscience 6 (66): 117. doi:10.3389/fnhum.2012.00066.

    • Search Google Scholar
    • Export Citation
  • Wallisch, Pascal. 2005. “Affective Responses to Moving Complex Audio-Visual Stimuli Are Better Predicted by ‘Simple’ Response Pools Than by Individual or Pooled Expert Predictors.” Paper presented at the Annual Meeting of the Midwestern Psychological Association, Chicago., 6 May.

    • Search Google Scholar
    • Export Citation
  • Wallisch, Pascal. 2015. “Brighter Than the Sun: Powerscape Visualizations Illustrate Power Needs in Neuroscience and Psychology.” arXiv:1512.09368.

    • Search Google Scholar
    • Export Citation
  • Wallisch, Pascal, and J. Anthony Movshon. 2008. “Structure and Function Come Unglued in the Visual Cortex.” Neuron 60: 195197. doi:10.1016/j.neuron.2008.10.008.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wallisch, Pascal, and Jake Alden Whritner. 2016. “Surprisingly High Concordance Rates in the Categorization of Movies.” Paper presented at the annual conference for the Society for Cognitive Studies of the Moving Image, Ithaca, NY, 1–4 June.

    • Search Google Scholar
    • Export Citation
  • Wang, Helena, Jeremy Freeman, Elisha Merriam, Uri Hasson, and David Heeger. 2012. “Temporal Eye Movement Strategies during Naturalistic Viewing.” Journal of Vision 12 (1): 127.

    • Search Google Scholar
    • Export Citation
  • Wanderer, Jules J. 2011a. “Scaling Professional Critics: Men and Women Rate Films.” Empirical Studies of the Arts 29 (2): 209223. doi:10.2190/EM.29.2.e.

    • Search Google Scholar
    • Export Citation
  • Wanderer, Jules J. 2011b. “When Film Critics Agree: Does Film Genre Matter?Empirical Studies of the Arts 29 (1): 3950. doi:10.2190/EM.29.1.c.

    • Search Google Scholar
    • Export Citation
  • Whritner, Jake Alden, and Pascal Wallisch. 2015. “A Neurocinematic Approach to the Appraisal of Film.” Paper presented at the annual conference for the Society for Cognitive Studies of the Moving Image, London, 17–20 June.

    • Search Google Scholar
    • Export Citation
  • Williams, Linda. 1991. “Film Bodies: Gender, Genre, and Excess.” Film Quarterly 44 (4): 213.

  • Zohary, Ehud, Michael N. Shadlen, and William T. Newsome. 1994. “Correlated Neuronal Discharge Tate and Its Implications for Psychophysical Performance.” Nature 370: 140143.

    • Crossref
    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 150 150 33
Full Text Views 37 37 0
PDF Downloads 9 9 0