Real-time rating (RTR) methods – also called “real-time response” or “continuous response measurement” (CRM) – collect judgments or evaluation data from a subject during media exposure. While questionnaires provide data about the outcome of a perception (e.g., television viewing), RTR focuses on the process of viewing. Besides the application to academic questions of reception research, it is widely used within the media industry for testing television programs or movies before they are aired or released. Applications range from commercial television or radio programs, advertising research, to measuring judgment processes in presidential debates. It can also be used for content analysis of television programs or speeches (e.g., measuring perceived degree of violence, dimensions of characters’ behavior, interaction, etc.).
Development: “Little Annie”
The idea of real-time response measurement dates back to the late 1930s and early 1940s when Paul F. Lazarsfeld and Frank Stanton introduced their “program analyzer,” which was later adopted by CBS and major advertising companies. Their first system, affectionately named “Little Annie,” consisted of two cylinders, about five inches long with a diameter of an inch. One cylinder had a red push-button at the end and the other a green button. The members of the audience held one in each hand with the thumb positioned to press on the button. Viewers were instructed to press the green button to indicate their liking of a radio program, and the red button if they were to feel uncomfortable about the program. Pushing neither button indicated indifference. Audience response charts on paper showed patterns of likers and dislikers for up to 10 persons at a time.
In the later years a set of 100 stations called “Big Annie” was developed. The first generations of these kinds of program analyzers were used for program and film testing, for example, at Columbia University’s Bureau of Applied Social Research, McCann Erickson, or CBS. In the years following the introduction of the program analyzer, the idea was adopted for a number of similar devices. The main differences were in the way participants’ judgments were collected: five- to ten-point scale push-button systems were developed as well as seven-point scale dialers, and ones with ten or more points. But basically the idea itself remained the same (Millard 1992).
The boom of microcomputers in the 1980s led to refinement in data collection, display, and analysis. Today participants in RTR studies signal their reactions to exposure mostly by means of a dialer, joystick, or slider using a predetermined scale (mood, judgment, attitudes, etc.). More or less any scale suitable for real-time reactions can be used. Hence, RTR can be considered as a continuous report on one repeated question (e.g., How much do I like what I see?), producing panel data that are dynamically sensitive to the subtle effects of the stimulus. Since results are nothing more than highly auto correlated time-series data, a wide range of data analysis techniques can be employed.
Compared to traditional surveys, the results of RTR are far more precise, letting the researcher pinpoint which parts of audiovisual stimuli are responsible for the so-called peaks or spikes during exposure. For example, one can describe how the appearance of a certain character influences judgments, how music can contribute to changes in evaluations, how humor moderates perception, how dynamic a plot is perceived to be, and more. Moreover the technique is also described as a measure of audience “attention” (Millard 1992), semantic processing, attitudes, or other psychological states or mental processes (Biocca et al. 1994). In general, Biocca et al. (1994) define the measure as subjects’ self-reports of changes in psychological state or judgment.
Furthermore RTR can also be used as a continuous measure of changing message content or a way to code communication behaviors. Measures of hedonic response (like/dislike) are found in a majority of the measures. Concerning validity, subjects report that the measure reflects their feelings about a program accurately. Moreover, RTR proves to be a sensitive indicator for attitudes toward a fictional or real character (often measured in the context of political or presidential debates). Finally, the measure allows for the investigation of whether our overall evaluation of a media stimulus is triggered largely by an overall judgment of that stimulus (mean-driven judgment), rather than by individual events or scenes of a program – whether it be cumulative, multiplicative, linear or nonlinear (e.g., peaks, spikes, punch-lines, appearance of important actors, peripeteia).
High attention to a program can be indicated by high frequency of use and wide range of movement of the dialer. Studies show a positive correlation between this kind of activity and memory recall of these parts of the program. However, critics point to the fact that high involvement in the program may lead to the subject forgetting to use the dialer, giving rise to a false interpretation of psychological deafness or boredom. The nonuse of the handheld device can therefore mean opposite things: being completely absorbed or completely bored. Implementing a secondary task reaction-time design into the research setting could address the problem, but would at the same time make the viewing situation more artificial and the task more complicated.
The participants could also be asked how involved they felt during a decisive part of the program and relate this answer to the individual real-time response data. If high reported involvement is accompanied by no scale movement, these subjects should be analyzed carefully, separately, or even eliminated from analysis. Another important issue in this context is that subjects utilize the given scale to a different degree. If the sample size is low, this scaling problem is often not leveled out sufficiently. Again, cross-checks with questionnaires – asking subjects about their highest and lowest scores in relation to their personal range – could help adjust RTR data. In some cases, it might even be helpful to standardize the mean series.
Validity Of The Method
One other major constraint of RTR measurement is that it is only one-dimensional. As the online evaluation task requires some cognitive effort, valid results can be expected only if subjects concentrate on one dimension only for evaluation. For example, it is relatively easy for subjects to indicate whether they like what they see or not. More difficult and less valid would be the question of whether a magazine, show, film, etc. is entertaining or informative. Moreover, reactivity of RTR measurements is often criticized, meaning that the task itself modifies the perception process or does not indicate what should be measured – hence, validity is violated. As stated above, if participants are asked to indicate their involvement in a movie by means of RTR, very high involvement might make viewers forget to use the dialer because they are too engaged in the program. Similarly, if viewers are continuously evaluating their state, they might not be able to become engaged with the media content. Hence, the task should be easy and participants be given time to get used to the method.
Compared to RTR measures a questionnaire can ask for retrospective judgment on a greater variety of criteria. However, a retrospective judgment may be quite different from an online judgment. After watching a movie, quite a lot is already forgotten; messages have already been processed on a higher level, rationalized, adjusted to one’s reference system, and often altered because of social desirability. If one wants to collect data on spontaneous, immediate impressions, RTR is the ideal measurement technique. Continuous audience response research is of high value when applied to pilot programs, TV commercials, movies, or political advertising at a point in production where they can still be changed. Taken together, the most effective means of audience research is a combination of several techniques. For audience profiling and ex post facto judgment, a questionnaire is employed. For continuous judgments RTR is needed, and for validation purposes and enrichment of RTR data, focus groups can be very useful.
- Biocca, F., David, P., & West, M. (1994). Continuous response measurement (CRM): A computerized tool for research on the cognitive processing of communication messages. In A. Lang (ed.), Measuring psychological responses to media messages. Hillsdale, NJ: Lawrence Erlbaum, pp. 15–64.
- Maier, J., Maurer, M., Reinemann, C., & Faas, T. (2006). Reliability and validity of real-time response measurement: A comparison of two studies of a televised debate in Germany. International Journal of Public Opinion Research, 19(1), 53–73.
- Millard, W. J. (1992). A history of handsets for direct measurement of audience response. International Journal of Public Opinion Research, 4(1), 1–17.