In many social empirical studies subjects are provoked by situational conditions to exhibit reactive behavior patterns. From these reactions (e.g., marking point 5 on a seven-point rating scale), the kind and the strength of the subjects’ behavioral dispositions (e.g., beliefs, personality traits or states, emotional feelings) may be inferred. For reliable and valid conclusions to be drawn from collected data, the subjects’ reactions should only depend on these inner states under study, and not be influenced by other factors that are irrelevant to the research topic. In other words, the methods chosen for the measurement of values in a study should provide data that resembles the “real values” of these inner states as close as possible.
The error part of measured values should also be minimized. An error occurs during measurement because values obtained are superimposed by a random distortion. In addition, values may be biased by systematic influences. For example, subjects participating in a social study tend to develop hypotheses concerning the goal of that study. As a consequence, they might react in accordance with their assumptions, and this response tendency causes a systematic error. A systematic bias also occurs if a subject in an interview gives answers that are formulated with respect to social desirability. As a rule, data-collection methods that are based on processes of verbal communication (i.e., interviews, questionnaires, and psychometric tests) are all more or less subjected to the influence of response sets. It should be the researcher’s intention to keep their negative biasing effects on collected data as small as possible.
Definition And Types Of Response Sets
A response set is defined as a subject’s tendency to react to the stimuli presented to him or her in an interview, a questionnaire, or a psychometric test (questions, test items) in a way that reveals a certain pattern. This pattern interferes with the values indicating the intensity or strength of the internal state or trait to be measured. As a consequence, it is made difficult to assess the ‘true’ values of the variables in the focus of research. Response sets reflect general human reaction tendencies, e.g., to answer questions in such a way that the partner will consider the person’s verbal utterance to be relevant to the ongoing communication (Grice 1975). Response sets may also be related to a person’s specific characteristics (e.g., not being able to say “no” to a communication partner).
Response sets that become obvious in characteristic answering patterns in social research are the acquiescence response set, the nay-saying response set, the central tendency, the tendency toward the extreme, and a random answering tendency. Two more response sets – faking and the tendency toward social desirability – are related to the content of the questions or test items. The acquiescence response set is based on a subject’s tendency to predominantly answer questions in an affirmative way. A nay-saying response set, by contrast, is effective if a subject tends to give mostly negative answers to questions. A central tendency describes an answering pattern that consists of subjects frequently selecting the middle points of a rating scale.
In contrast, a tendency toward the extreme will be observed if mainly extreme (i.e., the highest or the lowest) numbers on a rating scale are checked as answers. Guessing or answering at random occurs if a subject does not know the correct answer or does not want to deal with the question’s content. By faking, a person is intentionally giving answers that do not express what he or she really thinks or feels, but what he or she wants the experimenter to hear or read (e.g., when a person applies for a job and does not want the HR manager to know about their real opinion). The tendency toward social desirability is effective when a person is giving answers that are strongly oriented toward general norms and values as seen by the person.
Determinants Of Response Sets
As indicated, it has been argued whether response sets reflect a general tendency of subjects to react or whether they should be attributed to specific personal characteristics. For example, if test items for the measurement of a personal trait or state (e.g., anxiety) are constructed in such a way that affirmative answers are indicating stronger intensity of this trait or state, a higher test score might really indicate a higher intensity of that state or trait. However, a high test score may also simply be the consequence of an acquiescence tendency. As a third possibility, this acquiescence tendency could be related to the personal state or trait; as a consequence a high test score would reflect both strong intensity (i.e., anxiety) and an effective acquiescence tendency.
In empirical investigations, the acquiescence tendency has been found to be related to specific behavioral dispositions: subjects exhibiting a strong acquiescence tendency tend to act in a reserved and docile way toward other persons. In addition, persons with this tendency have been found to be less competent in social interaction. Contradictory results, however, have been provided by the observation that the acquiescence tendency varies considerably between tests and that only weak correlations between this tendency and behavior outside test situations have been found. Furthermore, a tendency toward faking as well as a social desirability tendency have been found especially for those questionnaire items that are related to norms and the value systems of the social environment the subject is living in.
Implications For Research Practices
Avoiding Response Sets
Response sets affecting measurements in social studies can be minimized in their effects by carefully constructing the methods used for data collection. In order to reduce the acquiescence and nay-saying tendency, the alternatives provided for the subject’s answers (e.g., in multiple-choice items) should be balanced out with respect to “yes” and “no” answers. For example, if the intensity of a person’s anxiety is to be measured, approximately half the “yes” and half the “no” answers should indicate a higher intensity of this state. Effects of central tendency can be limited by avoiding middle points on rating scales, and choosing an even number of points when constructing the scales. However, if people perceive the intensity or strength of the personal state or trait they are asked about as best represented by a value right in the middle, between the two extremes, they might be irritated by the omission of a central answering point. To avoid response sets in general the subjects’ cooperation should be asked for by providing comprehensible instructions for the test or questionnaire, as well as unambiguous formulations of test or questionnaire items. In that way subjects establishing unfounded assumptions concerning the intentions of the study can be avoided. As subjects tend to react in correspondence with their prevailing hypotheses about the study, obscure and incomprehensible texts in data-collection methods can intensify biasing tendencies and by that increase the error proportion of obtained data.
Controlling Response Sets
In addition to a careful design of instructions and questions or test items, the strength of some of the content-oriented response sets (i.e., social desirability, faking) can be estimated by the inclusion of scales specifically constructed for that purpose. These scales will make biasing tendencies obvious during data analysis and thus might indicate that the subject’s test results should be interpreted with great care. Moreover, if the correlations between these scales indicate that response sets and the variable(s) to be measured are known, a mathematical correction procedure can be applied to exclude biasing tendencies from received data.
For example, some personality tests contain so-called frankness or faking scales. The construction of items for these scales is based on the assumption that for some questions the correct answers be will known beforehand. If, for example, a subject says “no” to a question like “Have you ever been dishonest?” or “Have you ever lied?”, such an answer indicates a tendency of the subject to fake test results, assuming that everybody has lied at least once in their life. As soon as the frequency of positive answers to items of this faking scale exceeds a certain threshold, an interpretation of test results should be performed with care.
Social Processes Underlying Response Sets
Response sets should be appraised with respect to the fact that empirical studies in social sciences are conducted within social situations. A context is established, in which data based on the subject’s reaction is of importance for the experimenter as well as for the subject. An experimenter is interested in acquiring unbiased data, which does not restrict reliability too much and which allows for valid interpretations. For the subject the score achieved in a test or a questionnaire can have severe consequences if, for example, the chance to get a job depends on the results, and “wrong” answers (at least in the subject’s perception) are perceived as dramatically reducing the likelihood of success.
If a person is anxious to influence test or questionnaire results in a certain direction, that does not necessarily mean they intend to present themselves in a better light. In some cases subjects dissimulate, e.g., if a person is undergoing a testing procedure for the army and wants to avoid being selected. On the basis of a single answering sheet it is mostly impossible to decide whether a response set was active or not, and which response set has to be taken into account. Furthermore, a biasing tendency does not necessarily have to be a consequence of deliberate faking. Unconscious biases in answering can occur, e.g., if a subject is not experienced in filling out tests or answering questions in questionnaires and if they have difficulties in understanding the formulation of instructions, questions, or answering alternatives.
Given the likelihood of incompatible interests between the experimenter and the subject participating in a test, there is only a small chance that response sets will be avoided. Subjects tend to cooperate when they realize that the data collected in the study will be used to compute aggregated parameters like means and frequency tables, and that no values specific to individuals are made public. It is a further advantage if the goals of the study and the procedure are explained to the subjects at length and if they are explicitly asked for their cooperation. In general, it is important to make the relationship between the subjects’ cooperation, the goal of the study, and the quality of acquired data obvious to the subjects.
Response sets should be understood in the light of a general tendency of subjects to influence the results of studies they are involved in. Some subjects have been observed to compliantly follow experimenters’ instructions and to be surprisingly willing to oblige the experimenter and the study in general. Cooperation can also be based on a tendency to relent to social pressure in the testing situation or on the desire for social appreciation when the subjects are anxious to exhibit only behavior perceived to be acceptable to the peer group or the experimenter.
Subjects, however, have also been observed to demonstrate destructive or negativistic behavior that might result in abandoning their participation in the study. Such destructive behavior is intensified if the subjects consider the questions, answering alternatives, or scales as not being clear or useful to them. Negative behavior is also increased if subjects recognize a social pressure to react in a specific way, or if they feel misled in some way.
References:
- Bachman, J. G., & O’Malley, P. M. (1984). Yea-saying, nay-saying, and going to extremes: Black– white differences in response styles. Public Opinion Quarterly, 48, 491–509.
- Cronbach, L. J. (1950). Further evidence on response sets and test design. Educational and Psychological Measurement, 6, 474–494.
- Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (eds.), Speech acts. New York: Academic Press, pp. 41–58.
- Hamilton, D. L. (1968). Personality attributes associated with extreme response style. Psychological Bulletin, 69, 192–203.
- Hui, C. H., & Triandis, H. C. (1985). The instability of response sets. Public Opinion Quarterly, 49, 253–260.
- Meisels, M., & Ford, L. H. (1990). Social desirability, response set, and semantic differential evaluative judgments. Journal of Social Psychology, 78, 45–54.
- Schinka, J. A., Velicer, W. F., & Weiner, I. B. (eds.) (2003). Handbook of psychology: Research methods in psychology, vol. 2. Hoboken, NJ: John Wiley.
- Shaugnessy, J. J., Zechmeister, E. B., & Zechmeister, J. S. (2006). Research methods in psychology, 7th edn. Boston, NJ: McGraw-Hill.
- Wilcox, C., Siegelman, L., & Cook, E. (1989). Some like it hot: Individual differences in response to group feeling thermometers. Public Opinion Quarterly, 53, 246–257.