Process studies of inference behavior

UMEÅ PSYCHOLOGICAL REPORTS

S U P P L E M E N T S E R I E S

Department of Psychology University of Umeå

Supplement No. 2

PROCESS STUDIES OF INFERENCE BEHAVIOR

PROCESS STUDIES OF INFERENCE BEHAVIOR

Akademisk avhandling

som med tillstånd av rektorsämbetet vid Umeå universitet för vinnande av filosifie doktorsexamen framlägges till offentlig granskning vid

aulan, norra paviljongerna, Umeå universitet den 23 april 1976, kl 13.15

av

Bengt-Åke Armelius fil kand

The dissertation consists of this summary and the following six papers: I. Armelius, B-Â., & Armelius, K. Detection of cue intercorrelation

and cue validities in a multiple-cue judgment task with a suppressor cue. Umeå Psychological Reports No. 74, 1973. II. Armelius, K., & Armelius, B-Â. Note on detection of cue

inter-correlation in multiple-cue probability learning. Scandinavian Journal of Psychology, 1975, 16, 37-41. III. Armelius, B-A., & Armelius, K. Detection of cue intercorrelation

in multiple-cue probability learning. Umeå Psychological Reports No. 84, 1975.

IV. Armelius, B-A., & Armelius, K. Integration rules in a multiple-cue probability learning task with intercorrelated cues. Umeå Psychological Reports No. 80, 1975.

V. Armelius, B-A., & Armelius, K. Combination rules in multiple-cue probability learning. I« ^e effects of task character­ istics and performance. Umeå Psychological Reports No. 99, 1976.

VI. Armelius, B-A., & Armelius, K. Combination rules in multiple-cue probability learning. II. Performance, confidence and development of rules. Umeå Psychological Reports No. 101, 1976.

ACKNOWLEDGEMENTS

There are two persons who deserve special thanks for their contribution to this thesis. One of them is my wife, Kerstin, without whose faith­ fulness, patience, knowledge, skill, cooperation and love I doubt if this thesis would have been completed within the next ten years. The other person is Berndt Brehmer, ny supervisor, who has shared his knowl­ edge and wisdom with me during endless hours. He has taken pains to comment on details as well as more philosophical aspects of the work. Thanks also to all the people who helped us in the experimented work, students who served as subjects in the experiments, and assistants Ann Olofsson, Ingrid Sundberg and Maja Viklands who took care of the practical details.

Finally I would like to thank the people who typed the manuscripts and draw the figures, primarily Anita Âberg but also Marianne Larsson, Margareta Lindberg, Inger Olsson, MajLis Séhlstedt and Brita Westling as well as Stig Lindkvist and Harald Tjärnström who put the papers together into reports.

The research reported in this thesis was supported by grants from the Swedish Council for Social Science Research.

PROCESS STUDIES OF INFERENCE BEHAVIOR Bengt-Âke Armelius

Inference tasks require the subjects to learn to infer the state of a criterion variable from a set of predictor variables, called cues. The present dissertation is concerned with the question of hew this

is done, rather than how well it is done.

Usually, descriptions of inference processes are made in terms of mathematical equations which describe the functional relationship between the cues and the judgments. The adequacy of such equations is determined by the accuracy with which the equations represent the judgments. The most common mathematical equation within the field of inference behavior is a linear regression equation which defines the predicted judgments as a weighted sum of the cue values (see Dawes & Corrigan, 1974 for a review). Although nonlinear equations have been used (e.g., Wiggins & Hoffman, 1968; Einhorn, 1970) a regression equation or other linear composites (e.g., Anderson, 1968) usually are as good or better descriptions of the judgments.

Recently, the mathematical equations have recieved some competition from descriptive models of inference behavior based on verbal reports given by subjects (e.g., Bréhmer, 1974; Kleinmuntz, 1968). The possi­

bility to construct information processing models by computer programs (e.g., Newell & Simon, 1972) has opened up new perspectives for models

based on verbal reports.

The first three papers of the present dissertation are inspired by a modelof inferenoe behavior based on multiple regression statistics, while the last three papers utilize verbal reports and computer technology to simulate the inference process.

-2-PROCESS STUDIES BASED ON MULTIPLE REGRESSION STATISTICS

In multiple regression statistics the optimi prediction is made with a least squares criterion of optinality. In this sense the prediction constitutes the best possible linear combination of the predictor variables. The prediction can be written as a linear equation, which in standard score form is

J* = ß-|Z- + ß^Z« + _{1 1 2 2 n n}

where J1 = predicted value

ß^ = beta-weight for predictor variable i Z . = standard score of variable i _i

n = number of predictor variables

The least squares criterion means that the sum of the squared difference between the observed value, J, and the predicted value, J', is minimized. This is acconplished by finding the values of the beta-weights that

minimize the sum of the squared difference J - J'. The most conroon procedure for doing this is to solve the so called normal equations, which contain the coefficients of correlation among all the predictor variables, r^, and between the predictor variables and the criterion variable, r ., and the beta-weights, 8-. _{ei ° i}

One possible way that subjects learn multiple-cue probability learning (MCPL) tasks nay be described by analogy to the way the optimal prediction

is computed in multiple regression statistics. According to such a regression model subjects have to a)learn the cue-criterion corre lations, r b) learn the cue-intercorrelations, r.. and c) find the _{ei 13} beta-weights, ß^, through some operation on these two sets of corre­ lations.

In orthogonal MCPL tasks it is known that the values of the cue-judg­ ment correlations match the values of the cue-criterion correlations fairly well (e.g., Dudycha & Nay1er, 1966; Uhi, 1966). Such tasks

-3-are, however, not ideally suited to study hew subjects learn an in­ ference task. First, they are not representative of real world in­ ference tasks (Brunswik, 1952). Second, the cue-criterion correlations are equal to the cue-criterion beta-weights. Therefore it is not pos-ible to decide whether the subjects take r^j into account. In contrast, suppressor variable tasks are well suited to this purpose since r^j is not zero and the cue-criterion beta-weights are usually very dif­ ferent from the values of the cue-criterion correlations. The results from such tasks (e.g., Armelius & Armelius, 1974, 1976c) show that subjects performance is less optinai than in orthogonal tasks (Brehmer, 1976).

According to the regression model the suboptimal performance in sup­ pressor variable tasks nay be due to any or all three requirements listed above, i.e., subjects nay not have learned the necessary rela­ tions of the task or they may be unable to find the optinai weights on the basis of their knowledge.

The first experiment (Armelius & Armelius, 1973) was designed to study whether subjects know the values of r . and r.. in intercorrelated _{ei 13}

MCPL tasks. The subjects were required to reproduce r^ and r^ directly, rather than to make predictions, since it is impossible to tell whether subjects know the values of these correlations from their performance data only. After completion of learning tasks with different levels

re£ (rej_ = • «91 and .60; rg2 = .00 in all tasks) and different

levels of r^j (.00, .40 and .80) subjects were assigned to one of two test conditions. In the first condition subjects were shewn one of the two cues and asked to predict the valun of both the other cue and the criterion. Each cue was presented on every second tria l, and the purpose was to test the subjects' knowledge of r^. In the second condition subjects were asked to predict the value of the criterion only, otherwise the procedure was the same as in the first condition. The purpose of the second condition was to test the subjects' knowl­ edge of r ..

-4-The results showed that subjects had learned the value of r. • and r .

J i3 ei

almost perfectly in the tasks with r^j t .00. In the task with r^ = .00 subjects greatly overestimated both the value of r^j and the value of the low validity cue. The reason for this overestination may have been that the task was too easy since one cue had a corre­ lation of .99 with the criterion, and therefore subjects did not pay attention to other aspects of the task than that correlation. The results did not contradict the hypothesis that subjects learn the necessary correlations in an intercorrelated task, but failed to give support for the detection of different magnitudes of r^. The results cf the first experiment together with the result of an experiment by Knowles, Hammond, Stewart & Summers (1972), who also failed to give clear cut evidence that subjects actually detect the value of the cue intercorrelation were reasons to design the second experiment (Armelius & Armelius, 1975c). Knowles et al., had used a recognition method to study the detection of r-. Therefore it was

also desirable to compare the two methods to study detection of r^. In the second experiment two different tasks were used, one with r^ = .00 and one with r.. = .80. In addition, the task with r.. = .00

13 13

was made more difficult than in the first experiment by reducing the value of rg^. The results of this experiment showed that subjects de­

tect the value of r^j both with a reproduction and a recognition method. In order to test the generality of the conclusion from the second ex­ periment a third experiment on the detection of r^j was performed. In this experiment (Armelius & Armelius, 1975a) the recognition method was used. The values of r^ were spread out over a wide range in order to study the functional relation between subjective and objective values of r^j. In addition, the experiment was designed to test the effect of different task characteristics (task predictability, cue-criterion correlations and. sign of the cue intercorrelation) on de­ tection of r... The results were that the detection of r.• is inde-_{13 13} pendent of task characteristics. Subjects were able to reproduce different values of intercorrelation equally well, but the values

-5-of the subj ective intercarrelaticns were about half as high as the values of the actual correlations. Another important result from the experiment was that there was no relation between the detection of r^j and performance, a facit which raises serious doubts about at least the third requirement of the regression model for inference behavior.

The conclusion from these three experiments is that subjects have the necessary information to reach an almost optimal performance, and yet they fail to do so. In terms of the regression model therefore subjects are unable to give the adequate weight to the cues. The answer to the question of how subjects learn an inference task therefore is that we know what subjects do not do, i.e., they do not use their knowledge in an optimal way. In order to understand more about what subjects do when they learn an inference task it is necessary to study how subjects integrate the information given to them. One wey to do this is to ask subjects how they make their judgments and analyze the con­ sequences of their answers.

PROCESS STUDIES BASED ON VERBAL REPORTS

The fate of the regression model studied in the first three experiments is not unique to mathematical models of inference behavior. Ever since the book on clinical versus statistical prediction by Meehl appeared in 1954 and Hammond's paper in 1955 there have been studies on differ­ ent mathematical models of inference behavior. Besides multiple re­ gression there have been models based on analysis of variance (e.g., Hoffman, Slovic & Rohrer, 1968; Slovic 1969) functional measurement (e.g., Anderson, 1968) and other models (e.g., Wiggins & Hoffman, 1968;

Einhorn, 1970). The results of almost all experiments show, however, that the linear models account for most of the variance in subjects' judgments, and that the additional effects accounted for by nonlinear models are small, if any at all. This is of course due to the fact that the linear models are very powerful and give good approximations also to many nonlinear relations (e.g., Dawes & Corrigan, 1974).

-6-By now it is evident that it is relatively easy to set up a mathemati-cal model of inference behavior that accounts for the systematic vari­ ance in subjects' judgments. This is not, however, equal to a theoreti­ cal understanding of what subjects do when they learn an inference task. In the experiments above the multiple regression model probably would account for most of the systematic: variance that exist in sub­ jects' judgments. In spite of this, a ps;

the procedures of multiple regression dees not contribute very much to our understanding of inference béhavd.or. Thus, multiple regression has its primary value as a means to account for systematic aspects of the data, not as a paramorphic (Hoffman,

judgment.

Today, an alternative to the struggle between different mathematical the information processing in thinking and problem solv-models of inference behavior is given b}

approach by NeweXLand Simon (e.g., 1972!

ing. Thus, KLeinmuntz (1968) has shewn that inference behavior may be seen as a complex information processing; activity where subjects work with different rules and test hypotheses;. The same appraoch has been taken by Brehmer (1974) in single cue probability (SCPL) learning studies. In both these examples inference behavior is seen as a hypotheses testing activity and the use of verbal reports plays a central role to the understanding of how subjects make their infer­ ences.

19ß0) representation of human

The purpose of the last three experiments in this dissertation is to make an attempt to describe how subjects make their predictions in MCPL through the use of verbal reports given by subjects. An attempt will also be made to integrate multiple cue probability learning with the hypotheses testing model developed for inference behavior by Brehmer (1974).

-7-The purpose of the first experiment (Armelius & Armelius, 1975b) in this series was primarily methodological and no attenpt was made to study the effects of different task characteristics. Twenty four subjects learned a two-cue MCPL task with perfect predictability and after coupletion of the learning stage they were given a questionnaire with some open-ended questions. The answers to these questions were analyzed and categorized in four categories: a) not systematic, b) estimated weights, i.e. subjects had given weights to the cues, but no, or an incomplete verbal description of their rules, c) con­ figurai rule, i.e. more than one equation was needed to express the rule in mathematical terms, and d) linear rule, i.e. the rule could be expressed as a linear equation. The systematic verbal descrip­ tions (10 out of 24) were translated into mathematical equations and a set of judgments was generated to the cue value combinations in the last block of learning. These generated judgments were then correlated with the actual judgments for each subject and subjected to a polynomial regression analysis. Both the polynomial regression analysis and the size of the correlations (compared to the multiple correlations between cues and judgments) indicated that with a few exceptions, the verbal descriptions accounted for the systematic variance in subjects judg­ ments. The results suggested that subjects who had formulated system­ atic rules or gave weights to cues reached a higher level of perform­ ance than other subjects who could not describe their rule. The results were promising in the sense that the verbal reports given by nany sub­ jects at the end of learning seemed to be good descriptions of how they actually made their judgments.

In the second experiment (Armelius & Armelius 1976a) the effect of various task characteristics on subjects' formulation of combination rules was studied. The experimental tasks and subjects were the same as those in the third experiment on detection of r^. Subjects were asked to describe how they had made their judgments at the end of the experiment. The answers were analyzed and translated into pre­ diction equations whenever the verbal descriptions were systematic and consistent enough to allow such translation.

-8-The combination rules were classified as single rule or multiple rule dependent on whether one rule covered the complete cue matrix or whether a number of rules were used to cover the complete cue matrix. This

classification has the advantage of not confusing a single rule with a linear rule. As shown by Brehmer (1969) a single rule nay well form a configurai relation between cues and criterion. The classification is also a better description of the rules that subjects use. The multiple rules were usually a set of linear rules. The distinction between single rules and multiple rules seems to correspond to the distinction between "functional rules" and "classification rules" made in single cue probability learning (Brehmer, Kuylienstierna & Liljergren, 1974).

A statistical test of the goodness of fit of the combination rules was developed in the experiment. This was possible because the cue-and criterion values of block 5 cue-and 8 were identical. In this way an estimate of the amount of systematic variance in subjects judgments could be derived.

The results of the experiment showed that 53 of the 100 subjects formulated systematic combination rules and that in 41 of these

cases the verbal descriptions accounted for the systematic variance in subjects' judgments. There were no effects of the task character­ istics on the frequency with which subjects formulated combination rules. There were 46 multiple rules but only 7 single rules. This may have been due to the way the subjects were asked about their combination rules and should not be interpreted too strongly. The performance of subjects who formulated multiple combination rules was higher than for other subjects.

One methodological complication with the verbal reports is obvious after the second experiment, and that is the fact that only about 50 % of the subjects formulate systematic and complete combination rules. Very little is known about what the other 50 % are doing, or what the reasons are that they do not formulate systematic rules. One possible way to circumvent this problem is to ask subjects more frequently and perhaps make them think aloud.

-9-In the third experiment (Armelius & Armelius 1976b) an attenpt was made to integrate the studies on what rules subjects use in MCPL with the hypothesis testing approach to functional learning in single cue probability learning introduced by Brehmer, (1974). According to this appraoch learning of an inference task is a two-stage process. In the first stage subjects select a hypothesis about the rule relating cues and criterion from their preexperimental experience. The

selection is assumed to take place through sampling from a hierarchy of hypotheses that vary in strength. In the second stage subjects test their hypotheses against the task and learn to use them. This model has never been applied to MCPL, which means that nothing is known about subjects' preexperimental experiences and what rules they may have available. Similarly, nothing is known about how sub­ jects end up with the rule they do. This may occur either by repeated sampling until they find a rule that fits the task, or by successive adjustments of a certain rule that has been selected.

Through repeated questions to subjects in the third experiment (Arme­ lius "< Armelius, 1976b) it was hoped that some of these theoretical questions could be answered. In the first block no feedback was given, which made it possible to study the characteristics of subjects' preex­ perimental hypotheses about the relation between two cues and a cri-, terion variable. Five tasks with various degrees of predictability were used in the experiment.

The results of the experiment showed that the two-stage model developed for single cue probability learning also nay contribute to the under­ standing of what subjects do in multiple cue probability learning. All subjects but one formulated at least one combination rule at some tine during the experiment. An equal number of subjects formulated multiple and single rules, and about 30 % of the subjects fornulated inconsistent or incomplete rules. The rules accounted for the system­ atic variance in 86 % of the cases. Besides confirming the results of previous experiments; that performance is higher for subjects with systematic rules, it was also shown that subjects who formulated rules were more confident in their judgments.

-10-The first picture of a hierarchy of preexperimental hypotheses about the rule relating cues and criterion in MCPL showed that an average, sum or difference of the cue values are the most frequent hypotheses. The development of rules seemed to take place through modification of the rules rather than through repeated sampling. Especially inter­

esting was the development of multiple rules, which makes it possible for subjects to learn a certain part of the cue matrix by the use of heuristics. This is clearly similar to what the clinicians claim they do when they use e.g. MMPI in a configurai manner.

In sunntary, the present dissertation shows that it is necessary to use a variety of approaches to gain some understanding of inference behavior. The historical neglect for aspects of inference behavior other than performance has led to an unfortunate restriction of our

present knowledge. Many important areas such as the "incidental" learning of cue intercorrelation, what combination rules subjects use, subjects' conception of the weight of a cue, confidence, knowl­ edge of perfonrence, ability to use different rules, hypothesis testing and so on, are just beginning to be the focus of investi­ gations. This trend is consistent with Brunswik's claim for behavior-research isomorphism, i.e. "behavior-research may be said to have reached an adequate, functional, or molar level of complexity only if it parallells, and is thus capable of representing, behavior in all its essential features" (Brunswik, 1952, page 25).

The research program, of which this thesis is one result and the thesis by Kerstin Armelius (1976) another, has shewn that the lens model paradigm (Hursch, Hammond & Hursch, 1964) is well suited to represent the performance aspect of inference behavior. This paradigm is based on multiple regression statistics and therefore any attempts to study other aspects of inference behavior within this paradigm will be put into such terms. In order to represent some of the additional aspects, e.g., the combination rules used by subjects, it is necessary to develop new research paradigms. As shown in the present thesis it is possible to borrow paradigms f" m other areas of psychology. This also has the advantage of making the

-li-connection between multiple cue probability learning and areas such as problem solving and thinking evident. A first step towards the integration of different paradigms and problems of inference behavior has been taken in the present two theses.

-12-REFERENCES

Andersen, N. H. A simple model for information integration. In R. P. Abelson, E. Aronson, W. J. McGuire, T. M. Newcomb, M. J. Rosenberg, and P. H. Tannenbaum (Eds.), Theories of cogni­ tive consistency: A sourcebook. Chicago: Rand McNally, 1968. Armelius, K. Cue intercorrelation, redundancy, and inference behavior. Umeå Psychological Reports, Supplementary Report No. 1, 1976. Armelius, B-Â., & Armelius, K. Detection of cue intercorrelation and

cue validities in a multiple cue judgment task with a sup­ pressor cue. Umeå Psychological Reports No. 74, 1973. Armelius, B-Â., & Armelius, K. Utilization of redundancy in multiple

cue judgments: Data from a suppressor variable task. American Journal of Psychology, 1974, _3, 385-392.

Armelius, B-Å., & Armelius, K. Detection of cue intercorrelation in multiple-cue probability learning. Umeå Psychological Re­ ports No. 84, 1975 (a).

Armelius, B-Å., & Armelius, K. Integration rules in a multiple-cue probability learning task with intercorrelated cues. Umeå Psychological Reports No. 80, 1975 (b).

Armelius, K., & Armelius, B-Å. Note on detection of cue intercorrelation in multiple-cue probability learning. Scandinavian Journal of Psychology, 1975, 16, 37-41 (c).

Armelius, B-A., & Armelius, K. Combination rules in multiple-cue

probability learning. I. The effects of task characteristics and perforaance. Umeå Psychological Reports No. 99, 1976 (a).

-13-Armelius, B-Â., & -13-Armelius, K. Combination rules in multiple-cue probability learning. II. Performance, confidence and development of rules. Umeå Psychological Reports No. 101, 1976 (b).

Armelius, K., & Armelius, B-Â. The effect of cue-criterion correlations, cue intercorrelations and the sign of the cue intercorrelation on performance in suppressor variable tasks. Organizational Behavior and Human Performance, 1976 (c) (in press).

Brehmer, B. Cognitive dependence on additive and configurai cue-cri-ter ion relations. American Journal of Psychology, 1969, 82, 490-503.

Brehmer, B. Hypotheses about relations between scaled variables in the learning of probabilistic inference tasks. Organizational Behavior and Human Performance, 1974, LU 1-27.

Brehmer, B. Response consistency in probabilistic inference tasks. Umeå Psychological Reports No. 100, 1976.

Brehmer, B., Kuylenstierna, J., & Liljergren, J-E. Effects of function form and cue validity on the subjects' hypotheses in proba­ bilistic inference tasks. Organizational Behavior and Hunan Performance, 1974, U., 338-354.

Brunswik, E. Conceptual framework of psychology. Chicago: University of Chicago Press, 1952.

Dudycha, L., & Naylor, J. C. Characteristics of the human inference process in complex choice behavior situations. Organizational Behavior and Human Performance, 1966, 1, 110-128.

Dudycha, L., Dudycha, N., & Schmitt, N. Cue redundancy: Some overlooked analytical relationships in MCPL. Organizational Behavior and Human Performance, 1974, 1J., 222-234.

-14-Eirihorn, H. J. The use of nonlinear, noncompensatory models in decision making. Psychological Bulletin, 1970, 221-230.

Hammond, K. R. Probabilistic functioning and the clinical method. Psychological Review, 1955, _62, 255-262.

Hoffman, P. J. The paramorphic representation of clinical judgment. Psychological Bulletin, 1960, _57, 116-131.

Hursch, C., Hammond, K. R., & Hursch, J. L. Some methodological con­ siderations in multiple-cue probability studies. Psychologi­ cal Review, 1964, 71.» 42-60.

Dawes, R. M., & Corrigan, B. Linear models in decision making. Psycho­ logical Bulletin, 1974, J31, 95-106.

Hoffman, P. J., Slovic, P., & Rorer, L. G. An analysis-of-variance model for the assessment of configurai cue utilization in clinical judgment. Psychological Bulletin, 1968, _69, 338-349. KLeinmuntz, B. The processing of clinical information by nan and

machine. In KLeinmuntz, B. (Ed.), Formal representation of human judgment. New York: Wiley, 1968.

Krtcwles, B. A., Hammond, K. R., Stewart, T. R., & Summers, D. A. De­ tection of redundancy in multiple cue probability tasks. Journal of Experimental Psychology, 1972, 93, 425-427. Meehl, P. E. Clinical versus statistical prediction: A theoretical

analysis and review of the littérature. Minneapolis: Uni­ versity of Minnesota Press, 1954.

Newell, A., & Simon, H. A. Human problem solving. Englewood Cliffs: N. J.: Prentice-Hall, 1972.

-15-Slovio, P. Analyzing the expert judge: A descriptive study of a stockbrokers decision processes. Journal of Applied Psy­ chology, 1969, j>3, 255-263.

Uhi, C. N. Effects of multiple stimulus validity and criterion dis­ persion on learning of interval concepts. Journal of Experi­ mental Psychology, 1966, 72, 519-527.

Wiggins, N., & Hoffman, P. J. Three models of clinical judgment. Journal of Abnormal Psychology, 1968, T3> 70-77.