Requires data coding and measures of inter-rater reliability (like narrative methods and interviews).
Allows access to hidden mental behavior, but:
Relies on introspection - people are not so good at understanding their behavior.
Dangerous to rely too heavily on protocols.
Ways of collecting data
Observation without intervention
Naturalistic observation
Physical traces
Archival data
Observations without intervention are still theory-laden - observer just does not intervene
Observation with intervention
Experimental manipulations
Experimenter has control - systematically test whether a variable causes changes in measures
Test alternative hypotheses, different groups, infrequent events
Artificial, less ecological validity
Some issues
How "ecologically valid" are the observations?
What influence does the observer have on the situation?
Naturalistic observation
Why perform naturalistic observations?
Advantage: Ecological validity - observe behavior in naturalistic settings
Avoid ethical problems of causing noxious conditions on purpose - feral children (raised in isolation), Kitty Genovese case (stabbing)
Anyone can just observe - How is scientific observation different from nonscientific observation?
Precision
Definition of conditions - be clear about what you are observing
Non-participant observation - researcher is not part of the group being observed
Participant observation - researcher becomes part of the target group - an intensive involvement with people in their natural environment (religious or occupational groups)
Disguised observation
Experimenter "infiltrates" group and observes - behavior of a cult
Ethical problems with disguised observation
Deception
Group members will treat the experimenter as one of their own
Experimenter may (unwittingly) adopt attitudes of the group
Experimenter bias and expectancy
The experimenter may lose objectivity
Can the experimenter shape the behavior of the group? - Experimenter may inadvertently push the group
Suppose the cult wanted to commit a crime - should the experimenter talk them out o fit or prod them into it?
Can be dangerous
Undisguised observation
Observer follows the group around - Anthropology and cultural psychology
E.g., following the inner working of a political campaign - reactivity?
Problems with naturalistic observation
Interesting behaviors may occur infrequently
Some events do not occur in public
Hard to examine processes during observation
No control over circumstances - possible to find correlations but no cause-effect relations
Credit assignment problem
X correlated with Y does not mean X causes Y
Third variable problem
There may a third, unmeasured variable that causes the behavior
Instead of [ X --> Y ], maybe [ X <-- Z --> Y ]
Children in day care develop attachment problems - maybe parents cannot stand their children
Directionality problem (circular reasoning)
It is difficult to state what causes what
Instead of [ X --> Y ], maybe [ X <--> Y ]
Violent television and violent kids
No experimental control - How would you make this into a true experiment?
Observation with intervention
We can cause infrequent events - e.g., simulated emergencies
We can investigate the limits of an ability - e.g., how much information people can process
We can observe normally private events - e.g., personal behavior
Intervention allows control
Can vary the settings explicitly
Comparisons with different groups that you create
Can assign conditions to people
Eases the credit assignment problem - two groups differing only in a single factor
Allows repeated observation of the same behavior
Controlled settings
The experimenter will manipulate the world
We want to set up different situations that are as similar as possible
We want to be able to eliminate alternative explanations for behavior
Example: Diffusion of Responsibility
Naturalistic observation
Kitty Genovese case
Woman stabbed repeatedly - nobody intervened.
No bystander helped - Why? Cannot know for sure.
Possibilities? - Diffusion of responsibility by Darley and Latane
Test by a controlled experiment.
Diffusion of responsibility explanation - the more people present, the less likely that one will help - how could this be operationalized?
Perceived group size in a conversation.
2: Participant, victim
3: Participant, victim, and one bystander
6: Participant, victim, and four bystanders
Group Size
% responding by end of help
Time (seconds)
2
85
52
3
62
93
6
31
166
Variable types
Independent variables
Variables that are controlled by the experiment
E.g., training aid or not
Levels
How many hours you play with the training aids
Treatments (non-control) groups receive different levels of IV (i.e., different hours of training using the training aid)
Control groups receive no training aids
Dependent variables
The actual measure for the statistics and results
E.g., time to complete a set of tasks
Extraneous variables or confounds
Not interested in them but could affect the results
E.g., simply spending more time in front of a computer, not the training aid itself, may influence the result
Unrelated computer task for added control
Internal validity
Is the validity with which we infer that a relationship between two variables is causal
Does your experiment test what it is supposed to?
I.e., did changes in the independent variable cause the change in the dependent variable?
Confounds are threat to internal validity - even time of day, different experimenters, etc.
Whereas external validity is about:
Are participants representative?
Can I generalize?
Is there a point to my research?
Confounds - threat to internal validity - some examples
History: occurs when some event other than the planned treatment event occurs between the pretest and posttest (learn new things) - multiple observations may lead to learning effect, effect of previous items, etc.
Maturation: occurs when a physical or mental change occurs over time and affects performance on the dependent variable (even fatigue)
Testing: refers to any change on the second test as a result of having previously taken the test
Instrumentation: refers to any change that occurs in the way the dependent variable is measured (poor measurement, drifting scales)
Biased selection of participants: selecting participants that have different characteristics - e.g., selecting based on scores and statistical regression
Mortality: refers to participants dropping out of a study, differential loss of participants will lead to systematic missing values (more people dropping out of "harder" condition)
Other extraneous variables: time of day, different experimenters, etc.
Some control techniques for eliminating confounds
Random assignment
Matching
Holding the extraneous variables constant
Building the extraneous variables into the research design
Counterbalancing
Statistical control techniques - ANCOVA, multiple regression, partial correlation, etc.
Experimental design
Define variables
What to measure - dependent variables (sell more services)
What to control - independent variables (e.g., minimum account balance, availability of financial services)
Determine different levels
Quantitative: $25,000, $50,000, $100,000
Qualitative: investment service, financial planning service
Determine which design
Between-participants
Within-participants
Mixed
Single-participant
Consider extraneous variables
Error variance
Is variability of scores caused by something other than your independent variables
The effect of extraneous variables
Variables not including the independent variables that may influence the dependent measure
Cannot control for everything
Strange events do happen even with randomization
Reducing error variance
Controlled experimental setting to limit impact of extraneous variables
Hold extraneous variables constant if possible
Match participants on certain criteria
Equating stimuli in all groups
Naming experiment for frequency effects
Use all words or some class?
External validity
Increase the effectiveness of your non-extraneous variables
Strong manipulation of the independent variable - levels are testable and vary enough
More sensitive dependent variable
Better measurement techniques
Random assignment
To help assure groups are similar to each other prior to testing
Each participant has an equal chance to be assigned to each group
Should distribute error variance across groups
But strange things happen
Randomized between-participants design
Randomly assign different people to different groups
Condition 1: A, B, C, D
Condition 2: E, F, G, H
Responses from a given participant appear in only one group
Advantage: safest design - no carryover effects
Disadvantage: least power - large error variance (individual differences)
Matched group design
Equate participants on some criteria - e.g., GPA
Randomly assign a member of each pair to a condition
Advantage: potentially reduce error variance
Disadvantage: time consuming and how do you decide on relevant factors?
Within-participants design
The ultimate matched design
Condition 1: A, B, C, D
Condition 2: A, B, C, D
Each participant see every level of the independent variable
Advantage: lots of power - low error variance (control for individual differences)
Disadvantage: fatigue and carryover effects (e.g., order in which conditions are encountered)
Carryover effects
Effects that carry over from one experimental condition to another
Dangers of repeated measures (when people participate in more than one conditions as in within-subjects design)
E.g., rate of presentation on memory (conditions: one vs. two per second)
Practice effect: practice in the first condition make the second condition easier
Learning effect: people may learn something in one condition that helps then do better in another
Fatigue effect: people do worse on the later condition because they are worn out
Habituation effect: people become habituated and uninterested in performing the later task
Other examples
Sensitization effect: people may become highly sensitive after completing the initial condition
Contrast effect: people may develop an expectation in the initial condition (e.g., difficulty level, rating scales)
Adaptation effect: People may change during the experiment
Dealing with carryover effects
Counterbalancing
Spread carryover equally over all conditions
Complete counterbalancing
k! where k is the number of conditions (3 conditions = 3*2*1 = 6 orders, 5 conditions = 120 orders, 10 conditions = 3,628,800 orders!)
1 2 3
1 3 2
2 1 3
2 3 1
3 1 2
3 2 1
Randomly assign participants to each row
Randomized counterbalancing
Rely on chance to find different orders
Easy to do
Most used method
Partial counterbalancing
Latin Square
Each condition appears at each ordinal position
Each condition precedes and succeeds each other condition at least once
Constructing a Latin Square - assume you have N=5 conditions