Corey: So we have Joe Cesario back. Joe is an Associate Professor in the Department of Psychology here at Michigan State University. He studies police shootings, and he’s also a broad commentator on research methods in social psychology and broadly a critic of his own field, which makes going to conferences interesting. What I’m interested in hearing about from Joe, however, is that you’ve got pretty strong views about how your own field has conducted this research, and so I’m kind of interested because you’re both a critic of your field on certain topics. You think that there’s evidence that may be police bias in lesser uses of force and other engagements with the criminal justice system, which has presumably come out the research in social psychology, at least in part.

Corey: So you think some of the findings are valid, but you’re actually a general critic of what happens in your own field. I want to begin to look at that. Give us some idea of how a typical social psychology experiment is conducted and what you think might be wrong with both execution of the experiment and the interpretation.

Joe: Right. There’s a whole lot that would be wrong. Let’s start with a particular kind of experimental social psych study, which is one on stereotyping and bias. It’s given that that’s the most relevant to what we’ve been talking about. Usually, the way these things work is that we start by observing some disparity in the world, some group disparity in the world that we care a lot about, whatever it may be. Gender in STEM participation, racial differences in school suspensions and expulsions, any kind of disparity that we see.

Joe: As social psychologists, we start with the assumption that stereotypes matter for understanding that disparity or for explaining that disparity, which is just the sort of natural bias of people who study social psychology. And so to get at that, what we do to show that stereotype effects matter for those kind of disparities, that we would design an experimental social psychology study very, very standard, systematically designed study, where we create, let’s say, resumes of individuals who are exactly alike in every way except for their racial group or their sex group. Or in the case of fatal police shootings, we would say that by showing people hundreds of pictures of black males or white males, and they’re equally likely to be armed or unarmed. Okay? School suspensions, we give people vignettes to read about a child threw an eraser at the teacher. What’s the proper response? And the child’s black or white.

Corey: So let’s zero in just for concrete’s sake on, say the shooting. A typical experiment would have me or Steve here sitting in front of a computer.

Joe: That’s right.

Corey: And what am I looking at on screen?

Joe: You’re going to get a picture is going to appear on the screen. That picture is a static image, let’s say with black or white male holding a gun. Okay? And then in front of you is the keyboard. One key is labeled shoot and another is labeled don’t shoot, and you’ve got to hit those keys as quickly as possible. We might impose some sort of time limit, like you have to respond in 600 milliseconds or something like that. That happens-

Steve: And you should shoot if you see a gun, but if there’s no gun, you should not.

Joe: Typically. Yeah, the decision rule is always shoot with a gun, don’t shoot without a gun, which, of course, isn’t really the decision rule for police officers, which is its own problem, but…

Steve: You’re looking for the main piece of information on which the action is contingent like does he have a gun.

Joe: Right.

Steve: But then the question is whether the ethnicity of the person or the race of the person affects that decision.

Joe: Yeah. And so, you might make 200 judgments like that in rapid succession, one after the other, okay? What we’re going to find, like almost all stereotyping studies find in experimental psychology is that you respond differently to the targets depending on their race. In this case, you’re more likely to press shoot when the person is unarmed if they’re black relative to if they’re white.

Joe: In the school suspension example, you give stronger disciplinary recommendations for a black child relative to a white child, again for doing the same action.

Steve: So I think Corey wants to focus on the limitations of extrapolating from that experimental result to the real world, however, I even question whether the result is statistically real because with p-hacking, people might be running the study just long enough that they accumulate a p less than 0.05 significance effect and then quickly publish a paper. Then when some other group does some very similar experiment, again just within the lab, they find they don’t replicate even the findings of the first study. So how confident are you even in the purely laboratory effect that these people are identifying?

Joe: Yeah, yeah. That’s a great question and big picture problems with social psychology is where this comes in. In the case of race and shooting, I’m extremely confident that that’s a robust finding and a replicable finding. We’ve done dozens of studies. We get the effect every time. It’s really hard not to get that effect in that pure, narrow laboratory study. In other domains, it might not be the case.

Corey: So there are two effects, right? There’s, first of all, you’re more likely to shoot a black suspect who is unarmed, right?

Joe: And that’s the more robust effect, yeah.

Corey: And also, you shoot faster if the suspect is black. Is that correct?

Joe: When they’re armed.

Corey: That’s correct, yes.

Joe: That is often found also, although that’s somewhat contingent on how long you give people to respond to do that, but the case of being more likely to shoot the unarmed person if they’re black is a highly replicable effect.

Steve: And just to hark back to the previous episode, when you did this under more realistic circumstances with real police officers and more realistic videos showing real-life situations like a traffic stop, you found that effect when away. Is that right? They weren’t-

Joe: That’s- yeah.

Steve: They were not more likely to shoot a black suspect.

Joe: That’s exactly right. In this case, it would be a highly replicable, reliable effect that is more or less meaningless in my opinion, for understanding real world shootings.

Corey: Presumably, there’s some assumption in designing these studies that fails to hold in real life that prevents you from extrapolating the finding. What do you think is unrealistic about it that fails to have it be true?

Joe: Right. There’s one assumption and one serious methodological problem that gets at what you’re asking. The assumption and this goes back to the question of bias within the discipline. The assumption is an ideological assumption, which is that when order lies, it is that all groups are basically the same in their behaviors. Okay? And so, the justification for presenting you or Steve in this computer task with an equal number of armed black and white targets, and an equal number of unarmed black and white targets is the assumption that in the real world, black and white citizens are as likely to be armed in the presence of a police officer, okay?

Joe: In the case of the school disciplinary procedure, the reason why we describe in the justification for it, a black child and a white child as performing exactly the same behavior, is the belief that in the real world, black and white school children behave exactly the same. Gender and STEM, the reason why we would present a resume from a male or a female engineering applicant exactly in the same manner is the belief that actually there is equality across the groups.

Steve: So, I thought another issue here would be this is a different population. Instead of a bunch of undergraduates sitting in a classroom doing this, if you brought trained police officers to run in the same experiment, would they also show the effect?

Joe: Right. That’s now another problem beyond the kind of ideological assumption issue that we just talked about. That’s another problem that relates to the general methodological approach. In most of experimental social psychology, in fact, the far majority of it, probably over 90%, what you have are naïve undergraduates, untrained undergraduates, who I would argue don’t even know how to make these decisions. Okay? If you ask them, “What are the rules by which you could use deadly force?” They would have no clue whatsoever, just as if you asked them, “How would you decide whether to make a disciplinary judgment or judge one engineering applicant versus another?” They have no clue what that scope is even like. And so, it’s like a lack of understanding of the expertise literature.

Steve: I mean, there’s a general complaint that convenient sample of undergraduates may not actually reflect what is happening in the situation that you actually care about. Has there ever been a study, where in that still very narrow laboratory setting, where it’s gun, no gun, black or white, you bring in police officers? Has anybody actually just done that experiment?

Joe: Yeah, we did that. A few other people have done that as well, but we did that. We also then used cognitive modeling to understand how were people making decisions. What that allowed us to do was to say, “What are the actual processing differences among trained experts versus untrained undergraduates?” You see really striking differences in terms of how officers use information when they’re making the decision and how they are or are not affected by the person’s race.

Steve: But is the effect still there with cops?

Joe: No.

Steve: Just-

Joe: No, the effect is not there.

Steve: Even in the classroom.

Joe: That’s correct, yeah.

Steve: Okay, thank you.

Joe: Even in the simplified version, the effect is not there because what one thing that we see is that officers, trained officers, who by the way, have to make this decision about what’s in a person’s hand probably hundreds of times a day relative to an undergrad who has never had to do that, they can locate and process the object that’s in the person’s hand much, much more quickly than the untrained undergraduate can do it. And they can do it in a way that more or less blocks out the effect of race-

Steve: Race neutral.

Joe: … from distinguishing what’s in their hand. Right.

Corey: This is sort of fascinating because you’re kind of saying there are actually two dimensions to the explanation. First, you’re saying that there’s a base rate issue as far as how often groups are likely to be armed and you didn’t say it, but I assume underlying your comments is that black suspects are more likely to be armed in encountering police than white suspects. And you think that partly drives the actual bias you find in amongst certain groups. The second is that expertise, even though there is a base rate difference, can actually eliminate that. You can actually just see what’s happening in the video, forget whatever base rate information you had and operate purely upon your visuals in the simulator if you’re an experienced police officer.

Joe: Right. That’s exactly right. Actually, in that second point, even more than that is that in the experimental setups that we have, we aren’t including those factors in the real world that actually do impact people’s judgment. One of the things we do is we strip out all of the complexity of that decision for good reason. We want to try to hone in, for instance on racial bias in the decision to shoot, but what we do is we take out all of the factors that matter for an actual police officer making that decision. Okay? Which in the real world, we’ve shown have very strong impact on an officer’s decision.

Joe: One thing that we did in these kinds of really stripped down laboratory studies is that we just introduced dispatch information. Okay? Officers have dispatch information. If you look at police shootings, overwhelmingly in the real world, they have dispatch information. They’re called to a scene for a particular reason. I think we talked about this in the last episode where stopping someone randomly on a street is really unlikely to result in a fatal police shooting. Okay? Officers have dispatch information going into the situation. We did a study, where we just introduced dispatch information into that standard-

Corey: What’s in this dispatch information?

Joe: Just-

Corey: Give me an example.

Joe: Probabilistic weapon information.

Corey: Give me-

Joe: Oh, sure.

Corey: … how would it read.

Joe: “The suspect is armed.” Okay? “There was a report of a gun,” okay? Is actually what it would be because the officer never knows if the suspect is actually armed until they get there, but, “There’s a report of a gun,” or a race-based information, “A black male has reported having a gun. A white male has reported having a gun, and so on.” The race information is always correct and we manipulate the probabilistic likelihood that the weapon information is correct because sometimes it is for officers and sometimes it’s not for officers.

Joe: What we find is just that one piece of information being added back into the experiment totally wipes out the effect of race on people’s decisions, in this case, even untrained undergraduates, okay? Just completely eliminates the race bias effect. What we showed in the simulator is that things like the scenario that the officer is called to is at nighttime in a dark alley, all of those things have massive effects on the officer’s decisions. What happens is that those things really matter in the real world. When we bring people into the lab, we’ve stripped out all of those strong forces. What we’re left with is this racial bias.

Corey: I want to emphasize when you say, “We’ve stripped out,” you’re saying, normal, other researchers in your field, generally not you in fact. You tend to put other things-

Joe: That’s right.

Corey: … into your experiments. But the standard in the field is to strip everything away.

Joe: That’s right.

Corey: To run a fully controlled experiment with only a single difference.

Joe: Right, right. I guess I still classify myself as a social psychologist since I keep saying, “We,” but yeah, right. That’s right.

Corey: It’s interesting this is kind of a problem, I guess, with our idealized science, right? We often want to run the simplest possible experiment in order to get at the finding and-

Steve: Well, I think if you’re a sophisticated scientist, you can run that experiment and just say, “There was an effect, statically significant effect in the setting that we arrange.” What you infer from that about the real world is where you have to be extremely disciplined about, “Well, maybe these other factors matter.” Then subsequently, they’re shown to matter, right? So-

Joe: Right, right. Granted then, I think we kind of run into the issue of scientists and others trying to sell their findings, right? Scientists, and we in this office, right? Trying to help our scientists on campus sell their findings. What we’re seeing is there’s a huge incentive in fact for people to want to seel their findings beyond what they may actually validly show.

Steve: Yeah. I think there are two slightly different effects. One is that if there is a preconceived notion of reality within the researcher, the research community then obviously, you’re going to get a lot of results which reinforce that because you’ll have a hard time getting something published if it contradicts people’s priors. The other issue is just in order to get it into Nature as opposed to just another good journal, it has to be extremely sexy and have some twist to it. And so, you have a lot of people distorting themselves to try to get that twist so they can get into nature. Then consequently, a lot of the papers in Nature are actually maybe even more likely to be wrong than in some other roughly equivalent journal because of the sensationalist factor that is required to get into Nature.

Joe: That’s part of the bias in general in social psychology or the view in social psychology that we care about these surprising, aha findings, right? That’s what gets into Nature. Showing personality stability from childhood to adulthood, once that’s been shown once, repeatedly showing the importance of that is not going to get you into Nature.

Steve: Yeah, I think I often comment on this that in some disciplines like psychics, replication of someone else’s result or showing that their result is wrong is considered equally as important, almost as important as a big aha thing because people realize how the scientific method actually works. I’m always surprised in these other fields where they don’t seem to understand that and they won’t give people credit for if I show your result is wrong, that’s as important as producing a flashy, but wrong result, which actually just confuses availability in the field for a few years.

Joe: Right. That’s interesting. You don’t even get a publication for replicating, much less credit for it. Even a second-tier journal is going to be biased against publishing a straightforward replication, but it’s worse than that really, which is that you have the time order heuristic that I think Gilman talks about, where the thing that comes in first gets priority in people’s minds even if the failure later on is a much better study.

Steve: Yeah, it’s been shown that a lot of the citations are to paper that were subsequently shown to be wrong, but the people citing them still cite them as if they were correct. And so, there isn’t even good equilibration of knowledge within these fields sometimes.

Joe: I mean, it’s not true entirely. Often over time, the citations of wrong papers begin to go down over time.

Steve: Decade.

Joe: It varies. It varies quite a lot. I mean, one of the really interesting early cases was the case of identifying the AIDs virus, where initially both the American group and a French group that identified the virus, and I forgot the name of the guy, the American guy who identified it. But he basically used the virus from the French group in his paper. He was later accused of misconduct, but over time, citations of his paper went down and the French group continued. And eventually, the French group won The Nobel Prize over this.

Joe: I think there’s actually a third dimension to getting results accepted and this is by the public, right? We don’t just want our things accepted by a journal. If we want them to be taken by the public, and get some public discussion, and then you’ve got to make them even more sexier.

Steve: And not just sexier, but you have to also make that leap from, “I did this very controlled lab experiment. Oh, and this is how you can use it in your own life to stick to your diet plan.”

Joe: Right. There’s a way in which we’re talking about this in a really, not quite malicious, but something like that manner. But there is a more positive spin to it, which is that the reason why we don’t just want to say, “In this sample of undergraduates in this highly simplified task, they showed X,” is because we actually care about the real world. Okay? There’s that element to is as well. We care about real people being shot by the police.

Steve: We do, but we are, I think, insufficiently careful in extrapolating from the simplified laboratory setting to the actual setting that we care about.

Corey: I think I should say it was Gallo, the American with the AIDs virus, who had first claimed to have it. It was Montagnier who actually did identify it.

Corey: There’s one test you’re pretty critical of and that’s The Implicit Association Test. This is a test everyone can take. It’s up on Harvard’s website. It tests implicit bias. I’ve taken the test. Malcolm Gladwell took the test and said he was pretty disturbed to find that he was mildly biased against black people and four white people, which disturbed him because he’s half Jamaican. People made a lot out of this test and you don’t think it shows very much. So you can explain what the test is? I encourage all of our listeners to take it. The results are actually a little bit troubling. I won’t tell you what my biases are because I find them personally very uncomfortable. I’m not biased against black people, I want to… But I am biased against other groups. But what is the test and what’s wrong with the common interpretation of it?

Joe: Okay. The test is basically… Again, you should definitely go online and take it because this is one of these pictures is worth 1,000 words.

Steve: It’s called the IAT.

Joe: Yeah. The Implicit Association Test. Basically, what’s going to happen is it’s a test in which you’re given two different categories. One thing that’s useful about the test is the general. It has a very general structure that can be used essentially for any kind of category in any kind of attribute you can think of. We might say, “Males and females.” Okay? You’re going to have these two categories, males and females. Then you’re going to have a secondary set of categories, which is the attribute that you’re interested in. You might say, “Science versus art.” Okay? For males and females, science and art. You might have a good and bad for black and white, which is the really common one that Gladwell took and that many other people have taken. Good and bad, republican and Democrat. Any kind of categories and attributes you can think of, you can do an IAT with this. Okay?

Joe: You have these two categories, these two dimensions and then you are to make decisions about some set of words or pictures that appear on your screen. You’re going to categorize it in one of these two ways. A picture of a male might appear on the screen. You’ve got to push a button to categorize it in terms of male, all right? Or a picture of a female appears. You push a button that says female. The word experiment appears. You’re going to push a button that says science, right? And the word pottery appears, you’re going to push the button to categorize it as an art. Okay? You’re going to do this over, and over, and over again.

Joe: The key thing is that sometimes the two dimensions are paired in what you might think of as a stereotypic way, other times in a counter-stereotypical way is one way of describing it. You might have one set of trials where on one side of the screen, let’s say the left side of the screen, male and science are both there. And on the right side of the screen, female and art are both there, okay? But then there’s a different set of trials, a different block of trials where male and art are paired together, and female and science are paired together, okay?

Joe: What happens, the main effect, what is called the IAT effect is that it’s more difficult to do the task in those counter stereotypic pairs, okay? When the word experiment appears on the screen or the word microscope or whatever appears on the screen, it’s harder for you to call that a science if the word science is next to the word female, okay? Relative to when the word science is next to the word male, okay? It’s hard for you to say stance is an art when the word art is paired with female. Or, sorry. When the word art is on the same side of the screen as male relative to when it’s on female, okay? That difficulty that people experience when you compare one set of pairings with the other is the essence of the IAT effect.

Steve: So, I mean, if you want just take a very narrow interpretation of this, the whole point is that we do have stereotypes, right?

Joe: That’s right.

Steve: And so, just cognitively, it could be a little bit harder to disentangle yourself from that stereotype. It doesn’t necessarily mean that you have when it really counts, a bias in the real world, right? So-

Joe: That’s exactly right. That’s exactly right. What it is is a measurement of associations, okay? That’s why it’s that name. It’s implicit because it’s very hard to control that. If you tell people, “Don’t take longer with this pairing then that pairing,” it’s really difficult for them to do that, okay? It’s a kind of speeded… In fact, it’s not unlike the Stroop task in the kind of controllability element, okay? It’s very hard to do the Stroop task, which is a famous cognitive psychology test. Maybe the most replicated finding in all of psychology, which is a task in which you’re given words, and you’re to name the color that the words are written in. The word might be written in the color blue. It might be written in the color red and you have to name the color, but sometimes those words are color words. So the word blue might be written in red. It’s hard to do that than when the word blue is written in blue.

Steve: Right. I think there are simple numerical tasks too like if number X is greater than another number. And when they’re close together, your brain has to work harder.

Joe: That’s right, yeah.

Steve: So that you’re slower in figuring that out, but so it’s-

Joe: On its own-

Steve: … there’s some low-level functioning difference.

Joe: Absolutely. Yeah.

Steve: There’s no question about that.

Joe: That’s correct. On its own, there’s no question that people have associations. People have stereotypes. That’s not at issue. What’s at issue is in one case, what exactly is it measuring, and the other is does it matter for real-world disparities, okay? Can we say, “Because I associate male with science, that helps us explain gender disparities in STEM engagement.”

Corey: So why-

Joe: And it’s that leap.

Corey: Why do you think that leap’s invalid?

Joe: A couple reasons. One is related to what we’ve already talked about in terms of my general critique of experimental social psychology is that the landscape for what matters in an individual deciding to pursue STEM, and remaining engaged in STEM, and being good at STEM is in no way represented by that task, okay? Think about all the factors that go into you deciding, like Steve, you’re a STEM and you also, Corey. You’re both STEM, strong STEM successes. Think about every piece of you and every piece of the situation that went into you succeeding at STEM. None of that is represented at all in the IAT.

Corey: Well, here’s a question. I knew a woman who had claimed to me that when she wrote her actual name on her paper, on her exams, let’s call her Julie Stein, she got lower grades than when she wrote J. Stein.

Joe: And so-

Steve: Well, how strong is her finding? How strong are her statistics?

Corey: I honestly don’t know. Let’s suppose she-

Joe: No. We could suppose that’s true actually.

Corey: Suppose she’s right.

Joe: Suppose that’s true. It’s not the case, and I’m going to be clear about this. It’s not the case that I’m saying discrimination or bias or stereotyping effects doesn’t exist. It’s always going to exist, right? There are always going to be human beings on the planet who discriminate against others on some kind of surface characteristic. The question is how much can that explain the kinds of disparities that we see in outcomes.

Steve: The question is how much can this simplified test capture all of that complexity. Maybe it would have been fantastic if it turned out through subsequent analysis that this little tiny computer test actually did capture. If you had a stronger IAT effect in real life, you could show this person really was more sexist or racist, right?

Joe: Right.

Steve: That would be a phenomenal finding, but I think what you’re going to claim maybe is that further analysis doesn’t support that.

Joe: That’s right. In some of the studies that have actually looked at behavior, at real behavior in real situations, you find that the IAT effects a person’s individual level IAT bias. So Corey, however much you did like whatever group you didn’t like, and you’re not going to reveal here who it was, but however much you might not like this group as indexed by the IAT, according to the best data that we have, appears to matter not at all how you behave towards those actual group members, okay? And that’s because there’s a lot of reasons why just in terms of the general complexity, but one of the reasons is that you have time in most kinds of judgment situations to incorporate a lot more information into your interaction with the person or even into your decision about a person, okay?

Joe: What’s forgotten about the IAT and other similar kinds of effects is that if you give people long enough, they won’t show the effect, okay? Just like in the Stroop task, if you just give people two seconds to name the color instead of forcing them to do it in 500 milliseconds, there’s no Stroop effect, and same with the IAT, okay? I can categorize experiment as a science if you just give me a second and a half to do it instead of 600 milliseconds to do it. Then the question is how often in the real world do you have to make that decision in 600 milliseconds when it comes to you participating in STEM or you judging a student to participate in STEM and so on.

Steve: If we could encapsulate this as you have an idealized experiment, which shows an effect. There is an effect in the idealized experiment, but unclear what it predicts about real-world behavior.

Joe: Right, right.

Steve: But I want to go back to this Julie Stein, J. Stein question. A friend of mine, who we were both on the faculty at Yale at the same time. She’s a psychologist now at Cornell. She and I believe her husband published a paper in which they did, I think, very realistic studies with resumes and the actual details on the resume, the CVs were the same, and they just switched the names to something obviously from female to obviously male. These were reviewed by STEM hiring committees, okay? Actual hiring committees, I believe. They ranked the applicants and then they randomized whether the applicant was male or female without changing the actual details of the strength of the CV.

Steve: What they found was a two-to-one female, preference for hiring females by STEM hiring committee over male. It’s the opposite sign of Julie Stein. Different situation, of course. This wasn’t a term paper or something. Now, they took incredible heat for this finding, and immediately attacked for this finding. And probably if she had not been a woman, they probably been, maybe fired from Cornell. I’m kidding, for having this finding. But as an administrator, I actually understand that even if there’s no subconscious bias, we are telling all of our hiring committees in STEM to find women. Even if there’s no subconscious bias in favor of women, there is an explicit pressure on all these committees to hire women. Anyway, the result is not surprising to me at all, but apparently highly controversial in psychology.

Joe: Right, yeah, yeah.

Corey: But that’s, in fact, the opposite finding that you incur in as in corporate America from exactly the same experiment, where you take a female, exactly the same resume. You have a female name or a male name or you have an African-American name or a classically white name, and send this out to a hiring committee at say, the local investment bank. There, you find bias in favor of white men. You’re chomping at the bit.

Joe: No, no, no. Yeah. I wanted to let you finish. Yeah. I mean, those kinds of studies actually have the same problem as the experimental social psychology studies. In fact, they more or less come from experimental social psychology, but in economics, those kinds of, which there, I think they call them audit studies, were criticized almost two decades ago by, I think, it was by Heckman who pointed out that whether there’s market-wide discrimination against some group is irrelevant for understanding that group’s actual employment and actual employment discrimination because employment doesn’t happen across the market. Employment happens in some set of marginal companies that actual people interact with and get hired or not hired.

Joe: So in the case of an investment firm, even if you showed, for instance at that Goldman Sachs let’s say, showed strong discrimination against black applicants, okay? In this kind of audit study. We sent Goldman Sachs black applications or black resumes and white resumes, and they showed really strong racial discrimination. His argument was that that’s not going to tell you anything about actual racial discrimination in investment bank, let’s say, across the market because the question is, A, are black applicants applying to Goldman Sachs. If not, if that transaction isn’t occurring in the real world, then it doesn’t really matter whether Goldman Sachs is engaged in discrimination against hypothetical applicants or not.

Corey: What if I’m applying to Goldman Sachs? Doesn’t it suggest that I have a less chance of getting hired by Goldman Sachs than if you were to apply to Goldman Sachs, and it’d be equivalent qualifications?

Joe: That’s exactly right, exactly right. The question is how often is it the case that black and white applicants have equal resumes, okay? If it’s the case that there are differences in resumes, then whether Goldman Sachs is going to discriminate against you and I, when we’re hypothetically equally qualified is just irrelevant to the question of general labor market discrimination. That’s similar in the case to-

Corey: Hold up.

Joe: Yeah.

Corey: But we’re applying, right? And you’re going to get in and I’m not. You’re going to make a lot of money and I’m not. Maybe you can’t generalize this finding terribly far, but it does seem to suggest that a well-qualified black applicant coming out of, say Amherst College, right? May have trouble getting into Goldman Sachs compared to a white applicant coming out of Amherst College.

Joe: That’s right. There’s no question about that. Again, I’m not denying that and we can agree that Goldman Sachs is in fact discriminating…

Corey: We, in fact, don’t know.

Joe: I don’t want…

Corey: We don’t know. Let’s say-

Joe: Yeah. This is all hypothetical.

Corey: We know nothing about Goldman.

Steve: Right, right.

Corey: I’m a McKinsey alum. I see no evidence that McKinsey discriminates. Apparently, McKinsey and Goldman compete for the same people. So perhaps there’s no-

Joe: In a totally hypothetical world, we could agree actually that Goldman Sachs is engaged in discrimination against black applicants. That’s not at issue. If you and I have the same resume, we’re both coming out of the same colleges, and we’re both applying to Goldman Sachs, I agree with you then that that’s a problem with you that you’re not going to get the job or that I’m more likely to get the job than you are. My point is that that doesn’t tell us anything about population-wide disparities in employment for black and white applicants because if you’re the only black applicant who has the quality resume that you have, then whether Goldman Sachs is discriminating against other black applicants is essentially irrelevant because you are the one who’s qualified. And if that’s not a widespread market phenomenon in terms of equally qualified applicants, then it’s not going to do anything to explain those disparities.

Joe: And so again, this gets back to the assessment that underlies those kinds of audit studies as well as experimental social psychology that absent discrimination, all groups would look exactly the same in their outcomes because all groups are exactly the same in all of the qualifying, personality, and other characteristics that go into obtaining some outcome.

Corey: You were going to say, Steve?

Steve: I thought you were making a different point, Joe. I thought you were saying that because the CVs are seldom identical, it’s similar to the situation with the shooting experiments in the labs. There’s a lot of side information and the forcing them to focus on race maybe doesn’t extrapolate to the real-world situation where there are richer differences between the candidates.

Joe: That’s right, yeah, yeah.

Steve: I thought that’s what you were, the point.

Joe: That’s also part of it.

Steve: Okay.

Joe: Right, yeah.

Corey: I don’t know if you said this explicitly, but you’re basically accusing your own field of left-wing political bias. Actually, I want to get clarification. Are you accusing them of that kind of bias? Is that bias explicit or in fact, are these implicit bias researchers exhibiting implicit bias in their conduct of their research into implicit bias?

Joe: Sure. Let’s start with an anecdote. I had a senior, senior, one of the most well-known stereotyping researchers. I won’t say his name. He came and gave a talk here that I had invited him to give a talk a number of years ago. A very senior person, works in one of the top research institutions in the country, studies stereotyping and prejudice. With no hesitation when we were talking about these kinds of issues, he said that he would not take a conservative student into his lab, okay? And showed no sign at all that that was a problem to him. He said, “Look, I’m just going on…” I mean, the idea that he said this without any kind of self-awareness was incredible. He said, “I’m just going on likelihoods.” Okay? Honest to God, he said, “The likelihood that somebody who is conservative is going to come into my lab and not get along with the rich, diverse people that are in my lab is higher than if they’re a liberal student, so I simply would not take a conservative student if I knew that they were conservative.”

Steve: Wow.

Joe: Yeah. That’s an incredible thing that somebody would admit in public to say. Yeah.

Corey: The experiments you’ve been talking about, you think this is not just a singular data point?

Joe: No. I mean, if you look at the data, the data are utterly clear that there is just massive, ideological lopsidedness in all of the social sciences, but especially in social psychology.

Steve: I think there are well-known surveys taken at national meeting of social psychologists and they just ask, “Is anybody here not far left of center?” And it’s like 1% or something, one person in the room.

Joe: Oh, yeah. Well, Jonathan Haidt did this at one of our major social personality conferences, where there was something like 500 people in the room. When he asked who was conservative to raise their hand, I think three or four people raised their hadn’t, which is just an incredible lopsidedness.

Steve: Right.

Corey: Is there a response bias? People are afraid to raise their hand?

Steve: No, they can have anonymous surveys too, but…

Corey: Yeah, yeah, yeah. Okay. No, I didn’t [crosstalk 00:35:40] it.

Steve: I don’t think it’s disputed by anybody that many of these disciplines are overwhelmingly on one side ideologically and-

Joe: And then the question is does it matter.

Steve: I think there’s not a single department, like if you went to the mechanical engineering department, you might find some Republicans, but there’s not a single department on campus that is overwhelming on the right.

Corey: I think economics departments may be majority conservative in some schools.

Steve: No. Well, maybe at some schools like George Mason or something. Maybe they’re libertarians or something, but generally, you’ll still find more people left of center than right of center.

Corey: I think that’s definitely true in most fields. I think there was a book a few years ago on biased academia. They tend to find that the biases existed primarily in the social sciences and humanities. It’s not so much in engineering and not in medicine, for example.

Steve: Right. No.

Corey: What do you mean, for example?

Steve: No, I disagree. No, I disagree. I mean, the professoriate, in general, has shifted left. Even if you look at some field which you think is like, “Okay. This has nothing to do with ideology. We’re just studying quarks.” You’ll still find more people left of center than right of center because the professoriate, academia is shifting.

Corey: No. It may be shifting, but as far as I recall, the evidence suggested that it was pretty evenly split in medicine. At least this was as of 10 years ago, right? In medicine, it was pretty evenly split and engineering, it was fairly evenly split.

Steve: I definitely will grant you it’s closer to being balanced, but I still think it’s shifted to the left.

Joe: Right. No, I was going say, the question then, the second part you asked is whether that really matters at all for the practice of psychology or any of the social sciences. To me, it’s indisputable that it does, but I would just point to stereotyping research as a great example of this, where it took at least a half of a century before anybody thought to ask whether stereotypes were accurate, okay? For decades, and decades, and decades, researchers in social psychology simply assumed that stereotypes were inaccurate. Sometimes, actually by definition, they would by definition say that they were inaccurate beliefs about different social groups.

Joe: Lee Jussim, who’s the main person who does stereotype accuracy research has story after story of him trying to get this work published and just the incredible difficulty of doing that, even though as he claims, and I like this phrase, “Stereotype accuracy is the largest and most replicable finding in all of social psychology,” which is an incredible thing that it took us that long to even ask the question, much less to find out the answer to it.

Corey: You’re in social psychology and there’s another field, criminology, which studies something very, very similar to you, if not identical.

Steve: There’s definitely overlap.

Corey: Yeah. At least I’m talking his particular topic, right?

Steve: His topic, yes.

Corey: Do you find kind of biases that you describe in social psychology, do you find those in criminology or not?

Joe: My sense, and field-wide, I have much less experience with criminology or criminal justice, but my sense is that that bias is not quite as strong in criminal justice. If you look at the work on police shootings, decades and decades ago, they were asking questions about to what degree does neighborhood crime level predict somebody being shot by the police. Now, admittedly, the data weren’t as good decades ago as they are now for us to ask that question, but that question never comes up in experimental social psychology studies of police shootings. So I do think there’s disciplinary differences in terms of a greater appreciation for the role that lots of variables play in something like police shooting, whereas in social psychology, it is about demographic characteristics only.

Steve: Moving away from research a little bit to teaching, I don’t think that it’s controversial that most conservative students when they go to college, so imagine you just happen to be a republican and you go to college, and you go into your required social science course. Many of them have the feeling that they’re being preached at. The professor probably thinks that they’re conveying accurate, fair scientific results of social impact import top this group of students, but I think most of the students that are on the right would say that, “No, they’re trying to actually change my way of view the world, calling it science.” And it’s clearly unbalanced in the sense that they will go through their entire undergraduate career without meeting a social science professor who actually has the same social views as they do. I think that’s a common story among people who are right of center.

Joe: Yeah. I could tell you. I teach a class on stereotyping, prejudice, and discrimination. I try to have a very wide range of views represented in that class. One of the authors that we read a fair amount about because the class has a lot to do with group disparities is Thomas Sowell, okay? The first time I taught that class, I had a student come up to me, and this was the Spring semester of this student’s fourth year at college. He, at the end of the semester, he came up to me and said, “I just want to thank you. This is the first class I’ve ever had in any social science class where a view that was not on the far left was presented.” He went three and a half years without ever getting any kind of viewpoint-

Steve: And I’m not making-

Joe: … other than one.

Steve: I’m not making a normative statement or even a scientific statement about which side is right or wrong in these issues.

Joe: That’s right.

Steve: But it is true that half our population is passing through the schools and their view of what they’re being told by their professors is that it’s highly alien to what they and their families believe.

Joe: That’s right.

Steve: And so, I don’t think it’s healthy for our institutions in general for that to be the case.

Joe: The other half are passing through our institutions never getting a conflicting viewpoint.

Steve: Exactly. Never being challenged.

Joe: That’s right, yeah.

Corey: I think this is actually one of the best arguments from my former field, which is philosophy. When I used to teach, I taught it morals as a social issues class. And they make a very explicit point of presenting the pro-life position and arguing for it strenuously. I made a point of trying to make sure that people actually couldn’t tell what my ideological perspective was.

Steve: But now they would pick at your class for that.

Corey: I’m not sure. I think philosophy is sort of, perhaps abstract enough.

Steve: If it were-

Corey: What was-

Steve: If it were philosophy one and there were not a lot of majors, they’re taking it as a distribution, don’t you’d think you’d get very strong pushback for teaching, for example, that human rights began at conception and things like that?

Corey: I honestly don’t know. I haven’t been in a philosophy class for almost 20 years now, but what I can say is there was pushback from my giving the pro-life argument from pro-choice students, where I didn’t get that from the other side.

Steve: Again just to clarify, none of us are necessarily pro-life. We’re just talking about the argument given by pro-life people not being something acceptable in academia at this point. That’s my impression.

Corey: Honestly, I can’t say. I have to say, when I give the argument, I still give these arguments to people today, right? I can find arguments that people will find compelling until they find out they’re pro-life, and I can actually push people in that direction. When we get people-

Steve: No, I’m asking whether you would be picked, whether the students would-

Corey: So I don’t know.

Steve: Somebody would stand up in your class and-

Corey: So-

Steve: … say, “Hey. You’re making me feel unsafe because-

Corey: Well-

Steve: … you’re giving these arguments.”

Corey: Well, I’m not sure you or I know this, right? You and I are in an office where we actually don’t encounter undergraduates very often.

Steve: No, I know. I’m just asking whether you… My impression is that that could happen.

Corey: Yeah. I actually don’t think so. I think in philosophy, there may be a bias a philosophy, but I think it’s understood just as part of philosophy education, you do look at things from different perspectives. Whether people take these seriously or not is another matter.

Joe: My impression actually is that it’s highly dependent on the university.

Steve: Yes, correct.

Joe: I think if you did that at Yale, you are going to be picketed.

Steve: Correct. Yeah.

Joe: If you did that at MSU-

Steve: Low chance here.

Joe: … I don’t believe you would here. Yeah.

Steve: Yes. Less chance here.

Joe: It’s different approach, yeah.

Steve: Yeah.

Corey: Last, I want to emphasize, I think this may just be a problem for the humanities and the social sciences. You go through natural science class, you go through physics classes, you go through math classes, there’s never a political issue raised in any context at all.

Steve: Right, because it’s not really that relevant to the material, right?

Corey: That’s right, that’s right.

Steve: So yes.

Corey: It’s interesting is it’s actually not so clearly relevant to the material of psychology in general, although psychology’s kind of shifted in this direction over time, but if you’re still-

Steve: Social psychology.

Corey: Yeah. If you’re studying basic cognitive psychology perhaps, processing, looking at visual processing, looking at language, et cetera, et cetera, it’s not clear this is going to come up, although linguistics in my experience was very, very, very left-wing oriented.

Steve: Yeah.

Joe: But even in other areas, where it could possibly come up, those sub-areas in psychology, there’s a number of them that have somehow sidestepped those issues and have very good replicable findings that could have political implications, but I think have other features that prevent that kind of skew from happening. Personnel selection, for example, is an example of that. Personnel selection is great research that’s done in IO, really replicable research.

Steve: So-

Joe: We know a lot about that.

Steve: Be a little more concrete about that.

Joe: Sure. What kinds of personalities, let’s say, would perform better at different kinds of jobs? Okay. You want some battery of test that you could give to someone to find out where you should place them, let’s say in the army, for example, which is one of the origins of this research. That would be an example. Cognitive ability measurement is a highly replicable field, where… and that has its own problems, as we know well, but that’s preceded, where we have some the data on a topic like that. There are some fields, subareas in psychology, mostly… Personality assessment, for example, is another one where they have really good understanding of measurement and they do very careful measurement, where I think they’re doing highly replicable work.

Steve: Yeah. I think the field of psychometrics among the social sciences is on the strongest, I would say, empirical footing in terms of their models work, they predict reality, they’re used broadly in the military to college admissions, et cetera. I want to get back to these more controversial issues that are covered. I want to give the typical left of center professor’s defense of it, which is they would just say, “Look. We got all the science on our side.” Sure, some republican kid comes in from the sticks, “And I got to educate this kid. I got to tell him that a lot of stuff that he’s learning from his parents, and his church, and his elders, and the political party his dad belongs to are just wrong. I just got to tell him that. That’s our business at the university. We’re dealing in truth and I’m here to educate that kid.” I believe that’s the general attitude that most of these professors have, right or wrong. Do you agree with that?

Joe: I completely agree that that’s the general attitude, absolutely. Yeah.

Steve: Yeah. I mean, if some kid comes in my class and says, “I don’t believe in quarks.” Well, I’m going to say, “Okay. It’s my job to explain to you why we believe in quarks and why we can produce them at accelerators, and we know all their properties, and they’ve been precisely measured.” And I think I’m fully justified in, if the kid just won’t shut up about not believing in quarks, eventually I have to say, “Hey. Let the other kids learn. I can’t take any more of your interrupting my lecture. I know you don’t believe in quarks or evolution or whatever it is.” But on these very socially relevant controversial issues, I believe we’ve reached this point where the professoriate thinks it has the right, the scientifically correct view, and they’re okay jamming it down the throats of half their students who really don’t believe in what they’re saying. That’s my view of the situation.

Joe: I think that’s accurate and I think the difference between the quirk example and the social psychology example is that what the social psychology professors don’t realize is that their base of knowledge is just completely missing, okay? That it’s missing other kinds of information that actually undermine a lot of what they believe to be true about the world. And the general practice of social psychology, which we’ve hinted at, has a lot of problems in terms of being able to them claim that they have some true information.

Joe: In a standard social psychology study, you could open up any social psychology journal and find examples of this, “We think that this variable matters for some kind of outcome, so we measured it. We threw it into a model, a statistical model. And there’s a statically significant coefficient, okay? Where X is greater than Y, p less than 0.05.” Okay. They’re going to take that as truth that that thing matters, and that they’ve uncovered something real.

Joe: Independent of all the other p-hacking, and publication problems, and everything else that undermined actual claims to truth is that at no point do they say, “Oh, and by the way, this variable explains one half of 1% of variance in people’s responses.” Is that really meaningful in any way for actual human behavior? It just isn’t done.

Steve: I wanted to mention that there is a movement, I believe, among younger social scientists to basically improve the replicability and the strength of the scientific method as practiced in their fields. Now, I know that in very strong ways like preregistration of experimental designs, things meant to provide p-hacking. So statistical modifications or methodological modifications. There’s, I think, a lot of much support in the community for fixing these issues. But for this issue of ideological bias, which is a kind of separate thing, how much support is there for that issue among these people? So the people who are the most progressive in terms of improving the scientific method. Do they also understand the ideological issue as of concern or is that just a completely different thing for them?

Joe: It’s a completely different thing, but my anecdotal experiences anyway or not formal measurement of this is that there’s some fair degree of overlap there, where the people who are concerned about scientific practice issues do recognize that ideological lopsidedness is a problem as well for us getting to a better place in social psychology. My sense is those aren’t totally distinct populations.

Corey: Do we have affirmative action then coming for Republicans in social science? Joe, for example, if you had opportunity to willfully encourage as many conservatives to join your lab as possible just for, I’m not really joking, for ideological balance, and to provide a counterbalance to any potential left-wing bias, would you do that?

Joe: No. I’m more interested in early intervention programs, okay? If we had early intervention programs for conservative students, I think that would be actually an interesting program to pursue. In terms of at the hiring stage, I’m generally speaking against affirmative action hiring. And so, I think I would like to believe that I would apply that equally across different kinds of groups and say that I wouldn’t engage in that in hiring practices. But if we had some program to encourage conservatives to actually remain in academia at the early stage, at the undergraduate level or something, I’m not sure I would be 100% against that. Just as I’m not against other kinds of programs that do those early intervention.

Corey: So some extent, Steve, I think that social psychology has suffered from political bias probably because its evidential basis is weak, right? But you find a lot of other fields that aren’t so, where the experiments aren’t perhaps as straightforward as physics. Bias of any sort can really distort the field. I saw this in linguistics, right? You had these factions in linguistics. I was thinking as you were discussing having a conservative kid come into a social science class and a disbeliever in quarks come into a physics class. If you had someone who believed in say, lexical functional grammar, right? Or theory of linguistics did not involve movement and the underlying syntax into a Chomskyan class, you get almost exactly the same response. People maybe discuss it with them for a little bit, then probably get extremely angry, and mentally shut the person out.

Corey: There is a complete ideological homogeneity into these linguistic sub-camps because you can’t actually run a straightforward experiment that’s non-ideological. I mean, quant had the idea long ago, which suggested fundamentally that the data underdetermined the theory pretty radically in these fields. I think to some extent, you may find that in other fields. When that happens, you find ideological bias coming in because it’s going to shape what theory you’re going to build upon. This data that you could easily build another theory on.

Corey: Now, the question is with this in social science is are the data that really compelling to force a certain theoretical choice? And so I’m asking, do you often find the data can be interpreted multiple ways, and biases coming in, and how it’s being interpreted? Is it possible to find data that it’s just so solid that you can’t have another interpretation? You clearly have that in your very specific case of your simulators.

Steve: To take a very specific example, I know Lee Jussim’s work about stereotypes that actually quite often, some stereotype is derived by people cognitively, and does actually present statistical trends in a population group. And I think he has quite strong data about some of those stereotypes being quite accurate. Who in your field believes him? Does anybody believe him? If you just go to the school down the road, Michigan, and you just take a poll of the faculty in the psychology department there, who are in some adjacent area to his work, how many of them believe his results?

Joe: Yeah. My guess is that he’s still slow in winning coverts onto his position, even though, again, he would argue that he’s got the strongest data probably of anyone. You can look to his other similar work on the self-fulfilling prophecy, okay? And teach your expectations for student outcomes, where there also, he showed that if there’s any kind of self-fulfilling prophecy, it’s extremely small, an extremely small effect. Go open up your standard social psychology textbook and it still discusses and teaches the self-fulfilling prophecy as a major force on students’ outcomes. I think it’s very, very slow, even when in the case, Steve, where you bring up where you have really great data. It’s a very, very slow process.

Joe: Corey, your question about are the data, is it the case also though that the data are weak, that interpretation can come in, interpretative bias can come in. It’s absolutely the case. And so, in the case of, for instance, you have a stereotyping effect, that stereotyping effect or that categorical effect might explain a minute amount of people’s actual responses, okay? Less than 1% of people’s responses. The interpretative bias comes in in not paying attention to that fact, which has direct importance for understanding real-world behavior. It comes in, for instance, the data are abundantly clear that when you provide people with what’s called individuating data, so just data about you as an actual person, that categorical effects and stereotyping effects get completely wiped out. Okay. If I know something about you, the categorical influence is virtually zero, okay? Well, researchers don’t pay attention to that. So we craft our studies in ways where you don’t get any of that information so we can show that there’s a stereotyping effect.

Corey: So let’s be more specific. You’re saying if you know it’s me, then the fact that I’m black or I’m male or I’m from Massachusetts, what sort of thing you’re studying, that fades away-

Joe: That’s right. That’s right. If I had some-

Corey: … into the details of my personal-

Joe: That’s right. If I have details about you and your individual behavior, the likelihood of some category that you belong to impacting my judgment is very, very low, okay? I mean, we know that. But that information then, the bias comes in of not providing that kind of information, okay? And not interpreting the size of stereotyping effects in light of that kind of information.

Steve: I think the statement that in fields where the data underdetermine the theory, so empirically weka areas, whether it’s string theory or social psychology or whatever, that progress is hard and ideology can easily dominate. I think everybody agrees with that. What is more troubling to me is in the few cases where we actually have overwhelming strong, rigorous data that’s replicable, and still, people refuse to update their priors based on that information. That, to me, is extremely difficult to deal with, however even in physics, when quantum mechanics was first invented, that’s where this first quip came from that the field is going to advance one funeral at a time. Because these old, classical physicists could not accept quantum mechanics and the field advanced just basically as those guys died, they were replaced by people who had learned it when they were younger, and fully understood it, and accepted it. Maybe that’s what we’re in for. Do you ever get demoralized about the state of affairs?

Joe: Oh, yeah. All the time. Yeah. There’s a lot of demoralization, particularly when you’re trying to publish work that does go against people’s priors and you just see the evaluation of that work be highly skewed, but there is some optimism. I think the optimism is not to wish for people’s deaths, but that the optimism is in younger scientists. Again, this goes back also to the broader issues of reform in scientific practice is that the young people coming into the field are much more amenable than the senior people.

Steve: Even on these ideological issues.

Joe: I think so, yeah, yeah.

Steve: One of the things that I feel is that science or the academy, it’s a very monastic life. You’re basically taking a monastic vow to sacrifice all kinds of things to really push for the search for truth, but the search for truth is a collective effort. So the other people in your field, the other scholars have to be at least somewhat receptive to arguments and evidence. If they’re not, then it’s very easy to say, “Why am I taking this monastic vow when these people are not even engaging with my results in honest and good faith?” I can just easily see people get disillusioned by that. I think it takes incredible courage to fight an entire discipline that has some entrenched ideology that is in opposition to your findings.

Joe: Yeah. Well, it took Lee how many decades before he could start this publish and have any sort of acceptance and appreciation of his work.

Corey: It’s fascinating that Wilson actually found himself probably in a pretty similar situation back in the ’70s, and he went around the standard journal, basically the journal infrastructure to the public with these findings. He wrote books and that causes a debate that then penetrated back into the field.

Steve: Yeah. He would be an interesting guy for us to interview even though he’s pretty old now because he actually lived through this entire thing in one out, right? And so, now his views, which were made him a pariah, people were pouring water over him or throwing pies at him at his seminars to it being basically more or less accepted, right? A lot of things that he said in sociobiology. Yeah.

Joe: Right.

Corey: Well, Joe, look, as always, it’s been a lot of fun to talk to you.

Joe: All right. Thanks for having me.