Skip to main content

Replication Indoctrination – An Aim of Teaching in Political Science

Introduction


If positivists had a mantra, it would be “validation demands transparency and replication.”

The ideal is that if all the data from a study is posted online, and a thorough account given as to how that data was acquired, then other political scientists can examine everything, and determine how valid the method and its conclusions really are.

This mimicking of natural science methods may seem reasonable, at first, but in this, and succeeding, posts I will discusses its disastrous consequences for the political science profession. I will examine each element of the ideology. In general, my position is that this is the Wrong Conceptual Framework for the subject matter of political science – human beings engaged in political behavior. Why this paradigm persists in political science is another worthy matter of analysis. This post will consider the replication myth.

Is Replication Possible?

For natural science, replication is one of the keys to validation. Suppose an animal study is conducted on a sample of rats, dogs, or chimps that have been made sick with COVID-19. Some are left to see if they can heal on their own. Others are injected with bleach, a technique suggested by President Trump, to see if that is really an effective treatment.[1] The physiology of each animal in a species is very similar to the others in the species. So, a later replication of the study by other scientists, using a new set of critters, can either debunk or validate the claims. If validated, then perhaps volunteers for trials on humans can be solicited.

But political science does not deal with animal bodies. From its very inception, for example, public opinion survey research is far outside the positivistic model, and is actually a fully interpretive practice.

In the act of formulating survey questions, for instance, the political scientist must envision human respondents, and understand something of their culture and language. This requires sufficient empathy to assess the relevant characteristics of the anticipated respondents. Such as the capacity to grasp the meaning of the questions put to them. Already, the questioner is in a human-to-human relationship with his or her respondents, and is personally involved in the interpersonal exchange of meanings. The “objectivity” of observing hapless animals cannot exist in an interpersonal communicative relationship with other people.

The survey is planned and constructed by an individual person, working alone or within a group of other persons. Each person has his or her own unique characteristics. Some are more or less intelligent, more or less articulate, and more or less empathic. If planned and constructed by a group of colleagues, the combination of personal qualities creates a unique configuration that can never be “replicated.” 

Once the survey is administered, and the results are in, they are mere words and numbers on a computer screen until they are interpreted. “Coding” responses is an act of interpreting the meanings conveyed by respondents. Coders and respondents are in a human-to-human relationship. The coders exercise empathy to make sense of the responses. Being human, each coder exercises his or her unique combination of talent, experience, and judgment. 

Taking exit polls on Election Day also entails a human-to-human relationship. If respondents feel disrespected or otherwise resentful towards the poll taker, they may refuse to respond, or provide false information. The personal characteristics of the poll taker are a significant element of the study; yet, they cannot be replicated. For example, a “smart ass,” or a bored questioner will surely skew the responses.

Focus group studies are completely out of the realm of the positivistic replication ideal. Everything there is unique, and cannot be duplicated. Every person is a unique configuration of human qualities, from life experiences, to education, to intelligence, to temperament, religious beliefs, political philosophy, and much more. This applies with greater force to the leader of the focus group. He or she must have a significant degree of empathy to illicit cooperation from the group as a whole, and the talent to see when to hone in on a particular issue or member of the group. The multiple unique person-to-person relations involved in focus groups can never be replicated.

When the positivist façade is stripped from these three common practices in political science – survey research, exit polling, and focus groups – what is exposed is the deliberate effort of human researchers to adapt their methods to their subject matter, other humans. The uniqueness of each study can be obscured by injecting into the reports such terms as “transparency,” or “replication,” but to the interpretivist, such injections are no more effective covers than injecting bleach is for curing COVID-19.

Because unique studies cannot, by definition, be replicated, replication is an inappropriate professional standard. So too is the positivist conception of “validation.” I will return to that, and other positivist concepts, in other posts. But first, let us look a little more at “replication.”

Ethnography and Replication

Ethnography, or participant observation, is another method of studying public opinion. Because this is so clearly a personal activity, producing one-of-a-kind results in a non-replicable manner, positivists discourage its practice in political science. Indeed, it simply does not fit into the cookie cutter mold of “validation demands transparency and replication.” In most cases, the Positivism Police wouldn’t even send such studies out to like-minded peers for review.

The great Kathy Cramer is a pioneer in the Interpretivist study of public opinion. She has found that if she simply submits her research for publication in a journal, without forewarning editors of her unique approach, “the typical reviewer will quickly tune out and give the paper a resounding reject after the first few pages.” Her work does not fulfill positivistic expectations of talk about “variables,” “coefficients,” “data,” and “predictions.” Instead, as she says, “what I do is try to characterize in as rich a manner possible how people are creating contexts of meaning together.” [2]

In her reports, she does not strive for “objective transparency,” because “I do not think it is possible to remove me from the analysis.” As a person, she is a unique configuration of qualities. Hence, “I am against this push for making transcripts and fieldnotes publicly available ... Another scholar would not have all of the relevant data needed for replication unless he or she is me.”

Besides that, she engages her subjects as a trustworthy confidant, and could not betray their confidence by posting everything in an online repository. 

Also, presenting to people as a detached, objective professor “is just about the opposite of the approach needed in the kind of research I conduct.”


I am able to do the kind of work I do because I am willing to put myself out there and connect with people on a human level. And I am able to gather the data that I do because I can tell people verbally and through my behavior that what they see is what they get. If the people I studied knew that in fact they were not just connecting with me, but with thousands of anonymous others, I would feel like a phony, and frankly would not be able to justify doing this work.


In closing, let us consider a public policy study by another outstanding 

Interpretivist, Alice Goffman.  In her book, On the Run, Goffman tells 

of living with a group of inner city youths as a participant 

observer. [3] The study revealed the unintended consequences 

of an urban “get tough on crime” public policy. It exposed how the 

crime policy has been deleterious for a poor urban population living 

under the policy’s jurisdiction.

Her empathic interpretation was based on empirical observation, and shed light on actual facts; namely, the loss in the quality of life her subjects were being made to endure because they constantly feared the police. Family relationships were fractured, the development of interpersonal relationships was stymied, and steady employment was made impossible by the need to flee whenever one felt as the object of a police investigation. 

If Goffman’s study was held to the demands of the quantitativist mantra, it would be deemed unworthy of recognition as incorrect method, and too unique to be replicated or validated.

But her work was original, insightful, and could be very useful knowledge to concerned urban policy makers who might not realize what harm their policy is causing to individuals, families, and communities in their city.

On the Run is also an analogue for grad students and new PhDs. Knowing that the Positivism Police control who gets published in the top political science journals, and that career success requires publication, new scholars must do what is expected of them. Anxiety is widely shared that they will produce the Wrong Kind of Study. Then their work product might be frowned upon, or, worse, scorned as too maverick and out of the positivist coloring lines. Their career could suffer.

This pressure to conform stifles creativity and originality. It turns political science work into a mass production operation. If Good Political Science is measured by the standard of “validation demands transparency and replication,” then the best studies are the ones that look like those which passed muster before them. With each study a model of positivistic method, but saying little that is new or remarkable, political science will make of itself a socially useless sub-group of “academics” in the very worst sense of the word. Indeed, this is already happening, as I will show in a future post.

 

William J. Kelleher, Ph.D.

@InterpretivePo1 

References

[1] https://twitter.com/sarahcpr/status/1253474772702429189

[2] QMMR 2015 https://doi.org/10.5281/zenodo.893069

[3] Alice Goffman. On the Run: Fugitive Life in an American City (University of Chicago Press, 2014) Some of the implications of this book were discussed at the WPSA 2016 Pre-Conference Session entitled “Why Should We Believe You? Evidence and ‘Proof’ in Field and Other Interpretive Research.” (No link available)

 

 

Comments

Popular posts from this blog

How the “But For” Test for Causation in Law can be Adapted to Political Science

For social science, as for law, the concept of “causation” can  take on different forms. These forms of causation are “outside   the box” of the old Newtonian concept of causation. That is, a   one-on-one collision. For example, the “8 ball” in a pool game moved to the corner pocket because the cue ball struck it at the intended angle. The cue ball did that because Minnesota Fats hit the cue ball just right with his pool stick. This is a mechanistic model of causation. In that model, the list of causes prior to Minnesota Fats could go back endlessly; or at least to the Big Bang of 13.8 billion years ago, which, mechanistically, is thought to be the First Cause of everything. However, for the most part, social science, like law, envisions human behavior as conduct for which the actors are responsible. This need not be a “moral” responsibility, for which moral blame is attached. Instead, “causal responsibility” is simply a matter-of-fact, or practical, concept...

Executive Immunity – Beyond the Hype

                                           The media and fund raising emailers are having a ball screaming “the sky is falling! The Trump Supreme Court has given Trump ‘absolute immunity’ for whatever he wants to do as President. He’ll kill us all!” But that’s far from true. The United States v Trump The case is known as US v Trump . In its opinion, the Supreme Court didn’t give Trump anything. In fact, the Court remanded the case to the District Court, and told the lawyers for both sides to start all over again. This time, they have to consider three important points. Before explaining these, lets see how the case got to the High Court. As you probably know, Trump was indicted by a Federal Grand Jury for several criminal charges alleging a conspiracy to change the outcome of the 2020 presidential election. (SCt's Opinion, page 2. All references to the Opinion.) In the...

The Psychology of the Student Protests

University students have been in the news lately because of their protests on campuses and in the streets. Unfortunately, the mainstream media often convey false impressions about the intentions of the protesters. As a result of such misreporting, the meanings to the students of the encampments is widely misperceived. For instance, I recently heard one misinformed person talk about the encampments as if they were a colony of bonobos doing it all day in those tents. As a political scientist who studies all sorts of political behavior, I would like to add a little clarity to our understanding of the mental state, or intentions, of the protesters. Of utmost importance to keep in mind is that the feeling that is moving the majority of the student protesters is a strong compassion for their fellow human beings who are suffering so much in Gaza. The compassion felt by these protesters includes the feeling of such abhorrence for the slaughter they see going on in Gaza that they must...