A push for transparency in research

BOONE, N.C. — The field of political science is pushing toward greater data access and research transparency, or DA-RT. An Appalachian State University faculty member is among leaders identifying how best to achieve that, and the recommendations may impact other academic disciplines as well.

Dr. Ellen M. Key has published two articles in PS: Political Science and Politics, a journal of the American Political Science Association, calling for the need for improved DA-RT. One, published last month with colleagues at the California Institute of Technology (Caltech), identified specific recommendations to improve data organization, clarity and usability.

Providing data in the most clear, functional and well-documented way strengthens other researchers’ efforts to replicate studies — and replication is important to science of any kind, she said.

“Replication helps verify what we think we know. A lot of studies are probabilistic, so if we can replicate these studies and find findings that are consistent, we’re more confidant that it’s not just a chance finding and that our theories are correct,” said Key, assistant professor in the Department of Government and Justice Studies in the College of Arts and Sciences.

“If we want our research to be taken seriously and people are acting on what we recommend, then we’re affecting not just ourselves and the discipline, we’re impacting how other people conduct their lives and their work. And if that is false or a fluke of chance or in limited circumstances, then our research is having a negative impact.”

The motivation for her articles was a 2014 political science study published in, and later retracted from, Science magazine. It tested persuasion methods and supposedly showed that a brief conversation with gay political canvassers could sway voters to support same-sex marriage.

“If you’re into promoting tolerance, this (study) sounded fantastic,” Key said. But the findings turned out to be questionable.

In their editorial retraction, Science editors noted the article had misrepresented survey incentives, inaccurate information about research sponsors and statistical irregularities identified by independent researchers. In addition, the original researcher did not produce survey data from which other research could confirm whether the findings were valid.

Additional researchers wanting to test the same persuasion methods on other social attitudes could not because the original study was, in essence, falsified, Key said.

“That’s an example of something that wouldn’t have been identified without (the attempt at) replication, and it has a big effect on how the field is perceived and how people go about conducting campaigns and polling,” Key said.

She cited examples from other fields: economics, in which misplaced punctuation in an Excel file created faulty results in one study; and health, in which a study purportedly showing a connection between autism and vaccines hasn’t been successfully replicated but has been accepted as fact by many people.

“It could be your theory is wrong, or the way you came about the findings is human error,” she said about possible research flaws.

The specific recommendations in the article “Research Replication: Practical Considerations,” published with Caltech’s R. Michael Alvarez and Lucas Nunez, address organization, clarity and usability. The recommendations include preferred practices for:

• file naming;
• archiving;
• operation system compatibility; and
• documentation of how data is coded, among others.

“A simple rule for authors to follow when creating their replication materials is to put themselves in the place of another person whose only knowledge about the materials is what the author provides,” the authors wrote. “In the future, the construction of well-organized, clear, and usable replication materials will depend on the training of scholars so that research replication is effectively incorporated into their workflows.”

The recommendations are feasible for most quantitative researchers and possibly qualitative researchers as well, Key said.

“It’s easiest if you start your data collection with an eye toward ultimately making the information available to others. Not every recommendation will apply to every project. It’s not one size fits all, but we’ve tried to write the recommendations as general and useful as possible.”

The field of political science is working to figure out “where the burden should lie” — whether on researchers or journals, which set their own publication rules, Key said. Many political scientists believe it is a shared responsibility, she said.

“It would be great if this became a discipline-wide expectation that information would become available to everybody. And we’d like to see the data not stored on personal websites but on durable, institutional archives that could be maintained a long time, such as Harvard’s Dataverse Network,” she said.

Key sees the recommendations as applicable to other fields. “The recommendations in terms of how to build replication into your workflow, creating clearly labeled files and showing how these files are ordered and what code produces what analyses — all that should be applicable regardless of field.

“To put everything out in the sunshine, we’ll ultimately have better science,” Key said.

About the Department of Government and Justice Studies

Appalachian State University’s Department of Government and Justice Studies offers undergraduate programs in political science and criminal justice, and graduate programs in political science and public administration. Housed in the College of Arts and Sciences, the department has over 600 undergraduate majors and more than 70 graduate students. Learn more at https://gjs.appstate.edu.

About the College of Arts and Sciences

The College of Arts and Sciences is home to 16 academic departments, two stand-alone academic programs, two centers and one residential college. These units span the humanities and the social, mathematical and natural sciences. The College of Arts and Sciences aims to develop a distinctive identity built upon our university's strengths, traditions and unique location. Our values lie not only in service to the university and local community, but through inspiring, training, educating and sustaining the development of our students as global citizens. There are approximately 5,850 student majors in the college. As the college is also largely responsible for implementing Appalachian's general education curriculum, it is heavily involved in the education of all students at the university, including those pursuing majors in other colleges. Learn more at https://cas.appstate.edu.

About Appalachian State University

Appalachian State University, in North Carolina’s Blue Ridge Mountains, prepares students to lead purposeful lives as global citizens who understand and engage their responsibilities in creating a sustainable future for all. The transformational Appalachian experience promotes a spirit of inclusion that brings people together in inspiring ways to acquire and create knowledge, to grow holistically, to act with passion and determination, and embrace diversity and difference. As one of 17 campuses in the University of North Carolina System, Appalachian enrolls about 19,000 students, has a low student-to-faculty ratio and offers more than 150 undergraduate and graduate majors.