WRAPPING IT UP

Quality of Data: Psychometrics

• The essence of science is that conclusions should be based on data. Data can vary widely in quality; in personality psychology, the important dimensions of data quality are reliability, validity, and generalizability.
• Reliability refers to the stability or repeatability of measurements. Validity refers to the degree to which a measurement actually measures what it is trying to measure. Generalizability is a broader concept that subsumes both reliability and validity and refers to the kinds of other measurements to which a given measurement is related.

Research Design

• The plan for gathering psychological data is the research design. The three main types of design are case, experimental, and correlational.
• Case studies examine particular phenomena or individuals in detail and can be an important source of new ideas. To test these ideas, correlational and experimental studies are necessary.
• Each of the three methods has advantages and disadvantages. Case methods describe a phenomenon in detail but have uncertain generalizability. Experimental methods can establish causality but only for a particular method in a particular context. Correlational studies are more realistic but cannot directly establish causality.
• The key difference between the two research studies is that experiments demonstrate what can happen, whereas correlational studies demonstrate what does happen. A complete research program should include both methods.

Evaluating the Strength of a Finding

• The statistical significance of a result represents the probability that the data would have been obtained if the “null hypothesis” were true, but it is typically misinterpreted as indicating the probability that the substantive (non-null) hypothesis is true. Null-hypothesis significance testing (NHST) has many problems that are increasingly acknowledged. In particular, statistical significance is not the same as the strength or importance of the result.
• A better way to evaluate research than statistical significance is in terms of effect size, which describes numerically the degree to which one variable is related to another. One good measure of effect size is the correlation coefficient.
• The dependability of a research finding can only be evaluated, ultimately, through replication. This issue came to a head in recent years when some prominent findings were found to not be as well established as psychologists had assumed. No single study can establish the truth of any result, which is why researchers need to repeat and extend findings and always be open to the implications of new data.

Ethical Issues

• Some people are uncomfortable with the practice of personality assessment because they see it as undignified or unfair. However, because people inevitably judge each other’s personalities, the real issue is whether personality assessment should be based on informal intuitions or formalized techniques.
• Research must be careful to do nothing to harm participants. Potentials for harm include subjecting people to traumatic experiences, deceiving them, or violating their privacy. The potential for the violation of individuals’ privacy is a particularly important issue to be aware of for the future.
• Psychology, like any other science, can be used for benefit or for harm, so the methods and uses of research deserve careful thought.
• It is now widely acknowledged that the participants studied in psychological research are far from representative of the wider population, but the discrepancy might be even wider in researchers themselves. Increased diversity in researchers, over time, will lead to a wider diversity in the topics studied and findings obtained, but this goal will not be reached quickly or easily.
• Norms of “open science” encourage scientists to fully report all of their research methods and findings, including studies that fail to find the expected or hoped-for result, and to share their data with other scientists.

Conclusion

• As a citizen, it is important to keep close watch on the activities and methods of schools, police departments, doctors, businesses, governments, and scientists.

Online

Center for Open Science

The Center for Open Science provides many resources to make it easier to do good science. It is where a researcher can “pre-register” a study (state the predictions and planned analyses for a study before it begins), share and access data from other researchers, and share articles not yet published. The website is cos.io.

Society for the Improvement of Psychological Science

This society, founded in 2016, is already growing to be a major force in psychology. Its purpose is to develop and advocate for improved methods and practices. It holds annual meetings, and its website is improvingpsych.org.

Philosophy of Psychology Lectures

The late Paul Meehl, a longtime professor at the University of Minnesota, is probably the most respected methodologist in the history of personality psychology. His ideas about how to connect data with theory provide keen insights into modern controversies, such as issues of replicability and open science, discussed in this chapter. Lectures he gave in one of his graduate-level courses are available online. Although the course is called “Philosophy of Psychology,” as he points out in the first lecture the content is really the philosophy of how to do research in psychology (and other fields). You can watch and listen for free at meehl.umn.edu/recordings/philosophical-psychology-1989.

Print

Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals and meta-analysis. Taylor & Francis.

The most important of a new generation of statistics textbooks that go beyond conventional null-hypothesis significance testing to teach alternative methods for estimating effect sizes and confidence intervals, and accumulating research results over many studies. A lot of recent, interesting information on the “new statistics” is available at Cumming’s website: thenewstatistics.com/itns.