The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin (...) with an introduction to Type I error rates that distinguishes them from theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors. (shrink)

Scientists often adjust their significance threshold during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha (...) adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses. (shrink)

Kerr coined the term ‘HARKing’ to refer to the practice of ‘hypothesizing after the results are known’. This questionable research practice has received increased attention in recent years because it is thought to have contributed to low replication rates in science. The present article discusses the concept of HARKing from a philosophical standpoint and then undertakes a critical review of Kerr’s twelve potential costs of HARKing. It is argued that these potential costs are either misconceived, misattributed to HARKing, lacking evidence, (...) or that they do not take into account pre- and post-publication peer review and public availability to research materials and data. It is concluded that it is premature to conclude that HARKing has led to low replication rates. (shrink)

The replication crisis has caused researchers to distinguish between exact replications, which duplicate all aspects of a study that could potentially affect the results, and direct replications, which duplicate only those aspects of the study that are thought to be theoretically essential to reproduce the original effect. The replication crisis has also prompted researchers to think more carefully about the possibility of making Type I errors when rejecting null hypotheses. In this context, the present article considers the utility of two (...) types of Type I error probability: the Neyman–Pearson long run Type I error rate and the Fisherian sample-specific Type I error probability. It is argued that the Neyman–Pearson Type I error rate is inapplicable in social science because it refers to a long run of exact replications, and social science deals with irreversible units that make exact replications impossible. Instead, the Fisherian sample-specific Type I error probability is recommended as a more meaningful way to conceptualize false positive results in social science because it can be applied to each sample-specific decision about rejecting the same substantive null hypothesis in a series of direct replications. It is concluded that the replication crisis may be partly due to researchers’ unrealistic expectations about replicability based on their consideration of the Neyman–Pearson Type I error rate across a long run of exact replications. (shrink)

Studies on how one behavior affects subsequent behaviors find evidence for two opposite trends: Sometimes a first behavior increases the likelihood of engaging in additional behaviors that contribute to the same goal (positive behavioral spillover), and at other times a first behavior decreases this likelihood (negative spillover). A factor that may explain both patterns is attitude strength. A stronger (more favorable) attitude toward an issue may make the connections between related behaviors more salient and increase the motivation to work toward (...) the underlying goal. We predicted that people with a stronger (more favorable) attitude are more likely to engage in subsequent behaviors that address an issue they care about. Two experiments tested the prediction in the contexts of pro-environmental and health behavior. Study 1 (N = 378) provided some support for the predicted moderating role of attitude toward the environment when participants recalled either an environmentally friendly or unfriendly action: A strong attitude increased the likelihood, whereas a weak attitude decreased the likelihood of carrying out successive goal-conducive behaviors. When compared to a neutral control condition in Study 2 (N = 929), participants with a weak environmental attitude supported pro-environmental petitions less strongly after an environmentally harmful action. Support for such petitions did not waver, however, among participants with a strong environmental attitude: They consistently acted environmentally friendly. Contrary to the hypothesis, in neither study did strength of attitude toward personal health moderate the effect of an initial behavior in the expected direction. In sum, the two studies provided only limited evidence for behavioral spillover: Participants mostly acted in accordance with their attitude but were hardly affected by recalling previous actions. When behavioral spillover did occur, however, a strong environmental attitude tended to increase the likelihood of acting in an environmentally friendly way, whereas the behavior of those with a weak attitude was less predictable. This research contributes to a more nuanced theoretical understanding of the role of attitude in spillover, but provides only limited evidence for its role as a moderator. (shrink)

The present study examined how emotional fit with culture – the degree of similarity between an individual’ emotional response to the emotional response of others from the same culture – relates to well-being in a sample of Asian American and European American college students. Using a profile correlation method, we calculated three types of emotional fit based on self-reported emotions, facial expressions, and physiological responses. We then examined the relationships between emotional fit and individual well-being (depression, life satisfaction) as well (...) as collective aspects of well-being, namely collective self-esteem (one’s evaluation of one’s cultural group) and identification with one’s group. The results revealed that self-report emotional fit was associated with greater individual well-being across cultures. In contrast, culture moderated the relationship between self-report emotional fit and collective self-esteem, such that emotional fit predicted greater collective self-esteem in Asian Americans, but not in European Americans. Behavioral emotional fit was unrelated to well-being. There was a marginally significant cultural moderation in the relationship between physiological emotional fit in a strong emotional situation and group identification. Specifically, physiological emotional fit predicted greater group identification in Asian Americans, but not in European Americans. However, this finding disappeared after a Bonferroni correction. The current finding extends previous research by showing that, while emotional fit may be closely related to individual aspects of well-being across cultures, the influence of emotional fit on collective aspects of well-being may be unique to cultures that emphasize interdependence and social harmony, and thus being in alignment with other members of the group. (shrink)

Face perception and emotion categorization are widely investigated under laboratory conditions that are devoid of real social interaction. Using mobile eye-tracking glasses in a standardized diagnostic setting while applying the Autism Diagnostic Observation Schedule (ADOS-2), we had the opportunity to record gaze behavior of children and adolescents with and without Autism Spectrum Conditions (ASCs) during social interaction. The objective was to investigate differences in eye-gaze behavior between three groups of children and adolescents either (1) with ASC or (2) with unconfirmed (...) diagnosis of ASC or (3) with neurotypical development (NTD) during social interaction with an adult interviewer in a diagnostic standard situation using the ADOS-2. In a case control study, we used mobile eye-tracking glasses in an ecologically valid and highly standardized diagnostic interview to investigate suspected cases of ASC. After completion of the ASC diagnostic gold standard including the ADOS-2, the participants were assigned to two groups based on their diagnosis (ASC vs. non-ASC) and compared with a matched group of neurotypically developed controls. The primary outcome measure is the percentage of total dwell times assessed for different areas of interest (AOI) with regard to the face and body of a diagnostic interviewer and the surrounding space. Overall, 65 children and adolescents within an age range of 8.3–17.9 years were included in the study. The data revealed significant group differences, especially in the central-face area. Previous investigations under laboratory conditions gave preferential attention to the eye region during face perception to describe differences between ASC and NTD. In this study – using an ecologically valid setting within a standard diagnostic procedure – the results indicate that neurotypically developed controls seem to process faces and facial expressions in a holistic manner originating from the central-face region. Conversely, participants on the Autism Spectrum (tAS) seem to avoid the central-face region and show unsystematic gaze behavior, not using the preferred landing position in the central-face region as the Archimedean point of face perception. This study uses a new approach, and it will be important to replicate these preliminary findings in future research. (shrink)

Psychological sex differences have been studied scientifically for more than a century, yet linguists still debate about the existence, magnitude, and causes of such differences in language use. Advances in psychology and cognitive neuroscience have shown the importance of sex and sexual orientation for various psychobehavioural traits, but the extent to which such differences manifest in language use is largely unexplored. Using computerised text analysis, this study found substantial psycholinguistic sexual dimorphism in a large corpus of English-language novels by heterosexual (...) authors. The psycholinguistic sex differences largely aligned with known psychological sex differences, such as empathising–systemising, people–things orientation, and men’s more pronounced spatial cognitive styles and abilities. Furthermore, consistent with predictions from cognitive neuroscience, novels by lesbian authors showed minor signs of psycholinguistic masculinisation, while novels by homosexual men had a female-typical psycholinguistic pattern, supporting the gender shift hypothesis of homosexuality. The findings on this large corpus of 66.9 million words indicate how psychological group differences based on sex and sexual orientation manifest in language use in two centuries of literary art. (shrink)

One justification for preregistering research hypotheses, methods, and analyses is that it improves the transparent evaluation of the severity of hypothesis tests. In this article, I consider two cases in which preregistration does not improve this evaluation. First, I argue that, although preregistration can facilitate the transparent evaluation of severity in Mayo’s error statistical philosophy of science, it does not facilitate this evaluation in Popper’s theory-centric approach. To illustrate, I show that associated concerns about Type I error rate inflation are (...) only relevant in the error statistical approach and not in a theory-centric approach. Second, I argue that a preregistered test procedure that allows deviations in its implementation does not provide a more transparent evaluation of Mayoian severity than a non-preregistered procedure. In particular, I argue that sample-based validity-enhancing deviations cause an unknown inflation of the test procedure’s Type I (familywise) error rate and, consequently, an unknown reduction in its capability to license inferences severely. I conclude that preregistration does not improve the transparent evaluation of severity in Popper’s philosophy of science or when deviations are allowed. (shrink)