In recent years, there has been a heated debate about how to interpret findings that seem to show that humans rapidly and automatically calculate the visual perspectives of others. In the current study, we investigated the question of whether automatic interference effects found in the dot-perspective task (Samson, Apperly, Braithwaite, Andrews, & Bodley Scott, 2010) are the product of domain-specific perspective-taking processes or of domain-general “submentalizing” processes (Heyes, 2014). Previous attempts to address this question have done so by implementing inanimate controls, such as arrows, as stimuli. The rationale for this is that submentalizing processes that respond to directionality should be engaged by such stimuli, whereas domain-specific perspective-taking mechanisms, if they exist, should not. These previous attempts have been limited, however, by the implied intentionality of the stimuli they have used (e.g. arrows), which may have invited participants to imbue them with perspectival agency. Drawing inspiration from “novel entity” paradigms from infant gaze-following research, we designed a version of the dot-perspective task that allowed us to precisely control whether a central stimulus was viewed as animate or inanimate. Across four experiments, we found no evidence that automatic “perspective-taking” effects in the dot-perspective task are modulated by beliefs about the animacy of the central stimulus. Our results also suggest that these effects may be due to the task-switching elements of the dot-perspective paradigm, rather than automatic directional orienting. Together, these results indicate that neither the perspective-taking nor the standard submentalizing interpretations of the dot-perspective task are fully correct.