As Macnamara (1978) once asked, how can we talk about what we see? We report on a study manipulating realistic dynamic scenes and sentences aiming to understand the interaction between linguistic and visual representations in real-world situations. Specifically, we monitored participants’ eye movements as they watched video clips of everyday scenes while listening to sentences describing these scenes. We manipulated two main variables. The first was the semantic class of the verb in the sentence and the second was the action/motion (...) of the agent in the unfolding event. The sentences employed two verb classes—causatives (e.g., break) and perception/psychological (e.g., notice)—which impose different constraints on the nouns that serve as their grammatical complements. The scenes depicted events in which agents either moved towards a target object (always the referent of the verb-complement noun), away from it, or remained neutral performing a given activity (such as cooking). Scenes and sentences were synchronized such that the verb onset corresponded to the first video frame of the agent motion towards or away from the object. Results show effects of agent motion but weak verb-semantic restrictions: causatives draw more attention to potential referents of their grammatical complements than perception verbs only when the agent moves towards the target object. Crucially, we found no anticipatory verb-driven eye movements toward the target object, contrary to studies using non-naturalistic and static scenes. We propose a model in which linguistic and visual computations in real-world situations occur largely independent of each other during the early moments of perceptual input, but rapidly interact at a central, conceptual system using a common, propositional code. Implications for language use in real world contexts are discussed. (shrink)

How does the language we speak affect our perception? Here, we argue for linguistic relativity and present an explanation through “language-induced automatized stimulus-driven attention” : Our respective mother tongue automatically influences our attention and, hence, perception, and in this sense determines what we see. As LASA is highly practiced throughout life, it is difficult to suppress, and even shows in language-independent non-linguistic tasks. We argue that attention is involved in language-dependent processing and point out that automatic or stimulus-driven forms of (...) attention, albeit initially learned as serving a linguistic skill, account for linguistic relativity as they are automatized and generalize to non-linguistic tasks. In support of this possibility, we review evidence for such automatized stimulus-driven attention in language-independent non-linguistic tasks. We conclude that linguistic relativity is possible and in fact a reality, although it might not be as powerful as assumed by some of its strongest proponents. (shrink)

The science of mind wandering has rapidly expanded over the past 20 years. During this boom, mind wandering researchers have relied on self-report methods, where participants rate whether their minds were wandering. This is not an historical quirk. Rather, we argue that self-report is indispensable for researchers who study passive phenomena like mind wandering. We consider purportedly “objective” methods that measure mind wandering with eye tracking and machine learning. These measures are validated in terms of how well they predict self-reports, (...) which means that purportedly objective measures of mind wandering retain a subjective core. Mind wandering science cannot break from the cycle of self-report. Skeptics about self-report might conclude that mind wandering science has methodological foundations of sand. We take a rather more optimistic view. We present empirical and philosophical reasons to be confident in self-reports about mind wandering. Empirically, these self-reports are remarkably consistent in their contents and behavioral and neural correlates. Philosophically, self-reports are consistent with our best theories about the function of mind wandering. We argue that this triangulation gives us reason to trust both theory and method. (shrink)

Cognitive science depends on abstractions made from the complex reality of human behaviour. Cognitive scientists typically wish the abstractions in their theories to be universals, but seldom attend to the ontology of universals. Two sorts of universal, resulting from Galilean abstraction and materialist abstraction respectively, are available in the philosophical literature: the abstract universal—the one-over-many universal—is the universal conventionally employed by cognitive scientists; in contrast, a concrete universal is a material entity that can appear within the set of entities it (...) describes, of which it represents the essential, paradigmatic case. The potential role of concrete universals in cognitive science is discussed. (shrink)

Reingold, Reichle, Glaholt, and Sheridan reported a gaze-contingent eye-movement experiment in which survival-curve analyses were used to examine the effects of word frequency, the availability of parafoveal preview, and initial fixation location on the time course of lexical processing. The key results of these analyses suggest that lexical processing begins very rapidly and is supported by substantial parafoveal processing. Because it is not immediately obvious that these results are congruent with the theoretical assumption that words are processed and identified in (...) a strictly serial manner, we attempted to simulate the experiment using the E-Z Reader model of eye-movement control. These simulations were largely consistent with the empirical results, suggesting that parafoveal processing does play an important functional role by allowing lexical processing to occur rapidly enough to mediate direct control over when the eyes move during reading. (shrink)

Chinese readers' eye movements were simulated in the context of the E-Z Reader model, which was developed to account for the eye movements of readers of English. Despite obvious differences between English and Chinese, the model did a fairly good job of simulating the eye movements of Chinese readers. The successful simulation suggests that the control of eye movements in reading Chinese is similar to that in an alphabetic language such as English.

Cognitive Science, Volume 46, Issue 2, February 2022.

In this study, we investigate the processing of object-denoting emoji in sentences using eye tracking. We hypothesize that (a) such emoji are more difficult to process when used as word replacement; and (b) their processing is subject to ambiguity constraints similarly to what happens with words. We conduct two experiments in which participants have to read sentences in which an emoji either follows or replaces a word. Control stimuli not containing emoji are also tested. In the second experiment, the emoji (...) are presented in two different disambiguating contexts. First fixation duration, total visit duration, and total revisit duration are modeled in the various conditions using linear mixed models. Both our hypotheses are supported. We observe longer total visit time for non-redundant emoji, and higher values for all three measures for ambiguous emoji. We conclude that lexical access may be more difficult for emoji especially when they are used as word replacements and their meaning is not immediately clear. Furthermore, we also conclude that non-redundant emoji are more difficult to integrate in the processing of the sentence than the equivalent words, or emoji used in a redundant way. In turn, our results indicate that emoji may not always be as immediate and easy to process for readers in spite of their popularity. (shrink)

What are the effects of word-by-word predictability on sentence processing times during the natural reading of a text? Although information complexity metrics such as surprisal and entropy reduction have been useful in addressing this question, these metrics tend to be estimated using computational language models, which require some degree of commitment to a particular theory of language processing. Taking a different approach, this study implemented a large-scale cumulative cloze task to collect word-by-word predictability data for 40 passages and compute surprisal (...) and entropy reduction values in a theory-neutral manner. A separate group of participants read the same texts while their eye movements were recorded. Results showed that increases in surprisal and entropy reduction were both associated with increases in reading times. Furthermore, these effects did not depend on the global difficulty of the text. The findings suggest that surprisal and entropy reduction independently contribute to variation in reading times, as these metrics seem to capture different aspects of lexical predictability. (shrink)

A fundamental question in reading research concerns whether attention is allocated strictly serially, supporting lexical processing of one word at a time, or in parallel, supporting concurrent lexical processing of two or more words (Reichle, Liversedge, Pollatsek, & Rayner, 2009). The origins of this debate are reviewed. We then report three simulations to address this question using artificial reading agents (Liu & Reichle, 2010; Reichle & Laurent, 2006) that learn to dynamically allocate attention to 1–4 words to “read” as efficiently (...) as possible. These simulation results indicate that the agents strongly preferred serial word processing, although they occasionally attended to more than one word concurrently. The reason for this preference is discussed, along with implications for the debate about how humans allocate attention during reading. (shrink)

We report an eye movement experiment that investigates the effects of collocation strength and contextual predictability on the reading of collocative phrases by L2 English readers. Thirty-eight Chinese English as foreign language learners read 40 sentences, each including a specific two-word phrase that was either a strong or weak adjective-noun collocation and was either highly predictable or unpredictable from the previous sentence context. Eye movement measures showed that L2 reading times for the collocative phrases were sensitive to both collocation strength (...) and contextual predictability. However, an interaction effect between these factors, which appeared relatively late in the eye movement record, additionally revealed that contextual predictability more strongly influenced time spent reading weak compared with strong collocations. This was most likely because the greater familiarity of strong collocations facilitated their integration, even in the absence of strong contextual constraint. We discuss the findings in terms of the value of collocations in second language learning. (shrink)

We explore the idea that eye-movement strategies in reading are precisely adapted to the joint constraints of task structure, task payoff, and processing architecture. We present a model of saccadic control that separates a parametric control policy space from a parametric machine architecture, the latter based on a small set of assumptions derived from research on eye movements in reading (Engbert, Nuthmann, Richter, & Kliegl, 2005; Reichle, Warren, & McConnell, 2009). The eye-control model is embedded in a decision architecture (a (...) machine and policy space) that is capable of performing a simple linguistic task integrating information across saccades. Model predictions are derived by jointly optimizing the control of eye movements and task decisions under payoffs that quantitatively express different desired speed-accuracy trade-offs. The model yields distinct eye-movement predictions for the same task under different payoffs, including single-fixation durations, frequency effects, accuracy effects, and list position effects, and their modulation by task payoff. The predictions are compared to—and found to accord with—eye-movement data obtained from human participants performing the same task under the same payoffs, but they are found not to accord as well when the assumptions concerning payoff optimization and processing architecture are varied. These results extend work on rational analysis of oculomotor control and adaptation of reading strategy (Bicknell & Levy, ; McConkie, Rayner, & Wilson, 1973; Norris, 2009; Wotschack, 2009) by providing evidence for adaptation at low levels of saccadic control that is shaped by quantitatively varying task demands and the dynamics of processing architecture. (shrink)

We propose a framework for including information‐processing bounds in rational analyses. It is an application of bounded optimality (Russell & Subramanian, 1995) to the challenges of developing theories of mechanism and behavior. The framework is based on the idea that behaviors are generated by cognitive mechanisms that are adapted to the structure of not only the environment but also the mind and brain itself. We call the framework computational rationality to emphasize the incorporation of computational mechanism into the definition of (...) rational action. Theories are specified as optimal program problems, defined by an adaptation environment, a bounded machine, and a utility function. Such theories yield different classes of explanation, depending on the extent to which they emphasize adaptation to bounds, and adaptation to some ecology that differs from the immediate local environment. We illustrate this variation with examples from three domains: visual attention in a linguistic task, manual response ordering, and reasoning. We explore the relation of this framework to existing “levels” approaches to explanation, and to other optimality‐based modeling approaches. (shrink)

While positive psychology has drawn increasing interests among researchers in the SLA literature recently, little is known with respect to the relationship between positive psychology and mental processes during L2 reading. To bridge the gap, the present study investigated whether and how positive psychology (self-efficacy) influences word reading strategies during L2 sentence reading. Based on previous studies, eye-movement patterns with first-fixation locations closer to the beginning of a word can be characterized as an attempt to process the word with a (...) local strategy, whereas first-fixation locations farther away from the beginning and closer to the center of a word can be considered as an attempt to use a global strategy. Eye movements of a group of Japanese learners of English (N = 59) were monitored, and L2 reading self-efficacy was used to assess the participants’ positive belief towards their L2 reading skills. Based on B. L. Fredrickson’s (1998) broaden-and-build theory, we predicted an effect of L2 reading self-efficacy on participants’ first-fixation locations. Results from mixed-effects regression showed that while reading strategies depended in part on other factors such as L2 reading proficiency and word properties, L2 self-efficacy influenced reading strategy. The present data suggest that while more self-efficacious L2 readers prefer a more efficient global strategy, attempting to read the word as a whole word, less self-efficacious L2 readers tend to employ a local strategy, focusing more on sublexical information. These findings lend support to the broaden-and-build theory in the context of L2 processing. The present study has implications for how positive psychology works along with L2 proficiency in the development of strategic selection during reading. (shrink)

Parafoveal preview benefit is an implicit measure of lexical activation in reading. PB has been demonstrated for orthographic and phonological but not for semantically related information in English. In contrast, semantic PB is obtained in German and Chinese. We propose that these language differences reveal differential resource demands and timing of phonological and semantic decoding in different orthographic systems.

In the last decade, reading research has seen a paradigmatic shift. A new wave of computational models of orthographic processing that offer various forms of noisy position or context-sensitive coding have revolutionized the field of visual word recognition. The influx of such models stems mainly from consistent findings, coming mostly from European languages, regarding an apparent insensitivity of skilled readers to letter order. Underlying the current revolution is the theoretical assumption that the insensitivity of readers to letter order reflects the (...) special way in which the human brain encodes the position of letters in printed words. The present article discusses the theoretical shortcomings and misconceptions of this approach to visual word recognition. A systematic review of data obtained from a variety of languages demonstrates that letter-order insensitivity is neither a general property of the cognitive system nor a property of the brain in encoding letters. Rather, it is a variant and idiosyncratic characteristic of some languages, mostly European, reflecting a strategy of optimizing encoding resources, given the specific structure of words. Since the main goal of reading research is to develop theories that describe the fundamental and invariant phenomena of reading across orthographies, an alternative approach to model visual word recognition is offered. The dimensions of a possible universal model of reading, which outlines the common cognitive operations involved in orthographic processing in all writing systems, are discussed. (shrink)