Currently, production and comprehension are regarded as quite distinct in accounts of language processing. In rejecting this dichotomy, we instead assert that producing and understanding are interwoven, and that this interweaving is what enables people to predict themselves and each other. We start by noting that production and comprehension are forms of action and action perception. We then consider the evidence for interweaving in action, action perception, and joint action, and explain such evidence in terms of prediction. Specifically, we assume (...) that actors construct forward models of their actions before they execute those actions, and that perceivers of others' actions covertly imitate those actions, then construct forward models of those actions. We use these accounts of action, action perception, and joint action to develop accounts of production, comprehension, and interactive language. Importantly, they incorporate well-defined levels of linguistic representation. We show how speakers and comprehenders use covert imitation and forward modeling to make predictions at these levels of representation, how they interweave production and comprehension processes, and how they use these predictions to monitor the upcoming utterances. We show how these accounts explain a range of behavioral and neuroscientific data on language processing and discuss some of the implications of our proposal. (shrink)

Although researchers studying human speech recognition (HSR) and automatic speech recognition (ASR) share a common interest in how information processing systems (human or machine) recognize spoken language, there is little communication between the two disciplines. We suggest that this lack of communication follows largely from the fact that research in these related fields has focused on the mechanics of how speech can be recognized. In Marr's (1982) terms, emphasis has been on the algorithmic and implementational levels rather than on the (...) computational level. In this article, we provide a computational-level analysis of the task of speech recognition, which reveals the close parallels between research concerned with HSR and ASR. We illustrate this relation by presenting a new computational model of human spoken-word recognition, built using techniques from the field of ASR that, in contrast to current existing models of HSR, recognizes words from real speech input. (shrink)

Binding theory (e.g., Chomsky, 1981) has played a central role in both syntactic theory and models of language processing. Its constraints are designed to predict that the referential domains of pronouns and reflexives are nonoverlapping, that is, are complementary; these constraints are also thought to play a role in online reference resolution. The predictions of binding theory and its role in sentence processing were tested in four experiments that monitored participants' eye movements as they followed spoken instructions to have a (...) doll touch a picture belonging to another doll. The instructions used pronouns and reflexives embedded in picture noun phrases (PNPs) containing possessor phrases (e.g., Pick up Ken. Have Ken touch Harry's picture of himself). Although the interpretations assigned to pronouns were generally consistent with binding theory, reflexives were frequently assigned interpretations that violated binding theory. In addition, the timing and pattern of eye movements were inconsistent with models of language processing that assume that binding theory acts as an early filter to restrict the referential domain. The interpretations assigned to reflexives in PNPs with possessors suggest that they are binding‐theory‐exempt logophors, a conclusion that unifies the treatment of reflexives in PNPs. (shrink)

The sounds that make up spoken words are heard in a series and must be mapped rapidly onto words in memory because their elements, unlike those of visual words, cannot simultaneously exist or persist in time. Although theories agree that the dynamics of spoken word recognition are important, they differ in how they treat the nature of the competitor set—precisely which words are activated as an auditory word form unfolds in real time. This study used eye tracking to measure the (...) impact over time of word frequency and 2 partially overlapping competitor set definitions: onset density and neighborhood density. Time course measures revealed early and continuous effects of frequency (facilitatory) and on set based similarity (inhibitory). Neighborhood density appears to have early facilitatory effects and late inhibitory effects. The late inhibitory effects are due to differences in the temporal distribution of similarity within neighborhoods. The early facilitatory effects are due to subphonemic cues that inform the listener about word length before the entire word is heard. The results support a new conception of lexical competition neighborhoods in which recognition occurs against a background of activated competitors that changes over time based on fine-grained goodness-of-fit and competition dynamics. (shrink)

Pickering & Garrod's (P&G's) theory could be modified to describe how listeners rapidly incorporate context to generate predictions about speech despite inter-talker variability. However, in order to do so, the content of predicted percepts must be expanded to include phonetic information. Further, the way listeners identify and represent inter-talker differences and subsequently determine which prediction method to use would require further specification.

Our target article proposed that language production and comprehension are interwoven, with speakers making predictions of their own utterances and comprehenders making predictions of other people's utterances at different linguistic levels. Here, we respond to comments about such issues as cognitive architecture and its neural basis, learning and development, monitoring, the nature of forward models, communicative intentions, and dialogue.

Predicates like “coloring‐the‐star” denote events that have a temporal duration and a culmination point (telos). When combined with perfective aspect (e.g., “Valeria has colored the star”), a culmination inference arises implying that the action has stopped, and the star is fully colored. While the perfective aspect is known to constrain the conceptualization of the event as telic, many reading studies have demonstrated that readers do not make early commitments as to whether the event is bounded or unbounded. A few visual‐world (...) studies tested the processing of telic predicates during online sentence processing, demonstrating an early integration of aspectual and temporal cues. By employing the visual‐world paradigm, we tested the incremental processing of the perfective aspect in Italian in two eye‐tracking studies in which listeners heard durative predicates in the perfective form in a scenario showing a completed and a non‐completed event. Differently from previous studies, we compared telic durative predicates such as “coloring‐the‐star” to punctual predicates such as “lighting‐the‐candle.” While for punctual predicates, the inferences of telicity (the event has a telos) and of culmination (the telos is reached) are lexically encoded in the perfective verb, for durative predicates, the degree of event completion (visually encoded) needs to be integrated with perfective aspect (linguistically encoded) for the culmination inference derivation. By modulating the interaction of visual and linguistic stimuli across the two experiments, we show that the verb's perfective aspect triggers the culmination inference incrementally during sentence processing, offering novel evidence for the continuous integration of linguistic processing with real‐world visual information. (shrink)

Although researchers studying human speech recognition (HSR) and automatic speech recognition (ASR) share a common interest in how information processing systems (human or machine) recognize spoken language, there is little communication between the two disciplines. We suggest that this lack of communication follows largely from the fact that research in these related fields has focused on the mechanics of how speech can be recognized. In Marr's (1982) terms, emphasis has been on the algorithmic and implementational levels rather than on the (...) computational level. In this article, we provide a computational‐level analysis of the task of speech recognition, which reveals the close parallels between research concerned with HSR and ASR. We illustrate this relation by presenting a new computational model of human spoken‐word recognition, built using techniques from the field of ASR that, in contrast to current existing models of HSR, recognizes words from real speech input. (shrink)

Computational modelling has proven to be a valuable approach in developing theories of spoken-word processing. In this paper, we focus on a particular class of theories in which it is assumed that the spoken-word recognition process consists of two consecutive stages, with an `abstract' discrete symbolic representation at the interface between the stages. In evaluating computational models, it is important to bring in independent arguments for the cognitive plausibility of the algorithms that are selected to compute the processes in a (...) theory. This paper discusses the relation between behavioural studies, theories, and computational models of spoken-word recognition. We explain how computational models can be assessed in terms of the goodness of fit with the behavioural data and the cognitive plausibility of the algorithms. An in-depth analysis of several models provides insights into how computational modelling has led to improved theories and to a better understanding of the human spoken-word recognition process. (shrink)

Earlier studies have shown that musically trained individuals may have a benefit in adverse listening situations when compared to non-musicians, especially in speech-on-speech perception. However, the literature provides mostly conflicting results. In the current study, by employing different measures of spoken language processing, we aimed to test whether we could capture potential differences between musicians and non-musicians in speech-on-speech processing. We used an offline measure of speech perception, which reveals a post-task response, and online measures of real time spoken language (...) processing: gaze-tracking and pupillometry. We used stimuli of comparable complexity across both paradigms and tested the same groups of participants. In the sentence recall task, musicians recalled more words correctly than non-musicians. In the eye-tracking experiment, both groups showed reduced fixations to the target and competitor words’ images as the level of speech maskers increased. The time course of gaze fixations to the competitor did not differ between groups in the speech-in-quiet condition, while the time course dynamics did differ between groups as the two-talker masker was added to the target signal. As the level of two-talker masker increased, musicians showed reduced lexical competition as indicated by the gaze fixations to the competitor. The pupil dilation data showed differences mainly in one target-to-masker ratio. This does not allow to draw conclusions regarding potential differences in the use of cognitive resources between groups. Overall, the eye-tracking measure enabled us to observe that musicians may be using a different strategy than non-musicians to attain spoken word recognition as the noise level increased. However, further investigation with more fine-grained alignment between the processes captured by online and offline measures is necessary to establish whether musicians differ due to better cognitive control or sound processing. (shrink)

The sounds that make up spoken words are heard in a series and must be mapped rapidly onto words in memory because their elements, unlike those of visual words, cannot simultaneously exist or persist in time. Although theories agree that the dynamics of spoken word recognition are important, they differ in how they treat the nature of the competitor set—precisely which words are activated as an auditory word form unfolds in real time. This study used eye tracking to measure the (...) impact over time of word frequency and 2 partially overlapping competitor set definitions: onset density and neighborhood density. Time course measures revealed early and continuous effects of frequency (facilitatory) and on set based similarity (inhibitory). Neighborhood density appears to have early facilitatory effects and late inhibitory effects. The late inhibitory effects are due to differences in the temporal distribution of similarity within neighborhoods. The early facilitatory effects are due to subphonemic cues that inform the listener about word length before the entire word is heard. The results support a new conception of lexical competition neighborhoods in which recognition occurs against a background of activated competitors that changes over time based on fine-grained goodness-of-fit and competition dynamics. (shrink)

We identify a number of principles with respect to prediction that, we argue, underpin adult language comprehension: (a) comprehension consists in realizing a mapping between the unfolding sentence and the event representation corresponding to the real‐world event being described; (b) the realization of this mapping manifests as the ability to predict both how the language will unfold, and how the real‐world event would unfold if it were being experienced directly; (c) concurrent linguistic and nonlinguistic inputs, and the prior internal states (...) of the system, each drive the predictive process; (d) the representation of prior internal states across a representational substrate common to the linguistic and nonlinguistic domains enables the predictive process to operate over variable time frames and variable levels of representational abstraction. We review empirical data exemplifying the operation of these principles and discuss the relationship between prediction, event structure, thematic role assignment, and incrementality. (shrink)