A variety of mechanisms contribute to word learning. Learners can track co-occurring words and referents across situations in a bottom-up manner. Equally, they can exploit sentential contexts, relying on top–down information such as verb–argument relations and world knowledge, offering immediate constraints on meaning. When combined, CSWL and SLCL potentially modulate each other's influence, revealing how word learners deal with multiple mechanisms simultaneously: Do they use all mechanisms? Prefer one? Is their strategy context dependent? Three experiments conducted with adult learners reveal (...) that learners prioritize SLCL over CSWL. CSWL is applied in addition to SLCL only if SLCL is not perfectly disambiguating, thereby complementing or competing with it. These studies demonstrate the importance of investigating word-learning mechanisms simultaneously, revealing important characteristics of their interaction in more naturalistic learning environments. (shrink)

Currently, production and comprehension are regarded as quite distinct in accounts of language processing. In rejecting this dichotomy, we instead assert that producing and understanding are interwoven, and that this interweaving is what enables people to predict themselves and each other. We start by noting that production and comprehension are forms of action and action perception. We then consider the evidence for interweaving in action, action perception, and joint action, and explain such evidence in terms of prediction. Specifically, we assume (...) that actors construct forward models of their actions before they execute those actions, and that perceivers of others' actions covertly imitate those actions, then construct forward models of those actions. We use these accounts of action, action perception, and joint action to develop accounts of production, comprehension, and interactive language. Importantly, they incorporate well-defined levels of linguistic representation. We show how speakers and comprehenders use covert imitation and forward modeling to make predictions at these levels of representation, how they interweave production and comprehension processes, and how they use these predictions to monitor the upcoming utterances. We show how these accounts explain a range of behavioral and neuroscientific data on language processing and discuss some of the implications of our proposal. (shrink)

Disfluencies include editing terms such as uh and um as well as repeats and revisions. Little is known about how disfluencies are processed, and there has been next to no research focused on the way that disfluencies affect structure-building operations during comprehension. We review major findings from both computational linguistics and psycholinguistics, and then we summarize the results of our own work which centers on how the parser behaves when it encounters a disfluency. We describe some new research showing that (...) information associated with misarticulated verbs lingers, and which adds to the large body of data on the critical influence of verb argument structures on sentence comprehension. The paper also presents a model of disfluency processing. The parser uses a Tree Adjoining Grammar to build phrase structure. In this approach, filled and unfilled pauses affect the timing of Substitution operations. Repairs and corrections are handled by a mechanism we term “Overlay,” which allows the parser to overwrite an undesired tree with the appropriate, correct tree. This model of disfluency processing highlights the need for the parser to sometimes coordinate the mechanisms that perform garden-path reanalysis with those that do disfluency repair. The research program as a whole demonstrates that it is possible to study disfluencies systematically and to learn how the parser handles filler material and mistakes. It also showcases the power of Tree Adjoining Grammars, a formalism developed by Aravind Joshi which has yielded results in many different areas of linguistics and cognitive science. (shrink)

We identify a number of principles with respect to prediction that, we argue, underpin adult language comprehension: (a) comprehension consists in realizing a mapping between the unfolding sentence and the event representation corresponding to the real‐world event being described; (b) the realization of this mapping manifests as the ability to predict both how the language will unfold, and how the real‐world event would unfold if it were being experienced directly; (c) concurrent linguistic and nonlinguistic inputs, and the prior internal states (...) of the system, each drive the predictive process; (d) the representation of prior internal states across a representational substrate common to the linguistic and nonlinguistic domains enables the predictive process to operate over variable time frames and variable levels of representational abstraction. We review empirical data exemplifying the operation of these principles and discuss the relationship between prediction, event structure, thematic role assignment, and incrementality. (shrink)

Pickering & Garrod's (P&G's) framework assumes an efference copy based on the interlocutor's intentions. Yet, elaborate attribution of intentions may not always be necessary for online prediction. Instead, contextual cues such as speaker gaze can provide similar information with a lower demand on processing resources.

We show how an epistemology informed by cognitive science promises to shed light on an ancient problem in the philosophy of mathematics: the problem of exactness. The problem of exactness arises because geometrical knowledge is thought to concern perfect geometrical forms, whereas the embodiment of such forms in the natural world may be imperfect. There thus arises an apparent mismatch between mathematical concepts and physical reality. We propose that the problem can be solved by emphasizing the ways in which the (...) brain can transform and organize its perceptual intake. It is not necessary for a geometrical form to be perfectly instantiated in order for perception of such a form to be the basis of a geometrical concept. (shrink)

While verbs and argument structure constructions are essential for deriving sentence meaning, their roles in sentence processing remains less known. To address this issue, the present study explored how a verb’s lexical properties and the strength of verb–construction associations influence second language (L2) sentence processing. In two self-paced reading experiments, Korean-speaking learners of English and native English speakers read English argument structure constructions containing verbs with varying lexical properties and association strength. In both Experiment 1 (involving the prepositional dative construction) (...) and Experiment 2 (involving the caused-motion construction), only the L2-English learners, and not the native speakers, spent more time integrating the target constructions with verbs of weaker associations than those with stronger associations. Furthermore, the learners exhibited greater difficulty with verb–construction integration when the verbs were less frequent and less concrete; L2 proficiency did not modulate these effects. Our findings support the constructionist approach, suggesting that both verbs and verb–construction associations play an instrumental role in sentence comprehension. (shrink)

We present a neural dynamic model that perceptually grounds nested noun phrases, that is, noun phrases that contain further (possibly also nested) noun phrases as parts. The model receives input from the visual array and a representation of a noun phrase from language processing. It organizes a search for the denoted object in the visual scene. The model is a neural dynamic architecture of interacting neural populations which has clear interfaces with perceptual processes. It solves a set of theoretical challenges, (...) including the problem of keeping a nested structure in short-term memory in a way that solves the problem of 2 and massive binding problem emphasized by Jackendoff. The model organizes a search for the objects that are referenced in that structure. We motivate the model, demonstrate simulation results, and discuss how it differs from related models. (shrink)

Native language listeners engage in predictive processing in many processing situations and adapt their predictive processing to the statistics of the input. In contrast, second language listeners engage in predictive processing in fewer processing situations. The current study uses eye-tracking data from two experiments in bilinguals’ native language (L1) and second language (L2) to explore their predictive processing based on contrastive pitch accent cues, and their adaptation in the face of prediction errors. The results of the first experiment show inhibition (...) effects for unpredicted referents in both the L1 and the L2 that can be modeled with a Bayesian adaptation model, suggesting that bilinguals adapt their prediction in the face of prediction errors in a way that is compatible with the model. In contrast, the results of the second experiment, after a training phase that increased the predictive validity of the cue, show inhibition effects for unpredicted referents only in the L1, but not in the L2. In addition, the Bayesian adaptation model significantly predicts only the L1, but not the L2 data. The results are discussed with respect to adaptation to the statistical properties of the input. (shrink)

We investigated the effects of everyday language exposure on the prediction of orthographic and phonological forms of a highly predictable word during listening comprehension. Native Japanese speakers in Tokyo (Experiment 1) and Berlin (Experiment 2) listened to sentences that contained a predictable word and viewed four objects. The critical object represented the target word (e.g., /sakana/;fish), an orthographic competitor (e.g., /tuno/;horn), a phonological competitor (e.g., /sakura/;cherry blossom), or an unrelated word (e.g., /hon/;book). The three other objects were distractors. The Tokyo (...) group fixated the target and the orthographic competitor over the unrelated objects before the target word was mentioned, suggesting that they pre-activated the orthographic form of the target word. The Berlin group showed a weaker bias toward the target than the Tokyo group, and they showed a tendency to fixate the orthographic competitor only when the orthographic similarity was very high. Thus, prediction effects were weaker in the Berlin group than in the Tokyo group. We found no evidence for the prediction of phonological information. The obtained group differences support probabilistic models of prediction, which regard the built-up language experience as a basis of prediction. (shrink)

In general, we know that gesture accompanying spoken instruction can help children learn. The present study was conducted to better understand how gesture can support children’s comprehension of spoken instruction and whether the benefit of teaching though speech and gesture over spoken instruction alone depends on differences in cognitive profile – prior knowledge children have that is related to a to-be-learned concept. To answer this question, we explored the impact of gesture instruction on children’s analogical reasoning ability. Children between the (...) ages of 4 and 11 years solved scene analogy problems before or after speech alone or speech and gesture instruction while their visual attention was monitored. Our behavioral results suggest a marginal benefit of gesture instruction over speech alone, but only 5-year-old children showed a distinct advantage from speech+gesture instruction when solving the post-instruction trial, suggesting that at this age, children have the cognitive profile in place to utilize the added support of gesture. Furthermore, while speech+gesture instruction facilitated effective visual attention during instruction, directing attention away from featural matches, and towards relational information, was pivotal for younger children’s success post instruction. We consider how these results contribute to the gesture-for-learning literature and consider how the nuanced impact of gesture is informative for educators teaching tasks of analogy in the classroom. (shrink)

The majority of our linguistic exchanges, such as everyday conversations, are divided into turns; one party usually talks at a time, with only relatively rare occurrences of brief overlaps in which there are two simultaneous speakers. Moreover, conversational turn-taking tends to be very fast. We typically start producing our responses before the previous turn has finished, i.e., before we are confronted with the full content of our interlocutor’s utterance. This raises interesting questions about the nature of linguistic understanding. Philosophical theories (...) typically focus on linguistic understanding characterized either as an ability to grasp the contents of utterances in a given language or as outputs of this ability—mental states of one type or another. In this paper, I supplement these theories by developing an account of the process of understanding. I argue that it enables us to capture the dynamic and temporal aspect of understanding and reconcile philosophical investigations with empirical research on language comprehension. (shrink)

In this paper I focus on a recently discussed phenomenon illustrated by sentences containing predicates of taste: the phenomenon of " perspectival plurality " , whereby sentences containing two or more predicates of taste have readings according to which each predicate pertains to a different perspective. This phenomenon has been shown to be problematic for (at least certain versions of) relativism. My main aim is to further the discussion by showing that the phenomenon extends to other perspectival expressions than predicates (...) of taste and by proposing a general solution to the problem raised by it on behalf of the relativist. The core claim of the solution (" multiple indexing ") is that utterances of sentences containing perspectival expressions should be evaluated with respect to (possibly infinite) sequences of perspective parameters. (shrink)

Children and adults use established global knowledge to generate real‐time linguistic predictions, but less is known about how listeners generate predictions in circumstances that semantically conflict with long‐standing event knowledge. We explore these issues in adults and 5‐ to 10‐year‐old children using an eye‐tracked sentence comprehension task that tests real‐time activation of unexpected events that had been previously encountered in brief stories. Adults generated predictions for these previously unexpected events based on these discourse cues alone, whereas children overall did not (...) override their established global knowledge to generate expectations for semantically conflicting material; however, they do show an increased ability to integrate discourse cues to generate appropriate predictions for sentential endings. These results indicate that the ability to rapidly integrate and deploy semantically conflicting knowledge has a long developmental trajectory, with adult‐like patterns not emerging until later in childhood. (shrink)

Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cognitive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual‐world (...) eye‐tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences. (shrink)

The core niche for language use is in verbal interaction, involving the rapid exchange of turns at talking. This paper reviews the extensive literature about this system, adding new statistical analyses of behavioral data where they have been missing, demonstrating that turn-taking has the systematic properties originally noted by Sacks et al. (1974 ; hereafter SSJ). This system poses some significant puzzles for current theories of language processing: the gaps between turns are short (of the order of 200 ms), but (...) the latencies involved in language production are much longer (over 600 ms). This seems to imply that participants in conversation must predict (or ‘project’ as SSJ have it) the end of the current speaker’s turn in order to prepare their response in advance. This in turn implies some overlap between production and comprehension despite their use of common processing resources. Collecting together what is known behaviorally and experimentally about the system, the space for systematic explanations of language processing for conversation can be significantly narrowed, and we sketch some first model of the mental processes involved for the participant preparing to speak next. (shrink)

Beyond the observation that both speakers and listeners rapidly inspect the visual targets of referring expressions, it has been argued that such gaze may constitute part of the communicative signal. In this study, we investigate whether a speaker may, in principle, exploit listener gaze to improve communicative success. In the context of a virtual environment where listeners follow computer-generated instructions, we provide two kinds of support for this claim. First, we show that listener gaze provides a reliable real-time index of (...) understanding even in dynamic and complex environments, and on a per-utterance basis. Second, we show that a language generation system that uses listener gaze to provide rapid feedback improves overall task performance in comparison with two systems that do not use gaze. Aside from demonstrating the utility of listener gaze in situated communication, our findings open the door to new methods for developing and evaluating multi-modal models of situated interaction. (shrink)

Evidence from numerous studies using the visual world paradigm has revealed both that spoken language can rapidly guide attention in a related visual scene and that scene information can immediately influence comprehension processes. These findings motivated the coordinated interplay account (Knoeferle & Crocker, 2006) of situated comprehension, which claims that utterance‐mediated attention crucially underlies this closely coordinated interaction of language and scene processing. We present a recurrent sigma‐pi neural network that models the rapid use of scene information, exploiting an utterance‐mediated (...) attentional mechanism that directly instantiates the CIA. The model is shown to achieve high levels of performance (both with and without scene contexts), while also exhibiting hallmark behaviors of situated comprehension, such as incremental processing, anticipation of appropriate role fillers, as well as the immediate use, and priority, of depicted event information through the coordinated use of utterance‐mediated attention to the scene. (shrink)

Although for many years a sharp distinction has been made in language research between rules and words—with primary interest on rules—this distinction is now blurred in many theories. If anything, the focus of attention has shifted in recent years in favor of words. Results from many different areas of language research suggest that the lexicon is representationally rich, that it is the source of much productive behavior, and that lexically specific information plays a critical and early role in the interpretation (...) of grammatical structure. But how much information can or should be placed in the lexicon? This is the question I address here. I review a set of studies whose results indicate that event knowledge plays a significant role in early stages of sentence processing and structural analysis. This poses a conundrum for traditional views of the lexicon. Either the lexicon must be expanded to include factors that do not plausibly seem to belong there; or else virtually all information about word meaning is removed, leaving the lexicon impoverished. I suggest a third alternative, which provides a way to account for lexical knowledge without a mental lexicon. (shrink)

Dialog is a joint action at different levels. At the highest level, the goal of interlocutors is to align their mental representations. This emerges from joint activity at lower levels, both concerned with linguistic decisions (e.g., choice of words) and nonlinguistic processes (e.g., alignment of posture or speech rate). Because of the high‐level goal, the interlocutors are particularly concerned with close coupling at these lower levels. As we illustrate with examples, this means that imitation and entrainment are particularly pronounced during (...) interactive communication. We then argue that the mechanisms underlying such processes involve covert imitation of interlocutors’ communicative behavior, leading to emulation of their expected behavior. In other words, communication provides a very good example of predictive emulation, in a way that leads to successful joint activity. (shrink)

We examine two connectionist networks—a fractal learning neural network (FLNN) and a Simple Recurrent Network (SRN)—that are trained to process center-embedded symbol sequences. Previous work provides evidence that connectionist networks trained on infinite-state languages tend to form fractal encodings. Most such work focuses on simple counting recursion cases (e.g., anbn), which are not comparable to the complex recursive patterns seen in natural language syntax. Here, we consider exponential state growth cases (including mirror recursion), describe a new training scheme that seems (...) to facilitate learning, and note that the connectionist learning of these cases has a continuous metamorphosis property that looks very different from what is achievable with symbolic encodings. We identify a property—ragged progressive generalization—which helps make this difference clearer. We suggest two conclusions. First, the fractal analysis of these more complex learning cases reveals the possibility of comparing connectionist networks and symbolic models of grammatical structure in a principled way—this helps remove the black box character of connectionist networks and indicates how the theory they support is different from symbolic approaches. Second, the findings indicate the value of future, linked mathematical and empirical work on these models—something that is more possible now than it was 10 years ago. (shrink)

Vector-based models of word meaning have become increasingly popular in cognitive science. The appeal of these models lies in their ability to represent meaning simply by using distributional information under the assumption that words occurring within similar contexts are semantically similar. Despite their widespread use, vector-based models are typically directed at representing words in isolation, and methods for constructing representations for phrases or sentences have received little attention in the literature. This is in marked contrast to experimental evidence (e.g., in (...) sentential priming) suggesting that semantic similarity is more complex than simply a relation between isolated words. This article proposes a framework for representing the meaning of word combinations in vector space. Central to our approach is vector composition, which we operationalize in terms of additive and multiplicative functions. Under this framework, we introduce a wide range of composition models that we evaluate empirically on a phrase similarity task. (shrink)

We present computational modeling results based on a self‐paced reading study investigating number attraction effects in Eastern Armenian. We implement three novel computational models of agreement attraction in a Bayesian framework and compare their predictive fit to the data using k‐fold cross‐validation. We find that our data are better accounted for by an encoding‐based model of agreement attraction, compared to a retrieval‐based model. A novel methodological contribution of our study is the use of comprehension questions with open‐ended responses, so that (...) both misinterpretation of the number feature of the subject phrase and misassignment of the thematic subject role of the verb can be investigated at the same time. We find evidence for both types of misinterpretation in our study, sometimes in the same trial. However, the specific error patterns in our data are not fully consistent with any previously proposed model. (shrink)