Currently, production and comprehension are regarded as quite distinct in accounts of language processing. In rejecting this dichotomy, we instead assert that producing and understanding are interwoven, and that this interweaving is what enables people to predict themselves and each other. We start by noting that production and comprehension are forms of action and action perception. We then consider the evidence for interweaving in action, action perception, and joint action, and explain such evidence in terms of prediction. Specifically, we assume (...) that actors construct forward models of their actions before they execute those actions, and that perceivers of others' actions covertly imitate those actions, then construct forward models of those actions. We use these accounts of action, action perception, and joint action to develop accounts of production, comprehension, and interactive language. Importantly, they incorporate well-defined levels of linguistic representation. We show how speakers and comprehenders use covert imitation and forward modeling to make predictions at these levels of representation, how they interweave production and comprehension processes, and how they use these predictions to monitor the upcoming utterances. We show how these accounts explain a range of behavioral and neuroscientific data on language processing and discuss some of the implications of our proposal. (shrink)

Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves: (...) the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic/semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands: indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly context‐dependent, while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type/token distinction. (shrink)

Events—the experiences we think we are having and recall having had—are constructed; they are not what actually occurs. What occurs is ongoing dynamic, multidimensional, sensory flow, which is somehow transformed via psychological processes into structured, describable, memorable units of experience. But what is the nature of the redescription processes that fluently render dynamic sensory streams as event representations? How do such processes cope with the ubiquitous novelty and variability that characterize sensory experience? How are event‐rendering skills acquired and how do (...) event representations change with development? This review considers emerging answers to these questions, beginning with evidence that an implicit tendency to monitor predictability structure via statistical learning is key to event rendering. That is, one way that the experience of bounded events (e.g., actions within behavior, words within speech) arises is with the detection of “troughs” in sensory predictability. Interestingly, such troughs in predictability are often predictable; these regions of predictable‐unpredictability provide articulation points to demarcate one event from another in representations derived from the actual streaming information. In our information‐optimization account, a fluent event‐processor predicts such troughs and selectively attends to them—while suppressing attention to other regions—as sensory streams unfold. In this way, usage of attentional resources is optimized for efficient sampling of the most relevant, information‐rich portions of the unfolding flow of sensation. Such findings point to the development of event‐processing fluency—whether in action, language, or other domains—depending crucially on rapid and continual cognitive reorganization. As knowledge of predictability grows, attention is adaptively redeployed. Accordingly, event experiences undergo continuous alteration. (shrink)

A fundamental aspect of human cognition is the ability to parse our constantly unfolding experience into meaningful representations of dynamic events and to communicate about these events with others. How do we communicate about events we have experienced? Influential theories of language production assume that the formulation and articulation of a linguistic message is preceded by preverbal apprehension that captures core aspects of the event. Yet the nature of these preverbal event representations and the way they are mapped onto language (...) are currently not well understood. Here, we review recent evidence on the link between event conceptualization and language, focusing on two core aspects of event representation: event roles and event boundaries. Empirical evidence in both domains shows that the cognitive representation of events aligns with the way these aspects of events are encoded in language, providing support for the presence of deep homologies between linguistic and cognitive event structure. (shrink)

To understand the endogenous components of the event-related brain potential (ERP), we must use data about the components' antecedent conditions to form hypotheses about the information-processing function of the underlying brain activity. These hypotheses, in turn, generate testable predictions about the consequences of the component. We review the application of this approach to the analysis of the P300 component. The amplitude of the P300 is controlled multiplicatively by the subjective probability and the task relevance of the eliciting events, whereas its (...) latency depends on the duration of stimulus evaluation. These and other factors suggest that the P300 is a manifestation of activity occurring whenever one's model of the environment must be revised. Tests of three predictions based on this “context updating” model are reviewed. Verleger's critique is based on a misconstrual of the model as well as a partial and misleading reading of the relevant literature. (shrink)

Brains, it has recently been argued, are essentially prediction machines. They are bundles of cells that support perception and action by constantly attempting to match incoming sensory inputs with top-down expectations or predictions. This is achieved using a hierarchical generative model that aims to minimize prediction error within a bidirectional cascade of cortical processing. Such accounts offer a unifying model of perception and action, illuminate the functional role of attention, and may neatly capture the special contribution of cortical processing to (...) adaptive success. This target article critically examines this approach, concluding that it offers the best clue yet to the shape of a unified science of mind and action. Sections 1 and 2 lay out the key elements and implications of the approach. Section 3 explores a variety of pitfalls and challenges, spanning the evidential, the methodological, and the more properly conceptual. The paper ends (sections 4 and 5) by asking how such approaches might impact our more general vision of mind, experience, and agency. (shrink)

To describe phenomena that occur at different time scales, computational models of the brain must incorporate different levels of abstraction. At time scales of approximately 1/3 of a second, orienting movements of the body play a crucial role in cognition and form a useful computational level embodiment level,” the constraints of the physical system determine the nature of cognitive operations. The key synergy is that at time scales of about 1/3 of a second, the natural sequentiality of body movements can (...) be matched to the natural computational economies of sequential decision systems through a system of implicit reference called deictic in which pointing movements are used to bind objects in the world to cognitive programs. This target article focuses on how deictic bindings make it possible to perform natural tasks. Deictic computation provides a mechanism for representing the essential features that link external sensory data with internal cognitive programs and motor actions. One of the central features of cognition, working memory, can be related to moment-by-moment dispositions of body features such as eye movements and hand movements. (shrink)

Topics in Cognitive Science, Volume 13, Issue 1, Page 187-205, January 2021.

Human control of action in routine situations involves a flexible interplay between (a) task-dependent serial ordering constraints; (b) top-down, or intentional, control processes; and (c) bottom-up, or environmentally triggered, affordances. In addition, the interaction between these influences is modulated by learning mechanisms that, over time, appear to reduce the need for top-down control processes while still allowing those processes to intervene at any point if necessary or if desired. We present a model of the acquisition and control of goal-directed action (...) that goes beyond existing models by operationalizing an interface between two putative systems—a routine and a non-routine system—thereby demonstrating how explicitly represented goals can interact with the emergent task representations that develop through learning in the routine system. The gradual emergence of task representations offers an explanation for the transfer of control with experience from the non-routine goal-based system to the routine system. At the same time it allows action selection to be sensitive both to environmental triggers and to biasing from multiple levels within the goal system. (shrink)

Traditional approaches to human information processing tend to deal with perception and action planning in isolation, so that an adequate account of the perception-action interface is still missing. On the perceptual side, the dominant cognitive view largely underestimates, and thus fails to account for, the impact of action-related processes on both the processing of perceptual information and on perceptual learning. On the action side, most approaches conceive of action planning as a mere continuation of stimulus processing, thus failing to account (...) for the goal-directedness of even the simplest reaction in an experimental task. We propose a new framework for a more adequate theoretical treatment of perception and action planning, in which perceptual contents and action plans are coded in a common representational medium by feature codes with distal reference. Perceived events (perceptions) and to-be-produced events (actions) are equally represented by integrated, task-tuned networks of feature codes – cognitive structures we call event codes. We give an overview of evidence from a wide variety of empirical domains, such as spatial stimulus-response compatibility, sensorimotor synchronization, and ideomotor action, showing that our main assumptions are well supported by the data. Key Words: action planning; binding; common coding; event coding; feature integration; perception; perception-action interface. (shrink)

Shin and DuBrow propose that a key principle driving event segmentation relates to causal analyses: specifically, that experiences that are attributed as having the same underlying cause are grouped together into an event. This offers an alternative to accounts of segmentation based on prediction error.

McRae, Brown and Elman argue against the view that events are structured as frequently‐occurring sequences of world stimuli. They underline the importance of temporal structure defining event types and advance a more complex temporal structure, which allows for some variance in the component elements.

I address the commentators' calls for clarification of theoretical terms, discussion of similarities to other proposals, and extension of the ideas. In doing so, I keep the focus on the purpose of memory: enabling the organism to make sense of its environment so that it can take action appropriate to constraints resulting from the physical, personal, social, and cultural situations.

People tend to perceive ongoing continuous activity as series of discrete events. This partitioning of continuous activity may occur, in part, because events correspond to dynamic patterns that have recurred across different contexts. Recurring patterns may lead to reliable sequential dependencies in observers' experiences, which then can be used to guide perception. The current set of simulations investigated whether this statistical structure within events can be used 1) to develop stable internal representations that facilitate perception and 2) to learn when (...) to update such representations in a self-organizing manner. These simulations demonstrate that experience with recurring patterns enables a system to accurately predict upcoming stimuli within an event, to identify boundaries between such events based on transient increases in prediction error, and to use such boundaries to improve prediction about subsequent activities. (shrink)

Topics in Cognitive Science, Volume 13, Issue 1, Page 128-141, January 2021.

Looking times and gaze behavior indicate that infants can predict the goal state of an observed simple action event (e.g., object‐directed grasping) already in the first year of life. The present paper mainly focuses on infants’ predictive gaze‐shifts toward the goal of an ongoing action. For this, infants need to generate a forward model of the to‐be‐obtained goal state and to disengage their gaze from the moving agent at a time when information about the action event is still incomplete. By (...) about 6 months of age, infants show goal‐predictive gaze‐shifts, but mainly for familiar actions that they can perform themselves (e.g., grasping) and for familiar agents (e.g., a human hand). Therefore, some theoretical models have highlighted close relations between infants’ ability for action‐goal prediction and their motor development and/or emerging action experience. Recent research indicates that infants can also predict action goals of familiar simple actions performed by non‐human agents (e.g., object‐directed grasping by a mechanical claw) when these agents display agency cues, such as self‐propelled movement, equifinality of goal approach, or production of a salient action effect. This paper provides a review on relevant findings and theoretical models, and proposes that the impacts of action experience and of agency cues can be explained from an action‐event perspective. In particular, infants’ goal‐predictive gaze‐shifts are seen as resulting from an interplay between bottom‐up processing of perceptual information and top‐down influences exerted by event schemata that store information about previously executed or observed actions. (shrink)

Topics in Cognitive Science, Volume 13, Issue 1, Page 63-78, January 2021.

We welcome the proposal to use forward models to understand predictive processes in language processing. However, Pickering & Garrod (P&G) miss the opportunity to provide a strong framework for future work. Forward models need to be pursued in the context of learning. This naturally leads to questions about what prediction error these models aim to minimize.

Marr's levels of analysis—computational, algorithmic, and implementation—have served cognitive science well over the last 30 years. But the recent increase in the popularity of the computational level raises a new challenge: How do we begin to relate models at different levels of analysis? We propose that it is possible to define levels of analysis that lie between the computational and the algorithmic, providing a way to build a bridge between computational- and algorithmic-level models. The key idea is to push the (...) notion of rationality, often used in defining computational-level models, deeper toward the algorithmic level. We offer a simple recipe for reverse-engineering the mind's cognitive strategies by deriving optimal algorithms for a series of increasingly more realistic abstract computational architectures, which we call “resource-rational analysis.”. (shrink)