What is the role of language in social interaction? What does language bring to social encounters? We argue that language can be conceived of as a tool for interacting minds, enabling especially effective and flexible forms of social coordination, perspective-taking and joint action. In a review of evidence from a broad range of disciplines, we pursue elaborations of the language-as-a-tool metaphor, exploring four ways in which language is employed in facilitation of social interaction. We argue that language dramatically extends the (...) possibility-space for interaction, facilitates the profiling and navigation of joint attentional scenes, enables the sharing of situation models and action plans, and mediates the cultural shaping of interacting minds. (shrink)

Over two decades have passed since the publication of van Gelder's (1998) “dynamical hypothesis.” In that paper, van Gelder proposed that cognitive agents were not digital computers—per the representational computational approach—but dynamical systems. The evolution of the dynamical hypothesis was driven by parallel advances in three areas. Theoretically, a deeper understanding of genetics, biology, neuroscience, and cognitive science inspired questions about how systems within each domain dynamically interact and extend their effects across spatiotemporal scales. Methodologically, more sophisticated and domain-general tools (...) allowed researchers to discover, model, and quantify system dynamics, structure, and patterns across multiple scales to generate a more comprehensive system-level understanding of behaviors. Empirically, we can analyze a system's behavior while preserving its natural dynamics, revealing evidence that the reductionist approach leads to an incomplete understanding of the components and the overall system. Researchers have traditionally reduced a complex system into its component processes and assumed that the parts can be recombined to explain the whole. These three advances fundamentally altered our understanding of a “cognitive agent:” How their behaviors are driven by long-range coordination across multiple processes, how the interdependent and nested structure of interacting variables produces behaviors that are greater than the sum of its parts, and how environmental constraints shape adaptive yet stable behavioral patterns. (shrink)

Despite the challenges associated with virtually mediated communication, remote collaboration is a defining characteristic of online multiplayer gaming communities. Inspired by the teamwork exhibited by players in first-person shooter games, this study investigated the verbal and behavioral coordination of four-player teams playing a cooperative online video game. The game, Desert Herding, involved teams consisting of three ground players and one drone operator tasked to locate, corral, and contain evasive robot agents scattered across a large desert environment. Ground players could move (...) throughout the environment, while the drone operator’s role was akin to that of a “spectator” with a bird’s-eye view, with access to veridical information of the locations of teammates and the to-be-corralled agents. Categorical recurrence quantification analysis was used to measure the communication dynamics of teams as they completed the task. Demands on coordination were manipulated by varying the ground players’ ability to observe the environment with the use of game “fog.” Results show that catRQA was sensitive to changes to task visibility, with reductions in task visibility reorganizing how participants conversed during the game to maintain team situation awareness. The results are discussed in the context of future work that can address how team coordination can be augmented with the inclusion of artificial agents, as synthetic teammates. (shrink)

A key emergent property of group social dynamic is synchrony–the coordination of actions, emotions, or physiological processes between group members. Despite this fact and the inherent nested structure of groups, little research has assessed physiological synchronization between group members from a multi-level perspective, thus limiting a full understanding of the dynamics between members. To address this gap of knowledge we re-analyzed a large dataset comprising physiological and psychological data that were collected in two laboratory studies that involved two different social (...) group tasks. In both studies, following the group task, members reported their experience of group cohesion via questionnaires. We utilized a non-linear analysis method-multidimensional recurrence quantification analysis that allowed us to represent physiological synchronization in cardiological interbeat intervals between group members at the individual-level and at the group-level. We found that across studies and their conditions, the change in physiological synchrony from baseline to group interaction predicted a psychological sense of group cohesion. This result was evident both at the individual and the group levels and was not modified by the context of the interaction. The individual- and group-level effects were highly correlated. These results indicate that the relationship between synchrony and cohesion is a multilayered construct. We re-affirm the role of physiological synchrony for cohesion in groups. Future studies are needed to crystallize our understanding of the differences and similarities between synchrony at the individual-level and synchrony at the group level to illuminate under which conditions one of these levels has primacy, or how they interact. (shrink)

Previous research showed that sensorimotor information affects the perception of properties associated with implied perceptual context during language comprehension. Three experiments addressed a novel question of whether perceptual context may contribute to a simulation of information about such out‐of‐sight objects as cast shadows. In Experiment 1, participants read a sentence that implied a particular shadow cast on a target (blinds vs. an open window) and then verified the picture of the object onto which a shadow was cast. Responses were faster (...) when the shadow of blinds cast on the object matched that implied by the sentence. However, the data did not show the same matching effect for pictures with cast shadows from an open window. In Experiments 2 and 3, we found that verification times for pictures with no cast shadows were faster when preceded by an “open window” sentence, thus suggesting that reading the sentence does not elicit a visual simulation of any specific shadow. Experiment 3 showed that the objects superimposed with a cast shadow of the blinds and blinds themselves were verified faster after reading a “blinds” sentence. However, the results of an order analysis showed the temporal stability of the “blinds shadows” effect, but the disappearance of the “blinds” effect in the second half of the data. We conclude that the results are compatible, to a lesser or greater extent, with multiple accounts, and discuss our findings in the context of a mental imagery view, a mental simulation view, and an amodal representation view. (shrink)

It has been claimed that a sense of us is presupposed for shared intentions to be possible. Searle introduced this notion together with the notion of the sense of the other. in joint action. It argues that the sense of the other is a necessary condition for a sense of us. Whereas thisarticle distinguishes between the “sense of the other” and the “sense of us” and elaborates on their role the sense of the other is immediate and automatic, the sense (...) of us can (but need not) arise between people and can (a) develop over time, (b) depend on the situation, and (c) involves several sufficient but not necessary processes. The article relies on research on core knowledge to better understand the sense of the other. It elaborates the sense of us using insights from cognitive science and social psychology. The article shows that the sense of the other and the sense of us can contribute to our understanding of the perception of possibilities for joint action and how individuals can come to experience actions and intentions as shared, even if the participants lack common knowledge. This leads to the conclusion that people are ordinarily socially oriented rather than individually. (shrink)

Important work on alignment systems has been applied to philosophical work on joint action by Tollefsen and Dale. This paper builds from and expands on their work. The first aim of the paper is to spell out how the empirical research on alignment may be integrated into philosophical theories of joint action. The second aim is then to develop a successful characterization of joint action, which spells out the difference between genuine joint action and simpler forms of coordination based on (...) alignment. I begin by introducing the empirical research and two definitions of joint action. I then argue that instead of using this research in conjunction with Searle’s account of collective intentionality, as Tollefsen and Dale suggest, we would be better served by applying this research to Gilbert’s account of plural subjects. In the final sections I distinguish between alignment, coordination, and joint action, clarify the roles of joint commitment and sub-personal alignment in joint action, and argue that... (shrink)

This study investigates a specific aspect of joint attention, that is, the emergence of the sense that one is leading the attentional focus of others. Thirty participants were placed in front of two avatars and had to pay attention to objects that were also attended to by the avatars. Unbeknownst to the participant, the avatars’ gaze orientations were alternately controlled by the participant’s eyes. Eye-tracking data were collected and participants were enquired about their experience to account for their sense of (...) agency. Only five participants noticed that the avatars were following their gaze. The 25 participants who remained unaware that they controlled the avatars nevertheless modified their gazing behavior with an increase in direct gaze toward one of the avatars. This outcome highlights a dissociation between an implicit feeling that a change occurred and the explicit awareness of being the cause of this change. (shrink)

Beyond the observation that both speakers and listeners rapidly inspect the visual targets of referring expressions, it has been argued that such gaze may constitute part of the communicative signal. In this study, we investigate whether a speaker may, in principle, exploit listener gaze to improve communicative success. In the context of a virtual environment where listeners follow computer-generated instructions, we provide two kinds of support for this claim. First, we show that listener gaze provides a reliable real-time index of (...) understanding even in dynamic and complex environments, and on a per-utterance basis. Second, we show that a language generation system that uses listener gaze to provide rapid feedback improves overall task performance in comparison with two systems that do not use gaze. Aside from demonstrating the utility of listener gaze in situated communication, our findings open the door to new methods for developing and evaluating multi-modal models of situated interaction. (shrink)

Dialog is a joint action at different levels. At the highest level, the goal of interlocutors is to align their mental representations. This emerges from joint activity at lower levels, both concerned with linguistic decisions (e.g., choice of words) and nonlinguistic processes (e.g., alignment of posture or speech rate). Because of the high‐level goal, the interlocutors are particularly concerned with close coupling at these lower levels. As we illustrate with examples, this means that imitation and entrainment are particularly pronounced during (...) interactive communication. We then argue that the mechanisms underlying such processes involve covert imitation of interlocutors’ communicative behavior, leading to emulation of their expected behavior. In other words, communication provides a very good example of predictive emulation, in a way that leads to successful joint activity. (shrink)

According to the hypotheses of distributed and extended cognition, remembering does not always occur entirely inside the brain but is often distributed across heterogeneous systems combining neural, bodily, social, and technological resources. These ideas have been intensely debated in philosophy, but the philosophical debate has often remained at some distance from relevant empirical research, while empirical memory research, in particular, has been somewhat slow to incorporate distributed/extended ideas. This situation, however, appears to be changing, as we witness an increasing level (...) of interaction between the philosophy and the empirical research. In this editorial, we provide a high-level historical overview of the development of the debates around the hypotheses of distributed and extended cognition, as well as relevant theory and empirical research on memory, considering both the role of memory in theoretical debates around distributed/extended ideas and strands of memory research that resonate with those ideas; we emphasize recent trends towards increased interaction, including new empirical paradigms for investigating distributed memory systems. We then provide an overview of the special issue itself, drawing out a number of general implications from the contributions, and conclude by sketching promising directions for future research on distributed memory. (shrink)

Language is a rewarding field if you are in the prediction business. A reader who is fluent in English and who knows how academic papers are typically structured will readily come up with several possible guesses as to where the title of this section could have gone, had it not been cut short by the ellipsis. Indeed, in the more natural setting of spoken language, anticipatory processing is a must: performance of machine systems for speech interpretation depends critically on the (...) availability of a good predictive model of how utterances unfold in time (Baker, 1975; Jelinek, 1990; Goodman, 2001), and there is strong evidence that prospective uncertainty affects human sentence processing too (Jurafsky, 2003; Hale, 2006; Levy, 2008). The human ability to predict where the current utterance is likely to be going is just another adaptation to the general pressure to anticipate the future (Hume, 1748; Dewey, 1910; Craik, 1943), be it in perception, thinking, or action, which is exerted on all cognitive systems by evolution (Dennett, 2003). Look-ahead in language is, however, special in one key respect: language is a medium for communication, and in communication the most interesting (that is, informative) parts of the utterance that the speaker is working through are those that cannot be predicted by the listener ahead of time. (shrink)

Cognitive Science, Volume 45, Issue 11, November 2021.

In the past several decades, psychologists and linguists have begun experimentally investigating linguistic pragmatic phenomena. They share the assumption that the best way to study the use of language in context incorporates an experimental methodology, here understood to comprise controlled studies and careful field observations. This article surveys some key projects in experimental pragmatics and relates these projects to ongoing philosophical discussions.

A variety of theoretical frameworks predict the resemblance of behaviors between two people engaged in communication, in the form of coordination, mimicry, or alignment. However, little is known about the time course of the behavior matching, even though there is evidence that dyads synchronize oscillatory motions (e.g., postural sway). This study examined the temporal structure of nonoscillatory actions—language, facial, and gestural behaviors—produced during a route communication task. The focus was the temporal relationship between matching behaviors in the interlocutors (e.g., facial (...) behavior in one interlocutor vs. the same facial behavior in the other interlocutor). Cross-recurrence analysis revealed that within each category tested (language, facial, gestural), interlocutors synchronized matching behaviors, at temporal lags short enough to provide imitation of one interlocutor by the other, from one conversational turn to the next. Both social and cognitive variables predicted the degree of temporal organization. These findings suggest that the temporal structure of matching behaviors provides low-level and low-cost resources for human interaction. (shrink)

Although the neural systems that underlie spoken language are well-known, how they adapt to evolving social cues during natural conversations remains an unanswered question. In this work we investigate the neural correlates of face-to-face conversations between two individuals using functional near infrared spectroscopy and acoustical analyses of concurrent audio recordings. Nineteen pairs of healthy adults engaged in live discussions on two controversial topics where their opinions were either in agreement or disagreement. Participants were matched according to their a priori opinions (...) on these topics as assessed by questionnaire. Acoustic measures of the recorded speech including the fundamental frequency range, median fundamental frequency, syllable rate, and acoustic energy were elevated during disagreement relative to agreement. Consistent with both the a priori opinion ratings and the acoustic findings, neural activity associated with long-range functional networks, rather than the canonical language areas, was also differentiated by the two conditions. Specifically, the frontoparietal system including bilateral dorsolateral prefrontal cortex, left supramarginal gyrus, angular gyrus, and superior temporal gyrus showed increased activity while talking during disagreement. In contrast, talking during agreement was characterized by increased activity in a social and attention network including right supramarginal gyrus, bilateral frontal eye-fields, and left frontopolar regions. Further, these social and visual attention networks were more synchronous across brains during agreement than disagreement. Rather than localized modulation of the canonical language system, these findings are most consistent with a model of distributed and adaptive language-related processes including cross-brain neural coupling that serves dynamic verbal exchanges. (shrink)

Our everyday interactions increasingly involve both embodied face-to-face communication and various forms of mediated and distributed communication such as email, skype, and facebook. In daily face-to-face communications, we are connected in rhythm and synchrony at multiple levels ranging from the moment-by-moment continuity of timed syllables to emergent body and vocal rhythms of pragmatic sense-making. Our human capacity to synchronize with each other may be essential for our survival as social beings. Moving our bodies and voices together in time embodies a (...) potent pragmatic purpose that of being together. In this synchrony of self with other, witnessing and being present become part of each other. There is growing research into how rhythm and synchrony operate in embodied face-to-face interaction and this provides parameters for investigating the relations and differences in how we connect and are socially present in the embodied and distributed settings, and understanding the effect of one setting upon the other. This paper explores the arena of research into rhythm in human interaction, musical and linguistic, with a focus on the movements of body and voice. It draws together salient issues and ideas that would form the basis for a framework of rhythm in embodied interaction. (shrink)

In some areas of cognitive science we are confronted with ultrafast cognition, exquisite context sensitivity, and scale-free variation in measured cognitive activities. To move forward, we suggest a need to embrace this complexity, equipping cognitive science with tools and concepts used in the study of complex dynamical systems. The science of movement coordination has benefited already from this change, successfully circumventing analogous paradoxes by treating human activities as phenomena of self-organization. Therein, action and cognition are seen to be emergent in (...) ultrafast symmetry breaking across the brain and body; exquisitely constituted of the otherwise trivial details of history, context, and environment; and exhibiting the characteristic scale-free signature of self-organization. (shrink)

Pickering & Garrod's (P&G's) framework assumes an efference copy based on the interlocutor's intentions. Yet, elaborate attribution of intentions may not always be necessary for online prediction. Instead, contextual cues such as speaker gaze can provide similar information with a lower demand on processing resources.

Research on linguistic interaction suggests that two or more individuals can sometimes form adaptive and cohesive systems. We describe an “alignment system” as a loosely interconnected set of cognitive processes that facilitate social interactions. As a dynamic, multi-component system, it is responsive to higher-level cognitive states such as shared beliefs and intentions (those involving collective intentionality) but can also give rise to such shared cognitive states via bottom-up processes. As an example of putative group cognition we turn to transactive memory (...) and suggest how further research on alignment in these cases might reveal how such systems can be genuinely described as cognitive. Finally, we address a prominent critique of collective cognitive systems, arguing that there is much empirical and explanatory benefit to be gained from considering the possibility of group cognitive systems, especially in the context of small-group human interaction. (shrink)

Task-unrelated thought (TUT) occurs frequently in our daily lives and across a range of tasks, but we know little about how this phenomenon arises during and influences the way we communicate. Conversations also provide a novel opportunity to assess the alignment (or divergence) in TUT during dyadic interactions. We conducted a study to determine: (a) the frequency of TUT during conversation as well as how partners align/diverge in their rates of TUT, (b) the subjective and behavioral correlates of TUT and (...) TUT divergence during conversation, and (c) if perceived social group identity impacts TUT and TUT divergence during conversation. We used a minimal groups induction procedure to assign participants (N = 126) to either an ingroup, outgroup, or control condition. We then asked them to converse with one another via a computer-mediated text chat application for 10 min while self-reporting TUTs. On average, participants reported TUT about once every 2 min; however, this rate was lower for participants in the ingroup condition, compared to the control condition. Conversational pairs in the ingroup condition were also aligned more in their rates of TUT compared to the outgroup condition. Finally, we discuss subjective and behavioral correlates of TUT and TUT divergence in conversations, such as valence, turn-taking ratios, and topic shifts. (shrink)

During communication, we form assumptions about what our communication partners know and believe. Information that is mutually known between the discourse partners—their common ground—serves as a backdrop for successful communication. Here we present an introduction to the focus of this topic, which is the role of memory in common ground and language use. Two types of questions emerge as central to understanding the relationship between memory and common ground, specifically questions having to do with the representation of common ground in (...) memory, and the use of common ground during language processing. (shrink)

Journal Name: Cognitive Linguistics Issue: Ahead of print.

When people communicate, they coordinate a wide range of linguistic and non-linguistic behaviors. This process of coordination is called alignment, and it is assumed to be fundamental to successful communication. In this paper, we question this assumption and investigate whether disalignment is a more successful strategy in some cases. More specifically, we hypothesize that alignment correlates with task success only when communication is interactive. We present results from a spot-the-difference task in which dyads of interlocutors have to decide whether they (...) are viewing the same scene or not. Interactivity was manipulated in three conditions by increasing the amount of information shared between interlocutors. We use recurrence quantification analysis to measure the alignment between the scan-patterns of the interlocutors. We found that interlocutors who could not exchange feedback aligned their gaze more, and that increased gaze alignment correlated with decreased task success in this case. When feedback was possible, in contrast, interlocutors utilized it to better organize their joint search strategy by diversifying visual attention. This is evidenced by reduced overall alignment in the minimal feedback and full dialogue conditions. However, only the dyads engaged in a full dialogue increased their gaze alignment over time to achieve successful performances. These results suggest that alignment per se does not imply communicative success, as most models of dialogue assume. Rather, the effect of alignment depends on the type of alignment, on the goals of the task, and on the presence of feedback. (shrink)

Integrating different perspectives is a sophisticated strategy for developing constructive interactions in collaborative problem solving. However, cognitive aspects such as individuals’ knowledge and bias often obscure group consensus and produce conflict. This study investigated collaborative problem solving, focusing on a group member interacting with another member having a different perspective. It was predicted that mavericks might mitigate disadvantages and facilitate perspective taking during problem solving. Thus, 344 university students participated in two laboratory-based experiments by engaging in a simple rule-discovery task (...) that raised conflicts among perspectives. They interacted with virtual partners whose conversations were controlled by multiple conversational agents. Results show that when participants interacted with a maverick during the task, they were able to take others’ perspectives and integrate different perspectives to solve the problem. Moreover, when participants interacted in groups with a positive mood, groups with a maverick outperformed groups having several perspectives. (shrink)

We hypothesize that effective collaboration is facilitated when individuals and environmental components form a synergy where they work together and regulate one another to produce stable patterns of behavior, or regularity, as well as adaptively reorganize to form new behaviors, or irregularity. We tested this hypothesis in a study with 32 triads who collaboratively solved a challenging visual computer programming task for 20 min following an introductory warm‐up phase. Multidimensional recurrence quantification analysis was used to examine fine‐grained (i.e., every 10 (...) s) collective patterns of regularity across team members' speech rate, body movement, and team interaction with the shared user interface. We found that teams exhibited significant patterns of regularity as compared to shuffled baselines, but there were no systematic trends in regularity across time. We also found that periods of regularity were associated with a reduction in overall behavior. Notably, the production of irregular behavior predicted expert‐coded metrics of collaborative activity, such as teams' ability to construct shared knowledge and effectively negotiate and coordinate execution of solutions, net of overall behavioral production and behavioral self‐similarity. Our findings support the theory that groups can interact to form interpersonal synergies and indicate that information about system‐level dynamics is a viable way to understand and predict effective collaborative processes. (shrink)

Conversation is supported by the beliefs that people have in common and the perceptual experience that they share. The visual context of a conversation has two aspects: the information that is available to each conversant, and their beliefs about what is present for each other. In our experiment, we separated these factors for the first time and examined their impact on a spontaneous conversation. We manipulated the fact that a visual scene was shared or not and the belief that a (...) visual scene was shared or not. Participants watched videos of actors talking about a controversial topic, then discussed their own views while looking at either a blank screen or the actors. Each believed (correctly or not) that their partner was either looking at a blank screen or the same images. We recorded conversants’ eye movements, quantified how they were coordinated, and analyzed their speech patterns. Gaze coordination has been shown to be causally related to the knowledge people share before a conversation, and the information they later recall. Here, we found that both the presence of the visual scene, and beliefs about its presence for another, influenced language use and gaze coordination. (shrink)

In online lectures, unlike in face-to-face lectures, teachers lack access to (nonverbal) cues to check if their students are still “with them” and comprehend the lecture. The increasing availability of low-cost eye-trackers provides a promising solution. These devices measure unobtrusively where students look and can visualize these data to teachers. These visualizations might inform teachers about students’ level of “with-me-ness” (i.e., do students look at the information that the teacher is currently talking about) and comprehension of the lecture, provided that (...) (1) gaze measures of “with-me-ness” are related to comprehension, (2) people not trained in eye-tracking can predict students’ comprehension from gaze visualizations, (3) we understand how different visualization techniques impact this prediction. We addressed these issues in two studies. In Study 1, 36 students watched a video lecture while being eye-tracked. The extent to which students looked at relevant information and the extent to which they looked at the same location as the teacher both correlated with students’ comprehension (score on an open question) of the lecture. In Study 2, 50 participants watched visualizations of students’ gaze (from Study 1), using six visualization techniques (dynamic and static versions of scanpaths, heatmaps, and focus maps) and were asked to predict students’ posttest performance and to rate their ease of prediction. We found that people can use gaze visualizations to predict learners’ comprehension above chance level, with minor differences between visualization techniques. Further research should investigate if teachers can act on the information provided by gaze visualizations and thereby improve students’ learning. (shrink)

This study provides evidence that eye movements reflect the positions of objects while participants listen to a spoken description, retell a previously heard spoken description, and describe a previously seen picture. This effect is equally strong in retelling from memory, irrespective of whether the original elicitation was spoken or visual. In addition, this effect occurs both while watching a blank white board and while sitting in complete darkness. This study includes 4 experiments. The first 2 experiments measured eye movements of (...) participants looking at a blank white board. Experiment 1 monitors eye movements of participants on 2 occasions: first, when participants listened to a prerecorded spoken scene description; second, when participants were later retelling it from memory. Experiment 2 first monitored eye movements of participants as they studied a complex picture visually, and then later as they described it from memory. The second pair of experiments (Experiments 3 and 4) replicated Experiments 1 and 2 with the only difference being that they were executed in complete darkness. This method of analysis differentiated between eye movements that are categorically correct relative to the positions of the whole eye gaze pattern (global correspondence) and eye movements that are only locally correct (local correspondence). The discussion relates the findings to the current debate on mental imagery. (shrink)

People coordinate body postures and gaze patterns during conversation. We review literature showing that (1) action embodies cognition, (2) postural coordination emerges spontaneously when two people converse, (3) gaze patterns influence postural coordination, (4) gaze coordination is a function of common ground knowledge and visual information that conversants believe they share, and (5) gaze coordination is causally related to mutual understanding. We then consider how coordination, generally, can be understood as temporarily coupled neuromuscular components that function as a collective unit (...) known as a coordinative structure in the motor control literature. We speculate that the coordination of gaze and body sway found in conversation may be understood as a cross‐person coordinative structure that embodies the goals of the joint action system. (shrink)