Gestures and speech are clearly synchronized in many ways. However, previous studies have shown that the semantic similarity between gestures and speech breaks down as people approach transitions in understanding. Explanations for these gesture–speech mismatches, which focus on gestures and speech expressing different cognitive strategies, have been criticized for disregarding gestures’ and speech's integration and synchronization. In the current study, we applied three different perspectives to investigate gesture–speech synchronization in an easy and a difficult task: temporal alignment, semantic similarity, and (...) complexity matching. Participants engaged in a simple cognitive task and were assigned to either an easy or a difficult condition. We automatically measured pointing gestures, and we coded participant's speech, to determine the temporal alignment and semantic similarity between gestures and speech. Multifractal detrended fluctuation analysis was used to determine the extent of complexity matching between gestures and speech. We found that task difficulty indeed influenced gesture–speech synchronization in all three domains. We thereby extended the phenomenon of gesture–speech mismatches to difficult tasks in general. Furthermore, we investigated how temporal alignment, semantic similarity, and complexity matching were related in each condition, and how they predicted participants’ task performance. Our study illustrates how combining multiple perspectives, originating from different research areas (i.e., coordination dynamics, complexity science, cognitive psychology), provides novel understanding about cognitive concepts in general and about gesture–speech synchronization and task difficulty in particular. (shrink)

People frequently gesture when a word is on the tip of their tongue (TOT), yet research is mixed as to whether and why gesture aids lexical retrieval. We tested three accounts: the lexical retrieval hypothesis, which predicts that semantically related gestures facilitate successful lexical retrieval; the cognitive load account, which predicts that matching gestures facilitate lexical retrieval only when retrieval is hard, as in the case of a TOT; and the motor movement account, which predicts that any motor movements should (...) support lexical retrieval. In Experiment 1 (a between‐subjects study; N = 90), gesture inhibition, but not neck inhibition, affected TOT resolution but not overall lexical retrieval; participants in the gesture‐inhibited condition resolved fewer TOTs than participants who were allowed to gesture. When participants could gesture, they produced more representational gestures during resolved than unresolved TOTs, a pattern not observed for meaningless motor movements (e.g., beats). However, the effect of gesture inhibition on TOT resolution was not uniform; some participants resolved many TOTs, while others struggled. In Experiment 2 (a within‐subjects study; N = 34), the effect of gesture inhibition was traced to individual differences in verbal, not spatial short‐term memory (STM) span; those with weaker verbal STM resolved fewer TOTs when unable to gesture. This relationship between verbal STM and TOT resolution was not observed when participants were allowed to gesture. Taken together, these results fit the cognitive load account; when lexical retrieval is hard, gesture effectively reduces the cognitive load of TOT resolution for those who find the task especially taxing. (shrink)

The goal of the article is to offer a framework against which relations between gesture and sign can be systematically explored beyond the current literature. It does so by (a) reconstructing the history of the discussion in the field of gesture studies, focusing on three leading positions (Kendon, McNeill, and Goldin-Meadow); and (b) by formulating a position to illustrate how this can be achieved. The paper concludes by emphasizing the need for systematic cross-linguistic research on multimodal use of language in (...) its signed and spoken forms. (shrink)

It is argued that colour name strategy, object name strategy, and chunking strategy in memory are all aspects of the same general phenomena, called stereotyping, and this in turn is an example of a know-how representation. Such representations are argued to have their origin in a principle called the minimum duplication of resources. For most the subsequent discussions existence of colour name strategy suffices. It is pointed out that the BerlinA- KayA universal partial ordering of colours and the frequency of (...) traffic accidents classified by colour are surprisingly similar; a detailed analysis is not carried out as the specific colours recorded are not identical. Some consequences of the existence of a name strategy for the philosophy of language and mathematics are discussed: specifically it is argued that in accounts of truth and meaning it is necessary throughout to use real numbers as opposed to bi-valent quantities; and also that the concomitant label associated with sentences should not be of unconditional truth, but rather several real-valued quantities associated with visual communication. The implication of real-valued truth quantities is that the Continuum Hypothesis of pure mathematics is side-stepped, because real valued quantities occur ab initio. The existence of name strategy shows that thought/sememes and talk/phonemes can be separate, and this vindicates the assumption of thought occurring before talk used in psycho-linguistic speech production models. (shrink)

Chomskyian claims of a genetically hard-wired and cognitively autonomous “universal grammar” are being promoted by generative linguistics as facts about language to the present day. The related doctrine of an evolutionary discontinuity in language emergence, however, is based on misconceptions about the notions of homology and preadaptation. The obvious lack of equivalence between symbolic communicative capacities in existing nonhuman primates and human language does not preclude common roots. Normal and disordered language development is strongly influenced by the genome, but there (...) is no evidence for the existence of specific genes underlying “universal grammar.” In the mature brain, stages of language processing can be distinguished and “first-pass” syntactic analyses appear to precede semantic decoding. However, this partial seriality – as well as behavioral and clinical dissociations between lexical and functional categories – can be best described, not in terms of serially activated and discrete modules, but in terms of classes of cell assemblies that differ in their distributional properties. Whereas cell assemblies involved in semantic interpretation (“content word” assemblies) are widely distributed and are generally less vulnerable to focal lesion, those involved in structural decoding (functor assemblies) are primarily distributed within the left perisylvian cortices and are selectively vulnerable to left perisylvian lesion. These distributional differences are explained in terms of the perceptuomotor components involved in the acquisition of relevant representations. The emerging “motivated” or “toposemantic” brain regional specificity can only be accommodated with “soft” and maturational versions of modularity. The failure to reproduce double dissociations in current connectionist models is due to overly simple neuroscientific assumptions, notably that of overall equipotentiality. Linguistic models should not be expected to be “implemented” in the brain, but need to be constrained by neuroscientific evidence on how biological brains function. (shrink)

Because Müiller fails to distinguish between two senses of the term “autonomy,” there is a danger that his results will be misinterpreted by both linguists and neuroscientists. Although he may very well have been successful in refuting one sense of autonomy, he may actually have helped to provide an explanation for the correctness of autonomy in its other sense.

The belief that syntax is an innate, autonomous, species-specific module is highly questionable. Syntax demonstrates the mosaic nature of evolutionary change, in that it made use of (and led to the enhancement of) numerous preexisting neurocognitive features. It is best understood as an emergent characteristic of the explosion of semantic complexity that occurred during hominid evolution.

Both autonomy and local specificity are compatible with observed interconnectivity at the cell level when considering two different levels: cell assemblies and brain systems. Early syntactic structuring processes in particular are likely to representan autonomous module in the language/brain system.

The conclusion that language is not really innate or modular is based on several fallacies. I show that the target article confuses communicative skills with linguistic abilities, and that its discussion of brain/language relations is replete with factual errors. I also criticize its attempt to contrast biological and linguistic principles. Finally, I argue that no case is made for the “alternative” approach proposed here.

This article addresses the issue of non-verbal communication in the light of the Gricean conceptualisation of intentionally conveyed meanings. The first goal is to testify that non-verbal cues can be interpreted as nonnatural meanings and speaker meanings, which partake in intentional communication. Secondly, it is argued that non-verbal signals, exemplified by gestures, are similar to utterances which generate the communicator's what is said and/or conversational implicatures, together with their different subtypes and manifestations. Both of these objectives necessitate a critical overview (...) of Grice's work on the focal aspects of meaning and communication, also on the strength of neo-Gricean research. (shrink)

Two central assumptions of current models of language acquisition were addressed in this study: (1) knowledge of linguistic structure is "mapped onto" earlier forms of non-linguistic knowledge; and (2) acquiring a language involves a continuous learning sequence from early gestural communication to linguistic expression. The acquisition of the first and second person pronouns ME and YOU was investigated in a longitudinal study of two deaf children of deaf parents learning American Sign Language (ASL) as a first language. Personal pronouns in (...) ASL are formed by pointing directly to the addressee (YOU) or self (I or ME), rather than by arbitrary symbols. Thus, personal pronouns in ASL resemble paralinguistic gestures that commonly accompany speech and are used prelinguistically by both hearing and deaf children beginning around 9 months. This provides a means for investigating the transition from prelinguistic gestural to linguistic expression when both gesture and language reside in the same modality.\nThe results indicate that deaf children acquired knowledge of personal pronouns over a period of time, displaying errors similar to those of hearing children despite the transparency of the pointing gestures. The children initially (ages 10 and 12 months) pointed to persons, objects, and locations. Both children then exhibited a long avoidance period, during which one function of the pointing gesture (pointing to self and others) dropped out completely. During this period their language and cognitive development were otherwise entirely normal, and they continued to use other types of pointing (e.g., to objects). When pointing to self and others returned, it was marked with errors typical of hearing children; one child exhibited consistent pronoun reversal errors, thinking the YOU point referred to herself, while the other child exhibited reversal errors inconsistently. Evidence from experimental tasks conducted with the first child revealed that pronoun errors occurred in comprehension as well. Full control of the ME and YOU pronouns was not achieved until 25-27 months, around the same time when hearing children master these forms. Thus, the study provides evidence for a discontinuity in the child's transition from prelinguistic to linguistic communication. It is argued that aspects of linguistic structure and its acquisition appear to involve distinct, language-specific knowledge. (shrink)

This bold and brilliant book asks the ultimate question of the life sciences: How did the human mind acquire its incomparable power? In seeking the answer, Merlin Donald traces the evolution of human culture and cognition from primitive apes to the era of artificial intelligence, and presents an original theory of how the human mind evolved from its presymbolic form. In the emergence of modern human culture, Donald proposes, there were three radical transitions. During the first, our bipedal but still (...) apelike ancestors acquired "mimetic" skill—the ability to represent knowledge through voluntary motor acts—which made Homo erectus successful for over a million years. The second transition—to "mythic" culture —coincided with the development of spoken language. Speech allowed the large-brained Homo sapiens to evolve a complex preliterate culture that survives in many parts of the world today. In the third transition, when humans constructed elaborate symbolic systems ranging from cuneiforms, hieroglyphics, and ideograms to alphabetic languages and mathematics, human biological memory became an inadequate vehicle for storing and processing our collective knowledge. The modern mind is thus a hybrid structure built from vestiges of earlier biological stages as well as new external symbolic memory devices that have radically altered its organization. According to Donald, we are symbol-using creatures, more complex than any that went before us, and we may have not yet witnessed the final modular arrangement of the human mind. (shrink)

A common method of making a theory more understandable is to compare it to another theory that has been better developed. Radical interpretation is a theory that attempts to explain how communication has meaning. Radical interpretation is treated as another time-dependent theory and compared to the time-dependent theory of biological evolution. The main reason for doing this is to find the nature of the time dependence; producing analogs between the two theories is a necessary prerequisite to this and brings up (...) many problems. When the nature of the time dependence is better known it might allow the underlying mechanism to be uncovered. Several similarities and differences are uncovered, and there appear to be more differences than similarities. (shrink)

A common method of improving how well understood a theory is, is by comparing it to another theory which has been better developed. Radical interpretation is a theory which attempts to explain how communication has meaning. Radical interpretation is treated as another time dependent theory and compared to the time dependent theory of biological evolution. Several similarities and differences are uncovered. Biological evolution can be gradual or punctuated. Whether radical interpretation is gradual or punctuated depends on how the question is (...) framed: on the coarse-grained time scale it proceeds gradually, but on the fine-grained time scale it proceeds by punctuated equilibria. Biological evolution proceeds by natural selection, the counterpart to this is the increase in both correspondence and coherence. Exaption, mutations, and spandrels have counterparts metaphor, speech errors, and puns respectively. Homologous and analogs have direct counterparts in specific words. The most important differences originate from the existence of a unit of inheritance occurring in biological evolution - there is no such unit in language. (shrink)

The strong predominance of right-handedness appears to be a uniquely human characteristic, whereas the left-cerebral dominance for vocalization occurs in many species, including frogs, birds, and mammals. Right-handedness may have arisen because of an association between manual gestures and vocalization in the evolution of language. I argue that language evolved from manual gestures, gradually incorporating vocal elements. The transition may be traced through changes in the function of Broca's area. Its homologue in monkeys has nothing to do with vocal control, (...) but contains the so-called “mirror neurons,” the code for both the production of manual reaching movements and the perception of the same movements performed by others. This system is bilateral in monkeys, but predominantly left-hemispheric in humans, and in humans is involved with vocalization as well as manual actions. There is evidence that Broca's area is enlarged on the left side in Homo habilis, suggesting that a link between gesture and vocalization may go back at least two million years, although other evidence suggests that speech may not have become fully autonomous until Homo sapiens appeared some 170,000 years ago, or perhaps even later. The removal of manual gesture as a necessary component of language may explain the rapid advance of technology, allowing late migrations of Homo sapiens from Africa to replace all other hominids in other parts of the world, including the Neanderthals in Europe and Homo erectus in Asia. Nevertheless, the long association of vocalization with manual gesture left us a legacy of right-handedness. Key Words: cerebral dominance; gestures; handedness; hominids; language evolution; primates; speech; vocalization. (shrink)

This article presents an interview method which enables us to bring a person, who may not even have been trained, to become aware of his or her subjective experience, and describe it with great precision. It is focused on the difficulties of becoming aware of one’s subjective experience and describing it, and on the processes used by this interview technique to overcome each of these difficulties. The article ends with a discussion of the criteria governing the validity of the descriptions (...) obtained, and then with a brief review of the functions of these descriptions. (shrink)

Research on interaction in speaking assessment suggests that both verbal and nonverbal interaction are integral parts of the construct of interactional competence (Galaczi & Taylor, 2018; Plough et al., 2018; Young, 2011). However, little has been done to investigate which features significantly contribute to interactional competence scores. This study, therefore, examined which interaction features that raters noticed in individual scripted interview and paired discussion tasks to gain an insight into the interactional competence construct, providing validity evidence for an inclusion of (...) interactional competence in speaking assessment. Sixty-eight student performances were rated based on interaction rating scales. Exploratory factor analysis revealed four factors: nonverbal communication, topic management, interactional management, and interactive listening. Logistic regressions showed that while raters attended to more topic management features in the individual scripted interview task, they noticed more interactional management features in the paired discussion task. Simple regressions showed that nonverbal communication and topic management features predicted interactional competence scores in the individual scripted interview task, whereas nonverbal communication, topic management, interactional management, and interactive listening features were predictors of scores in the paired discussion task. The findings suggest that both nonverbal and verbal interaction features are important in the interactional competence construct with the paired task providing test-takers with more opportunities to demonstrate their interactional ability. (shrink)

This paper pursues the idea that human and non-human great apes share a common set of directive (imperative) gestures and their meanings. We investigate gestures that are multifunctional, in that they have different effects in different contexts, focusing on non-human ape gestures that communicate “Stop that” in some contexts, and “Move away” in others. What may superficially appear to be lexical ambiguity can be derived from a single abstract lexical entry, “Not X!”, concluded to be a candidate for a universal (...) building block of meaning, shared by human and non-human great apes, reflections of which may also be found in the pragmatic gestures in humans. (shrink)

Investigations of iconicity in language, whereby interactants coordinate meaningful bodily actions to create resemblances, are prevalent across the human communication sciences. However, when it comes to analysing and comparing iconicity across different interactions and modes of communication, it is not always clear we are looking at the same thing. For example, tokens of spoken ideophones and manual depicting actions may both be analysed as iconic forms. Yet spoken ideophones may signal depictive and descriptive qualities via speech, while manual actions may (...) signal depictive, descriptive, and indexical qualities via the shape, movement, and placement of the hands in space. Furthermore, each may co-occur with other semiotics articulated with the face, hands, and body within composite utterances. The paradigm of iconicity as a single property is too broad and coarse for comparative semiotics, as important details necessary for understanding the range of human communicative potentialities may be masked. Here, we draw on semiotic approaches to language and communication, including the model of language as signalled via describing, indicating and/or depicting and the notion of non-referential indexicality, to illustrate the multidimensionality of iconicity in co-present interactions. This builds on our earlier proposal for analysing how different methods of semiotic signalling are combined in multimodal language use. We discuss some implications for the language and communication sciences and explain how this approach may inform a theory of biosemiotics. (shrink)

People with aphasia use gestures not only to communicate relevant content but also to compensate for their verbal limitations. The Sketch Model (De Ruiter, 2000) assumes a flexible relationship between gesture and speech with the possibility of a compensatory use of the two modalities. In the successor of the Sketch Model, the AR‐Sketch Model (De Ruiter, 2017), the relationship between iconic gestures and speech is no longer assumed to be flexible and compensatory, but instead iconic gestures are assumed to express (...) information that is redundant to speech. In this study, we evaluated the contradictory predictions of the Sketch Model and the AR‐Sketch Model using data collected from people with aphasia as well as a group of people without language impairment. We only found compensatory use of gesture in the people with aphasia, whereas the people without language impairments made very little compensatory use of gestures. Hence, the people with aphasia gestured according to the prediction of the Sketch Model, whereas the people without language impairment did not. We conclude that aphasia fundamentally changes the relationship of gesture and speech. (shrink)

People with aphasia use gestures not only to communicate relevant content but also to compensate for their verbal limitations. The Sketch Model (De Ruiter, 2000) assumes a flexible relationship between gesture and speech with the possibility of a compensatory use of the two modalities. In the successor of the Sketch Model, the AR‐Sketch Model (De Ruiter, 2017), the relationship between iconic gestures and speech is no longer assumed to be flexible and compensatory, but instead iconic gestures are assumed to express (...) information that is redundant to speech. In this study, we evaluated the contradictory predictions of the Sketch Model and the AR‐Sketch Model using data collected from people with aphasia as well as a group of people without language impairment. We only found compensatory use of gesture in the people with aphasia, whereas the people without language impairments made very little compensatory use of gestures. Hence, the people with aphasia gestured according to the prediction of the Sketch Model, whereas the people without language impairment did not. We conclude that aphasia fundamentally changes the relationship of gesture and speech. (shrink)

One can disagree with Müller that it is neurobiologically questionable to suppose that human language is innate, specialized, and species-specific, yet agree that the precise brain mechanisms controlling language in any individual will be influenced by epigenesis and genetic variability, and that the interplay between inherited and acquired aspects of linguistic capacity deserves to be investigated.

Segregated neural circuits that effect particular domain-specific behaviors can be differentiated from neuroanatomical structures implicated in many different aspects of behavior. The basal ganglionic components of circuits regulating nonlinguistic motor behavior, speech, and syntax all function in a similar manner. Hence, it is unlikely that special properties and evolutionary mechanisms are associated with the neural bases of human language.

Neurobiological models of language need a level of analysis that can account for the typical range of language phenomena. Because linguistically motivated models have been successful in explaining numerous language properties, it is premature to dismiss them as biologically irrelevant. Models attempting to unify neurobiology and linguistics need to be sensitive to both sources of evidence.

Evidence is presented from a polyglot savant to suggest that double dissociations between linguistic and nonverbal abilities are more important than Müller's target article implies. It is also argued that the special nature of syntax makes its assimilation to other aspects of language or to nonhuman communication systems radically implausible.

Müller misconstrues autonomy to mean strict locality of brain function, something quite different from the functional autonomy that linguists claim. Similarly, he misperceives the interaction of learned and innate components hypothesized in current generative models. Evidence from sign languages, Creole languages, and neurological studies of rare forms of aphasia also argues against his conclusions.

The study of signed languages has inspired scientific' speculation regarding foundations of human language. Relationships between the acquisition of sign language in apes and man are discounted on logical grounds. Evidence from the differential hreakdown of sign language and manual pantomime places limits on the degree of overlap between language and nonlanguage motor systems. Evidence from functional magnetic resonance imaging reveals neural areas of convergence and divergence underlying signed and spoken languages.

The concepts of the innateness, universality, species-specificity, and autonomy of the human language capacity have had an extreme impact on the psycholinguistic debate for over thirty years. These concepts are evaluated from several neurobiological perspectives, with an emphasis on the emergence of language and its decay due to brain lesion and progressive brain disease.Evidence of perceptuomotor homologies and preadaptations for human language in nonhuman primates suggests a gradual emergence of language during hominid evolution. Regarding ontogeny, the innate component of language (...) capacity is likely to be polygenic and shared with other developmental domains. Dissociations between verbal and nonverbal development are probably rooted in the perceptuomotor specializations of neural substrates rather than the autonomy of a grammar module. Aphasiologicaldata often assumed to suggest modular linguistic subsystems can be accounted for in terms of a neurofunctional model incorporating perceptuomotor-based regional specializationsand distributivity of representations. Thus, dissociations between grammatical functors and content words are due to different conditions of acquisition and resulting differences in neural representation. Human brains are characterized by multifactorial interindividual variability, and strict universality of functional organization is biologically unrealistic.A theoretical alternative is proposed according to which (1) linguistic specialization of brain areas is due to epigenetic and probabilistic maturational events, not to genetic ”hard-wiring,” and (2) linguistic knowledge is neurally represented in distributed cell assemblies whose topography reflects the perceptuomotor modalities involved in the acquisition and use of a given item of knowledge. (shrink)

The tradeoff hypothesis in the speech–gesture relationship claims that (a) when gesturing gets harder, speakers will rely relatively more on speech, and (b) when speaking gets harder, speakers will rely relatively more on gestures. We tested the second part of this hypothesis in an experimental collaborative referring paradigm where pairs of participants (directors and matchers) identified targets to each other from an array visible to both of them. We manipulated two factors known to affect the difficulty of speaking to assess (...) their effects on the gesture rate per 100 words. The first factor, codability, is the ease with which targets can be described. The second factor, repetition, is whether the targets are old or new (having been already described once or twice). We also manipulated a third factor, mutual visibility, because it is known to affect the rate and type of gesture produced. None of the manipulations systematically affected the gesture rate. Our data are thus mostly inconsistent with the tradeoff hypothesis. However, the gesture rate was sensitive to concurrent features of referring expressions, suggesting that gesture parallels aspects of speech. We argue that the redundancy between speech and gesture is communicatively motivated. (shrink)

This study investigates whether there is a relation between the semantics of linguistic expressions that indicate temporal distance and the spatial properties of their co-speech gestures. To this date, research on time gestures has focused on features such as gesture axis, direction, and shape. Here we focus on a gesture property that has been overlooked so far: the distance of the gesture in relation to the body. To achieve this, we investigate two types of temporal linguistic expressions are addressed: proximal (...) (e.g., near future, near past) and distal (e.g., distant past, distant future). Data was obtained through the NewsScape library, a multimodal corpus of television news. A total of 121 co-speech gestures were collected and divided into the two categories. The gestures were later annotated in terms of gesture space and classified in three categories: (i) center, (ii) periphery, and (iii) extreme periphery. Our results suggest that gesture and language are coherent in the expression of temporal distance: when speakers locate an event far from them, they tend to gesture further from their body; similarly, when locating an event close to them, they gesture closer to their body. These results thus reveal how co-speech gestures also reflect a space-time mapping in the dimension of distance. (shrink)

I enlarge on the theme that the brain mechanisms required for languageand other aspects of the human mind evolved through selective changes in the regulatory genes governing growth. Extension of the period of postnatal growth increases the role of the environment in structuring the brain, and spatiotemporal programming (heterochrony) ofgrowth might explain hierarchical representation, hemispheric specialization, and perhaps sex differences.

Speakers frequently produce elaborate hand movements during talk that have been shown to serve a communicative function. Nevertheless, two-thirds of the semantic content of these hand movements is encoded linguistically elsewhere in the discourse . The present experiment demonstrated that while 62.9% of semantic information in gesture was elsewhere, most gestures retained at least one semantic feature that was never represented linguistically. Semantic features were more explicit when they occurred in gesture than when represented linguistically. Even in cases where the (...) imagistic gesture appeared somewhat redundant, gesture at the narrative level preserves a discernable communicative function. (shrink)

The present study aims to explore the status of filled pauses as pragmatic markers by taking into account their accompanying visual and gestural behavior. This aspect has not yet been widely explored, and the current study breaks new ground by demonstrating that the analysis of gaze and gesture can shed substantial light on the pragmatic functions of filled pauses and other pausing phenomena. Filled pauses (FPs) serve several pragmatic functions in speech, mainly planning but also turn-holding and emphasis, and their (...) use is also highly determined by register and setting. This research explores the different pragmatic functions of FPs by analyzing their distribution in two different communication settings (conversation vs presentation setting), combining a quantitative and a qualitative methodology, following Kosmala & Crible’s (2021) study on the same data. Particular attention was paid to the co-occurring gestural activity of uh/ums and gaze behavior. Analyses show that the pragmatic functions of FPs are also embodied in kinetic activities which differ according to the setting: more pragmatic and referential ones were found during FPs in conversation than in the presentation setting, as well as more eye-contact, which reflects their potential communicative role during interactional sequences. (shrink)