Sculptors, architects, and painters are three professional groups that require a comprehensive understanding of how to manipulate spatial structures. While it has been speculated that they may differ in the way they conceive of space due to the different professional demands, this has not been empirically tested. To achieve this, we asked architects, painters, sculptors, and a control group questions about spatially complex pictures. Verbalizations elicited were examined using cognitive discourse analysis. We found significant differences between each group. Only painters (...) shifted consistently between 2D and 3D concepts, architects were concerned with paths and spatial physical boundedness, and sculptors produced responses that fell between architects and painters. All three differed from controls, whose verbalizations were generally less elaborate and detailed. Thus, for the case of sculptors, architects, and painters, profession appears to relate to a different spatial conceptualization manifested through a systematically contrasting way of talking about space. (shrink)

Multimodal Critical Discourse Analysis (MCDA) as a sub-discipline of Critical Discourse Analysis (CDA) emerged from the availability of social semiotic frameworks describing multimodal meaning making. However, weaknesses of these frameworks have raised concerns and prompted recent methodological reflections in MCDA. Inspired by these reflections, this paper critically assesses MCDA research on advertising persuasion and identifies a lack of attention in studies to account for the social and ideological impact of advertising. This shortcoming is argued to be attributable to the weak (...) empirical grounding of social semiotic frameworks which in turn is rooted in a ‘discursive gap’ (Bateman, Citation2019, p. 300) in multimodality as a discipline. The authors then propose methodological solutions involving eye tracking experiments and higher order data analytics which can strengthen the empirical basis of theoretical assumptions made in MCDA research on advertising. Drawing on methodological guidance for empirical multimodal research, the authors discuss specific issues in realizing this proposal, focusing on a strategy addressing operationalization challenges in experimental design and the affordance of a data cluster algorithm that calculates the common viewing path of a group of viewers. (shrink)

We present a corpus-based study of coherence in multimodal documents. We concern ourselves with the types of relationships between graphs and tables and the text of the document in which they appear. In order to understand and categorize the types of relations across modalities, we are making use of Rhetorical Structure Theory, and propose that this can adequately describe these types of relations. We analyzed a corpus comprising three different genres, and consisting of about 1500 pages of material and almost (...) 600 figures, tables and graphs. We show that figures stand in both presentational and subject matter relations to the text, and that the relationship between figures and text is one of a small set out of the larger possible rhetorical relations. We also discuss several issues that arise in the treatment of multimodal material, such as the potential for multiple connections between figure and text. (shrink)

This thesis investigates the relationship between eye movements, mental imagery and memory retrieval in four studies based on eye-tracking experiments. The first study is an investigation of eye movements during mental imagery elicited both visually and verbally. The use of complex stimuli and the development of a novel method where eye movements are recorded concurrently with verbal data enabled the above-mentioned relationship to be studied to an extent going beyond what previous research had been able to do. Eye movements were (...) found to closely reflect content and spatial layout while participants were listening to a spoken scene description, while they were describing the same scene from memory, and while they were describing a picture they had previously seen. This effect was equally strong during recall from memory irrespective of whether the scene visualised had originally been inspected visually by the participants or whether it was constructed whole-cloth from long-term memory. It was also found that eye movements "to nothing" appeared both when the participants were visualising scenes while looking at a blank screen and when they were doing so in complete darkness. The second study explored an effect frequently observed in the first study, involving a "scaling-down" during recall of participants' gaze patterns to an area smaller than that occupied by the stimulus encoded. It was found that this scaling effect correlated with spatial-imagery ability: the gaze patterns of participants with weaker spatial-imagery ability were closer in size to the encoded scene than the gaze patterns of those stronger in spatial-imagery ability. In the third study, the role of eye movements during mental imagery was investigated in four experiments where eye movements were prohibited during either the encoding phase or the recall phase. Experiments 1 and 2 showed that maintaining central fixation during visual or auditory encoding, respectively, had no effect on how eye movements were executed during recall. Thus, oculomotor events during recall are not reproductions of those produced during encoding. In Experiments 3 and 4, central fixation was instead maintained during recall. This turned out to alter and impair scene recollection, irrespective of the modality of encoding. Finally, in the fourth study, the functional role of eye movements in relation to memory retrieval was further investigated by means of direct eye-movement manipulation in the retrieval phase of an episodic-memory task. Four conditions were used: free viewing on a blank screen, maintaining central fixation, viewing within a square congruent with the location of the objects to be recalled, and viewing within a square incongruent with the location of the objects to be recalled. The results obtained show that gaze position plays an active and facilitatory role during memory retrieval. The findings from these studies are discussed in the light of current theories regarding eye movements during mental imagery and memory retrieval. (shrink)