Recent buzzes around big data, data science and artificial intelligence portray a data-driven future for healthcare. As a response, Europe's key players have stimulated the use of big data technologies to make healthcare more efficient and effective. Critical Data Studies and Science and Technology Studies have developed many concepts to reflect on such overly positive narratives and conduct critical policy evaluations. In this study, we argue that there is also much to be learned from studying how professionals in the healthcare (...) field affectively engage with this strong European narrative in concrete big data projects. We followed twelve hospital-based big data pilots in eight European countries and interviewed 145 professionals between 2018 and 2020. In this study, we introduce the metaphor of dreams to describe how professionals link the big data promises to their own frustrations, ideas, values and experiences with healthcare. Our research answers the question: how do professionals in concrete data-driven initiatives affectively engage with European Union's data hopes in their ‘dreams’ – and with what consequences? We describe the dreams of being seen, of timeliness, of connectedness and of being in control. Each of these dreams emphasizes certain aspects of the grand narrative of big data in Europe, makes particular assumptions and has different consequences. We argue that including attention to these dreams in our work could help shine an additional critical light on the big data developments and stimulate the development of responsible data-driven healthcare. (shrink)

There is a problematic tradition of dualistic and reductionist thinking in artificial intelligence (AI) research, which is evident in AI storytelling and imaginations as well as in public debates about AI. Dualistic thinking is based on the assumption of a fixed reality and a hierarchy of power, and it simplifies the complex relationships between humans and machines. This commentary piece argues that we need to work against the grain of such logics and instead develop a thinking that acknowledges AI–human interconnectedness (...) and the complexity in such relations. To learn how to live better with AI in futures to come, the paper suggests an AI politics that turns to practices of serious attentiveness to help us re-imagine our machines and re-configure AI–human relations. (shrink)

This paper reports on a two-year, field-based study set in a charter management organization, a not-for-profit educational organization that operates 18 public schools exclusively in the Black and Latinx communities of South and East Los Angeles. At CMO-LAX, the nine-member Data Team pursues the organization's avowed mission of making public schools data-driven, primarily through the aggregation, analysis, and visualization of digital data derived from quotidian educational activities. This paper draws on the theory of racialized organizations to characterize aspects of data-driven (...) management of public education as practiced by CMO-LAX. I explore two examples of how CMO-LAX shapes data to support racial projects: the reconstruction of the figure of chronic truants and the incorporation of this figure in a calculative regime of student accomplishment. Organizational uses of data support a strategy I call productive myopia, a way of pursuing racial projects via seemingly independent, objective quantifications. This strategy allows the organization to claim to mitigate racial projects and, simultaneously, to accommodate them. This paper concludes by arguing for approaches to research and practice that center racial projects, particularly when data-intensive tools and platforms are incorporated into the provision of public goods and services such as education. (shrink)

A common theme in social science studies of algorithms is that they are profoundly opaque and function as “black boxes.” Scholars have developed several methodological approaches in order to address algorithmic opacity. Here I argue that we can explicitly enroll algorithms in ethnographic research, which can shed light on unexpected aspects of algorithmic systems—including their opacity. I delineate three meso-level strategies for algorithmic ethnography. The first, algorithmic refraction, examines the reconfigurations that take place when computational software, people, and institutions interact. (...) The second strategy, algorithmic comparison, relies on a similarity-and-difference approach to identify the instruments’ unique features. The third strategy, algorithmic triangulation, enrolls algorithms to help gather rich qualitative data. I conclude by discussing the implications of this toolkit for the study of algorithms and future of ethnographic fieldwork. (shrink)

This paper contributes to our understanding of farm data value chains with assistance from 54 semi-structured interviews and field notes from participant observations. Methodologically, it includes individuals, such as farmers, who hold well-known positionalities within digital agriculture spaces—platforms that include precision farming techniques, farm equipment built on machine learning architecture and algorithms, and robotics—while also including less visible elements and practices. The actors interviewed and materialities and performances observed thus came from spaces and places inhabited by, for example, farmers, crop (...) scientists, statisticians, programmers, and senior leadership in firms located in the U.S. and Canada. The stability of “the” artifacts followed for this project proved challenging, which led to me rethinking how to approach the subject conceptually. The paper is animated by a posthumanist commitment, drawing heavily from assemblage thinking and critical data scholarship coming out of Science and Technology Studies. The argument’s understanding of “chains” therefore lies on an alternative conceptual plane relative to most commodity chain scholarship. To speak of a data value chain is to foreground an orchestrating set of relations among humans, non-humans, products, spaces, places, and practices. The paper’s principle contribution involves interrogating lock-in tendencies at different “points” along the digital farm platform assemblage while pushing for a varied understanding of governance depending on the roles of the actors and actants involved. (shrink)

Two definitions of algorithm, their uses, and their implied models of computing in society, are reviewed. The first, termed the structural programming definition, aligns more with usage in computer science, and as the name suggests, the intellectual project of structured programming. The second, termed the systemic definition, is more informal and emerges from ethnographic observations of discussions of software in both professional and everyday settings. Specific examples of locating algorithms within modern codebases are shared, as well as code directly impacting (...) social and ethical concerns. The structural distinction between algorithms and social concerns is explained as mirroring the engineering construct of algorithms and data structures. It is proposed that, rather than this separation being an attempt to enforce a professional boundary and evade social responsibility, it is a crucial technical distinction within code which makes it clearer and more transparent. The power structures reinforced by the broader, cultural interpretations of algorithm are reconsidered, along with what it would mean for software to have an inclusive design culture. (shrink)

The power of algorithms has become a familiar topic in society, media, and the social sciences. It is increasingly common to argue that, for instance, algorithms automate inequality, that they are biased black boxes that reproduce racism, or that they control our money and information. Implicit in many of these discussions is that algorithms are permeated with normativities, and that these normativities shape society. The aim of this editorial is double: First, it contributes to a more nuanced discussion about algorithms (...) by discussing how we, as social scientists, think about algorithms in relation to five theoretical ideal types. For instance, what does it mean to go under the hood of the algorithm and what does it mean to stay above it? Second, it introduces the contributions to this special theme by situating them in relation to these five ideal types. By doing this, the editorial aims to contribute to an increased analytical awareness of how algorithms are theorized in society and culture. The articles in the special theme deal with algorithms in different settings, ranging from farming, schools, and self-tracking to AIDS, nuclear power plants, and surveillance. The contributions thus explore, both theoretically and empirically, different settings where algorithms are intertwined with normativities. (shrink)

Since the inception of recorded music there has been a need for standards and reliability across sound formats and listening environments. The role of the audio mastering engineer is prestigious and akin to a craft expert combining scientific knowledge, musical learning, manual precision and skill, and an awareness of cultural fashions and creative labour. With the advent of algorithms, big data and machine learning, loosely termed artificial intelligence in this creative sector, there is now the possibility of automating human audio (...) mastering processes and radically disrupting mastering careers. The emergence of dedicated products and services in artificial intelligence-driven audio mastering poses profound questions for the future of the music industry, already having faced significant challenges due to the digitalization of music over the past decades. The research reports on qualitative and ethnographic inquiry with audio mastering engineers on the automation of their expertise and the potential for artificial intelligence to augment or replace aspects of their workflows. Investigating audio mastering engineers' awareness of artificial intelligence, the research probes the importance of criticality in their labour. The research identifies intuitive performance and critical listening as areas where human ingenuity and communication pose problems for simulation. Affective labour disrupts speculation of algorithmic domination by highlighting the pragmatic strategies available for humans to adapt and augment digital technologies. (shrink)

Connecting corporate software work in the United States and Germany, this essay tracks the racialization of mostly male Indian software engineers through the casualization of their labor. In doing so, I show the connections between overt, anti-immigrant violence today and the ongoing use of race to sediment divisions of labor in the industry as a whole. To explain racialization in the tech industry, I develop the concept of race-as-algorithm as a device to unpack how race is made productive within digital (...) economies and to show the flexibility of race as it works to create orders of classification that are sensitive to context. Using evidence collected through observation in tech offices and through interviews with programmers over five years, I track race as an essential but continually disavowed variable within the construction of global tech economies. Historical racializations of casual labor in plantation economies illuminates how casualness marks laborers whose rights can be muted and allows corporations to deny their culpability in promoting discrimination within and outside of the tech industry. These denials occur across a political field that divides “good” from “bad” migrants. Using the ethnographic symptoms that Indian tech workers identify in their environments, this essay reads these signs as an antidote to these continued denials. (shrink)

Policy discussions and corporate strategies on machine learning are increasingly championing data reuse as a key element in digital transformations. These aspirations are often coupled with a focus on responsibility, ethics and transparency, as well as emergent forms of regulation that seek to set demands for corporate conduct and the protection of civic rights. And the Protective measures include methods of traceability and assessments of ‘good’ and ‘bad’ datasets and algorithms that are considered to be traceable, stable and contained. However, (...) these ways of thinking about both technology and ethics obscure a fundamental issue, namely that machine learning systems entangle data, algorithms and more-than-human environments in ways that challenge a well-defined separation. This article investigates the fundamental fallacy of most data reuse strategies as well as their regulation and mitigation strategies that data can somehow be followed, contained and controlled in machine learning processes. Instead, the article argues that we need to understand the reuse of data as an inherently entangled phenomenon. To examine this tension between the discursive regimes and the realities of data reuse, we advance the notion of reuse entanglements as an analytical lens. The main contribution of the article is the conceptualization of reuse that places entanglements at its core and the articulation of its relevance using empirical illustrations. This is important, we argue, for our understanding of the nature of data and algorithms, for the practical uses of data and algorithms and our attitudes regarding ethics, responsibility and regulation. (shrink)

Machine learning algorithms pervade contemporary society. They are integral to social institutions, inform processes of governance, and animate the mundane technologies of daily life. Consistently, the outcomes of machine learning reflect, reproduce, and amplify structural inequalities. The field of fair machine learning has emerged in response, developing mathematical techniques that increase fairness based on anti-classification, classification parity, and calibration standards. In practice, these computational correctives invariably fall short, operating from an algorithmic idealism that does not, and cannot, address systemic, Intersectional (...) stratifications. Taking present fair machine learning methods as our point of departure, we suggest instead the notion and practice of algorithmic reparation. Rooted in theories of Intersectionality, reparative algorithms name, unmask, and undo allocative and representational harms as they materialize in sociotechnical form. We propose algorithmic reparation as a foundation for building, evaluating, adjusting, and when necessary, omitting and eradicating machine learning systems. (shrink)

Predictive uses of data are becoming widespread in institutional settings as actors seek to anticipate people and their activities. Predictive modeling is increasingly the subject of scholarly and public criticism. Less common, however, is scrutiny directed at the data that inform predictive models beyond concerns about homogenous training data or general epistemological critiques of data. In this paper, I draw from a qualitative case study set in higher education in the United States to investigate the making of data. Data analytics (...) projects at universities have become more pervasive and intensive to better understand and anticipate undergraduate student bodies. Drawing from 12 months of ethnographic research at a large public university, I analyze the ways data personnel at the institution—data scientists, administrators, and programmers—sort student data into “attributes” and “behaviors,” where “attributes” are demographic data that students “can’t change.” “Behaviors,” in contrast, are data defined as reflective of what students can choose: attending and paying attention in class, studying on campus, among other data which personnel categorize as what students have control over. This discursive split enables the institution nudge students to make responsible choices according to behavior data that correlate with success in the predictive model. In discussing how personnel type, sort, stabilize, and nudge on behavior data, this paper examines the contingencies of data making processes and implications for the application of student data. (shrink)

This article explores how the shift from print to electronic calendars materializes and exacerbates a distinctively quantitative, “spreadsheet” orientation to time. Drawing on interviews with engineers, I argue that calendaring systems are emblematic of a larger design rationale in Silicon Valley to mechanize human thought and action in order to make them more efficient and reliable. The belief that technology can be profitably employed to control and manage time has a long history and continues to animate contemporary sociotechnical imaginaries of (...) what automation will deliver. In the current moment we live in the age of the algorithm and machine learning, so it is no wonder, then, that the contemporary design of digital calendars is driven by a vision of intelligent time management. As I go on to show in the second part of the article, this vision is increasingly realized in the form of intelligent digital assistants whose tracking capacities and behavioral algorithms aim to solve life’s existential problem—how best to organize the time of our lives. This article contributes to STS scholarship on the role of technological artifacts in generating new temporalities that shape people’s perception of time, how they act in the world, and how they understand themselves. (shrink)

Algorithmic vision, the computational process of making meaning from digital images or visual information, has changed the relationship between the image and the human subject. In this paper, I explicate on the role of algorithmic vision as a technique of algorithmic governance, the organization of a population by algorithmic means. With its roots in the United States post-war cybernetic sciences, the ontological status of the computational image undergoes a shift, giving way to the hegemonic use of automated facial recognition technologies (...) towards predatory policing and profiling practices. By way of example, I argue that algorithmic vision reconfigures the philosophical links between vision, image, and truth, paradigmatically changing the way a human subject is represented through imagistic data. With algorithmic vision, the relationship between subject and representation challenges the humanistic discourse around images, calling for a critical displacement of the human subject from the center of an analysis of how computational images make meaning. I will explore the relationship between the operative image, the image that acts but is not seen by human eyes, and what Louise Amoore calls an “emergent subject,” a subject that is made visible through algorithmic techniques. Algorithmic vision reveals subjects to power in a mode that requires a new approach towards analyzing the entanglement and invisiblization of the human in automated decision-making systems. (shrink)

This paper seeks to bypass assumptions that researchers in critical algorithmic studies and urban studies find it difficult to study algorithmic systems due to their black-boxed nature. In addition, it seeks to work against the assumption that advocating for transparency in algorithms is, therefore, the key for achieving an enhanced understanding of the role of algorithmic technologies on modern life. Drawing on applied assemblage thinking via the concept of the urban assemblage, I demonstrate how the notion of urban assemblage can (...) work as an alternative way to explore the distributed and potential dimensions of what has been termed as Urban AI phenomena. Rather than seeing Urban AI phenomena as black-boxed, unknown and opaque, the notion of urban assemblage locates such entities within the wider contests of the city: urban places, communities and politics, where human-algorithmic relationships gather and disperse. In addition, this approach focuses on the potentialities of Urban AI phenomena—how algorithmic systems can operate differently through different aspects of the city—which can be seen to manifest new forms of resistance, collective actions and democracy. I use a case study of an algorithmic system designed to facilitate digital democracy—vTaiwan—to exemplify how assemblage methodology foregrounds the role of cities as spaces and places for exploring the democratic possibilities of algorithmic systems. This paper concludes with discussion of how the assemblage methodology contributes to serve as a bridge between critical algorithm studies and recent studies of platform urbanism. (shrink)

Transparency is now a fundamental principle for data processing under the General Data Protection Regulation. We explore what this requirement entails for artificial intelligence and automated decision-making systems. We address the topic of transparency in artificial intelligence by integrating legal, social, and ethical aspects. We first investigate the ratio legis of the transparency requirement in the General Data Protection Regulation and its ethical underpinnings, showing its focus on the provision of information and explanation. We then discuss the pitfalls with respect (...) to this requirement by focusing on the significance of contextual and performative factors in the implementation of transparency. We show that human–computer interaction and human-robot interaction literature do not provide clear results with respect to the benefits of transparency for users of artificial intelligence technologies due to the impact of a wide range of contextual factors, including performative aspects. We conclude by integrating the information- and explanation-based approach to transparency with the critical contextual approach, proposing that transparency as required by the General Data Protection Regulation in itself may be insufficient to achieve the positive goals associated with transparency. Instead, we propose to understand transparency relationally, where information provision is conceptualized as communication between technology providers and users, and where assessments of trustworthiness based on contextual factors mediate the value of transparency communications. This relational concept of transparency points to future research directions for the study of transparency in artificial intelligence systems and should be taken into account in policymaking. (shrink)

Over recent years, the stakes and complexity of online content moderation have been steadily raised, swelling from concerns about personal conflict in smaller communities to worries about effects on public life and democracy. Because of the massive growth in online expressions, automated tools based on machine learning are increasingly used to moderate speech. While ‘design-based governance’ through complex algorithmic techniques has come under intense scrutiny, critical research covering algorithmic content moderation is still rare. To add to our understanding of concrete (...) instances of machine moderation, this article examines Perspective API, a system for the automated detection of ‘toxicity’ developed and run by the Google unit Jigsaw that can be used by websites to help moderate their forums and comment sections. The article proceeds in four steps. First, we present our methodological strategy and the empirical materials we were able to draw on, including interviews, documentation, and GitHub repositories. We then summarize our findings along five axes to identify the various threads Perspective API brings together to deliver a working product. The third section discusses two conflicting organizational logics within the project, paying attention to both critique and what can be learned from the specific case at hand. We conclude by arguing that the opposition between ‘human’ and ‘machine’ in speech moderation obscures the many ways these two come together in concrete systems, and suggest that the way forward requires proactive engagement with the design of technologies as well as the institutions they are embedded in. (shrink)

This paper examines folk theories of algorithmic recommendations on Spotify in order to make visible the cultural specificities of data assemblages in the global South. The study was conducted in Costa Rica and draws on triangulated data from 30 interviews, 4 focus groups with 22 users, and the study of “rich pictures” made by individuals to graphically represent their understanding of algorithmic recommendations. We found two main folk theories: one that personifies Spotify and another one that envisions it as a (...) system full of resources. Whereas the first theory emphasizes local conceptions of social relations to make sense of algorithms, the second one stresses the role of algorithms in providing a global experience of music and technology. We analyze why people espouse either one of these theories and how these theories provide users with resources to enact different modalities of power and resistance in relation to recommendation algorithms. We argue that folk theories thus offer a productive way to broaden understanding of what agency means in relation to algorithms. (shrink)

Aborda a modulação algorítmica e sua relação com a mediação da informação e o processo de filtragem da informação a partir do efeito “filtro-bolha”. Tem como objetivos discutir a informação no contexto da sociedade algoritimizada; debater a vigilância, o monitoramento e a filtragem da informação; e destacar a modulação algorítmica e a mediação da informação. A abordagem é qualitativa e utiliza a pesquisa bibliográfica sobre a temática proposta e a observação direta junto às plataformas digitais que fazem uso de filtragem. (...) Como resultado, este artigo apresenta uma revisão dos principais conceitos que norteiam a filtragem da informação com o auxílio da modulação algorítmica. Conclui-se que a filtragem da informação e o “filtro-bolha” desenvolvem um ambiente controlado de informações mediante a propósitos escusos, gerando influências nos tecidos informacional, social, cultural e político da sociedade. (shrink)

The social impacts of computer technology are often glorified in public discourse, but there is growing concern about its actual effects on society. In this article, we ask: how does “consent” as an analytical framework make visible the social dynamics and power relations in the capture, extraction, and labor of data science knowledge production? We hypothesize that a form of boundary violation in data science workplaces—gender harassment—may correlate with the ways humans’ lived experiences are extracted to produce Big Data. The (...) concept of consent offers a useful way to draw comparisons between gender relations in data science and the means by which machines are trained to learn and reason. Inspired by how Big Tech leaders describe unsupervised machine learning, and the co-optation of “revolutionary” rhetoric they use to do so, we introduce a concept we call “techniques of invisibility.” Techniques of invisibility are the ways in which an extreme imbalance between exposure and opacity, demarcated along fault lines of power, are fabricated and maintained, closing down the possibility for bidirectional transparency in the production and applications of algorithms. Further, techniques of invisibility, which we group into two categories—epistemic injustice and the Brotherhood—include acts of subjection by powerful actors in data science designed to quell resistance to exploitative relations. These techniques may be useful in making further connections between epistemic violence, sexism, and surveillance, sussing out persistent boundary violations in data science to render the social in data science visible, and open to scrutiny and debate. (shrink)

The central thesis of this paper is that human unconscious processes influence the behavior and design of artificial intelligence (AI). This thesis is discussed through the case study of a chatbot called Replika, which intends to provide psychological assistance and friendship but has been accused of inciting murder and suicide. Replika originated from a trauma and a work of mourning lived by its creator. The traces of these unconscious dynamics can be detected in the design of the app and the (...) narratives about it. Therefore, a process of de-psychologization and de-humanization of the unconscious takes place through AI. This psychosocial approach helps criticize and overcome the so-called “standard model of intelligence” shared by most AI researchers. It facilitates a new interpretation of some classic problems in AI, such as control and responsibility. (shrink)

All datasets emerge from and are enmeshed in power-laden semiotic systems. While emerging data ethics curriculum is supporting data science students in identifying data biases and their consequences, critical attention to the cultural histories and vested interests animating data semantics is needed to elucidate the assumptions and political commitments on which data rest, along with the externalities they produce. In this article, I introduce three modes of reading that can be engaged when studying datasets—a denotative reading, a connotative reading, and (...) a deconstructive reading. I then outline how I have taught students to engage these methods when analyzing three datasets in Data and Society—a course designed to cultivate student competency in politically aware data analysis and interpretation. I show how combined, the reading strategies prompt students to grapple with the double binds of perceiving contemporary problems through systems of representation that are always situated, incomplete, and inflected with diverse politics. While I introduce these methods in the context of teaching, I argue that the methods are integral to any data practice in the conclusion. (shrink)

In the age of ubiquitous computing and artificially intelligent applications, social machines serves as a powerful framework for understanding and interpreting interactions in socio-algorithmic ecosystems. Although researchers have largely used it to analyze the interactions of individuals and algorithms, limited attempts have been made to investigate the politics in social machines. In this study, I claim that social machines are per se political machines, and introduce a five-point framework for classifying influence processes in socio-algorithmic ecosystems. By drawing from scholars from (...) political theory, I use a notion of influence that functions as a meta-concept for connecting and comparing different conceptions of politics. In this way, I can associate multiple political aspects of social machines from a cybernetic perspective. I show that the framework efficiently categorizes dimensions of influence that shape interactions between individuals and algorithms. These categories are symbolic influence, political conduct, algorithmic influence, design, and regulatory influence. Using case studies, I describe how they interact with each other on online social networks and in algorithmic decision-making systems and illustrate how the framework is able to guide scientists in further research. (shrink)

Drawing on the analytic of the “colonial matrix of power” developed by Aníbal Quijano within the Latin American modernity/coloniality research program, this article theorises how a system of coloniality underpins the structuring logic of artificial intelligence (AI) systems. We develop a framework for critiquing the regimes of global labour exploitation and knowledge extraction that are rendered invisible through discourses of the purported universality and objectivity of AI. ​​Through bringing the political economy literature on AI production into conversation with scholarly work (...) on decolonial AI and the modernity/coloniality research program, we advance three main arguments. First, the global economic and political power imbalances in AI production are inextricably linked to the continuities of historical colonialism, constituting the colonial supply chain of AI. Second, this is produced through an international division of digital labour that extracts value from majority world labour for the benefit of Western technology companies. Third, this perpetuates hegemonic knowledge production through Western values and knowledge that marginalises non-Western alternatives within AI’s production and limits the possibilities for decolonising AI. By locating the production of AI systems within the colonial matrix of power, we contribute to critical and decolonial literature on the legacies of colonialism in AI and the hierarchies of power and extraction that shape the development of AI today. (shrink)

The development of numerical models to produce realistic prospective scenarios for the evolution of biological diversity is essential. Only integrative impact assessment models are able to take into account the diverse and complex interactions embedded in social-ecological systems. The knowledge used is objective, the procedure of their integration is rigorous and the data massive. Nevertheless, the technical choices made at each stage of the development of models and scenarios are mostly circumstantial, depending on both the skills of modellers on a (...) project and the means available to them. In the end, the scenarios selected and the way they are simulated limit the futures explored, and the options offered to decision makers and stakeholders to act. The ethical implications of these circumstantial choices are generally not documented, explained or even perceived by modellers. Applied ethics propose a coherent set of principles to guide a critical reflection on the social and environmental consequences of integrative modelling and simulation of biodiversity scenarios. Such reflection should be incorporated into the actual modelling process, in a broad participatory framework, and foster effective moral involvement of modellers, policy-makers and stakeholders, in preference to the application of fixed ethical rules. (shrink)

Algorithms silently structure our lives. Algorithms can determine whether someone is hired, promoted, offered a loan, or provided housing as well as determine which political ads and news articles consumers see. Yet, the responsibility for algorithms in these important decisions is not clear. This article identifies whether developers have a responsibility for their algorithms later in use, what those firms are responsible for, and the normative grounding for that responsibility. I conceptualize algorithms as value-laden, rather than neutral, in that algorithms (...) create moral consequences, reinforce or undercut ethical principles, and enable or diminish stakeholder rights and dignity. In addition, algorithms are an important actor in ethical decisions and influence the delegation of roles and responsibilities within these decisions. As such, firms should be responsible not only for the value-laden-ness of an algorithm but also for designing who-does-what within the algorithmic decision. As such, firms developing algorithms are accountable for designing how large a role individual will be permitted to take in the subsequent algorithmic decision. Counter to current arguments, I find that if an algorithm is designed to preclude individuals from taking responsibility within a decision, then the designer of the algorithm should be held accountable for the ethical implications of the algorithm in use. (shrink)

How can one ‘see’ the operationalization of contemporary visual culture, given the imperceptibility and apparent automation of so many processes and dimensions of visuality? Seeing – as a position from a singular mode of observation – has become problematic since many visual elements, techniques, and forms of observing are highly distributed through data practices of collection, analysis and prediction. Such practices are subtended by visual cultural techniques that are grounded in the development of image collections, image formatting and hardware design. (...) In this article, we analyze recent transformations in forms of prediction and data analytics associated with spectacular performances of computation. We analyze how transformations in the collection and accumulation of images as ensembles by platforms have a qualitative and material effect on the emergent sociotechnicality of platform ‘life’ and ‘perception’. Reconstructing the visual transformations that allow artificial intelligence assemblages to operate allows some sense of their heteronomous materiality and contingency. (shrink)

This paper examines the paradoxical transparency involved in training machine-learning models. Existing literature typically critiques the opacity of machine-learning models such as neural networks or collaborative filtering, a type of critique that parallels the black-box critique in technology studies. Accordingly, people in power may leverage the models’ opacity to justify a biased result without subjecting the technical operations to public scrutiny, in what Dan McQuillan metaphorically depicts as an “algorithmic state of exception”. This paper attempts to differentiate the black-box abstraction (...) that wraps around complex computational systems from the opacity of machine-learning models. It contends that the degree of asymmetry in knowledge is greater in the former than the latter. In the case of software systems, software codes are difficult to understand as only software experts with sufficient domain knowledge are equipped to formulate a sound critique. In contrast, the meanings of trained parameters in a machine-learning model are obscure even to the data scientists who configure and train the model. Hence, the asymmetry of knowledge lies only in how data examples are collected, the choice and configuration of machine-learning models, and the specification of features in model design. Under the trend of algorithmic decision-making proliferating with machine-learning heuristics, the paper contends that the more symmetric distribution of knowledge in machine learning could lead to a more transparent production process if proper policies are in place. (shrink)

As states increasingly use algorithms to improve the legibility of society, particularly during the COVID-19 pandemic, it is common for concerns about the expanding power of the algorithm or the state to be raised in a deterministic manner. However, how are the algorithms for states’ legibility projects enacted, contested, and reconfigured? Drawing on interviews and media data, this study fills this gap by examining Health Code, the Chinese contact tracing and risk assessment algorithmic system that serves as the COVID-19 health (...) passport. I first explore the intensive and invisible work and infrastructures that enact and stabilize Health Code’s sociotechnical assemblage. I then show how this assemblage is frequently challenged and destabilized by errors, breakdowns, and exclusions. Facing unintended engagements from heterogeneous social actors, local interests, and power hierarchies, Health Code reassembles into multiple and contradictory assemblages at different periods and social localities. Finally, I examine how people game and bypass the algorithm’s surveillance with their agencies. Recognizing this messiness and heterogeneity contributes to a more nuanced and realistic understanding of states’ use of algorithms, including the risks. Doing so also urges us to rethink the politics of citizenship and inequality in the digital age beyond inclusion. (shrink)

Organizations increasingly rely on algorithm-based HR decision-making to monitor their employees. This trend is reinforced by the technology industry claiming that its decision-making tools are efficient and objective, downplaying their potential biases. In our manuscript, we identify an important challenge arising from the efficiency-driven logic of algorithm-based HR decision-making, namely that it may shift the delicate balance between employees’ personal integrity and compliance more in the direction of compliance. We suggest that critical data literacy, ethical awareness, the use of participatory (...) design methods, and private regulatory regimes within civil society can help overcome these challenges. Our paper contributes to literature on workplace monitoring, critical data studies, personal integrity, and literature at the intersection between HR management and corporate responsibility. (shrink)

In recent years, scholars in the social sciences and humanities have turned their attention to how the rise of digital technologies is reshaping political life in contemporary society. Here, we analyze this issue by distinguishing between two classification technologies typical of pre-digital and digital eras that differently constitute the relationship between individuals and groups. In class-based systems, characteristic of the pre-digital era, one’s status as an individual is gained through membership in a group in which salient social identities are shared (...) in common with other group members. In attribute-based systems, characteristic of the digital era, one’s status as an individual is determined by virtue of possession of a set of attributes that need not be shared with others. We argue that differences between these two types of classification technologies have important implications for how persons attach (or fail to attach) to groups, and therefore what kinds of political mobilization are possible. We illustrate this argument by examining contention over the use of gender as a variable in the pricing of risk in insurance and credit – two markets in which individuals directly encounter class-based and attribute-based systems of classification, respectively. (shrink)

This article deals with choice-inducing algorithms––algorithms that are explicitly designed to affect people’s choices. Based on an ethnographic account of three Israeli data analytics companies, I explore how algorithms are being designed to drive people into choice-making and examine their co-constitution by an assemblage of specifically positioned human and nonhuman agents. I show that the functioning, logic, and even ethics of choice-inducing algorithms are deeply influenced by the epistemologies, meaning systems, and practices of the individuals who devise and use them (...) and that such algorithms are similarly affected by interorganizational relationships, various nonhuman agents, and changing geopolitical contexts. I conclude by discussing the flexibility of choice-inducing algorithms and by arguing that such algorithms are not programmed to induce specific choices but to more generally convert people into choosers, and thus, to algorithmically create the modern need to choose. This article contributes to the growing literature on algorithms and culture and to our understanding of choice-making in contemporary life. At the same time, it provides a new vocabulary that offers to critically engage with algorithms and their power without losing sight of the often very specific contexts from which they arise. (shrink)

This paper focuses on the conception and use of machine-learning algorithms for marketing. In the last years, specialized service providers as well as in-house data scientists have been increasingly using machine learning to predict consumer behavior for large companies. Predictive marketing thus revives the old dream of one-to-one, perfectly adjusted selling techniques, now at an unprecedented scale. How do predictive marketing devices change the way corporations know and model their customers? Drawing from STS and the sociology of quantification, I propose (...) to study the original ambivalence that characterizes the promise of a mass personalization, i.e. algorithmic processes in which the precise adjustment of prediction to unique individuals involves the computation of massive datasets. By studying algorithms in practice, I show how the active embedding of local preexisting consumer knowledge and punctual de-personalization mechanisms are keys to the epistemic and organizational success of predictive marketing. This paper argues for the study of algorithms in their contexts and suggests new perspectives on algorithmic objectivity. (shrink)

Research on algorithms tends to focus on American companies and on the effects their algorithms have on Western users, while such algorithms are in fact developed in various geographical locations and used in highly diverse socio-cultural contexts. That is, the spatial trajectories through which algorithms operate and the distances and differences between the people who develop such algorithms and the users their algorithms affect remain overlooked. Moreover, while the power of big data algorithms has been recently compared to colonialism (Couldry (...) and Mejias 2019), the move from the colonial gaze (Yegenoglu 1998) to the algorithmic gaze (Graham 2010) has yet to be fully discussed. This article aims to fill these gaps by exploring the attempts to algorithmically conceptualize “the Other”. Based on the case study of an Israeli user-profiling company and its attempts to sell its services to East Asian corporations, I show that the algorithmic gaze—algorithms’ ability to characterize, conceptualize, and affect users—stems from a complex combination between opposing-but-complimentary perspectives: that it is simultaneously a continuation of the colonial gaze and its complete opposite. The ways in which algorithms are being programmed to see the Other, the ways algorithmic categories are named to depict the Other, and the ways people who design such algorithms describe and understand the Other are all different but deeply interrelated factors in how algorithms “see.” I accordingly argue that the story of algorithms is an intercultural one, and that the power of algorithms perpetually flows back and forth—between East and West, South and North. (shrink)

The harvesting of data about people, organizations, and things and their transformation into a form of capital is often described as a process of “accumulation by dispossession,” a pervasive loss of rights buttressed by predatory practices and legal violence. Yet this argument does not square well with the fact that enrollment into digital systems is often experienced as a much more benign process: signing up for a “free” service, responding to a “friend’s” invitation, or being encouraged to “share” content. In (...) this paper, we focus on the centrality of gifting and reciprocity to the business model and cultural imagination of digital capitalism. Relying on historical narratives and in-depth interviews with the designers and critics of digital systems, we explain the cultural genesis of these “give-to-get” relationships and analyze the socio-technical channels that structure them in practice. We suggest that the economic relation that develops as a result of a digital gift offering not only masks the structural asymmetry between giver and gifted but also permits the creation of the new commodity of personal data, obfuscates its true value, and naturalizes its private appropriation. We call this unique regime “accumulation by gift.”. (shrink)

Building on critical approaches that understand algorithms in terms of communication, culture and organization, this paper offers the supplementary conceptualization of algorithms as organizational figuration, defined as material and meaningful sociotechnical arrangements that develop in spatiotemporal processes and are shaped by multiple enactments of affordance–agency relations. We develop this conceptualization through a case study of a Danish fintech start-up that uses machine learning to create opportunities for sustainable pensions investments. By way of ethnographic and literary methodology, we provide an in-depth (...) analysis of the dynamic trajectory in and through which the organization gives shape to and takes shape from its key algorithmic tool, mapping the shifting sociotechnical arrangements of the start-up, from its initial search for a viable business model through the development of the algorithm to the public launch of its product. On this basis, we argue that conceptualizing algorithms as organizational figuration enables us to detail not only what algorithms do but also what they are. (shrink)

This article demonstrates how meditation apps, such as Headspace and Calm, are imbricated within public discourse about technology addiction, exploring the consequences of this discourse on contemporary mental life. Based on ethnographic research with designers and users of meditation apps, I identify a promise put forth by meditation app companies that I call attention by design: a discursive strategy that frames attention as an antidote to technology addiction, which is ostensibly made possible when design is done right. I argue that (...) attention by design is a promise unfulfilled. Meditation app companies construct attention as socially valuable by endlessly pointing out its purported opposite, technology addiction. Attention by design is promissory in that it keeps promising even when it doesn’t deliver what it promises, compelling the user to return to a practice that represents socially desirable traits that can never be fully acquired—and that often recede further from reach as the person becomes distracted by other obligations and communication mediated through the smartphone. Despite this broken promise, users believe they are becoming more attentive. The promissory attention designed into meditation apps reflects a new form of governmentality, in which users receive a mental nudge to reinterpret similarly designed experiences as different. (shrink)

In science and technology studies today, there is a troubling tendency to portray actors in the biosciences as “cultural dopes” and technology as having monolithic qualities with predetermined outcomes. To remedy this analytical impasse, this article introduces the concept styles of valuation to analyze how actors struggle with valuing technology in practice. Empirically, this article examines how actors in a bioscientific laboratory struggle with valuing the properties and qualities of algorithms in a high-throughput setting and identifies the copresence of several (...) different styles. The question that the actors struggle with is what different configurations of algorithms, devices, and humans are “good bioscience,” that is, what do the actors perform as a good distribution of agency between algorithms and humans? A key finding is that algorithms, robots, and humans are valued in multiple ways in the same setting. For the actors, it is not apparent which configuration of agency and devices is more authoritative nor is it obvious which skills and functions should be redistributed to the algorithms. Thus, rather than tying algorithms to one set of values, such as “speed,” “precision,” or “automation,” this article demonstrates the broad utility of attending to the multivalence of algorithms and technology in practice. (shrink)

Framing datafication as new form of knowledge production has become a trope in both academic and commercial contexts. This special theme examines and ultimately rejects the familiar grand claims of datafication, to instead pay attention to emergent conversations that seek to take a more nuanced stock of the status and nature of datafied knowledge production. The articles in this special theme thus engage with datafied knowledge production through elaborate explorations of how datafied knowledge depends on the contexts of its production (...) and the forms of knowledge production that precede it in those contexts. Our basic argument is that while the resources, material features and analytical operations involved in datafied knowledge production may be different, many fundamental concerns about epistemology, ontology and methods remain relevant to understand what shapes it. We still need to understand and explicate the assumptions, operations and consequences of emergent forms of knowledge production. If datafied knowledge production is neither a clean revolutionary break with past forms of knowledge production nor a balloon of pure hype, the articles in this special theme ask: what does the phenomenon of datafied knowledge production look like? Which digital and datafied infrastructures support its future development? And what potentialities and limits do such forms of analysis and knowledge production contain? (shrink)

This article examines the construction of image recognition algorithms for the classification of plant pathology problems. Rooted in science and technology studies research on the effects of agricultural big data and agricultural algorithms, the study ethnographically examines how algorithms for the recognition of plant pathology are made. To do this, the article looks at the case of a German agtech startup developing image recognition algorithms for an app that aims to help small-scale farmers diagnose plant damages based on digital images (...) of their symptoms. The study posits that the construction of these algorithms can be grasped as a succession of layers, at each of which the startup’s employees carry out different selection practices. It is argued that these practices gradually inscribe a selective recognition of the phenomenon of plant pathology into the algorithms of the app. This selective recognition is reflected in the fact that the emerging algorithms are effective in identifying isolated plant damages on isolated crops, while ignoring most agroecosystem-related actors and relations through which plant damages arise. The article concludes that the app’s selective recognition of plant pathology is likely to perpetuate or even exacerbate the pesticide use of its users. This is because the app’s view of plant pathology as a phenomenon comprised of isolated plant damages on isolated crops is more compatible with the use of chemical pesticides than with approaches to crop protection that strive for a more moderate use of these substances (e.g., integrated pest management, agroecology, organic agriculture). (shrink)

Machine learning models are becoming increasingly prevalent in algorithmic trading and investment management. The spread of machine learning in finance challenges existing practices of modelling and model use and creates a demand for practical solutions for how to manage the complexity pertaining to these techniques. Drawing on interviews with quants applying machine learning techniques to financial problems, the article examines how these people manage model complexity in the process of devising machine learning-powered trading algorithms. The analysis shows that machine learning (...) quants use Ockham’s razor – things should not be multiplied without necessity – as a heuristic tool to prevent excess model complexity and secure a certain level of human control and interpretability in the modelling process. I argue that understanding the way quants handle the complexity of learning models is a key to grasping the transformation of the human’s role in contemporary data and model-driven finance. The study contributes to social studies of finance research on the human–model interplay by exploring it in the context of machine learning model use. (shrink)

This paper explores how calculative cultures shape perceptions of models and practices of model use in the financial industry. A calculative culture comprises a specific set of practices and norms concerning data and model use in an organizational setting. Drawing on interviews with model users working in algorithmic securities trading, I argue that the introduction of complex machine-learning models changes the dynamics in calculative cultures, which leads to a displacement of human judgment in quantitative finance. In this paper, I distinguish (...) between three calculative cultures: an idealistic culture of undivided trust in models, a pragmatic culture of skepticism toward model accuracy, and a pragmatic idealist culture of early stage skepticism and implementation and production-phase idealism. Based on the empirical material, the analysis engages with examples of each of the three calculative cultures. The study contributes to the social studies of finance and science and technology studies more broadly by showing how perceptions of models shape and are shaped through model work in data-intensive, computerized finance. (shrink)

Governments and citizens of nearly every nation have been compelled to respond to COVID-19. Many measures have been adopted, including contact tracing and risk assessment algorithms, whereby citizen whereabouts are monitored to trace contact with other infectious individuals in order to generate a risk status via algorithmic evaluation. Based on 38 in-depth interviews, we investigate how people make sense of Health Code, the Chinese contact tracing and risk assessment algorithmic sociotechnical assemblage. We probe how people accept or resist Health Code (...) by examining their ongoing, dynamic, and relational interactions with it. Participants display a rich variety of attitudes toward privacy and surveillance, ranging from fatalism to the possibility of privacy to trade-offs for surveillance in exchange for public health, which is mediated by the perceived effectiveness of Health Code and changing views on the intentions of institutions who deploy it. We show how perceived competency varies not just on how well the technology works, but on the social and cultural enforcement of various non-technical aspects like quarantine, citizen data inputs, and cell reception. Furthermore, we illustrate how perceptions of Health Code are nested in people’s broader interpretations of disease control at the national and global level, and unexpectedly strengthen the Chinese authority’s legitimacy. None of the Chinese public, Health Code, or people’s perceptions toward Health Code are predetermined, fixed, or categorically consistent, but are co-constitutive and dynamic over time. We conclude with a theorization of a relational perception and methodological reflections to study algorithmic sociotechnical assemblages beyond COVID-19. (shrink)

Scholars and practitioners across domains are increasingly concerned with algorithmic transparency and opacity, interrogating the values and assumptions embedded in automated, black-boxed systems, particularly in user-generated content platforms. I report from an ethnography of infrastructure in Wikipedia to discuss an often understudied aspect of this topic: the local, contextual, learned expertise involved in participating in a highly automated social–technical environment. Today, the organizational culture of Wikipedia is deeply intertwined with various data-driven algorithmic systems, which Wikipedians rely on to help manage (...) and govern the “anyone can edit” encyclopedia at a massive scale. These bots, scripts, tools, plugins, and dashboards make Wikipedia more efficient for those who know how to work with them, but like all organizational culture, newcomers must learn them if they want to fully participate. I illustrate how cultural and organizational expertise is enacted around algorithmic agents by discussing two autoethnographic vignettes, which relate my personal experience as a veteran in Wikipedia. I present thick descriptions of how governance and gatekeeping practices are articulated through and in alignment with these automated infrastructures. Over the past 15 years, Wikipedian veterans and administrators have made specific decisions to support administrative and editorial workflows with automation in particular ways and not others. I use these cases of Wikipedia’s bot-supported bureaucracy to discuss several issues in the fields of critical algorithms studies; critical data studies; and fairness, accountability, and transparency in machine learning—most principally arguing that scholarship and practice must go beyond trying to “open up the black box” of such systems and also examine sociocultural processes like newcomer socialization. (shrink)

In this article, we develop the concept of Transparency by Design that serves as practical guidance in helping promote the beneficial functions of transparency while mitigating its challenges in automated-decision making environments. With the rise of artificial intelligence and the ability of AI systems to make automated and self-learned decisions, a call for transparency of how such systems reach decisions has echoed within academic and policy circles. The term transparency, however, relates to multiple concepts, fulfills many functions, and holds different (...) promises that struggle to be realized in concrete applications. Indeed, the complexity of transparency for ADM shows tension between transparency as a normative ideal and its translation to practical application. To address this tension, we first conduct a review of transparency, analyzing its challenges and limitations concerning automated decision-making practices. We then look at the lessons learned from the development of Privacy by Design, as a basis for developing the Transparency by Design principles. Finally, we propose a set of nine principles to cover relevant contextual, technical, informational, and stakeholder-sensitive considerations. Transparency by Design is a model that helps organizations design transparent AI systems, by integrating these principles in a step-by-step manner and as an ex-ante value, not as an afterthought. (shrink)