This article considers the issue of opacity as a problem for socially consequential mechanisms of classification and ranking, such as spam filters, credit card fraud detection, search engines, news trends, market segmentation and advertising, insurance or loan qualification, and credit scoring. These mechanisms of classification all frequently rely on computational algorithms, and in many cases on machine learning algorithms to do this work. In this article, I draw a distinction between three forms of opacity: opacity as intentional corporate or state (...) secrecy, opacity as technical illiteracy, and an opacity that arises from the characteristics of machine learning algorithms and the scale required to apply them usefully. The analysis in this article gets inside the algorithms themselves. I cite existing literatures in computer science, known industry practices, and do some testing and manipulation of code as a form of lightweight code audit. I argue that recognizing the distinct forms of opacity that may be coming into play in a given application is a key to determining which of a variety of technical and non-technical solutions could help to prevent harm. (shrink)

Clear-cut cases of decision-making under risk (known probabilities) are unusual in real life. The gambler’s decisions at the roulette table are as close as we can get to this type of decision-making. In contrast, decision-making under uncertainty (unknown probabilities) can be exemplified by a decision whether to enter a jungle that may contain unknown dangers. Life is usually more like an expedition into an unknown jungle than a visit to the casino. Nevertheless, it is common in decision-supporting disciplines to proceed (...) as if reasonably reliable probability estimates were available for all possible outcomes, i.e. as if the prevailing epistemic conditions were analogous to those of gambling at the roulette table. This mistake can be called the tuxedo fallacy . It is argued that traditional engineering practices such as safety factors and multiple safety barriers avoid this fallacy and that they therefore manage uncertainty better than probabilistic risk analysis (PRA). PRA is a useful tool, but it must be supplemented with other methods in order not to limit the analysis to dangers that can be assigned meaningful probability estimates. (shrink)

Clear-cut cases of decision-making under risk (known probabilities) are unusual in real life. The gambler’s decisions at the roulette table are as close as we can get to this type of decision-making. In contrast, decision-making under uncertainty (unknown probabilities) can be exemplified by a decision whether to enter a jungle that may contain unknown dangers. Life is usually more like an expedition into an unknown jungle than a visit to the casino. Nevertheless, it is common in decision-supporting disciplines to proceed (...) as if reasonably reliable probability estimates were available for all possible outcomes, i.e. as if the prevailing epistemic conditions were analogous to those of gambling at the roulette table. This mistake can be called the tuxedo fallacy. It is argued that traditional engineering practices such as safety factors and multiple safety barriers avoid this fallacy and that they therefore manage uncertainty better than probabilistic risk analysis (PRA). PRA is a useful tool, but it must be supplemented with other methods in order not to limit the analysis to dangers that can be assigned meaningful probability estimates. (shrink)

The question of when people may impose risks on each other is of fundamental moral importance. Forms of “quantified risk assessment,” especially risk cost-benefit analysis, provide one powerful approach to providing a systematic answer. It is also well known that such techniques can show that existing resources could be used more effectively to reduce risk overall. Thus it is often argued that some current practices are irrational. On the other hand critics of quantified risk assessment argue that it cannot adequately (...) capture all relevant features, such as “societal concern” and so should be abandoned. In this paper I argue that current forms of quantified risk assessment are inadequate, and in themselves, therefore, insufficient to demonstrate that current practices are irrational. In particular, I will argue that insufficient attention has been given to the cause of a hazard, which needs to be treated as a primary variable in its own right. However rather than reject quantified risk assessment I wish to supplement it by proposing a framework to make explicit the role causation plays in the understanding of risk, and how it interacts with factors which influence perception of risks and other attitudes to risk control. Once an improved description of risk perception is available it will become possible to have a more informed debate about the normative question: how safety should be regulated. (shrink)

How are we to appraise new technological developments that may bring revolutionary social changes? Currently this is often done by trying to predict or anticipate social consequences and to use these as a basis for moral and regulatory appraisal. Such an approach can, however, not deal with the uncertainties and unknowns that are inherent in social changes induced by technological development. An alternative approach is proposed that conceives of the introduction of new technologies into society as a social experiment. An (...) ethical framework for the acceptability of such experiments is developed based on the bioethical principles for experiments with human subjects: non-maleficence, beneficence, respect for autonomy, and justice. This provides a handle for the moral and regulatory assessment of new technologies and their impact on society. (shrink)

How are we to appraise new technological developments that may bring revolutionary social changes? Currently this is often done by trying to predict or anticipate social consequences and to use these as a basis for moral and regulatory appraisal. Such an approach can, however, not deal with the uncertainties and unknowns that are inherent in social changes induced by technological development. An alternative approach is proposed that conceives of the introduction of new technologies into society as a social experiment. An (...) ethical framework for the acceptability of such experiments is developed based on the bioethical principles for experiments with human subjects: non-maleficence, beneficence, respect for autonomy, and justice. This provides a handle for the moral and regulatory assessment of new technologies and their impact on society. (shrink)

The promise of scientific and technological innovation – particularly in fields such as nanotechnology – is increasingly set against what has been articulated as a deficit in public trust in both the new technologies and regulatory mechanisms. Whilst the development of new technology is cast as providing contributions to both quality of life and national competitiveness, what has been termed a ‘legitimacy crisis’ is seen as threatening the vitality of this process. However in contrast to the risk debates that dominated (...) the technological controversies of the late 1990s the vitality of technological innovation is now cast as vulnerable to lack of public confidence and trust in the regulatory and governance structures upon which such innovation depends. In order to address this deficit in public trust, science policy has increasingly turned to the social sciences, suggesting that public values might be incorporated into the development of nanotechnology at an early stage. Public ambivalence therefore constitutes the problem addressed by the increasingly central role that public engagement and participation play in contemporary science policy. Although the recent proliferation of public engagement activities is premised on the need to address this ambivalence through direct engagement, we re-interpret ambivalence as an engaged – rather than passive – mode of relating to technological determinism. Whilst the move toward forms of direct public engagement might be regarded as symptomatic of the emergence of affective mode of governance we interpret public ambivalence as a nested set of enthusiasms and anxieties. Accordingly we suggest that public engagement might be re-thought, utilising ambivalence as a creative resource, rather than as the problem. (shrink)