Decision makers in banking, insurance or employment mitigate many of their risks by telling “good” individuals and “bad” individuals apart. Laws codify societal understandings of which factors are legitimate grounds for differential treatment —or are considered unfair discrimination, including gender, ethnicity or age. Discrimination-aware data mining implements the hope that information technology supporting the decision process can also keep it free from unjust grounds. However, constraining data mining to exclude a fixed enumeration of potentially discriminatory features is insufficient. We argue (...) for complementing it with exploratory DADM, where discriminatory patterns are discovered and flagged rather than suppressed. This article discusses the relative merits of constraint-oriented and exploratory DADM from a conceptual viewpoint. In addition, we consider the case of loan applications to empirically assess the fitness of both discrimination-aware data mining approaches for two of their typical usage scenarios: prevention and detection. Using Mechanical Turk, 215 US-based participants were randomly placed in the roles of a bank clerk or a citizen / policy advisor. They were tasked to recommend or predict the approval or denial of a loan, across three experimental conditions: discrimination-unaware data mining, exploratory, and constraint-oriented DADM. The discrimination-aware tool support in the eDADM and cDADM treatments led to significantly higher proportions of correct decisions, which were also motivated more accurately. There is significant evidence that the relative advantage of discrimination-aware techniques depends on their intended usage. For users focussed on making and motivating their decisions in non-discriminatory ways, cDADM resulted in more accurate and less discriminatory results than eDADM. For users focussed on monitoring for preventing discriminatory decisions and motivating these conclusions, eDADM yielded more accurate results than cDADM. (shrink)

This essay warns of eroding accountability in computerized societies. It argues that assumptions about computing and features of situations in which computers are produced create barriers to accountability. Drawing on philosophical analyses of moral blame and responsibility, four barriers are identified: 1) the problem of many hands, 2) the problem of bugs, 3) blaming the computer, and 4) software ownership without liability. The paper concludes with ideas on how to reverse this trend.

Increasing numbers of decisions about everyday life are made using algorithms. By algorithms we mean predictive models (decision rules) captured from historical data using data mining. Such models often decide prices we pay, select ads we see and news we read online, match job descriptions and candidate CVs, decide who gets a loan, who goes through an extra airport security check, or who gets released on parole. Yet growing evidence suggests that decision making by algorithms may discriminate people, even if (...) the computing process is fair and well-intentioned. This happens due to biased or non-representative learning data in combination with inadvertent modeling procedures. From the regulatory perspective there are two tendencies in relation to this issue: (1) to ensure that data-driven decision making is not discriminatory, and (2) to restrict overall collecting and storing of private data to a necessary minimum. This paper shows that from the computing perspective these two goals are contradictory. We demonstrate empirically and theoretically with standard regression models that in order to make sure that decision models are non-discriminatory, for instance, with respect to race, the sensitive racial information needs to be used in the model building process. Of course, after the model is ready, race should not be required as an input variable for decision making. From the regulatory perspective this has an important implication: collecting sensitive personal data is necessary in order to guarantee fairness of algorithms, and law making needs to find sensible ways to allow using such data in the modeling process. (shrink)

KDD (Knowledge Discovery in Databases) confronts us withphenomena that can intuitively be grasped as highly problematic, but arenevertheless difficult to understand and articulate. Many of theseproblems have to do with what I call the ``deindividualization of theperson'': a tendency of judging and treating persons on the basis ofgroup characteristics instead of on their own individual characteristicsand merits. This tendency will be one of the consequences of theproduction and use of group profiles with the help of KDD. Currentprivacy law and regulations, (...) as well as current ethical theoryconcerning privacy, start from too narrow a definition of ``personaldata'' to capture these problems. In this paper, I introduce the notionof ``categorical privacy'' as a starting point for a possible remedy forthe failures of the current conceptions of privacy. I discuss some waysin which the problems relating to group profiles definitely cannot besolved and I suggest a possible way out of these problems. Finally, Isuggest that it may take us a step forward if we would begin to questionthe predominance of privacy norms in the social debate on informationtechnologies and if we would be prepared to introduce normativeprinciples other than privacy rules for the assessment of newinformation technologies. If we do not succeed in articulating theproblems relating to KDD clearly, one day we may find ourselves in asituation where KDD appears to have undermined the methodic andnormative individualism which pervades the mainstream of morality andmoral theory. (shrink)