Statistical learning theory is often associated with the principle of Occam’s razor, which recommends a simplicity preference in inductive inference. This paper distills the core argument for simplicity obtainable from statistical learning theory, built on the theory’s central learning guarantee for the method of empirical risk minimization. This core “means-ends” argument is that a simpler hypothesis class or inductive model is better because it has better learning guarantees; however, these guarantees are model-relative and so the theoretical push towards simplicity is (...) checked by our prior knowledge. (shrink)

The no-free-lunch theorems promote a skeptical conclusion that all possible machine learning algorithms equally lack justification. But how could this leave room for a learning theory, that shows that some algorithms are better than others? Drawing parallels to the philosophy of induction, we point out that the no-free-lunch results presuppose a conception of learning algorithms as purely data-driven. On this conception, every algorithm must have an inherent inductive bias, that wants justification. We argue that many standard learning algorithms should rather (...) be understood as model-dependent: in each application they also require for input a model, representing a bias. Generic algorithms themselves, they can be given a model-relative justification. (shrink)

Hilary Kornblith (1993) has recently offered a reliabilist defense of the use of the Law of Small Numbers in inductive inference. In this paper I argue that Kornblith's defense of this inferential rule fails for a number of reasons. First, I argue that the sort of inferences that Kornblith seeks to justify are not really inductive inferences based on small samples. Instead, they are knowledge-based deductive inferences. Second, I address Kornblith's attempt to find support in the work of Dorrit Billman (...) and I try to show that close attention to the workings of her computational model reveals that it does not support Kornblith's argument. While the knowledge required to ground the inferences in question is perhaps inductively derived, Billman's work does not support the notion that small samples provide a reliable basis for our generalizing inferences. (shrink)

The problem of valid induction could be stated as follows: are we justified in accepting a given hypothesis on the basis of observations that frequently confirm it? The present paper argues that this question is relevant for the understanding of Machine Learning, but insufficient. Recent research in inductive reasoning has prompted another, more fundamental question: there is not just one given rule to be tested, there are a large number of possible rules, and many of these are somehow confirmed by (...) the data — how are we to restrict the space of inductive hypotheses and choose effectively some rules that will probably perform well on future examples? We analyze if and how this problem is approached in standard accounts of induction and show the difficulties that are present. Finally, we suggest that the explanation-based learning approach and related methods of knowledge intensive induction could be, if not a solution, at least a tool for solving some of these problems. (shrink)