Nelson Goodman’s new riddle of induction forcefully illustrates a challenge that must be confronted by any adequate theory of inductive inference: provide some basis for choosing among alternative hypotheses that fit past data but make divergent predictions. One response to this challenge is to distinguish among alternatives by means of some epistemically significant characteristic beyond fit with the data. Statistical learning theory takes this approach by showing how a concept similar to Popper’s notion of degrees of testability is linked to (...) minimizing expected predictive error. In contrast, formal learning theory appeals to Ockham’s razor, which it justifies by reference to the goal of enhancing efficient convergence to the truth. In this essay, I show that, despite their differences, statistical and formal learning theory yield precisely the same result for a class of inductive problems that I call strongly VC ordered , of which Goodman’s riddle is just one example. (shrink)

Howson's critique of my essay on Hume's problem of induction levels two main charges. First, Howson claims that I have attributed to him an error that he never made, and in fact which he warned against in the very text that I cite. Secondly, Howson argues that my proposed solution to Hume's problem is flawed on technical and philosophical grounds. In response to the first charge, I explain how Howson's text justifies attributing to him the claim that the principle of (...) induction is shown to be inconsistent by Goodman's riddle. In regards to the second, I show that Howson's objections rest on misunderstandings of formal learning theory and on conflating the problem of induction with the problem of unconceived alternatives. (shrink)

In this paper I compare parametric and nonparametric regression models with the help of a simulated data set. Doing so, I have two main objectives. The first one is to differentiate five concepts of simplicity and assess their respective importance. The second one is to show that the scope of the existing philosophical literature on simplicity and model selection is too narrow because it does not take the nonparametric approach into account, S112–S123, 2002; Forster and Sober in The British Journal (...) for the Philosophy of Science 45, 1–35, 1994; Forster, 2001, in Philosophy of Science 74, 588–600, 2007; Hitchcock and Sober in The British Journal for the Philosophy of Science 55, 1–34, 2004; Mikkelson in Philosophy of Science 73, 440–447, 2006; Baker 2013). More precisely, I point out that a measure of simplicity in terms of the number of adjustable parameters is inadequate to characterise nonparametric models and to compare them with parametric models. This allows me to weed out false claims about what makes a model simpler than another. Furthermore, I show that the importance of simplicity in model selection cannot be captured by the notion of parametric simplicity. ‘Simplicity’ is an umbrella term. While parametric simplicity can be ignored, there are other notions of simplicity that need to be taken into consideration when we choose a model. Such notions are not discussed in the previously mentioned literature. The latter therefore portrays an incomplete picture of why simplicity matters when we choose a model. Overall I support a pluralist view according to which we cannot give a general and interesting justification for the importance of simplicity in science. (shrink)

Ockham’s razor is the principle that, all other things being equal, scientists ought to prefer simpler theories. In recent years, philosophers have argued that simpler theories make better predictions, possess theoretical virtues like explanatory power, and have other pragmatic virtues like computational tractability. However, such arguments fail to explain how and why a preference for simplicity can help one find true theories in scientific inquiry, unless one already assumes that the truth is simple. One new solution to that problem is (...) the Ockham efficiency theorem, which states that scientists who heed Ockham’s razor retract their opinions less often and sooner than do their non-Ockham competitors. The theorem neglects, however, to consider competitors following random strategies and in many applications random strategies are known to achieve better worst-case loss than deterministic strategies. In this paper, we describe two ways to extend the result to a very general class of random, empirical strategies. The first extension concerns expected retractions, retraction times, and errors and the second extension concerns retractions in chance, times of retractions in chance, and chances of errors. (shrink)

Classic conceptions of model simplicity for machine learning are mainly based on the analysis of the structure of the model. Bayesian, Frequentist, information theoretic and expressive power concepts are the best known of them, which are reviewed in this work, along with their underlying assumptions and weaknesses. These approaches were developed before the advent of the Big Data deluge, which has overturned the importance of structural simplicity. The computational simplicity concept is presented, and it is argued that it is more (...) encompassing and closer to actual machine learning practices than the classic ones. In order to process the huge datasets which are commonplace nowadays, the computational complexity of the learning algorithm is the decisive factor to assess the viability of a machine learning strategy, while the classic accounts of simplicity play a surrogate role. Some of the desirable features of computational simplicity derive from its reliance on the learning system concept, which integrates key aspects of machine learning that are ignored by the classic concepts. Moreover, computational simplicity is directly associated with energy efficiency. In particular, the question of whether the maximum possibly achievable predictive accuracy should be attained, no matter the economic cost of the associated energy consumption pattern, is considered. (shrink)

Ockham’s razor is the characteristic scientific penchant for simpler, more testable, and more unified theories. Glymour’s early work on confirmation theory eloquently stressed the rhetorical plausibility of Ockham’s razor in scientific arguments. His subsequent, seminal research on causal discovery still concerns methods with a strong bias toward simpler causal models, and it also comes with a story about reliability—the methods are guaranteed to converge to true causal structure in the limit. However, there is a familiar gap between convergent reliability and (...) scientific rhetoric: convergence in the long run is compatible with any conclusion in the short run. For that reason, Carnap suggested that the proper sense of reliability for scientific inference should lie somewhere between short-run reliability and mere convergence in the limit. One natural such concept is straightest possible convergence to the truth, where straightness is explicated in terms of minimizing reversals of opinion and cycles of opinion prior to convergence. We close the gap between scientific rhetoric and scientific reliability by showing that Ockham’s razor is necessary for cycle-optimal convergence to the truth, and that patiently waiting for information to resolve conflicts among simplest hypotheses is necessary for reversal-optimal convergence to the truth. (shrink)

Synchronic norms of theory choice, a traditional concern in scientific methodology, restrict the theories one can choose in light of given information. Diachronic norms of theory change, as studied in belief revision, restrict how one should change one’s current beliefs in light of new information. Learning norms concern how best to arrive at true beliefs. In this paper, we undertake to forge some rigorous logical relations between the three topics. Concerning, we explicate inductive truth conduciveness in terms of optimally direct (...) convergence to the truth, where optimal directness is explicated in terms of reversals and cycles of opinion prior to convergence. Concerning, we explicate Ockham’s razor and related principles of choice in terms of the information topology of the empirical problem context and show that the principles are necessary for reversal or cycle optimal convergence to the truth. Concerning, we weaken the standard principles of agm belief revision theory in intuitive ways that are also necessary for reversal or cycle optimal convergence. Then we show that some of our weakened principles of change entail corresponding principles of choice, completing the triangle of relations between,, and. (shrink)

Synchronic norms of theory choice, a traditional concern in scientific methodology, restrict the theories one can choose in light of given information. Diachronic norms of theory change, as studied in belief revision, restrict how one should change one’s current beliefs in light of new information. Learning norms concern how best to arrive at true beliefs. In this paper, we undertake to forge some rigorous logical relations between the three topics. Concerning, we explicate inductive truth conduciveness in terms of optimally direct (...) convergence to the truth, where optimal directness is explicated in terms of reversals and cycles of opinion prior to convergence. Concerning, we explicate Ockham’s razor and related principles of choice in terms of the information topology of the empirical problem context and show that the principles are necessary for reversal or cycle optimal convergence to the truth. Concerning, we weaken the standard principles of agm belief revision theory in intuitive ways that are also necessary for reversal or cycle optimal convergence. Then we show that some of our weakened principles of change entail corresponding principles of choice, completing the triangle of relations between,, and. (shrink)

Over the past two decades, several consistent procedures have been designed to infer causal conclusions from observational data. We prove that if the true causal network might be an arbitrary, linear Gaussian network or a discrete Bayes network, then every unambiguous causal conclusion produced by a consistent method from non-experimental data is subject to reversal as the sample size increases any finite number of times. That result, called the causal flipping theorem, extends prior results to the effect that causal discovery (...) cannot be reliable on a given sample size. We argue that since repeated flipping of causal conclusions is unavoidable in principle for consistent methods, the best possible discovery methods are consistent methods that retract their earlier conclusions no more than necessary. A series of simulations of various methods across a wide range of sample sizes illustrates concretely both the theorem and the principle of comparing methods in terms of retractions. (shrink)

Machine learning is a scientific discipline that can be divided into two main branches: supervised machine learning and unsupervised machine learning. In this paper, we aim to show just how simplicity matters in unsupervised contexts. This is important because unsupervised machine learning algorithms have barely received any attention in philosophy. Yet, there is a direct link between simplicity and truth in unsupervised contexts that we do not find in their supervised counterparts. This has thus far evaded philosophical discussions on simplicity.