The nature of probability theory. The sample space. Elements of combinatorial analysis. Fluctuations in coin tossing and random walks. Combination of events. Conditional probability, stochastic independence. The binomial and the Poisson distributions. The Normal approximation to the binomial distribution. Unlimited sequences of Bernoulli trials. Random variables, expectation. Laws of large numbers. Integral valued variables, generating functions. Compound distributions. Branching processes. Recurrent events. Renewal theory. Random walk and ruin problems. Markov chains. Algebraic treatment of finite Markov chains. The simplest time-dependent stochastic (...) processes. Answer to problems. Index. (shrink)

The mathematical theory of communication.

In the last quarter of the nineteenth century, Ludwig Boltzmann explained how irreversible macroscopic laws, in particular the second law of thermodynamics, originate in the time-reversible laws of microscopic physics. Boltzmann’s analysis, the essence of which I shall review here, is basically correct. The most famous criticisms of Boltzmann’s later work on the subject have little merit. Most twentieth century innovations – such as the identification of the state of a physical system with a probability distribution on its phase space, (...) of its thermodynamic entropy with the Gibbs entropy of , and the invocation of the notions of ergodicity and mixing for the justification of the foundations of statistical mechanics – are thoroughly misguided. (shrink)

A paper on how to adapt your probabilisitc beliefs when learning a conditional.

This essay is, primarily, a discussion of four results about the principle of maximizing entropy (MAXENT) and its connections with Bayesian theory. Result 1 provides a restricted equivalence between the two: where the Bayesian model for MAXENT inference uses an "a priori" probability that is uniform, and where all MAXENT constraints are limited to 0-1 expectations for simple indicator-variables. The other three results report on an inability to extend the equivalence beyond these specialized constraints. Result 2 established a sensitivity of (...) MAXENT inference to the choice of the algebra of possibilities even though all empirical constraints imposed on the MAXENT solution are satisfied in each measure space considered. The resulting MAXENT distribution is not invariant over the choice of measure space. Thus, old and familiar problems with the Laplacian principle of Insufficient Reason also plague MAXENT theory. Result 3 builds upon the findings of Friedman and Shimony (1971; 1973) and demonstrates the absence of an exchangeable, Bayesian model for predictive MAXENT distributions when the MAXENT constraints are interpreted according to Jaynes's (1978) prescription for his (1963) Brandeis Dice problem. Lastly, Result 4 generalizes the Friedman and Shimony objection to cross-entropy (Kullback-information) shifts subject to a constraint of a new odds-ratio for two disjoint events. (shrink)

The technique of minimizing information (infomin) has been commonly employed as a general method for both choosing and updating a subjective probability function. We argue that, in a wide class of cases, the use of infomin methods fails to cohere with our standard conception of rational degrees of belief. We introduce the notion of a deceptive updating method and argue that non-deceptiveness is a necessary condition for rational coherence. Infomin has been criticized on the grounds that there are no higher (...) order probabilities that ‘support’ it, but the appeal to higher order probabilities is a substantial assumption that some might reject. Our elementary arguments from deceptiveness do not rely on this assumption. While deceptiveness implies lack of higher order support, the converse does not, in general, hold, which indicates that deceptiveness is a more objectionable property. We offer a new proof of the claim that infomin updating of any strictly-positive prior with respect to conditional-probability constraints is deceptive. In the case of expected-value constraints, infomin updating of the uniform prior is deceptive for some random variables but not for others. We establish both a necessary condition and a sufficient condition (which extends the scope of the phenomenon beyond cases previously considered) for deceptiveness in this setting. Along the way, we clarify the relation which obtains between the strong notion of higher order support, in which the higher order probability is defined over the full space of first order probabilities, and the apparently weaker notion, in which it is defined over some smaller parameter space. We show that under certain natural assumptions, the two are equivalent. Finally, we offer an interpretation of Jaynes, according to which his own appeal to infomin methods avoids the incoherencies discussed in this paper. (shrink)

Information theory and statistical mechanics II.

The strategy of updating credences by minimizing the relative entropy has been questioned by many authors, most strongly by means of the Judy Benjamin puzzle. I present a new analysis of Judy Benjamin–like forms of new information and defend the thesis that in general the rational posterior is indeterminate, meaning that a family of posterior credence functions rather than a single one is the rational response when that type of information becomes available. The proposed thesis extends naturally to all cases (...) in which new information is traditionally handled by minimizing the relative entropy. (shrink)

Many statistical problems, including some of the most important for physical applications, have long been regarded as underdetermined from the standpoint of a strict frequency definition of probability; yet they may appear wellposed or even overdetermined by the principles of maximum entropy and transformation groups. Furthermore, the distributions found by these methods turn out to have a definite frequency correspondence; the distribution obtained by invariance under a transformation group is by far the most likely to be observed experimentally, in the (...) sense that it requires by far the least “skill.” These properties are illustrated by analyzing the famous Bertrand paradox. On the viewpoint advocated here, Bertrand's problem turns out to be well posed after all, and the unique solution has been verified experimentally. We conclude that probability theory has a wider range of useful applications than would be supposed from the standpoint of the usual frequency definitions. (shrink)

Information theory and statistical mechanics.