Learning the structure of event sequences is a ubiquitous problem in cognition and particularly in language. One possible solution is to learn a probabilistic generative model of sequences that allows making predictions about upcoming events. Though appealing from a neurobiological standpoint, this approach is typically not pursued in connectionist modeling. Here, we investigated a sequential version of the restricted Boltzmann machine, a stochastic recurrent neural network that extracts high-order structure from sensory data through unsupervised generative learning and can encode contextual (...) information in the form of internal, distributed representations. We assessed whether this type of network can extract the orthographic structure of English monosyllables by learning a generative model of the letter sequences forming a word training corpus. We show that the network learned an accurate probabilistic model of English graphotactics, which can be used to make predictions about the letter following a given context as well as to autonomously generate high-quality pseudowords. The model was compared to an extended version of simple recurrent networks, augmented with a stochastic process that allows autonomous generation of sequences, and to non-connectionist probabilistic models. We conclude that sequential RBMs and stochastic simple recurrent networks are promising candidates for modeling cognition in the temporal domain. (shrink)

This paper introduces a special issue of Cognitive Science initiated on the 25th anniversary of the publication of Parallel Distributed Processing (PDP), a two-volume work that introduced the use of neural network models as vehicles for understanding cognition. The collection surveys the core commitments of the PDP framework, the key issues the framework has addressed, and the debates the framework has spawned, and presents viewpoints on the current status of these issues. The articles focus on both historical roots and contemporary (...) developments in learning, optimality theory, perception, memory, language, conceptual knowledge, cognitive control, and consciousness. Here we consider the approach more generally, reviewing the original motivations, the resulting framework, and the central tenets of the underlying theory. We then evaluate the impact of PDP both on the field at large and within specific subdomains of cognitive science and consider the current role of PDP models within the broader landscape of contemporary theoretical frameworks in cognitive science. Looking to the future, we consider the implications for cognitive science of the recent success of machine learning systems called “deep networks”—systems that build on key ideas presented in the PDP volumes. (shrink)

The cerebral cortex is a rich and diverse structure that is the basis of intelligent behavior. One of the deepest mysteries of the function of cortex is that neural processing times are only about one hundred times as fast as the fastest response times for complex behavior. At the very least, this would seem to indicate that the cortex does massive amounts of parallel computation.This paper explores the hypothesis that an important part of the cortex can be modeled as a (...) connectionist computer that is especially suited for parallel problem solving. The connectionist computer uses a special representation, termed value unit encoding, that represents small subsets of parameters in a way that allows parallel access to many different parameter values. This computer can be thought of as computing hierarchies of sensorimotor invariants. The neural substrate can be interpreted as a commitment to data structures and algorithms that compute invariants fast enough to explain the behavioral response times. A detailed consideration of this model has several implications for the underlying anatomy and physiology. (shrink)

This paper is a commentary on the target article by Dana H. Ballard, “Cortical connections and parallel processing: Structure and function”, in the same issue of the journal, pp. 67–120. -/- I raise some issues about the connectionist or neural-network implementation of information and information processing. Issues include the sharing of information by different parts of a connectionist/neural network, the copying of complex information from one place to another in a network, the possibility of connection weights not being synaptic weights, (...) and the possibility of fast communication within single neurons as complex systems in their own right. (shrink)

The paper is a commentary on the target article by Christine A. Skarda & Walter J. Freeman, “How brains make chaos in order to make sense of the world”, in the same issue of the journal, pp.161–195. -/- I confine my comments largely to some philosophical claims that Skarda & Freeman make and to the relationship of their model to connectionism. Some of the comments hinge on what symbols are and how they might sit in neural systems.

The appropriate methodology for psychological research depends on whether one is studying mental algorithms or their implementation. Mental algorithms are abstract specifications of the steps taken by procedures that run in the mind. Implementational issues concern the speed and reliability of these procedures. The algorithmic level can be explored only by studying across-task variation. This contrasts with psychology's dominant methodology of looking for within-task generalities, which is appropriate only for studying implementational issues.The implementation-algorithm distinction is related to a number of (...) other “levels” considered in cognitive science. Its realization in Anderson's ACT theory of cognition is discussed. Research at the algorithmic level is more promising because it is hard to make further fundamental scientific progress at the implementational level with the methodologies available. Protocol data, which are appropriate only for algorithm-level theories, provide a richer source than data at the implementational level. Research at the algorithmic level will also yield more insight into fundamental properties of human knowledge because it is the level at which significant learning transitions are defined.The best way to study the algorithmic level is to look for differential learning outcomes in pedagogical experiments that manipulate instructional experience. This provides control and prediction in realistically complex learning situations. The intelligent tutoring paradigm provides a particularly fruitful way to implement such experiments.The implications of this analysis for the issue of modularity of mind, the status of language, research on human/computer interaction, and connectionist models are also examined. (shrink)

Although the connectionist approach has lead to elegant solutions to a number of problems in cognitive science and artificial intelligence, its suitability for dealing with problems in knowledge representation and inference has often been questioned. This paper partly answers this criticism by demonstrating that effective solutions to certain problems in knowledge representation and limited inference can be found by adopting a connectionist approach. The paper presents a connectionist realization of semantic networks, that is, it describes how knowledge about concepts, their (...) properties, and the hierarchical relationship between them may be encoded as an interpreter‐free massively parallel network of simple processing elements that can solve an interesting class of inheritance and recognition problems extremely fast—in time proportional to the depth of the conceptual hierarchy. The connectionist realization is based on an evidential formulation that leads to principled solutions to the problems of exceptions and conflicting multiple inheritance situations during inheritance, and the best‐match or partial‐match computation during recognition. The paper also identifies constraints that must be satisfied by the conceptual structure in order to arrive at an efficient parallel realization. (shrink)

Previous neural network models of concept learning were mainly implemented with supervised learning schemes. However, studies of human conceptual memory have shown that concepts may be learned without a teacher who provides the category name to associate with exemplars. A modular neural network architecture that realizes concept acquisition through two functionally distinct operations, categorizing and naming, is proposed as an alternative. An unsupervised algorithm realizes the categorizing module by constructing representations of categories compatible with prototype theory. The naming module associates (...) category names to the output of the categorizing module in a supervised mode. In such a modular architecture, the interface between the modules can be conceived of as an “information relay” that encodes, constrains, and propagates important information. Five experiments were conducted to analyze the relationships among internal conceptual codes and simple conceptual and lexical development. The first two experiments show a prototype effect and illustrate some basic characteristics of the system. The third experiment presents a bottom‐up model of the narrowing down of children's early lexical categories that honors mutual exclusivity. The fourth experiment introduces top‐down constraints on conceptual coding. The fifth experiment exhibits how hierarchical relationships between concepts are learned by the architecture, and also demonstrates how a spectrum of conceptual expertise may gradually emerge as a consequence of experiencing more with certain categories than with others. (shrink)

This paper presents a general computational treatment of how mammals are able to deal with visual objects and environments. The model tries to cover the entire range from behavior and phenomenological experience to detailed neural encodings in crude but computationally plausible reductive steps. The problems addressed include perceptual constancies, eye movements and the stable visual world, object descriptions, perceptual generalizations, and the representation of extrapersonal space.The entire development is based on an action-oriented notion of perception. The observer is assumed to (...) be continuously sampling the ambient light for information of current value. The central problem of vision is taken to be categorizing and locating objects in the environment. The critical step in this process is the linking of visual information to symbolic object descriptions; this is calledindexing, from the analogy of identifying a book from index terms. The system must also identifysituationsand use this knowledge to guide movement and other actions in the environment. The treatment focuses on the different representations of information used in the visual system.The four representational frames capture information in the following forms: retinotopic, head-based, symbolic, and allocentric. The functional roles of the four frames, the communication among them, and their suggested neurophysiological realization constitute the core of the paper. The model is perforce crude, but appears to be consistent with all relevant findings. (shrink)

The pivotal problem of comorbidity research lies in the psychometric foundation it rests on, that is, latent variable theory, in which a mental disorder is viewed as a latent variable that causes a constellation of symptoms. From this perspective, comorbidity is a (bi)directional relationship between multiple latent variables. We argue that such a latent variable perspective encounters serious problems in the study of comorbidity, and offer a radically different conceptualization in terms of a network approach, where comorbidity is hypothesized to (...) arise from direct relations between symptoms of multiple disorders. We propose a method to visualize comorbidity networks and, based on an empirical network for major depression and generalized anxiety, we argue that this approach generates realistic hypotheses about pathways to comorbidity, overlapping symptoms, and diagnostic boundaries, that are not naturally accommodated by latent variable models: Some pathways to comorbidity through the symptom space are more likely than others; those pathways generally have the same direction (i.e., from symptoms of one disorder to symptoms of the other); overlapping symptoms play an important role in comorbidity; and boundaries between diagnostic categories are necessarily fuzzy. (shrink)

The majority of commentators agree on one thing: Our network approach might be the prime candidate for offering a new perspective on the origins of mental disorders. In our response, we elaborate on refinements (e.g., cognitive and genetic levels) and extensions (e.g., to Axis II disorders) of the network model, as well as discuss ways to test its validity.

The commentaries reflect three core themes that pertain not just to our theory, but to the enterprise of connectionist modeling more generally. The first concerns the relationship between a cognitive theory and an implemented computer model. Specifically, how does one determine, when a model departs from the theory it exemplifies, whether the departure is a useful simplification or a critical flaw? We argue that the answer to this question depends partially upon the model's intended function, and we suggest that connectionist (...) models have important functions beyond the commonly accepted goals of fitting data and making predictions. The second theme concerns perceived in-principle limitations of the connectionist approach to cognition, and the specific concerns these perceived limitations raise for our theory. We argue that the approach is not in fact limited in the ways our critics suggest. One common misconception, that connectionist models cannot address abstract or relational structure, is corrected through new simulations showing directly that such structure can be captured. The third theme concerns the relationship between parallel distributed processing (PDP) models and structured probabilistic approaches. In this case we argue that there the difference between the approaches is not merely one of levels. Our PDP approach differs from structured statistical approaches at all of Marr's levels, including the characterization of the goals of cognitive computations, and of the representations and algorithms used. (shrink)

In this prcis we focus on phenomena central to the reaction against similarity-based theories that arose in the 1980s and that subsequently motivated the approach to semantic knowledge. Specifically, we consider (1) how concepts differentiate in early development, (2) why some groupings of items seem to form or coherent categories while others do not, (3) why different properties seem central or important to different concepts, (4) why children and adults sometimes attest to beliefs that seem to contradict their direct experience, (...) (5) how concepts reorganize between the ages of 4 and 10, and (6) the relationship between causal knowledge and semantic knowledge. The explanations our theory offers for these phenomena are illustrated with reference to a simple feed-forward connectionist model. The relationships between this simple model, the broader theory, and more general issues in cognitive science are discussed. (shrink)

This paper critically examines the claim that parallel distributed processing (PDP) networks are autonomous learning systems. A PDP model of a simple distributed associative memory is considered. It is shown that the 'generic' PDP architecture cannot implement the computations required by this memory system without the aid of external control. In other words, the model is not autonomous. Two specific problems are highlighted: (i) simultaneous learning and recall are not permitted to occur as would be required of an autonomous system; (...) (ii) connections between processing units cannot simultaneously represent current and previous network activation as would be required if learning is to occur. Similar problems exist for more sophisticated networks constructed from the generic PDP architecture. We argue that this is because these models are not adequately constrained by the properties of the functional architecture assumed by PDP modelers. It is also argued that without such constraints, PDP researchers cannot claim to have developed an architecture radically different from that proposed by the Classical approach in cognitive science. (shrink)

A set of hypotheses is formulated for a connectionist approach to cognitive modeling. These hypotheses are shown to be incompatible with the hypotheses underlying traditional cognitive models. The connectionist models considered are massively parallel numerical computational systems that are a kind of continuous dynamical system. The numerical variables in the system correspond semantically to fine-grained features below the level of the concepts consciously used to describe the task domain. The level of analysis is intermediate between those of symbolic cognitive models (...) and neural models. The explanations of behavior provided are like those traditional in the physical sciences, unlike the explanations provided by symbolic models. (shrink)

This article responds to two unresolved and crucial problems of cognitive science: (1) What is actually accomplished by functions of the nervous system that we ordinarily describe in the intentional idiom? and (2) What makes the information processing involved in these functions semantic? It is argued that, contrary to the assumptions of many cognitive theorists, the computational approach does not provide coherent answers to these problems, and that a more promising start would be to fall back on mathematical communication theory (...) and, with the help of evolutionary biology and neurophysiology, to attempt a characterization of the adaptive processes involved in visual perception. Visual representations are explained as patterns of cortical activity that are enabled to focus on objects in the changing visual environment by constantly adjusting to maintain levels of mutual information between pattern and object that are adequate for continuing perceptual control. In these terms, the answer proposed to (1) is that the intentional functions of vision are those involved in the establishment and maintenance of such representations, and to (2) that semantic features are added to the information processes of vision with the focus on objects that these representations accomplish. The article concludes with proposals for extending this account of intentionality to the higher domains of conceptualization and reason, and with speculation about how semantic information-processing might be achieved in mechanical systems. (shrink)

Computational complexity is a discipline of computer science and mathematics which classifies computational problems depending on their inherent difficulty, i.e. categorizes algorithms according to their performance, and relates these classes to each other. P problems are a class of computational problems that can be solved in polynomial time using a deterministic Turing machine while solutions to NP problems can be verified in polynomial time, but we still do not know whether they can be solved in polynomial time as well. A (...) solution for the so-called NP-complete problems will also be a solution for any other such problems. Its artificial-intelligence analogue is the class of AI-complete problems, for which a complete mathematical formalization still does not exist. In this chapter we will focus on analysing computational classes to better understand possible formalizations of AI-complete problems, and to see whether a universal algorithm, such as a Turing test, could exist for all AI-complete problems. In order to better observe how modern computer science tries to deal with computational complexity issues, we present several different deep-learning strategies involving optimization methods to see that the inability to exactly solve a problem from a higher order computational class does not mean there is not a satisfactory solution using state-of-the-art machine-learning techniques. Such methods are compared to philosophical issues and psychological research regarding human abilities of solving analogous NP-complete problems, to fortify the claim that we do not need to have an exact and correct way of solving AI-complete problems to nevertheless possibly achieve the notion of strong AI. (shrink)

Complex systems are to be seen as typically having multiple levels of organization. For instance, in the behavioural and cognitive sciences, there has been a long lasting trend, promoted by the seminal work of David Marr, putting focus on three distinct levels of analysis: the computational level, accounting for the What and Why issues, the algorithmic and the implementational levels specifying the How problem. However, the tremendous developments in neuroscience knowledge about processes at different scales of organization together with the (...) complexity of today cognitive theories suggest that there will hardly be only three levels of explanation. Instead, there will be many different degrees of commitments corresponding to the different granularities—from high-level (behavioural) models to low-level (neural and molecular) models of the cognitive research program. For instance, Bayesian approaches, that are usually advocated for formalizing Marr’s computational level and rational behaviour, have even been adopted to model synaptic plasticity and axon guidance by molecular gradients. As a result, we can consider the behavioural scientist as dealing with models at a multiplicity of levels. The purpose of this Research Topic in Frontiers in Theoretical and Philosophical Psychology is to promote an approach to the role of the levels and explanation and models which is of interest for cognitive scientists, neuroscientists, psychologists, behavioural scientists, and philosophers of science. (shrink)