This paper introduces a special issue of Cognitive Science initiated on the 25th anniversary of the publication of Parallel Distributed Processing (PDP), a two-volume work that introduced the use of neural network models as vehicles for understanding cognition. The collection surveys the core commitments of the PDP framework, the key issues the framework has addressed, and the debates the framework has spawned, and presents viewpoints on the current status of these issues. The articles focus on both historical roots and contemporary (...) developments in learning, optimality theory, perception, memory, language, conceptual knowledge, cognitive control, and consciousness. Here we consider the approach more generally, reviewing the original motivations, the resulting framework, and the central tenets of the underlying theory. We then evaluate the impact of PDP both on the field at large and within specific subdomains of cognitive science and consider the current role of PDP models within the broader landscape of contemporary theoretical frameworks in cognitive science. Looking to the future, we consider the implications for cognitive science of the recent success of machine learning systems called “deep networks”—systems that build on key ideas presented in the PDP volumes. (shrink)

A number of ways of taxonomizing human learning have been proposed. We examine the evidence for one such proposal, namely, that there exist independent explicit and implicit learning systems. This combines two further distinctions, between learning that takes place with versus without concurrent awareness, and between learning that involves the encoding of instances versus the induction of abstract rules or hypotheses. Implicit learning is assumed to involve unconscious rule learning. We examine the evidence for implicit learning derived from subliminal learning, (...) conditioning, artificial grammar learning, instrumental learning, and reaction times in sequence learning. We conclude that unconscious learning has not been satisfactorily established in any of these areas. The assumption that learning in some of these tasks is predominantly based on rule abstraction is questionable. When subjects cannot report the “implicitly learned” rules that govern stimulus selection, this is often because their knowledge consists of instances or fragments of the training stimuli rather than rules. In contrast to the distinction between conscious and unconscious learning, the distinction between instance and rule learning is a sound and meaningful way of taxonomizing human learning. We discuss various computational models of these two forms of learning. (shrink)

This article addresses the ability of Parallel Distributed Processing (PDP) networks to generate stagewise cognitive development in accordance with Piaget's theory of cognitive epigenesis. We carried out a replication study of the simulation experiments by McClelland (1989) and McClelland and Jenkins (1991) in which a PDP network learns to solve balance scale problems. In objective tests motivated from catastrophe theory, a mathematical theory of transitions in epigenetical systems, no evidence for stage transitions in network performance was found. It is concluded (...) that PDP networks lack the ability to recover the positive outcomes of analogous catastrophe analyses of real cognitive developmental data. In an attempt to further characterize the learning behaviour of PDP networks, we carried out a second simulation study using the discrimination‐shift paradigm. The results thus obtained indicate that PDP learning is compatible with the learning of stimulus‐response relationships, not with the acquisition of mediating rules such as conceived in (neo‐)Piagetian theory. In closing, we speculate about the feasibility of simulating stagewise development with alternative network architectures. (shrink)

A central controversy in cognitive science concerns the roles of rules versus similarity. To gain some leverage on this problem, we propose that rule‐ versus similarity‐based processes can be characterized as extremes in a multidimensional space that is composed of at least two dimensions: the number of features (Pothos, 2005) and the physical presence of features. The transition of similarity‐ to rule‐based processing is conceptualized as a transition in this space. To illustrate this, we show how a neural network model (...) uses input features (and in this sense produces similarity‐based responses) when it has a low learning rate or in the early phases of training, but it switches to using self‐generated, more abstract features (and in this sense produces rule‐based responses) when it has a higher learning rate or is in the later phases of training. Relations with categorization and the psychology of learning are pointed out. (shrink)

The commentaries reflect three core themes that pertain not just to our theory, but to the enterprise of connectionist modeling more generally. The first concerns the relationship between a cognitive theory and an implemented computer model. Specifically, how does one determine, when a model departs from the theory it exemplifies, whether the departure is a useful simplification or a critical flaw? We argue that the answer to this question depends partially upon the model's intended function, and we suggest that connectionist (...) models have important functions beyond the commonly accepted goals of fitting data and making predictions. The second theme concerns perceived in-principle limitations of the connectionist approach to cognition, and the specific concerns these perceived limitations raise for our theory. We argue that the approach is not in fact limited in the ways our critics suggest. One common misconception, that connectionist models cannot address abstract or relational structure, is corrected through new simulations showing directly that such structure can be captured. The third theme concerns the relationship between parallel distributed processing (PDP) models and structured probabilistic approaches. In this case we argue that there the difference between the approaches is not merely one of levels. Our PDP approach differs from structured statistical approaches at all of Marr's levels, including the characterization of the goals of cognitive computations, and of the representations and algorithms used. (shrink)

In this prcis we focus on phenomena central to the reaction against similarity-based theories that arose in the 1980s and that subsequently motivated the approach to semantic knowledge. Specifically, we consider (1) how concepts differentiate in early development, (2) why some groupings of items seem to form or coherent categories while others do not, (3) why different properties seem central or important to different concepts, (4) why children and adults sometimes attest to beliefs that seem to contradict their direct experience, (...) (5) how concepts reorganize between the ages of 4 and 10, and (6) the relationship between causal knowledge and semantic knowledge. The explanations our theory offers for these phenomena are illustrated with reference to a simple feed-forward connectionist model. The relationships between this simple model, the broader theory, and more general issues in cognitive science are discussed. (shrink)

A number of ways of taxonomizing human learning have been proposed. We examine the evidence for one such proposal, namely, that there exist independent explicit and implicit learning systems. This combines two further distinctions, (1) between learning that takes place with versus without concurrent awareness, and (2) between learning that involves the encoding of instances (or fragments) versus the induction of abstract rules or hypotheses. Implicit learning is assumed to involve unconscious rule learning. We examine the evidence for implicit learning (...) derived from subliminal learning, conditioning, artificial grammar learning, instrumental learning, and reaction times in sequence learning. We conclude that unconscious learning has not been satisfactorily established in any of these areas. The assumption that learning in some of these tasks (e.g., artificial grammar learning) is predominantly based on rule abstraction is questionable. When subjects cannot report the rules that govern stimulus selection, this is often because their knowledge consists of instances or fragments of the training stimuli rather than rules. In contrast to the distinction between conscious and unconscious learning, the distinction between instance and rule learning is a sound and meaningful way of taxonomizing human learning. We discuss various computational models of these two forms of learning. (shrink)

Fodor's and Pylyshyn's stand on systematicity in thought and language has been debated and criticized. Van Gelder and Niklasson, among others, have argued that Fodor and Pylyshyn offer no precise definition of systematicity. However, our concern here is with a learning based formulation of that concept. In particular, Hadley has proposed that a network exhibits strong semantic systematicity when, as a result of training, it can assign appropriate meaning representations to novel sentences (both simple and embedded) which contain words in (...) syntactic positions they did not occupy during training. The experience of researchers indicates that strong systematicity in any form is difficult to achieve in connectionist systems.Herein we describe a network which displays strong semantic systematicity in response to Hebbian, connectionist training. During training, two-thirds of all nouns are presented only in a single syntactic position (either as grammatical subject or object). Yet, during testing, the network correctly interprets thousands of sentences containing those nouns in novel positions. In addition, the network generalizes to novel levels of embedding. Successful training requires a, corpus of about 1000 sentences, and network training is quite rapid. The architecture and learning algorithms are purely connectionist, but ‘classical’ insights are discernible in one respect, viz, that complex semantic representations spatially contain their semantic constituents. However, in other important respects, the architecture is distinctly non-classical. (shrink)

A computational model of inference during story comprehension is presented, in which story situations are represented distributively as points in a high‐dimensional “situation‐state space.” This state space organizes itself on the basis of a constructed microworld description. From the same description, causal/temporal world knowledge is extracted. The distributed representation of story situations is more flexible than Golden and Rumelhart's [Discourse Proc 16 (1993) 203] localist representation.A story taking place in the microworld corresponds to a trajectory through situation‐state space. During the (...) inference process, world knowledge is applied to the story trajectory. This results in an adjusted trajectory, reflecting the inference of propositions that are likely to be the case. Although inferences do not result from a search for coherence, they do cause story coherence to increase. The results of simulations correspond to empirical data concerning inference, reading time, and depth of processing.An extension of the model for simulating story retention shows how coherence is preserved during retention without controlling the retention process. Simulation results correspond to empirical data concerning story recall and intrusion. (shrink)

Although for many years a sharp distinction has been made in language research between rules and words—with primary interest on rules—this distinction is now blurred in many theories. If anything, the focus of attention has shifted in recent years in favor of words. Results from many different areas of language research suggest that the lexicon is representationally rich, that it is the source of much productive behavior, and that lexically specific information plays a critical and early role in the interpretation (...) of grammatical structure. But how much information can or should be placed in the lexicon? This is the question I address here. I review a set of studies whose results indicate that event knowledge plays a significant role in early stages of sentence processing and structural analysis. This poses a conundrum for traditional views of the lexicon. Either the lexicon must be expanded to include factors that do not plausibly seem to belong there; or else virtually all information about word meaning is removed, leaving the lexicon impoverished. I suggest a third alternative, which provides a way to account for lexical knowledge without a mental lexicon. (shrink)

People tend to perceive ongoing continuous activity as series of discrete events. This partitioning of continuous activity may occur, in part, because events correspond to dynamic patterns that have recurred across different contexts. Recurring patterns may lead to reliable sequential dependencies in observers' experiences, which then can be used to guide perception. The current set of simulations investigated whether this statistical structure within events can be used 1) to develop stable internal representations that facilitate perception and 2) to learn when (...) to update such representations in a self-organizing manner. These simulations demonstrate that experience with recurring patterns enables a system to accurately predict upcoming stimuli within an event, to identify boundaries between such events based on transient increases in prediction error, and to use such boundaries to improve prediction about subsequent activities. (shrink)

I propose a new approach to the constitutive problem of psychology ‘what is mind?’ The first section introduces modifications of the received scope, methodology, and evaluation criteria of unified theories of cognition in accordance with the requirements of evolutionary compatibility and of a mature science. The second section outlines the proposed theory. Its first part provides empirically verifiable conditions delineating the class of meaningful neural formations and modifies accordingly the traditional conceptions of meaning, concept and thinking. This analysis is part (...) of a theory of communication in terms of inter-level systems of primitives that proposes the communication-understanding principle as a psychological invariance. It unifies a substantial amount of research by systematizing the notions of meaning, thinking, concept, belief, communication, and understanding and leads to a minimum vocabulary for this core system of mental phenomena. Its second part argues that written human language is the key characteristic of the artificially natural human mind. Overall, the theory both supports Darwin’s continuity hypothesis and proposes that the mental gap is within our own species. (shrink)

Leech et al.'s analysis adds to an emerging consensus of the role of priming in analogy-making. However, their model cannot scale up to adult-level performance because not all relations can be cast as functions. One-size-fits-all accounts cannot capture the richness of analogy. Proportional analogies and transitive inferences can be made by nonstructural mechanisms. Therefore, these tasks do not generalize to tasks that require structure mapping.

The target article demonstrates how neurocognitive modellers should not be intimidated by challenges such as Jackendoff's and should explore neurally plausible implementations of linguistic constructs. The next step is to take seriously insights offlered by neuroscience, including the robustness allowed by analogue computation with distributed representations and the power of attractor dynamics in turning analogue into nearly discrete operations.

Despite its strengths, Leech et al.'s model fails to address the important benefits that derive from self-explanation and task feedback in analogical reasoning development. These components encourage explicit, self-reflective processes that do not necessarily link to knowledge accretion. We wonder, therefore, what mechanisms can be included within a connectionist framework to model self-reflective involvement and its beneficial consequences.

Theories concerning the structure, or format, of mental representation should (1) be formulated in mechanistic, rather than metaphorical terms; (2) do justice to several philosophical intuitions about mental representation; and (3) explain the human capacity to predict the consequences of worldly alterations (i.e., to think before we act). The hypothesis that thinking involves the application of syntax-sensitive inference rules to syntactically structured mental representations has been said to satisfy all three conditions. An alternative hypothesis is that thinking requires the construction (...) and manipulation of the cognitive equivalent of scale models. A reading of this hypothesis is provided that satisfies condition (1) and which, even though it may not fully satisfy condition (2), turns out (in light of the frame problem) to be the only known way to satisfy condition (3). (shrink)

Along with the increasing popularity of connectionist language models has come a number of provocative suggestions about the challenge these models present to Chomsky's arguments for nativism. The aim of this paper is to assess these claims. We begin by reconstructing Chomsky's argument from the poverty of the stimulus and arguing that it is best understood as three related arguments, with increasingly strong conclusions. Next, we provide a brief introduction to connectionism and give a quick survey of recent efforts to (...) develop networks that model various aspects of human linguistic behavior. Finally, we explore the implications of this research for Chomsky's arguments. Our claim is that the relation between connectionism and Chomsky's views on innate knowledge is more complicated than many have assumed, and that even if these models enjoy considerable success the threat they pose for linguistic nativism is small. (shrink)

In their provocative 1988 paper, Fodor and Pylyshyn issued a formidable challenge to connectionists, i.e. to provide a non‐classical explanation of the empirical phenomenon of systematicity in cognitive agents. Since the appearance of F&P's challenge, a number of connectionist systems have emerged which prima facie meet this challenge. However, Fodor and McLaughlin (1990) advance an argument, based upon a general principle of nomological necessity, to show that one of these systems (Smolensky's) could not satisfy the Fodor‐Pylyshyn challenge. Yet, if Fodor (...) and McLaughlin's analysis is correct, it is doubtful whether any existing connectionist system would fare better than Smolensky's. In the view of Fodor and McLaughlin, humans and classical architectures display systematicity as a matter of nomological necessity (necessity by virtue of natural law), but connectionist architectures do not. However, I argue that the Fodor‐Pylyshyn‐McLaughlin appeal to nomological necessity is untenable. There is a sense in which neither classical nor connectionist architectures possess nomological (or‘nomic’) necessity. However, the sense in which classical architectures do possess nomic necessity applies equally well to at least some connectionist architectures. Representational constituents can have causal efficacy within both classical and connectionist architectures. (shrink)

At present, the prevailing Connectionist methodology forrepresenting rules is toimplicitly embody rules in neurally-wired networks. That is, the methodology adopts the stance that rules must either be hard-wired or trained into neural structures, rather than represented via explicit symbolic structures. Even recent attempts to implementproduction systems within connectionist networks have assumed that condition-action rules (or rule schema) are to be embodied in thestructure of individual networks. Such networks must be grown or trained over a significant span of time. However, arguments (...) are presented herein that humanssometimes follow rules which arevery rapidly assignedexplicit internal representations, and that humans possessgeneral mechanisms capable of interpreting and following such rules. In particular, arguments are presented that thespeed with which humans are able to follow rules ofnovel structure demonstrates the existence of general-purpose rule following mechanisms. It is further argued that the existence of general-purpose rule following mechanisms strongly indicates that explicit rule following is not anisolated phenomenon, but may well be a common and important aspect of cognition. The relationship of the foregoing conclusions to Smolensky''s view of explicit rule following is also explored. The arguments presented here are pragmatic in nature, and are contrasted with thekind of arguments developed by Fodor and Pylyshyn in their recent, influential paper. (shrink)

A computational model of inference during story comprehension is presented, in which story situations are represented distributively as points in a high‐dimensional “situation‐state space.” This state space organizes itself on the basis of a constructed microworld description. From the same description, causal/temporal world knowledge is extracted. The distributed representation of story situations is more flexible than Golden and Rumelhart's [Discourse Proc 16 (1993) 203] localist representation.A story taking place in the microworld corresponds to a trajectory through situation‐state space. During the (...) inference process, world knowledge is applied to the story trajectory. This results in an adjusted trajectory, reflecting the inference of propositions that are likely to be the case. Although inferences do not result from a search for coherence, they do cause story coherence to increase. The results of simulations correspond to empirical data concerning inference, reading time, and depth of processing.An extension of the model for simulating story retention shows how coherence is preserved during retention without controlling the retention process. Simulation results correspond to empirical data concerning story recall and intrusion. (shrink)

A distributed neural network model called SPEC for processing sentences with recursive relative clauses is described. The model is based on separating the tasks of segmenting the input word sequence into clauses, forming the case‐role representations, and keeping track of the recursive embeddings into different modules. The system needs to be trained only with the basic sentence constructs, and it generalizes not only to new instances of familiar relative clause structures but to novel structures as well. SPEC exhibits plausible memory (...) degradation as the depth of the center embeddings increases, its memory is primed by earlier constituents, and its performance is aided by semantic constraints between the constituents. The ability to process structure is largely due to a central executive network that monitors and controls the execution of the entire system. This way, in contrast to earlier subsymbolic systems, parsing is modeled as a controlled high‐level process rather than one based on automatic reflex responses. (shrink)

Connectionist attention to variables has been too restricted in two ways. First, it has not exploited certain ways of doing without variables in the symbolic arena. One variable-avoidance method, that of logical combinators, is particularly well established there. Secondly, the attention has been largely restricted to variables in long-term rules embodied in connection weight patterns. However, short-lived bodies of information, such as sentence interpretations or inference products, may involve quantification. Therefore short-lived activation patterns may need to achieve the effect of (...) variables. The paper is mainly a theoretical analysis of some benefits and drawbacks of using logical combinators to avoid variables in short-lived connectionist encodings without loss of expressive power. The paper also includes a brief survey of some possible methods for avoiding variables other than by using combinators. (shrink)

Many studies of language, whether in philosophy, linguistics, or psychology, have focused on highly developed human languages. In their highly developed forms, such as are employed in scientific discourse, languages have a unique set of properties that have been the focus of much attention. For example, descriptive sentences in a language have the property of being "true" or "false," and words of a language have senses and referents. Sentences in a language are structured in accord with complex syntactic rules. Theorists (...) focusing on language are naturally led to ask questions such as what constitutes the meanings of words and sentences and how are the principles of syntax encoded in the heads of language users. While there is an important function for inquiries into the highly developed forms of these cultural products (Abrahamsen, 1987), such a focus can be quite misleading when we want to explain how these products have arisen or the human capacity to use language. The problem is that focusing on its most developed forms makes linguistic ability seem to be a _sui generis_ phenomenon, not related to, and hence not explicable in terms of other cognitive capacities. Chomsky's (1980) postulation of a specific language module equipped with specialized resources needed to process language and possessed only by hum ans is not a surprising result. (shrink)

The reemergence of connectionism² has profoundly altered the philosophy of mind. Paul Churchland has argued that it should equally transform the philosophy of science. He proposes that connectionism offers radical and useful new ways of understanding theories and explanations.

Much of traditional AI exemplifies the explicit representation paradigm, and during the late 1980''s a heated debate arose between the classical and connectionist camps as to whether beliefs and rules receive an explicit or implicit representation in human cognition. In a recent paper, Kirsh (1990) questions the coherence of the fundamental distinction underlying this debate. He argues that our basic intuitions concerning explicit and implicit representations are not only confused but inconsistent. Ultimately, Kirsh proposes a new formulation of the distinction, (...) based upon the criterion ofconstant time processing.The present paper examines Kirsh''s claims. It is argued that Kirsh fails to demonstrate that our usage of explicit and implicit is seriously confused or inconsistent. Furthermore, it is argued that Kirsh''s new formulation of the explicit-implicit distinction is excessively stringent, in that it banishes virtually all sentences of natural language from the realm of explicit representation. By contrast, the present paper proposes definitions for explicit and implicit which preserve most of our strong intuitions concerning straightforward uses of these terms. It is also argued that the distinction delineated here sustains the meaningfulness of the abovementioned debate between classicists and connectionists. (shrink)