{"title": "Learning Sequential Structure in Simple Recurrent Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 643, "page_last": 652, "abstract": "", "full_text": "643 \n\nLEARNING SEQUENTIAL STRUCTURE \nIN SIMPLE RECURRENT NETWORKS \n\nDavid Servan-Schreiber. Axel Cleeremans. and James L. McClelland \n\nDeparttnents of Computer Science and Psycholgy \n\nCarnegie Mellon University \n\nPittsburgh, PA 15213 \n\nABSTRACT \n\nWe explore a network architecture introduced by Elman (1988) for \npredicting successive elements of a sequence. The network uses the \npattern of activation over a set of hidden units from time-step t-l, \ntogether with element t, to predict element t+ 1. When the network is \ntrained with strings from a particular finite-state grammar, it can learn \nto be a perfect finite-state recognizer for the grammar. Cluster analyses \nof the hidden-layer patterns of activation showed that they encode \nprediction-relevant information about the entire path traversed through \nthe network. We illustrate the phases of learning with cluster analyses \nperformed at different points during training. \n\nSeveral connectionist architectures that are explicitly constrained to capture \nsequential infonnation have been developed. Examples are Time Delay \nNetworks (e.g. Sejnowski & Rosenberg. 1986) -- also called 'moving \nwindow' paradigms -- or algorithms such as back-propagation in time \n(Rumelhart. Hinton & Williams. 1986), Such architectures use explicit \nrepresentations of several consecutive events. if not of the entire history of \npast inputs. Recently. Elman (1988) has introduced a simple recurrent \nnetwork (SRN) that has the potential to master an infinite corpus of \nsequences with the limited means of a learning procedure that is completely \nlocal in time (see Figure I.). \n\nCONIlSrr UNI'l'S \n\nFigure 1. The simple recurrent network (Elman, 1988) \n\n\f644 \n\nServan-Schreiber, Cleeremans and McClelland \n\nthe pattern of activation on the hidden units at time t-I. \nIn the SRN. \ntogether with the new input pattern. is allowed to influence the pattern of \nactivation at time t . This is achieved by copying the pattern of activation on \nthe hidden layer at time t\u00b7I to a set of input units -- called the 'context units' \n-- at time t. The forward connections in the network are subject to training \nvia back-propagation. but there is no backpropagation through time. \n\nIn this paper. we show that the SRN can learn to mimic closely a flnite state \nautomaton. both in its behavior and in its state representations. In particular. \nwe show that it can learn to process an infinite corpus of strings based on \nexperience with afinite set of training exemplars. We then describe the \nphases through which the appropriate internal representations are discovered \nduring training. \n\nMASTERING A FINITE STATE GRAMMAR \n\nIn our first experiment. we asked whether the network could learn the \ncontingencies implied by a small flnite state grammar (see Figure 2). The \nnetwork was presented with strings derived from this grammar. and was \nrequired to try to predict the next letter at every step. These predictions are \ncontext dependent since each letter appears twice in the grammar and is \nfollowed in each case by different successors. \n\nA single unit on the input layer represented a given letter (six input units in \ntotal; flve for the letters and one for a begin symbol 'B'). Similar local \nrepresentations were used on the output layer (with the 'begin' symbol \nbeing replaced by an end symbol 'E'). There were three hidden units. \n\n.tart -~ \n\ns \n\nT \n\nFigure 2. The small fmite-state grammar (Reber. 1967) \n\nTraining. On each of 60.000 training trials. a string was generated from \nthe grammar. starting with 'B'. Successive arcs were selected randomly \nfrom the 2 possible continuations with a probability of 0.5. Each letter was \n\n\fLearning Sequential Structure in Simple Recurrent Networks \n\n645 \n\nthen presented sequentially to the network. The activations of the context \nunits were reset to 0.5 at the beginning of each string. After each letter, the \nerror between the network's prediction and the actual successor specified \nby the string was computed and back-propagated. The 60,000 randomly \ngenerated strings ranged from 3 to 30 letters (mean: 7; sd: 3.3). \n\nPerformance. Three tests were conducted. First, we examined the \nnetwork's predictions on a set of 70,000 random strings. During this test, \nthe network is first presented with the start signal, and one of the five letters \nor E is then selected at random as a successor. If that letter is predicted by \nthe network as a legal successor (Le, activation is above 0.3 for the \ncorresponding unit), it is then presented to the input layer on the next time \nstep, and another letter is drawn at random as its successor. This procedure \nis repeated as long as each letter is predicted as a legal successor until the \nend signal is selected as the next letter. The procedure is interrupted as soon \nas the actual successor generated by the random procedure is not predicted \nby the network, and the string of letters is then considered 'rejected'. A \nstring is considered 'accepted' if all its letters have been predicted as \npossible continuations up to, and including, the end signal. Of the 70,000 \nrandom strings, 0.3 % were grammatical, and 99.7 % were ungrammatical. \nThe network performed flawlessly, accepting all the grammatical strings and \nrejecting all the others. In a second test, we presented the network with \n20,000 generated at random from the grammar, i.e, all these strings were \ngrammatical. Using the same criterion as above, all of these strings were \ncorrectly 'accepted'. Finally, we constructed a set of very long grammatical \nstrings -- more than 100 letters long -- and verified that at each step the \nnetwork correcdy predicted all the possible successors (activations above \n0.3) and none of the other letters in the grammar. \n\nAnalysis of internal representations. What kind of internal representations \nhave developed over the set of hidden units that allow the network to \nassociate the proper predictions to intrinsically ambiguous letters? One way \nto answer this question is to record the hidden units' activation patterns \ngenerated in response to the presentation of individual letters in different \ncontexts. These activation vectors can then be used as input to a cluster \nanalysis program. Figure 3.A. shows the results of such an analysis \nconducted on a small random set of grammatical strings. The patterns of \nactivation are grouped according to the nodes of the grammar: all the \npatterns that are used to predict the successors of a given node are grouped \ntogether independently of the current letter. This observation sheds some \nlight on the behavior of the network: at each point in a sequence, the pattern \nof activation stored over the context units provides information about the \ncurrent node in the grammar. Together with information about the current \nletter (represented on the input layer), this contextual information is used to \nproduce a new pattern of activation over the hidden layer, that uniquely \nspecifies the next node. In that sense, the network closely approximates the \nfinite-state automaton that would encode the grammar from which the \ntraining exemplars were derived. However, a closer look at the cluster \nanalysis reveals that within a cluster corresponding to a particular node, \npatterns are further divided according to the path traversed before the node \nis reached. For example, looking at the bottom cluster -- node #5 -- patterns \nproduced by a 'VV', 'PS', 'XS' or 'SXS' ending are grouped separately: \n\n\f646 \n\nServan-Schreiber, Cleeremans and McClelland \n\nI \n\nI \n\nI \n\n'~'--------~----~------------\n\nI \n\n.. \n! \n\n\u2022 \u2022 Ie \n\n\u2022 \u2022 Ie \n\n\u2022 \u2022 ... \n\nI \n\nFigure 3. A. Hieruchical cluster analysis of the hidden unit activation patternS after \n\n60.000 presentations of Slrings generated at random from the finite-Slate grammar. \n\nB. Cluster analysis of the H.U. activaDon pauans following 2000 epochs of \n\n1raining 011 a set of 22 strings with a maximum length of eightlettets. \n\n\fLearning Sequential Structure in Simple Recurrent Networks \n\n64 7 \n\nthey are more similar to each other than to the abstract prototype of node #5. \nThis tendency to preserve information about the path is not a characteristic \nof traditional finite-state automata. \n\nENCODING PATH INFORMATION \n\nIn a different set of experiments, we asked whether the SRN could learn to \nuse the infonnation about the path that is encoded in the hidden units' \npatterns of activation. In one of these experiments, we tested whether the \nnetwork could master length constraints. When strings generated from the \nsmall finite-state grammar may only have a maximum of 8 letters, the \nprediction following the presentation of the same letter in position number \nsix or seven may be different. For example, following the sequence \n'TSSSXXV', 'V' is the seventh letter and only another 'V' would be a legal \nsuccessor. In contrast, following the sequence 'TSSXXV', both 'V' and \n'P' are legal successors. A network with 15 hidden units was trained on a \nsmall set of length-limited (max. 8 letters) grammatical strings. It was able \nto use the small activation differences present over the context units - and \ndue to the slightly different sequences presented - to master contingencies \nsuch as those illustrated above (see table 1). \n\nTable 1. Activation of each output unit following the presentation \n\nof 'Y' as the 6th or 7th letter in the string \n\ntssxxV \ntsssxxV \n\nT \n0.0 \n0.0 \n\nS \n0.0 \n0.0 \n\nX \n\nP \n0.54 0.0 \n0.02 0.0 \n\nV \nE \n0.48 0.0 \n0.97 0.0 \n\nA cluster analysis of all the patterns of activation on the hidden layer \ngenerated by each letter in each sequence demonstrates how the influence of \nthe path is reflected in these patterns (see figure 3.B.)*. We labeled the arcs \naccording to the letter being presented (the 'current letter') and its position in \nthe grammar defined by Reber. Thus 'VI' refers to the f11'st 'V' in the \ngrammar and 'V2' to the second 'V' which immediately precedes the end of \nthe string. 'Early' and 'Late' refer to whether the letter occurred early or late \nin the sequence (for example in 'PT . .' 'T2' occurs early; in 'PVPXT . .' it \noccurs late). Finally, in the left margin we indicated what predictions the \ncorresponding patterns yield on the output layer (e.g, the hidden unit pattern \ngenerated by 'BEGIN' predicts 'T' or 'P'). \n\nFrom the figure, it can be seen that the patterns are grouped according to \nthree distinct principles: (1) according to similar predictions, (2) according \nto similar letters presented on the input units, and (3) according to similar \npaths. These factors do not necessarily overlap since several occurrences of \nthe same letter in a sequence usually implies different predictions and since \nsimilar paths also lead to different predictions depending on the current \nletter. For example, the top cluster in the figure corresponds to all \noccurrences of the letter 'V' and is further subdivided among 'V I' and 'V2\" \n\n* Information about the leaves of the cluster analyses in this and the remaining figures is \navailable in Servan-Schreiber. Cleeremans and McCleUand (1988). \n\n\f648 \n\nServan-Schreiber, Cleeremans and McClelland \n\nThe 'V l' cluster is itself further divided between groups where 'V l' occurs \nearly in the sequence (e.g, 'pV .. .') and groups where it occurs later (e.g, \n'tssxxV .. .' and 'pvpxV .. .'). Note that the division according to the path \ndoes not necessarily correspond to different predictions. For example, 'V 2' \nalways predicts 'END' and always with maximum certainty. Nevertheless, \nsequences up to 'V 2' are divided according to the path traversed. \n\nPHASES OF LEARNING \n\nHow can information about the path be progressively encoded in the hidden \nlayer patterns of activation? To clarify how the network learns to use the \ncontext of preceding letters in a sequence, we will illustrate the different \nphases of learning with cluster analyses of the hidden layer patterns \ngenerated at each phase. To make the analyses simpler, we used a smaller \ntraining set than the training set mentioned previously. The corresponding \nfinite-state grammar is shown in Figure 4. In this simpler grammar, the \nmain difference -- besides the reduced number of patterns -- is that the \nletters 'P' and 'T' appear only once. \n\ns \n\n.11It --4IF-----~ \n\nx \n\n----~Eu \n\nT \n\nFigure 4. The reduced rmite-state grammar from which 12 \n\nstrings were generated for training \n\nDiscovering letters. At epoch 0, before the network has received any \ntraining, the hidden unit patterns clearly show an organization by letter: to \neach letter corresponds an individual cluster. These clusters are already \nsubdivided according to preceding sequences -- the 'path'. This fact \nillustrates how a pattern of activation on the context units naturally tends to \nencode the path traversed so far independently of any error correcting \nprocedure. The average distance between the different patterns -- the \n'contrast' as it were -- is nonetheless rather small; the scale only goes up to \n0.6 (see Figure 5.A.)**. But this is due to the very small initial random \n\n** In all the following figures. the scaJe was automatically detennined by the cluster \nanalysis program. It is important to keep this in mind when comparing the figures to \n\n\fLearning Sequential Structure in Simple Recurrent Netw:,l.'\" \n\n649 \n\n. . \n\n: \n\nI \n\nFigure 5. Cluster Analyses of the H.U. aclivation pattmls obcained with the reduced \nset of strings: A. before lraining. B. After 100 epochs of lraining. C. After 700 epochs \n\nof training. \n\n\f650 \n\nServan-Schreiber, Cleeremans and McClelland \n\nvalues of the weights from the input and context layers to the hidden layer. \nLarger initial values would enhance the network's tendency to capture path \ninfonnation in the hidden unit patterns before training is even started \n\nAfter 100 epochs of training, an organization by letters is still prevalent, \nhowever letters have been regrouped according to similar predictions. \n'START', 'P' and'S' all make the common prediction of 'X or S' \n(although'S' also predicts 'END'); 'T' and 'V' make the common \nprediction of 'V' (although 'V' also predicts 'END' and 'P'). The path \ninformation has been almost eliminated: there is very little difference \nbetween the patterns generated by two different occurrences of the same \nletter (see Figure 5.B.). For example, the hidden layer pattern generated by \n'S I' and the corresponding output pattern are almost identical to the patterns \ngenerated by 'S2' (see table 2). \n\nTable 2. Activation of each output unit fOllowing the \n\npresentation of the flrst S in the grammar (SI) \nor the second S (S2> after 100 epochs of training \n\nT \n0.0 \n0.0 \n\nP \n\nS \n0.36 0.0 \n0.37 0.0 \n\nX \n0.33 \n0.33 \n\nE \n\nV \n0.16 0.17 \n0.16 0.17 \n\nSI \nS2 \n\nIn this phase, the network is learning to ignore the pattern of activation on \nthe context units and to produce an output pattern appropriate to the letter \n'S' in any context. This is a direct consequence of the fact that the patterns \nof activation on the hidden layer -- and hence the context layer -- are \ncontinuously changing from one epoch to the next as the weights from the \ninput units (the letters) to the hidden layer are modified. Consequently, \nadjustments made to the weights from the context layer to the hidden layer \nare inconsistent from epoch to epoch and cancel each other. In contrast, the \nnetwork is able to pick up the stable association between each letter and all \nof its possible successors. \n\nDiscovering arcs. At the end of this phase, individual letters consistently \ngenerate a unique pattern of activation on the hidden layer. This is a crucial \nstep in developing a sensitivity to context: patterns copied onto the context \nlayer have become a unique code designating which letter immediately \npreceded the current letter. The learning procedure can now exploit the \nregular association between the pattern on the context layer and the desired \noutput. Around epoch 700, the cluster analysis shows that the network has \nused this infonnation to differentiate clearly between the fust and second \noccurrence of the same letter (Figure 5.C.). The pattern generated by 'S2' -(cid:173)\nwhich predicts 'END' -- clusters with the pattern generated by 'V2\" which \nalso predicts 'END'. The overall difference between all the hidden layer \npatterns has also more than roughly doubled, as indicated by the change in \nscale. \n\nEncoding the path. During the last phase of learning, the network learns \nto make different predictions to the same occurrence of a letter (e.g, 'V I') \n\neach other. \n\n\fLearning Sequential Structure in Simple Recurrent Networks \n\n651 \n\non the basis of the previous sequence. For example, it learns to differentiate \nbetween 'ssxxV' which predicts either 'P' or 'V', and 'sssxxV' which \npredicts only 'V' by exploiting the small difference between the activation \npatterns generated by X 2 in the two different contexts. \n\nThe process through which path information is encoded can be \nconceptualized in the following way: As the initial papers about back(cid:173)\npropagation pointed out, the hidden unit patterns of activation represent an \n'encoding' of the features of the input patterns that are relevant to the task. \nIn the recurrent network, the hidden layer is presented with information \nabout the current letter, but also -- on the context layer -- with an encoding \nof the relevant features of the previous letter. Thus, a given hidden layer \npattern can come to encode information about the relevant features of two \nconsecutive letters. When this pattern is fed back on the context layer, the \nnew pattern of activation over the hidden units can come to encode \ninformation about three consecutive letters, and so on. In this manner, the \ncontext layer patterns can allow the network to maintain prediction-relevant \nfeatures of an entire sequence. However, it is important to note that \ninformation about the path that is not relevant locally (Le, that does not \ncontribute to predicting successors of the current letter) tends not to be \nencoded in the next hidden layer pattern. It may then be lost for subsequent \nprocessing. This tendency is lessened when the network has extra degrees \nof freedom -- i.e, extra hidden units -- so as to allow small and locally \nuseless differences to survive for several processing steps. \n\nCONCLUSION \n\nWe have shown that the network architecture first proposed by Elman \n(1988) is capable of mastering an infinite coIpus of strings generated from a \nfinite-state grammar after training on a finite set of exemplars with a learning \nalgorithm that is local in time. The network develops internal representations \nthat correspond to the nodes of the grammar and closely approximates the \ncorresponding minimal finite-state recognizer. We have also shown that the \nsimple recurrent network is able to encode information about contingencies \nthat are not local to a given letter and its immediate predecessor, such as \nthose implied by a length constraint on the strings. Encoding of sequential \nstructure in the patterns of activation over the hidden layers proceeds in \nstages. The network first develops stable hidden-layer representations for \nindividual letters, and then for individual arcs in the grammar. Finally, the \nnetwork is able to exploit slight differences in the patterns of activation \nwhich denote a specific path through the grammar. Our current work is \nexploring the relevance of this architecture to the processing of embedded \nsequences typical of natural language. The results of some preliminary \nexperiments are available in Servan-Schreiber, Cleeremans and McClelland \n(1988). \n\n\f652 \n\nServan-Schreiber, Cleeremans and McClelland \n\nReferences \n\nElman. J.L. (1988). Finding structure in time. CRL Technical report 9901. Center for \n\nResearch in Language. University of California. San Diego. \n\nReber. A.S. (1967). Implicit learning of artificial grammars. Journal of Verbal Learning \n\nand Verbal Behavior. S. 855-863. \n\nRumelhart. D.E .\u2022 Hinton. G.E .\u2022 and Williams. R.I. (1986). Learning internal \n\nrepresentations by backpropagating errors. Nature 323:533-536. \n\nSejnowski. T J. and Rosenberg C. (1986). NETta1k: A parallel network that learns to read \n\naloud. Technical Report.lohns Hopkins University lHU-EECS-86-01. \n\nServan-Schreiber D. Cleeremans A. and McClelland JL (1988) Encoding sequential \nstructure in simple recurrent networks. Technical Report CMU-CS-88-183. \nComputer Science Department. Carnegie Mellon University. Pittsburgh. PA \n15213. \n\nWilliams. R.J. and Zipser. D. (1988). A learning algorithm for continually running fully \nrecurrent neural networks. ICS Technical report 8805. Institute for Cognitive \nScience. UCSD. La lolla. CA 92093. \n\n\f", "award": [], "sourceid": 188, "authors": [{"given_name": "David", "family_name": "Servan-Schreiber", "institution": null}, {"given_name": "Axel", "family_name": "Cleeremans", "institution": null}, {"given_name": "James", "family_name": "McClelland", "institution": null}]}