{"title": "Learning Statistically Neutral Tasks without Expert Guidance", "book": "Advances in Neural Information Processing Systems", "page_first": 73, "page_last": 79, "abstract": null, "full_text": "Learning Statistically Neutral Tasks \n\nwithout Expert Guidance \n\nTon Weijters \n\nInformation Technology, \nEindhoven University, \n\nThe Netherlands \n\nAntal van den Bosch \n\nILK, \n\nTilburg University, \nThe Netherlands \n\nAbstract \n\nEric Postma \n\nComputer Science, \n\nUniversiteit Maastricht, \n\nThe Netherlands \n\nIn this paper, we question the necessity of levels of expert-guided \nabstraction in learning hard, statistically neutral classification \ntasks. We focus on two tasks, date calculation and parity-12, that \nare claimed to require intermediate levels of abstraction that must \nbe defined by a human expert. We challenge this claim by demon(cid:173)\nstrating empirically that a single hidden-layer BP-SOM network can \nlearn both tasks without guidance. Moreover, we analyze the net(cid:173)\nwork's solution for the parity-12 task and show that its solution \nmakes use of an elegant intermediary checksum computation. \n\n1 \n\nIntroduction \n\nBreaking up a complex task into many smaller and simpler subtasks facilitates \nits solution. Such task decomposition has proved to be a successful technique in \ndeveloping algorithms and in building theories of cognition. In their study and \nmodeling of the human problem-solving process, Newell and Simon [1] employed \nprotocol analysis to determine the subtasks human subjects employ in solving a \ncomplex task. Even nowadays, many cognitive scientists take task decomposition, \nLe., the necessity of explicit levels of abstraction, as a fundamental property of \nhuman problem solving. Dennis Norris' [2] modeling study on the problem-solving \ncapacity of autistic savants is a case in point. In the study, Norris focuses on the \ndate-calculation task (Le., to calculate the day of the week a given date fell on), \nwhich some autistic savants have been reported to perform flawlessly [3]. In an \nattempt to train a multi-layer neural network on the task, Norris failed to get a \nsatisfactory level of generalization performance. Only by decomposing the task into \nthree sub-tasks, and training the separate networks on each of the sub-tasks, the \ndate-calculation task could be learned. Norris concluded that the date-calculation \ntask is solvable (learnable) only when it is decomposed into intermediary steps using \nhuman assistance [2]. \nThe date-calculation task is a very hard task for inductive learning algorithms, \nbecause it is a statistically neutral task: all conditional output probabilities on \nany input feature have chance values. Solving the task implies decomposing it, \nif possible, into subtasks that are not statistically neutral. The only suggested \ndecomposition of the date-calculation task known to date involves explicit assistance \n\n\f74 \n\nT. Weijters, A. v. d. Bosch and E. Postma \n\nMFN \n\nSOM \n\n\u2022 \n\n- class A elements \n\no - class B elements \nI!I - unlabelled element \n\nFigure 1: An example BP-SOM network. \n\nfrom a human supervisor [2J. This paper challenges the decomposition assumption \nby showing that the date-calculation task can be learned in a single step with a \nappropriately constrained single hidden-layer neural network. In addition, another \nstatistically neutral task, called the parity-n task (given an n-Iength bit string of \n1 's and O's, calculate whether the number of 1 's is even or odd) is investigated. \nIn an experimental study by Dehaene, Bossini, and Giraux [4], it is claimed that \nhumans decompose the parity-n task by first counting over the input string, and \nthen perform the even/odd decision. In our study, parity-12 is shown to be learnable \nby a network with a single hidden layer. \n\n2 BP-SOM \n\nBelow we give a brief characterization of the functioning of BP-SOM. For details we \nrefer to [5J. The aim of the BP-SOM learning algorithm is to establish a coopera(cid:173)\ntion between BP learning and SOM learning in order to find adequately constrained \nhidden-layer representations for learning classification tasks. To achieve this aim, \nthe traditional MFN architecture [6J is combined with SOMS [7]: each hidden layer of \nthe MFN is associated with one SOM (See Figure 1). During training of the weights in \nthe MFN, the corresponding SOM is trained on the hidden-unit activation patterns. \n\nAfter a number of training cycles of BP-SOM learning, each SOM develops a two(cid:173)\ndimensional representation, that is translated into classification information, i.e., \neach SOM element is provided with a class label (one of the output classes of the \ntask). For example, let the BP-SOM network displayed in Figure 1 be trained on \na classification task which maps instances to either output class A or B. Three \ntypes of elements can be distinguished in the SOM: elements labelled with class A, \nelements labelled with class B, and unlabelled elements (no winning class could be \nfound). The two-dimensional representation of the SOM is used as an addition to \nthe standard BP learning rule [6J. Classification and reliability information from the \nSOMS is included when updating the connection weights of the MFN. The error of \na hidden-layer vector is an accumulation of the error computed by the BP learning \nrule, and a SOM-error. The SOM-error is the difference between the hidden-unit \nactivation vector and the vector of its best-matching element associated with the \nsame class on the SOM. \n\nAn important effect of including SOM information in the error signals is that clusters \nof hidden-unit activation vectors of instances associated with the same class tend \nto become increasingly similar to each other. On top of this effect, individual \nhidden-unit activations tend to become more streamlined, and often end up having \nactivations near one of a limited number of discrete values. \n\n\fLearning Statistically Neutral Tasks without Expert Guidance \n\n75 \n\n3 The date-calculation task \n\nThe first statistically neutral calculation task we consider is the date-calculation \ntask: determining the day of the week on which a given date fell. (For instance, \nOctober 24, 1997 fell on a Friday.) Solving the task requires an algorithmic approach \nthat is typically hard for human calculators and requires one or more intermediate \nsteps. It is generally assumed that the identity of these intermediate steps follows \nfrom the algorithmic solution, although variations exist in the steps as reportedly \nused by human experts [2] . We will show that such explicit abstraction is not \nneeded, after reviewing the case for the necessity of \"human assistance\" in learning \nthe task. \n\n3.1 Date calculation with expert-based abstraction \n\nNorris [2] attempted to model autistic savant date calculators using a multi-layer \nfeedforward network (MFN) and the back-propagation learning rule [6]. He intended \nto build a model mimicking the behavior of the autistic savant without the need \neither to develop arithmetical skills or to encode explicit knowledge about reg(cid:173)\nularities in the structure of dates. A standard multilayer network trained with \nbackpropagation [6] was not able to solve the date-calculation task. Although the \nnetwork was able to learn the examples used for training, it did not manage to \ngeneralize to novel date-day combinations. In a second attempt Norris split up the \ndate-calculation task in three simpler subtasks and networks. \n\nU sing the three-stage learning strategy Norris obtained a nearly perfect performance \non the training material and a performance of over 90% on the test material (errors \nare almost exclusively made on dates falling in January or February in leap years). \nHe concludes with the observation that \"The only reason that the network was able \nto learn so well was because it had some human assistance.\" [2, p.285]. In addition, \nNorris claims that \"even if the [backpropagation] net did have the right number of \nlayers there would be no way for the net to distribute its learning throughout the \nnet such that each layer learned the appropriate step in computation.\" [2, p. 290]. \n\n3.2 Date calculation without expert-based abstraction \n\nWe demonstrate that with the BP-SOM learning rule, a single hidden-layer feedfor(cid:173)\nward network can become a successful date calculator. Our experiment compares \nthree types of learning: standard backpropagation learning (BP, [6]), backpropa(cid:173)\ngation learning with weight decay (BPWD, [8]), and BP-SOM learning. Norris used \nBP learning in his experiment which leads to overfitting [2] (a considerably lower \ngeneralization accuracy on new material as compared to reproduction accuracy on \ntraining material); BPWD learning was included to avoid overfitting. \nThe parameter values for BP (including the number of hidden units for each task) \nwere optimized by performing pilot experiments with BP. The optimal learning-rate \nand momentum values were 0.15 and 0.4, respectively. BP, BPWD, and BP-SOM were \ntrained for a fixed number of cycles m = 2000. Early stopping, a common method \nto prevent overfitting, was used in all experiments with BP, BPWD, and BP-SOM [9]. \n\nIn our experiments with BP-SOM, we used the same interval of dates as used by \nNorris, i.e., training and test dates ranged from January 1, 1950 to December 31, \n1999. We generated two training sets, each consisting of 3,653 randomly selected \ninstances, i.e., one-fifth of all dates. We also generated two corresponding test sets \nand two validation sets (with 1,000 instances each) of new dates within the same \n50-year period. In all our experiments, the training set, test set, and validation set \n\n\f76 \n\nT. Weijters, A. v. d. Bosch and E. Postma \n\nTable 1: Average generalization performances (plus standard deviation, after '\u00b1'; \naveraged over ten experiments) in terms of incorrectly-processed training and test \ninstances, of BP, BPWD, and BP-SOM, trained on the date-calculation task and the \nparity-12 task. \n\nhad empty intersections. We partitioned the input into three fields, representing \nthe day of the month (31 units), the month (12 units) and the year (50 units). The \noutput is represented by 7 units, one for each day of the week. The MFN contained \none hidden layer with 12 hidden units for BP, and 25 hidden units for BPWD and \nBP-SOM. The SOM of the BP-SOM network contained 12 x 12 elements. Each of the \nthree learning types was tested on two different data sets. Five runs with different \nrandom weight initializations were performed on each set, yielding ten runs per \nlearning type. The averaged classification errors on the test material are reported \nin Table 1. \n\nFrom Table 1 it follows that the average classification error of BP is high: on test \ninstances BP yields a classification error of 28.8%, while the classification error of \nBP on training instances is 20.8%. Compared to the classification error of BP, the \nclassification errors on both training and test material of BPWD and BP-SOM are \nmuch lower. However, BPWD'S generalization performance on the test material is \nconsiderably worse than its performance on the training material: a clear indication \nof overfitting. We note in passing that the results of BPWD contrast with Norris' \n[2J claim that BP is unable to learn the date-calculation task when it is not decom(cid:173)\nposed into subtasks. The inclusion of weight decay in BP is sufficient for a good \napproximation of the performance results of Norris' decomposed network. \n\nThe results in Table 1 also show that the performance of BP-SOM on test mate(cid:173)\nrial is significantly better than that of BPWD (t(19)=7.39, p