{"title": "Applications of Error Back-Propagation to Phonetic Classification", "book": "Advances in Neural Information Processing Systems", "page_first": 206, "page_last": 214, "abstract": null, "full_text": "206 \n\nAPPLICATIONS OF \n\n~RROR BACK-PROPAGATION \n\nTO PHONETIC CLASSIFICATION \n\nHong C. Leung & Victor W. Zue \nSpoken Language Systems Group \nLaboratory for Computer Science \n\nMassachusetts Institute of Technology \n\nCambridge, MA 02139 \n\nABSTRACT \n\nThis paper is concerced with the use of error back-propagation \nin phonetic classification. Our objective is to investigate the ba(cid:173)\nsic characteristics of back-propagation, and study how the frame(cid:173)\nwork of multi-layer perceptrons can be exploited in phonetic recog(cid:173)\nnition. We explore issues such as integration of heterogeneous \nsources of information, conditioll~ that can affect performance of \nphonetic classification, internal representations, comparisons with \ntraditional pattern classification techniques, comparisons of differ(cid:173)\nent error metrics, and initialization of the network. Our investiga(cid:173)\ntion is performed within a set of experiments that attempts to rec(cid:173)\nognize the 16 vowels in American English independent of speaker. \nOur results are comparable to human performance. \n\nEarly approaches in phonetic recognition fall into two major extremes: heuristic \nand algorithmic. Both approaches have their own merits and shortcomings. The \nheuristic approach has the intuitive appeal that it focuses on the linguistic informa(cid:173)\ntion in the speech signal and exploits acoustic-phonetic knowledge. HO'fever, the \nweak control strategy used for utilizing our knowledge has been grossly inadequate. \nAt the other extreme, the algorithmic approach relies primarily on the powerful con(cid:173)\ntrol strategy offered by well-formulated pattern recognition techniques. However, \nrelatively little is known about how our speech knowledge accumulated over the \npast few decades can be incorporated into the well-formulated algorithms. We feel \nthat artificial neural networks (ANN) have some characteristics that can potentially \nenable them to bridge the gap between these two extremes. On the one hand, our \nspeech knowledge can provide guidance to the structure and design of the network. \nOn the other hand, the self-organizing mechanism of ANN can provide a control \nstrategy for utilizing our knowledge. \n\nIn this paper, we extend our earlier work on the use of artificial neural networks \nfor phonetic recognition [2]. Specifically, we focus our investigation on the following \nsets of issues. First, we describe the use of the network to integrate heterogeneous \nsources of information. We will see how classification performance improves as more \n\n\fError Back-Propagation to Phonetic Classification \n\n207 \n\ninformation is available. Second, we discuss several important factors that can sub(cid:173)\nstantially affect the performance of phonetic classification. Third, we examine the \ninternal representation of the network. Fourth, we compare the network with two \ntraditional classification techniques: K-nearest neighbor and Gaussian classifica(cid:173)\ntion. Finally, we discuss our specific implementations of back-propagation that \nyield improved performance and more efficient learning time. \n\nEXPERIMENTS \n\nOur investigation is performed within the context of a set of experiments that \nattempts to recognize the 16 vowels in American English independent of speaker. \nThe vowels are excised from continuous speech and they can be preceded and fol(cid:173)\nlowed by any phonemes, thus providing a rich environment to study contextual \ninfluence. We assume that the locations of the vowels have been detected. Given a \ntime region, the network determines which one of the 16 vowels was spoken. \n\nCORPUS \n\nAs Table 1 shows, our training set consists of 20,000 vowel tokens, excised from \n2,500 continuous sentences spoken by 500 male and female speakers. The test set \nconsists of about 2,000 vowel tokens, excised from 250 sentences spoken by 50 dif(cid:173)\nferent speakers. All the data are extracted from the TIMIT database, which has a \nwide range of American dialectical variations [1]. The speech signal is represented \nby spectral vectors obtained from an auditory model [4]. Speaker and energy nor(cid:173)\nmalization are also performed [5]. \n\nTraining \nTesting \n\nTokens Sentences Speakers (M/F) \n20,000 \n500 (350/150) \n2,000 \n\n2500 \n250 \n\n50 (33/17) \n\nTable 1: Corpus extracted from the TIMIT database. \n\nNETWORK STRUCTURE \n\nThe structure of the network we have examined most extensively has 1 hidden \nlayer as shown in Figure 1. It has 16 output units, with one unit for each of the 16 \nvowels. In order to capture dynamic information, the vowel region is divided into \nthree equal subregions. An average spectrum is then computed in each subregion. \nThese 3 average spectra are then applied to the first 3 sets of input units. Additional \nsources of information, such as duration and local phonetic contexts, can also be \nmade available to the network. While spectral and durational inputs are continuous \nand numerical, the contextual inputs are discrete and symbolic. \n\n\f208 \n\nLeung and Zue \n\noutput from auditory model \n(synchrony spectrogram) \n\n-.......... -... -... ~.~j:.I._._ .... ---.......... -... -.. ~ \n\nFigure 1: Basic structure of the network. \n\nHETEROGENEOUS INFORMATION INTEGRATION \n\nIn our earlier study, we have examined the integration of the Synchrony En(cid:173)\n\nvelopes and the phonetic contexts [2]. The Synchrony Envelopes, an output of \nthe auditory model, have been shown to enhance the formant information. In this \nstudy, we add additional sources of information. Figure 2 shows the performance \nas heterogeneous sources of information are made available to the network. The \nperformance is about 60% when only the Synchrony Envelopes are available. The \nperformance improves to 64% when the Mean Rate Response, a different output of \nthe auditory model which has been shown to enhance the temporal aspects of the \nspeech signal, is also available. We can also see that the performance improves con(cid:173)\nsistently to 77% as durational and contextual inputs are provided to the network. \nThis experiment suggests that the network is able to make use of heterogeneous \nsources of information, which can be numerical and/or symbolic. \n\n\fError Back-Propagation to Phonetic Classification \n\n209 \n\nOne may ask how well human listeners can recognize the vowels. Experiments \nhave been performed to study how well human listeners agree with each other when \nthey can only listen to sequences of 3 phonemes, i.e. the phoneme before the vowel, \nthe vowel itself, and the phoneme after the vowel [3]. Results indicate that the \naverage agreement among the listeners on the identities of the vowels is between \n65% and 70%. \n\n80\u00b7 \n\n70\u00b7 \n\n60-\n\n5lr \n\nSynchrony \nEnvelopes Mean Rate Duration \n\nAdd \n\nAdd \n\nResponse \n\n: \n\nAdd \n\nPhonetic \nContext \n\nSources of Information \n\nFigure 2: Integration of heterogeneous sources of information. \n\nPERFORMANCE RESULTS \n\nWe have seen that one of the important factors for the network performance \nis the amount of information available to the network. To gain additional insights \nabout how the network performs under different conditions, several experiments \nwere conducted using different databases. In these and the subsequent experiments \nwe describe in this paper, only the Synchrony Envelopes are available to the network. \n\nTable 2 shows the performance results for several recognition tasks. In each \nof these tasks, the network is trained and tested with independent sets of speech \ndata. The first task recognizes vowels spoken by one speaker and excised from the \nfbf-vowel-ftf environment, spoken in isolation. This recognition task is relatively \nstraightforward, resulting in perfect performance. In the second experiment, vowel \ntokens are extracted from the same phonetic context, but spoken by 17 male and \nfemale speakers. Due to inter-speaker variability, the accuracy degrades to 86%. \nThe third task recognizes vowels spoken by one speaker and excised from an un(cid:173)\nrestricted context, spoken continuously. We can see that the accuracy decreases \nfurther to 70%. Finally, data from the TIM IT database are used, spoken by multi(cid:173)\nple speakers. The accuracy drops to 60%. These results indicate that a substantial \ndifference in performance can be expected under different conditions, depending on \nwhether the task is speaker-independent, what is the restriction on the phonetic \n\n\f210 \n\nLeung and Zue \n\nSpeakers(M/F) Context Training Percent \nTokens Correct \n\n1(1/0) \n17(8/9) \n1(1/0) \n\n500(350/150) \n\nb -\nt \nb - t \n* \n* \n* \n* \n-\n\n64 \n256 \n3,000 \n20,000 \n\n100 \n86 \n70 \n60 \n\nRemark \n\nisolated \nisolated \n\ncontinuous \ncontinuous \n\nTable 2: Performance for different tasks, using only the synchrony spectral infor(cid:173)\nmation. \"*,, stands for any phonetic contexts. \n\ncontexts, whether the speech material is spoken continuously, and how much data \nare used to train the network. \n\nINTERNAL REPRESENTATION \n\nTo understand how the network makes use of the input information, we exam(cid:173)\n\nined the connection weights of the network. A vector is formed by extracting the \nconnections from all the hidden units to one output unit as shown in Figure 3a. The \nsame process is repeated for all output units to obtain a total of 16 vectors. The \ncorrelations among these vectors are then examined by measuring the inner prod(cid:173)\nucts or the angles between them. Figure 3b shows the distribution of the angles \nafter the network is trained, as a function of the number of hidden units. The circles \nrepresent the mean of the distribution and the vertical bars stand for one standard \ndeviation away from the mean. As the number of hidden units increases, the distri(cid:173)\nbution becomes more and more concentrated and the vectors become increasingly \northogonal to each other. \n\nThe correlations of the connection weights before training were also examined, \nas shown in Figure 3c. Comparing parts (b) and (c) of Figure 3, we can see that \nthe distributions before and after training overlap more and more as the number of \nhidden units increases. With 128 hidden units, the two distributions are actually \nquite similar. This leads us to suspect that perhaps the connection weights between \nthe hidden and the output layer need not be trained if we have a sufficient number \nof hidden units. \n\nFigure 4a shows the performance of recognizing the 16 vowels using three differ(cid:173)\n\nent techniques: (i) train all the connections in the network, (ii) fix the connections \nbetween the hidden and output layers after random initialization and train only \nthe connections between the input and hidden layers, and (iii) fix the connections \nbetween the input and hidden layers and train only the connections between the \nhidden and output layers. We can see that with enough hidden units, training only \nthe connections between the input and the hidden layers achieves almost the same \nperformance as training all the connections in the network. We can also see that \n\n\fError Back-Propagation to Phonetic Classification \n\n211 \n\nfor the same number of hidden units, training only the connections between the \ninput and the hidden layer can achieve higher performance than training only the \nconnections between the hidden and the output layer. \n\nFigure 4b compares the three training techniques for 8 vowels, resulting in 8 \noutput units only. We can see similar characteristics in both parts (a) and (b) of \nFigure 4. \n\nOutput \nLayer \n\nHidden \nLayer \n\nCl\" Y \"j \u2022 \u2022 \u00a3r\"Y\"J' \n\n'~',~, \">-<.\"-----';,.. ......... :., \n\nInput \nLayer \n\n150 \n\nw \n\n-\n130 \nl110 \n! \nJa 70 \n~ < 50 \n30 \n\n90 I \n\n1 \n\nI f Iff I \n\n10 \n\n100 \n\nNumber of Hidden Units \n\n(a) \n\n150 \n-;;- 130 \nl 110 \n! \nw u \nCib c < 50 \n\n90 \n70 \n\ntoOO \n\n30 \n\n1 \n\n10 \n\n100 \n\n1000 \n\nNumber of Hidden Units \n\n(b) \n\n(c) \n\nFigure 3: (a) Correlations of the vectors from the hidden to output layers are \nexamined. (b) Distribution of the angles between these vectors after training. (c) \nDistribution of the angles between these vectors before training. \n\nCOMPARISONS WITH TRADITIONAL TECHNIQUES \n\nOne of the appealing characteristics of back-propagation is that it does not as(cid:173)\nsume any probability distributions or distance metrics. To gain further insights, we \ncompare with two traditional pattern classification techniques: K-nearest neighbor \n(KNN) and multi-dimensional Gaussian classifiers. \n\n\f212 \n\nLeung and Zue \n\n70 \n\n60 \n... 50 \n~ \n... \n8 40 \n1: \n~ \n\n30 \n\nCD \nQ. \n\n20 \n\n10 \n\n1 \n\n(i) \n\ny .. o ..\u2022. o-.. \u2022 \u00b7~D \n\n/ \nI \ni \no \n\n/ \ncI \n\n/ \n\n..p/ \n/--C) \n\nIII \n\n\"\" \n\n\\(\") \n\nII \n\n10 \n\n100 \n\nNumber of Hidden Units \n\n90 \n\n~ 70 \n... \n... \n8 50 \n1: \n... \nCD \nu \nCD \nQ. 30 \n\n10 \n\n1 \n\n1000 \n\n(i) __ ..0 .\u2022\u2022\u2022\u2022 0-._. . \u2022.. \n\n\". \u2022.. 0.\u00b7\u00b7\u00b7.0 \n\np.... \nI \n\nI \n\n, \n\n~ \n\n~.c//\\ \n\u2022 \n\n(iii) \n\n~. \n\n,....-0/ \no \n\n'Qi) \n\n10 \n\n100 \n\n1000 \n\nNumber of Hidden Units \n\n(a) \n\n(b) \n\nFigure 4: Performance of recognizing (a) 16 vowels, (b) 8 vowels when (i) all the \nconnections in the network are trained, (ii) only the connections between the input \nand hidden layers are trained, and (iii) only the connections between the hidden \nand output layers are trained. \n\nFigure 5a compares the performance results of the network with those of KNN, \nfor different amounts of training tokens. Again, only the Synchrony Envelopes \nare made available to the network, resulting in input vectors of 100 dimensions. \nEach cluster of crosses corresponds to performance results of ten networks, each \none randomly initialized differently. Due to different initialization, a fluctuation of \n2% to 3% is observed even for the same training size. For comparison, we perform \nKNN using the Euclidean distance metric. For each training size, we run KNN 6 \ntimes, each one with a different K, which is chosen to be proportional to the square \nroot of the number of training tokens, N. For simplicity, Figure 5a shows results for \nonly 3 different values of K: (i) K = Vii, (ii) K = 10Vii, and (iii) K = 1. In this \nexperiment, we have found that the performance is the best when K = ..fFi and is \nthe worst when K = 1. We have also found that up to 20,000 training tokens, the \nnetwork consistently compares favorably to KNN. It is possible that the network is \nable to find its own distance metric to achieve better performance. \n\nSince the true underlying probability distribution is unknown, we assume multi(cid:173)\ndimensional Gaussian distribution in the second experiment. (i) We use the full \ncovariance matrix, which has 100zl00 elements. To avoid problems with singularity, \nwe obtain results only for large number of training tokens. (ii) We use the diagonal \ncovariance matrix which has non-zero elements only along the diagonal. We can \nsee from Figure 5b that the network compares favorably to the Gaussian classifiers. \nOur results also suggest that the Gaussian assumption is invalid. \n\n\fError Back-Propagation to Phonetic Classification \n\n213 \n\n60 \n\n50 \n\n40 \n\n~ \n~ \n\niii \n\n(i) \n\n! . ..! \n........... \n\n.t. \n~--....-, (ii) \n....... \nj.. ..... ..-:. \u2022. _ ..... \nJ\"\"\".' ........ \n/\"Jr/ \n(.-\n~ \n\n... / \n\n(iii) \n\n60 \n\n] 50 \n'e! \n~ \n\n40 \n\nQ. \n\n\u2022 \u2022 \nI ~~:~) \nD\u00b7\u00b7\u00b7\u00b7 \n)..-/ \nI \n\n. .... \n\n30~----------------------------------~ \n\n10000 \n\n100000 \n\n100 \n\n1000 \n\n~~------------------------------------~ \n100 \n\n10000 \n\n1000 \n\n100000 \n\nNumber of Training Tokens \n\nNumber of Training Tokens \n\n(a) \n\n(b) \n\nFigure 5: (a) Comparison with KNN for different values of K (See text). \n(b) \nComparison with Gaussian classification when using the (i) full covariance matrix, \nand (ii) diagonal covariance matrix. Each cluster of 10 crosses corresponds to the \nresults of 10 different networks, each one randomly initialized. \n\nERROR METRIC AND INITIALIZATION \n\nIn order to take into account the classification performance of the network more \n\nexplicitly, we have introduced a weighted mean square error metric [2]. By modu(cid:173)\nlating the mean square error with weighting factors that depend on the classifica(cid:173)\ntion performance, we have shown that the rank order statistics can be improved. \nLike simulated annealing, gradient descent takes relatively big steps when the per(cid:173)\nformance is poor, and takes smaller and smaller steps as the performance of the \nnetwork improves. \n\nResults also indicate that it is more likely for a unit output to be initially in the \nsaturation regions of the sigmoid function if the network is randomly initialized. \nThis is not desirable since learning is slow when a unit output is in a saturation \nregion. Let the sigmoid function goes from -1 to 1. If the connection weights \nbetween the input and the hidden layers are initialized with zero weights, then all \nthe hidden unit outputs in the network will initially be zero, which in turn results in \nzero output values for all the output units. In other words, all the units will initially \noperate at the center of the transition region of the sigmoid function, where learning \nis the fastest. We call this method center initialization (CI). \n\nParts (a) and (b) of Figure 6 compare the learning speed and performance, \nrespectively, of the 3 different techniques: (i) mean square error (MSE), (ii) weighted \nmean square error (WMSE), and (iii) center initialization (CI) with WMSE. We can \nsee that both WMSE and CI seem to be effective in improving the learning time \nand the performance of the network. \n\n\f214 \n\nLeung and Zue \n\n(iii) \n\nK'\" \n\n(i) \n\n(ii) \n\n30~--~----------------~ \n50 \n\n30 \n\n40 \n\n10 \n\n20 \n\no \n\nNumber of Training Iterations \n\n30~4-------------~----~ \n\n100 \n\n1000 \n\n10000 \n\n100000 \n\nNumber of Training Tokens \n\n(a) \n\n(b) \n\nFigure 6: Comparisons of the (a) learning characteristics and, (b) performance \nresults, for the 3 different techniques: (i) MSE, (ii) WMSE, and (iii) CI with WMSE. \nEach point corresponds to the average of 10 different networks, each one initialized \nrandomly. \n\nSUMMARY \n\nIn summary, we have described a set of experiments that were designed to help \n\nus get a better understanding of the use of back-propagation in phonetic classifica(cid:173)\ntion. Our results are encouraging and we are hopeful that artificial neural networks \nmay provide an effective framework for utilizing our acoustic-phonetic knowledge \nin speech recognition. \n\nReferences \n\n[1] Fisher, W.E., Doddington, G.R., and Goudie-Marshall, K.M., \"The DARPA \n\nSpeech Recognition Research Database: Specifications and Status,\" Proceed(cid:173)\nings of the DARPA Speech Recognition Workshop Report No. SAIC-86/1546, \nFebruary, 1986. \n\n[2] Leung, H.C., \"Some phonetic recognition experiments using artificial neural \n\nnets/' ICASSP-88, 1988. \n\n[3] Phillips, M.S., \"Speaker independent classification of vowels and diphthongs \nin continuous speech,\" Proc. of the 11th International Congress of Phonetic \nSciences, Estonia, USSR, 1987. \n\n[4] Seneff S., \"A computational model for the peripheral auditory system: appli(cid:173)\n\ncation to speech recognition research,\" Proc. ICASSP, Tokyo, 1986. \n\n[5] Seneff S., \"Vowel recognition based on 'line-formants' derived from an auditory(cid:173)\n\nbased spect(al representation,\" Proc. of the 11th International Congress of \nPhonetic Sciences, Estonia, USSR, 1987. \n\n\f", "award": [], "sourceid": 109, "authors": [{"given_name": "Hong", "family_name": "Leung", "institution": null}, {"given_name": "Victor", "family_name": "Zue", "institution": null}]}