{"title": "Operational Fault Tolerance of CMAC Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 340, "page_last": 347, "abstract": null, "full_text": "340 \n\nCarter, Rudolph and Nucci \n\nOperational Fault Tolerance \n\nof CMAC Networks \n\nMichael J. Carter \n\nFranklin J. Rudolph \n\nAdam J. Nucci \n\nIntelligent Structures Group \n\nDepartment of Electrical and Computer Engineering \n\nUniversity of New Hampshire \n\nDurham, NH 03824-3591 \n\nABSTRACT \n\nThe performance sensitivity of Albus' CMAC network was studied for \nthe scenario in which faults are introduced into the adjustable weights \nafter training has been accomplished.  It was found that fault sensitivity \nwas reduced with increased generalization when \"loss of weight\" faults \nwere considered,  but sensitivity  was increased for  \"saturated weight\" \nfaults. \n\n1 \n\nINTRODUCTION \n\nFault-tolerance is often cited as an inherent property of neural networks, and is thought by \nmany  to be a  natural consequence of \"massively parallel\"  computational architectures. \nNumerous  anecdotal  reports  of  fault-tolerance  experiments,  primarily  in  pattern \nclassification tasks, abound in the literature.  However, there has been surprisingly little \nrigorous investigation of the fault-tolerance properties of various network architectures in \nother application  areas.  In  this  paper we  investigate the  fault-tolerance  of the  CMAC \n(Cerebellar Model Arithmetic Computer) network [Albus 1975] in a systematic manner. \nCMAC  networks  have  attracted  much  recent  attention  because  of their  successful \napplication in  robotic  manipulator control  [Ersu  1984, Miller  1986, Lane  1988].  Since \nfault-tolerance is a key concern in critical control tasks, there is added impetus to study \n\n\fOperational Fault Tolerance of CMAC Networks \n\n341 \n\nthis  aspect of CMAC  performance.  In  particular.  we  examined the  effect on  network \nperformance of faults introduced into the adjustable weight layer after training has been \naccomplished in a fault-free environment  The degradation of approximation error due to \nfaults was studied for the task of learning simple real functions of a single variable.  The \ninfluence of receptive field width and total CMAC memory size on the fault sensitivity of \nthe network was evaluated by means of simulation. \n\n2  THE  CMAC  NETWORK ARCHITECTURE \n\nThe CMAC network shown in Figure  1 implements a form  of distributed table lookup. \nIt consists of two  parts:  1)  an  address  generator module.  and  2)  a  layer of adjustable \nweights.  The address generator is a fixed algorithmic transformation from  the input space \nto the space of weight addresses.  This transformation  has  two  important properties:  1) \nOnly a fixed number  C of weights are activated in response to any particular input. and \nmore  importantly,  only  these  weights  are  adjusted  during  training;  2)  It  is  locally \ngeneralizing. in the sense that any two input points separated by Euclidean distance less \nthan some threshold produce activated weight subsets that are close in Hamming distance, \ni.e. the  two  weight  subsets  have  many  weights  in  common.  Input  points  that  are \nseparated by more than the threshold distance produce non-overlapping activated weight \nsubsets.  The first property gives rise  to the extremely fast  training  times noted by all \nCMAC investigators.  The number of weights activated by  any input is referred to as  the \n\"generalization parameter\".  and  is typically  a  small  number  ranging  from  4  to  128  in \npractical applications [Miller 1986].  Only the activated weights are summed to form  the \nresponse to the current input.  A simple delta rule adjustment procedure is used to update \nthe activated weights in response to the presentation of an input-desired output exemplar \npair.  Note  that there  is no adjustment of the  address generator transformation  during \nlearning.  and  indeed,  there  are  no  \"weights\"  available  for  adjustment  in  the  address \ngenerator.  It should also be noted that the hash-coded mapping is in general  necessary \nbecause there are many more resolution  cells in  the  input space  than  there  are  unique \nfinite  combinations  of  weights  in  the  physical  memory.  As  a  result,  the  local \ngeneralization  property  will  be  disturbed  because  some  distant  inputs  share  common \nweight addresses in their activated weight subsets due to hashing collisions. \n\nWhile  the  CMAC  network readily  lends  itself to the  task  of learning and  mimicking \nmultidimensional nonlinear transformations. the investigation of network fault-tolerance \nin  this setting is daunting!  For reasons discussed in  the next section. we opted to  study \nCMAC fault-tolerance  for simple one-dimensional input and output spaces without the \nuse of the hash-coded mapping. \n\n3 \n\n.FAULT-TOLERANCE  EXPERIMENTS \n\nWe distinguish between  two types of fault-tolerance  in  neural  networks  [Carter  1988]: \noperational fault-tolerance and learning fault-tolerance.  Operational fault-tolerance deals \nwith the sensitivity of network performance to faults inttoduced after learning has been \n\n\f342 \n\nCarter, Rudolph and Nucci \n\naccomplished in a fault-free environment.  Learning fault-tolerance refers to the sensitivity \nof network performance to faults (either permanent or transient) which are present during \ntraining.  It  should be  noted  that the  term  fault-tolerance as used  here  applies only  to \nfaults that represent perturbations in network parameters or topology, and does not refer \nto  noisy  or  censored  input  data. \nIndeed,  we  believe  that  the  latter  usage  is  both \ninappropriate and inconsistent with  conventional  usage  in  the computer fault-tolerance \ncommunity. \n\n3.1 \n\nEXPERIMENT  DESIGN  PHILOSOPHY \n\nSince the CMAC network is widely used for learning nonlinear functions (e.g. the motor \ndrive  voltage  to  joint angle  transformation  for  a  multiple  degree-of-freedom  robotic \nmanipulator),  the  obvious  measure  of network performance is  function  approximation \nerror.  The sensitivity of approximation error to faults is the subject of this paper.  There \nare several types of faults that are of concern in the CMAC architecture.  Faults that occur \nin  the  address  generator  module  may  ultimately  have  the  most  severe  impact  on \napproximation error since the selection of incorrect weight addresses will likely produce a \nbad response.  On the other hand, since the address generator is an algorithm rather than a \ntrue  network of simple computational  units,  the  fault-tolerance  of any  serial  processor \nimplementation of the algorithm will be difficult to  study.  For this reason  we  initially \nelected to study the fault sensitivity of the adjustable weight layer only. \n\nThe choice of fault types and fault placement strategies for neural network fault tolerance \nstudies  is  not at all  straightforward.  Unlike classical fault-tolerance  studies in  digital \nsystems which  use \"stuck-at-zero\" and \"stuck-at-one\"  faults,  neural  networks which  use \nanalog or mixed analog/digital implementations may suffer from a host of fault types.  In \norder to make some progress, and to  study the fault tolerance of the CMAC  network at \nthe  architectural  level rather than at the  device  level,  we  opted  for  a  variation on the \n\"stuck-at\" fault model of digital systems.  Since this study was concerned only with  the \nadjustable weight layer, and since we assumed that weight storage is  most likely  to be \ndigital  (though  this  will  certainly  change  as  good  analog  memory  technologies  are \ndeveloped), we considered two fault models which are admittedly severe.  The first is  a \n\"loss of weight\"  fault  which  results in  the selected weight being set to zero,  while the \nsecond  is  a  \"saturated  weight\"  fault  which  might  correspond  to  the  situation  of  a \nstuck-at-one fault in the most significant bit of a single weight register. \n\nThe question of fault placement is also problematic.  In  the absence of a specific circuit \nlevel  implementation  of  the  network,  it  is  difficult  to  postulate  a  model  for  fault \ndistribution.  We  adopted a somewhat perverse outlook in the hope of characterizing the \nnetwork's fault tolerance under a worst-case fault placement strategy.  The insight gained \nwill  still  prove  to  be  valuable in  more  benign fault placement tests  (e.g.  random  fault \nplacement), and in addition, if one can devise network modifications which  yield good \nfault-tolerance in this extreme case, there is hope of still better performance in more \n\n\fOperational Fault Tolerance of CMAC Networks \n\n343 \n\ntypical  instances of circuit failure.  When placing  \"loss of weight\"  faults,  we  attacked \nlarge magnitude weight locations fast, and continued to add more such faults to locations \nranked in descending order of weight magnitude.  Likewise, when placing saturated weight \nfaults  we  attacked  small  magnitude  weight  locations  first,  and successive  faults  were \nplaced in locations ordered by ascending weight magnitude.  Since the activated weights \nare simply summed to form a response in CMAC, faults of both types create an  error in \nthe  response  which  is equal  to  the  weight change  in  the  faulted  location.  Hence,  our \nstrategy was designed to produce the maximum output error for a given number of faults. \nIn  placing  faults  of either type,  however,  we  did  not place  two  faults  within  a  single \nactivated weight subset.  Our strategy was thus not an  absolute worst-case strategy, but \nwas still more stressful than  a purely random fault placement strategy.  Finally, we did \nnot mix fault types in any single experiment. \n\nThe fault  tolerance experiments presented in  the next section all had  the same general \nstructure.  The  network under study  was  trained  to  reproduce  (to a  specified  level  of \napproximation error) a real function of a single variable, y=f(x), based upon presentation \nof (x,y) exemplar pairs.  Faults of the types described previously were  then  introduced, \nand  the  resulting degradation  in  approximation  error was logged versus the  number of \nfaults.  Many such  experiments were conducted with varying CMAC memory size and \ngeneralization  parameter while  learning  the  same  exemplar  function.  We  considered \nsmoothly varying  functions  (sinusoids of varying spatial frequency)  and discontinuous \nfunctions (step functions) on a bounded interval. \n\n3.2 \n\nEXPERIMENT  RESULTS  AND  DISCUSSION \n\nIn this section we present the results of experiments in which the function to be learned is \nheld  fixed,  while  the  generalization  parameter of the  CMAC  network  to  be  tested  is \nvaried.  The total  number of weights  (also referred to here as memory locations) is the \nsame  in  each  batch  of  experiments.  Memory  sizes  of  50,  250,  and  1000  were \ninvestigated, but only the results for  the case N=250 are presented here.  They exemplify \nthe trends observed for all memory sizes. \n\nFigure 2 shows the dependence of RMS  (root mean square) approximation error on the \nnumber of loss-of-weight faults  injected for generalization parameter values C=4, 8,  16. \nThe  task  was  that  of reproducing  a  single cycle  of a  sinusoidal  function  on  the  input \ninterval.  Note that approximation error was diminished with increasing generalization at \nany  fault  severity  level.  For  saturated  weight  faults,  however,  approximation  error \nincr.eased  with  increasing  generalization!  The  reason  for  this  contrasting  behavior \nbecomes clear upon examination of Figure 3.  Observe also in Figure 2 that the increase \nin  RMS  error due  to the  introduction  of a  single  fault  can  be as  much  as an  order of \nmagnitude.  This  is  somewhat  deceptive  since  the  scale  of the  error  is  rather  small \n(typically  10-3 or so), and so it may  not seem of great consequence.  However, as one \nmay  note  in  Figure  3,  the  effect  of  a  single  fault  is  highly  localized,  so  RMS \napproximation error may be a poor choice of performance measure in selected \n\n\f344 \n\nCarter, Rudolph and Nucci \n\napplications.  In particular, saturated weight faults in nominally small weight magnitude \nlocations create a large relative response error, and this may be devastating in real-time \ncontrol applications.  Loss-of-weight faults  are more  benign, and  their impact may  be \ndiluted by  increasing generalization.  The penalty  for  doing  so, however,  is increased \nsensitivity to saturated weight faults  because larger regions of the network mapping are \naffected by a single fault \n\nFigure  4  displays  some  of the  results  of fault-tolerance  tests  with  a  discontinuous \nexemplar  function.  Note  the  large  variation  in  stored  weight  values  necessary  to \nreproduce the step  function.  When  a large  magnitude  weight needed  to  form  the step \ntransition  was  faulted,  the result was a double step (Figure 4(b\u00bb  or a shifted transition \npoint (Figure 4(c\u00bb.  The extent of the  fault  impact  was  diminished  with  decreasing \ngeneralization.  Since  pattern  classification  tasks  are  equivalent  to  learning  a \ndiscontinuous function over the input feature space, this finding suggests that improved \nfault-tolerance in  such tasks might be obtained by reducing the generalization parameter \nC.  This  would limit the shifting of pattern class boundaries in  the  presence of weight \nfaults.  Preliminary experiments,  however,  also showed  that learning of discontinuous \nexemplar functions proceeded much more slowly with small values of the generalization \nparameter. \n\n4  CONCLUSIONS  AND  OPEN  QUESTIONS \n\nThe CMAC network is well-suited to applications that demand fast learning of unknown \nmultidimensional, static mappings (such as those arising in nonlinear control and signal \nprocessing systems).  The results of the preliminary investigations reported here suggest \nthat  the  fault-tolerance  of conventional  CMAC  networks  may  not be as great as  one \nmight hope on the basis of anecdotal evidence in  the prior literature with other network \narchitectures.  Network fault sensitivity does not seem to be uniform, and the location of \nparticularly  sensitive  weights  is  very  much  dependent on  the  exemplar  function  to  be \nle3f!led.  Furthermore, the  obvious fault-tolerance  enhancement technique of increasing \ngeneralization (i.e. distributing the response computation over more weight locations) has \nthe  undesirable  effect of increasing sensitivity  to  saturated weight faults.  While  the \nlocal generalization feature of CMAC has  the desirable attribute of limiting the region of \nfault impact, it suggests that global approximation error measures may  be  misleading. \nA low value of RMS  error degradation may  in fact mask a  much more severe response \nerror over a  small  region  of the  mapping.  Finally,  one  must  be  cautious  in  making \nassessments of the fault-tolerance of a fixed  network on the basis of tests using  a single \nmapping.  Discontinuous exemplar functions produce stored  weight distributions which \nare much more fault-sensitive than those associated with smoothly varying functions, and \nsuch functions are clearly of interest in  pattern classification. \n\nMany  important open  questions remain concerning the  fault-tolerance properties of the \nCMAC  network.  The effect of faults  on  the  address  generator  module  has  yet to  be \ndetermined.  Collisions in the hash-coded mapping effectively propagate weight faults to \n\n\fOperational Fault Tolerance of CMAC Networks \n\n345 \n\nremote  regions  of  the  input  space,  and  the  impact  of  this  phenomenon  on  overall \nfault-tolerance  has  not  been  assessed.  Much  more  work  is  needed  on  the  role  that \nexemplar function smoothness plays in detennining the fault-tolerance of a fIxed  topology \nnetwork. \n\nAcknowledgements \n\nThe authors would like to thank Tom Miller, Fil Glanz. Gordon Kraft. and Edgar An for \nmany helpful discussions on  the CMAC network architecture.  This work was supported \nin part by an Analog Devices Career Development Professorship and by a General Electric \nFoundation Young Faculty Grant awarded to MJ. Carter. \n\nReferences \n\nJ.S.  Albus.  (1975)  \"A  new  approach  to  manipulator  control:  the  Cerebellar  Model \nArticulation Controller  (CMAC),\"  Trans.  ASME- 1.  Dynamic Syst ..  Meas ..  Contr.  97 ; \n220-227. \n\nMJ.  Carter.  (1988)  \"The  illusion  of fault-tolerance  in  neural  networks  for  pattern \nrecognition and signal processing,\" Proc.  Technical Session on Fault-Tolerant Integrated \nSystems. Durham, NH:  University of New Hampshire. \n\nE.  Ersu  and  J.  Militzer.  (1984)  \"Real-time  implementation  of  an  aSSOC13Uve \nmemory-based learning control scheme for non-linear multivariable processes,\" Proc. 1 st \nMeasurement  and  Control  Symposium  on  Applications  of Multivariable  Systems \nTechniques;  109-119. \n\nS.  Lane,  D.  Handelman, and J.  Gelfand.  (1988)  \"A neural network computational  map \napproach  to reflexive motor control,\"  Proc.  IEEE Intelligent Control  Conf. Arlington, \nVA. \n\nW.T.  Miller.  (1986)  \"A  nonlinear learning  controller  for  robotic  manipulators,\"  Proc. \nSPIE: Intelligent Robots and Computer Vision  726; 416-423. \n\nC addrc:a elo:c:liaa \n\nIiDu \n\n--~ ~ ..----0 \n\nx \n\ny \n\n\f346 \n\nCarter, Rudolph and Nucci \n\n.. ~------------------------~ \n\u2022 \n\u2022  w\u00b7 \noo \n~ \u2022 \n\u2022. ~~--------------.....-. \n\n~4 \n- I  _1' \n\nc \n\n\u2022 \n\nFigure 2: Sinusoid Approximation Enor vs. Number of \"Loss-of-Weight\" Faults \n\nu \n\n\u00b7 \u2022 \n\n!  U \n\n---... \n\n'i. \u2022 \u2022 \u00b7 \u00b7 i4.1 \u2022 \n.... 1--.-----.......------...... ----\n\u2022 \n\u2022\u2022 \n\n. ~ \n\nIi \n\n-\n\nzea \n\n...... \n\nloU \n\n_ _  \n\n~  u \u2022 \u2022 \u2022 \u00b7 \u00b7 \n......... -----...... -----...... ---\n\u2022 \nu ,. \n\u00b7 \u2022 \nt \n\u2022  u \n!  u~----------~--________ --\n::,.:~ \n.... \n-.,~  -\n.... +-----------...-(cid:173)\n\u2022 \n\nII. \n\nFigure 3: Network Response and Stored Weight Values. a) single lost weight. \ngeneralization C=4; b) single lost weight, C=16; c) single saturated weight, C=16. \n\n\fOperational Fault Tolerance of CMAC Networks \n\n347 \n\nv \n'0 \n:I \n\n-~ e \n< \nv .: \n:s \nV a: \n\n1.5 \n\n1.0 \n\n0.5 \n\n0.0 \n\n-<1.5 \n\n~ \n\n\u00b71.0 \n\n\u00b71.5 \n\na \n\n0.8 \n\n-<1.2 \n\n.. '0 \n\n:I \n\n~ :: \n\u00ab \n.. \n'\" \n~ .. \n= \n\n\u00b71.2 \n\na \n\n2 \n\n~ -\u00ab  a \n\n\u00b72 \n\na \n\nneIWOrk response \nstOn:d weighlS \n\n\u2022 \n\n100 \n\nMemory Localioa \n\n200 \n\n\u2022 \n\n.-\n\nnetWOrk response \nstored WCIg/l1S \n\n0 \n\n100 \n\nMemory Loatioa \n\n200 \n\nSatUrated Fault ~ \n\n.--\n\nnetwork response \nstOred wcighlS \n\n0 \n\n100 \n\nMemory Locatioa \n\n200 \n\nFigure 4: Network Response and Stored Weight Values. a) no faults. transition at \nlocation  125. C=8; b) single lost weight, C=16; c) single saturated weight, C=16. \n\n\f", "award": [], "sourceid": 283, "authors": [{"given_name": "Michael", "family_name": "Carter", "institution": null}, {"given_name": "Franklin", "family_name": "Rudolph", "institution": null}, {"given_name": "Adam", "family_name": "Nucci", "institution": null}]}