{"title": "Explorations with the Dynamic Wave Model", "book": "Advances in Neural Information Processing Systems", "page_first": 549, "page_last": 555, "abstract": null, "full_text": "Explorations with the Dynamic Wave \n\nModel \n\nThomas  P.  Rebotier \n\nDepartment of Cognitive Science \n\nUCSD,  9500  Gilman Dr \n\nLA JOLLA CA 92093-0515 \n\nrebotier@cogsci.ucsd.edu \n\nJeffrey  L.  Elman \n\nDepartment of Cognitive Science \n\nUCSD,  9500  Gilman  Dr \n\nLA JOLLA  CA 92093-0515 \n\nelman@cogsci.ucsd.edu \n\nAbstract \n\nFollowing  Shrager  and  Johnson  (1995)  we  study  growth  of  logi(cid:173)\ncal  function  complexity  in  a  network  swept  by  two  overlapping \nwaves:  one  of pruning, and  the  other of Hebbian  reinforcement  of \nconnections.  Results  indicate  a  significant  spatial  gradient  in  the \nappearance  of  both  linearly  separable  and  non  linearly  separable \nfunctions of the two inputs of the network; the n.l.s.  cells are  much \nsparser  and  their slope of appearance  is  sensitive  to  parameters  in \na  highly  non-linear  way. \n\n1 \n\nINTRODUCTION \n\nBoth  the  complexity of the  brain  (and  concomittant  difficulty  encoding  that  C0111-\nplexity  through  any  direct  genetic  mapping).  as  well  as  the apparently  high  degree \nof  cortical  plasticity  suggest  that  a  great  deal  of  cortical  structure  is  emergent \nrather  than  pre-specified.  Several  neural  models  have  explored  the  emergence  of \ncomplexity.  Von  der  Marlsburg  (1973)  studied  the  grouping  of orientation  selec(cid:173)\ntivity  by  competitive  Hebbian  synaptic  modification.  Linsker  (1986.a,  1986 .b  and \n1986.c) showed how spatial selection cells  (off-center on-surround), orientation selec(cid:173)\ntive  cells,  and finally  orientation columns, emerge  in  successive layers  from  random \ninput by  simple,  Hebbian-like learning  rules .  ~[iller (1992,  1994) studied  the  emer(cid:173)\ngence of orientation selective columns from activity dependant competition between \non-center  and off-center  inputs. \n\nKerzsberg, Changeux and Dehaene (1992) studied a model with a dual-aspect learn(cid:173)\ning  mechanism:  Hebbian  reinforcement  of the  connection  strengths  in  case  of cor(cid:173)\nrelated  activity,  and gradual pruning of immature connections.  Cells  in  this  model \nwere  organized on a  2D  grid , connected  to each other according to a  probability ex(cid:173)\nponentially decreasing with distance , and received inputs from two different sources, \n\n\f550 \n\nT. P. REBOTIER, J.  L.  ELMAN \n\nA  and  B,  which  might or  might  not  be  correlated.  The analysis of the  network  re(cid:173)\nvealed  17  different  kinds of cells:  those  whose ou tpu t  after several  cycles  depended \non  the  network's  initial state,  and  the  16  possible  logical  functions  of two  inputs. \nKerzsberg  et  al.  found  that  learning  and  pruning created  different  patches  of cells \nimplementing common  logical functions,  with strong  excitation  within  the  patches \nand inhibition between  patches. \n\nShrager and Johnson (1995)  extended  that work by giving the network structure in \nspace  (structuring  the inputs in intricated stripes)  or in time,  by  having a  Hebbian \nlearning  occur  in  a  spatiotemporal  wave  that  passed  through  the  network  rather \nthan  occurring  everywhere  simultaneously.  Their  motivation  was  to  see  if  these \nlearning  conditions  might  create  a  cascade  of increasingly  complex functions.  The \napproach  was  also  motivated  by  developmental  findings  in  humans  and  monkeys \nsuggesting  a  move  of  the  peak  of  maximal  plasticity  from  the  primary  sensory \nand  motor  areas  to\\vards  parietal  and  then  frontal  regions.  Shrager  and  Johnson \nclassified  the logical functions into three groups:  the constants (order 0),  those  that \ndepend  on  one  input  only  (order  1),  those  that  depend  on  both  inputs  (order  2). \nThey  found  that  a  slow  wave  favored  the  growth  of order  2  cells,  whereas  a  fast \nwave favored order  1 cells.  However,  they only varied  the  connection  reinforcement \n(the growth Trophic Factor), so  that the still diffuse  pruning affected  the  rightmost \nconnections  before  they  could stabilize,  resulting  in  an overall  decrease  which  had \nto  be  compensated for  in  the  analysis. \n\nIn  this  work,  v,,'e  followed  Shrager  and  Johnson  in  their  study  of the  effect  of  a \ndynamic wave of learning.  We present  three novel features.  Firstly, both the growth \ntrophic  factor  (hereafter,  TF)  and  the  probability of pruning  (by  analogy,  \"death \nfactor\",  DF) travel in gaussian-shaped  waves.  Second,  we  classify  the cells  in 4,  not \n3,  orders:  order  3 is  made  of the  non-linearly separable  logical  functions,  whereas \nthe  order  2 is  now  restricted  to  linearly separable  logical functions  of both  inputs. \nThird. we  use  an overall measure of network  performance:  the  slope of appearance \nof units  of a  given  order.  The  density  is  neglected  as  a  measure  not  related  to  the \nspecific  effects  we  are looking for,  namely, spatial changes in  complexity.  Thus, each \nrun  of our  network  can  be  analyzed  using  4  values:  the slopes  for  units  of order  0, \n1,  2  and  3  (See  Table  1.).  This  extreme  summarization of functional  information \nallows  us  to  explore  systematically  many  parameters  and  to  study  their  influence \nover  how  complexity grows  in  space. \n\nTable  1:  Orders of logical complexity \n\nORDER \no \n1 \n2 \n3 \n\nFUNCTIONS \n\nTrue  False \nA  !A  B  !B \nA.B  !A.B  A.!B  !A.!B  AvB  !AvB  Av!B  !Av!B \nA  xor  B,  A==B \n\n2  METHODS \n\nOur  basic  network  consisted  of 4  columns  of 50  units  (one  simulation  verified  the \nscaling  up  of results,  see  section  3.2).  Internal  connections  had  a  gaussian  band(cid:173)\nwidth and did not wrap around .  All initial connections were of weight 1,  so that the \nconnectivity  weights  given  as  parameters  specified  a  number  of labile connections. \nEarly  investigations were  made with  a  set  of manually chosen  parameters  (\" MAN-\n\n\fExplorations  with the Dynamic Wave  Model \n\n551 \n\nUAL\").  Afterwards, two sets of parameters were determined by a Genetic Algorithm \n(see  Goldberg  1989):  the  first,  \"SYM\",  by  maximizing  the  slope  of appearance  of \norder 3 units only, the second,  \" ASY\" , byoptimizing jointly the appearance of order \n2  and  order  3  units  (\" ASY\").  The  \"SYM\"  network  keeps  a  symmetrical  rate  of \npresentation  between  inputs  A  and  B.  In  contrast,  the\" ASY\"  net  presents  input \nB  much  more  often  than  input  A.  Parameters  are  specified  in  Table  1 and,  are  in \n\"natural\"  units:  bandwidths  and  distances  are  in  \"cells  apart\",  trophic  factor  is \nhomogenous  to  a  weight,  pruning  is  a  total  probability.  Initial  values  and  prun(cid:173)\ning  necessited  random  number  generation.  \\Ve  used  a  linear congruence  generator \n(see  p284 in  Press  1988), so  that given the same seed,  two  different  machines could \nproduce  exactly  the same  run.  All  the  points of each  Figure  are  means  of several \n(usually 40)  runs  with  different  random seeds  and share  the same series  of random \nseeds. \n\nTable 2:  Default  parameters \n\nMAN.  SYM.  ASY.  name \n\ndescription \n\n8.5 \n6.5 \n8.5 \n6.5 \n5.0 \n3.5 \n0.2 \n7.0 \n7.0 \n0.7 \n1.5 \n9.87 \n0.6 \n3.5 \n0.6 \n0.65 \n0.5 \n0.5 \n0.00 \n\n6.20 \n5.2 \n8.5 \n6.5 \n6.5 \n1.24 \n0.20 \n1.26 \n2.86 \n0.68 \n3.0 \n17.6 \n0.6 \n1.87 \n0.64 \n0.62 \n0.5 \n0.5 \n0.00 \n\nWae  mean ini.  weight  of A excitatory  connections \n12 \nWai  mean ini.  weight  of A  inhibitory connections \n9.7 \n13.4  Wbe  mean ini.  weight  of B excitatory  connections \n\\Vbi  mean ini.  weight  of B  inhibitory connections \n14.1 \nWne  m.ini.  density  of internal excitatory  connections \n9.9 \n12.4  Wni  m.ini.  density  of internal inhibitory connections \nDW \n0.28 \nBne \n0.65 \nBni \n0.03 \nCdw \n0.98 \n-3.2 \nDdw \n16.4  Wtf \nBtf \n0.6 \nTst \n3.3 \nBdf \n0.5 \n0.12 \nPdf \nPa \n0.06 \nPb \n0.81 \n0.00 \nPab \n\nrelative variation in  initial weights \nbandwidth of internal excitatory  connections \nbandwidth of internal inhibitory connections \ncelerity of dynamic  wave \ndistance between  the  peaks  of both waves \nbase level  of TF (=highest  available  weight) \nbandwidth of TF dynamic  wave \nThreshold of stabilisation (pruning stop) \nband .. vidth of DF  dynamic  wave \nbase  level of DF  (total proba.  of degeneration) \nprobability of A  alone  in  the stimulus set \nprobability of B alone  in  the stimulus set \nprobability of simultaneous s  A and  B \n\n3  RESULTS \n\n3.1  RESULTS  FORMAT \n\nAll Figures have the same format and summarize 40  runs per point unless otherwise \nspecified.  The  top  graph  presents  the  mean  slope  of appearance  of all  4  orders \nof complexity  (see  Table  1)  on  the  y  axis ,  as  a  function  of different  values  of the \nexperimentally manipulated parameter, on the x axis.  The bottom left graph shows \nthe mean slope for  order 2,  surrounded by  a gray area one standard deviation below \nand  above.  The  bottom  right  graph  shows  the  mean  slope  for  order  3,  also  with \na  I-s.d.  surrounding  area.  The  slopes  have  not  been  normalized,  and  come  from \nnetworks  whose  columns are 50  units  high, so  that  a slope of 1.0 indicates  that the \nnumber  of such  units  increase  in  average  by  one  unit  per  columns,  ie,  by  3  units \n\n\f", "award": [], "sourceid": 1142, "authors": [{"given_name": "Thomas", "family_name": "Rebotier", "institution": null}, {"given_name": "Jeffrey", "family_name": "Elman", "institution": null}]}