{"title": "An Input Output HMM Architecture", "book": "Advances in Neural Information Processing Systems", "page_first": 427, "page_last": 434, "abstract": null, "full_text": "An Input  Output  HMM Architecture \n\nDept.  Informatique et  Recherche \n\nDipartimento di Sistemi e Informatica \n\nPaolo Frasconi \n\nUniversita di Firenze  (Italy) \n\npaoloOmcculloch.ing.unifi.it \n\nYoshua Bengio* \n\nOperationnelle \n\nUniversite de  Montreal,  Qc  H3C-3J7 \n\nbengioyOIRO.UMontreal.CA \n\nAbstract \n\nWe  introduce  a  recurrent  architecture  having  a  modular structure \nand we formulate a training procedure based on the EM  algorithm. \nThe resulting model has similarities to hidden  Markov models, but \nsupports  recurrent  networks  processing style and allows  to exploit \nthe supervised  learning paradigm while using maximum likelihood \nestimation. \n\nINTRODUCTION \n\n1 \nLearning problems involving sequentially structured data cannot be effectively dealt \nwith static models such as feedforward networks.  Recurrent networks allow to model \ncomplex dynamical systems  and  can  store  and  retrieve  contextual  information in \na  flexible  way.  Up  until  the  present  time,  research  efforts  of supervised  learning \nfor  recurrent  networks  have  almost  exclusively  focused  on  error  minimization  by \ngradient  descent  methods.  Although  effective  for  learning  short  term  memories, \npractical  difficulties  have  been  reported  in  training  recurrent  neural  networks  to \nperform  tasks  in  which  the  temporal  contingencies  present  in  the  input/output \nsequences  span long intervals  (Bengio et al.,  1994;  Mozer,  1992). \nPrevious work on alternative training algorithms (Bengio et al.,  1994) could suggest \nthat  the  root  of the  problem  lies  in  the  essentially  discrete  nature  of the  process \nof storing information for  an  indefinite  amount of time.  Thus,  a  potential solution \nis  to  propagate,  backward  in  time,  targets  in  a  discrete  state  space  rather  than \ndifferential error information.  Extending previous work  (Bengio &  Frasconi,  1994a), \nin this paper we  propose a statistical approach to target propagation, based on the \nEM  algorithm.  We consider a parametric dynamical system with discrete states and \nwe introduce a modular architecture,  with subnetworks associated to discrete states. \nThe architecture can be interpreted as a statistical model and can be trained by the \nEM  or  generalized  EM  (GEM)  algorithms (Dempster et  al.,  1977), considering the \ninternal  state  trajectories  as  missing  data.  In  this  way  learning  is  decoupled  into \n\n\u2022 also,  AT&T Bell  Labs,  Holmdel,  N J  07733 \n\n\f428 \n\nYoshua  Bengio,  Paolo  Frasconi \n\na  temporal  credit  assignment  subproblem  and  a  static  learning  subproblem  that \nconsists of fitting parameters to the next-state and output mappings defined  by the \nestimated trajectories .  In order to iteratively tune parameters with the EM or GEM \nalgorithms, the system propagates forward and backward a discrete distribution over \nthe  n  states,  resulting  in  a  procedure  similar  to  the  Baum-Welch  algorithm  used \nto  train standard  hidden  Markov models  (HMMs)  (Levinson  et  al.,  1983) .  HMMs \nhowever  adjust  their  parameters  using  unsupervised  learning, whereas  we  use  EM \nin  a  supervised  fashion .  Furthermore,  the  model  presented  here  could  be  called \nInput/Output  HMM,  or  IOHMM,  because  it  can  be  used  to  learn  to  map  input \nsequences  to  output  sequences  (unlike  standard  HMMs,  which  learn  the  output \nsequence  distribution) .  This  model  can  also  be  seen  as  a  recurrent  version  of the \nMixture of Experts  architecture  (Jacobs et al. , 1991) , related  to the model already \nproposed in (Cacciatore and Nowlan, 1994).  Experiments on artificial tasks  (Bengio \n&  Frasconi ,  1994a)  have  shown  that  EM  recurrent  learning  can  deal  with  long \nterm dependencies  more effectively  than  backpropa~ation through  time and  other \nalternative algorithms.  However,  the model used  in  (Bengio &  Frasconi,  1994a)  has \nvery  limited representational capabilities and can only map an input sequence  to a \nfinal discrete  state.  In the present  paper we  describe an extended  architecture  that \nallows  to fully  exploit  both input  and output  portions of the data,  as  required  by \nthe supervised  learning  paradigm.  In  this  way, general  sequence  processing  tasks, \nsuch  as  production, classification, or prediction, can be dealt  with. \n\n2  THE PROPOSED  ARCHITECTURE \nWe  consider  a  discrete  state dynamical system  based  on  the following state space \ndescription: \n(1) \n\n) \nt-l,  t \n\n-\n-\n\nf(x \n\nx \nU \nt \nYt  =  9(Xt, Ut) \n\nwhere  Ut  E  R m  is  the  input  vector  at  time t,  Yt  E  R r  is  the  output  vector,  and \nXt  E  {I , 2, .. . , n}  is  a  discrete  state.  These  equations  define  a  generalized  Mealy \nfinite state machine, in which inputs and outputs may take on continuous values.  In \nthis paper, we  consider  a  probabilistic version  of these  dynamics, where  the current \ninputs and the current state distribution are used to estimate the state distribution \nand the output distribution for  the next time step.  Admissible state transitions will \nbe specified  by  a directed  graph 9  whose  vertices  correspond  to the model 's states \nand  the set  of successors  for  state j  is  Sj . \n\nThe system defined  by  equations  (1)  can  be modeled by  the recurrent  architecture \ndepicted  in  Figure  l(a) .  The  architecture  is  composed  by  a  set  of state  n etworks \nN j, j  =  1 .. . n  and a  set  of output  networks  OJ, j  =  1 . .. n.  Each  one of the state \nand  output  networks  is  uniquely  associated  to  one  of the  states,and  all  networks \nshare the same input Ut .  Each state network M has the task of predicting the next \nstate distribution, based  on  the  current  input  and  given  that  Xt-l  =  j .  Similarly, \neach  output network OJ  predicts  the output of the system, given  the current  state \nand  input.  All  the  subnetworks  are  assumed  to be static  and  they  are  defined  by \nmeans of smooth mappings Nj (Ut ; 9j ) and OJ (Ut; 1J j), where 9 j  and 1J j  are vectors \nof adjustable  parameters  (e.g.,  connection  weights).  The  ranges  of the  functions \nNj  0 may  be  constrained  in  order  to  account  for  the  underlying  transition  graph \n9 .  Each  output  'Pij ,t  of the state  subnetwork Nj  (at  time t)  is  associated  to  one \nof the successors  i  of state  j .  Thus  the  last  layer  of M has  as  many  units  as  the \ncardinality of Sj.  For convenience of notation, we suppose that 'Pij,t  are defined  for \neach  i, j  =  1, ... , n  and  we  impose the condition  'Pij ,t  = 0 for  each  i  not  belonging \nto S j .  The softmax function is used  in the last layer:  'Pij ,t  = e a,j ,t ILlEsj  ea l j,t,  j  = \ni  E Sj  where  aij, t  are intermediate variables that can be thought of as the \n1, ... , n , \n\n\fAn Input Output HMM Architecture \n\n429 \n\ncurrent \u2022\u2022 pectod output, \ngiven PIlat Input  Mquenc. \n\n11 t  EIYt  lull \n\ncurrent atilt. dlatrtbutton \n\nCt= Pl' t I Ul ) \n\ncu ... nt Input  '-'t \n\n(a) \n\nXt \n\nXt-l \n\n... \n... \n1  1 \n\nYt-l \n\nYt \n\nXt+l \n\n... \n..  .... 1  HMM \n\nYt+l \n\nXt-l \n\nX( \n\nXt+l \n\n\\.Yt-l  \\Y t  \n\n\\.Yt+l \n\nI \n\nI \n\nI \n\nIOHMM \n\nUt-l \n\nUt \n\nUt+l \n\n(b) \n\nFigure 1:  (a):  The proposed IOHMM  architecture.  (b):  Bottom:  Bayesian network \nexpressing  conditional dependencies  for  an  IOHMM;  top:  Bayesian  network  for  a \nstandard HMM \n\nactivations of the output  units of subnetwork N j .  In  this  way  L:7=1 'Pij ,t  =  1 Tij,t. \n\nThe vector 't E R n  represents  the internal state of the model and it is  computed as \n\na linear combination of the  outputs of the state  networks,  gated  by  the  previously \ncomputed internal  state: \n\nn \n\nwhere  IPj,t  = ['PIj,t, ... , 'Pnj,t]'. \noutput of the system 1Jt  E R r : \n\n't = L ( j ,t-IIPj,t \nOutput  networks  compete  to  predict  the  global \n\n(2) \n\nj=1 \n\nn \n\n1Jt  = L (jt1Jjt \n\nj=1 \n\n(3) \n\nwhere  1Jjt  E  R r  is  the  output  of subnetwork  OJ.  At  this  level ,  we  do  not  need \nto  further  specify  the  internal  architecture  of the  state  and  output  subnetworks. \nDepending  on  the  task,  the  designer  may decide  whether  to  include  hidden  layers \nand  what activation  rule  to use  for  the hidden  units. \nThis connectionist  architecture can  be also interpreted  as  a  probability model.  Let \nus  assume  a  multinomial distribution  for  the state  variable  Xt  and  let  us  consider \n\n't, the  main variable of the  temporal recurrence  (2).  If we  initialize the vector '0 \nprobabilities.  In  general,  we  obtain  relation  (it  = P(Xt  = i I un, having  denoted \nto  positive numbers summing to  1,  it can  be  interpreted  as  a  vector  of initial state \nwith ui  the subsequence  of inputs from  time 1 to t,  inclusively.  Equation  (2)  then \nhas  the following  probabilistic interpretation: \n\nP(Xt =  i  lui) =  L  P(Xt  =  i I Xt-I  =  j, ut}P(Xt-1 =  j  lui-I) \n\nn \n\n(4) \n\ni.e., the subnetworks N j  compute transition probabilities conditioned on the  input \nsequence  Ut: \n(5) \n\n. )  \n\n. I \n\nP( \n\n'Pij,t  = \n\nXt  =  ~  Xt-l = ), Ut \n\nj=l \n\nAs  in  neural  networks  trained  to  minimize  the  output  squared  error,  the  output \n1Jt  of  this  architecture  can  be  interpreted  as  an  expected  \"position  parameter\" \nfor  the  probability  distribution  of the  output  Yt.  However,  in  addition  to  being \nconditional on  an  input Ut,  this expectation  is  also conditional on the state  Xt,  i.e. \n\n\f430 \n\nYoshua  Bengio,  Paolo  Frasconi \n\n7]t  =  E[Yt I Xt,Ut] .  The actual form  of the output density, denoted  !Y(Yt ; 7]t),  will \nbe chosen  according to the task.  For example a multinomial distribution is suitable \nfor  sequence  classification,  or for  symbolic mutually exclusive  outputs.  Instead,  a \nGaussian  distribution  is  adequate  for  producing  continuous  outputs.  In  the  first \ncase we  use  a softmax function at the output of subnetworks OJ; in the second case \nwe  use  linear output units for  the subnetworks  OJ. \nIn order to reduce the amount of computation, we introduce an independency model \namong the variables involved in the probabilistic interpretation of the architecture. \nWe  shall  use  a  Bayesian  network  to  characterize  the  probabilistic  dependencies \namong  these  variables.  Specifically,  we  suppose  that  the  directed  acyclic  graph \n9  depicted  at  the  bottom of Figure  1 b  is  a  Bayesian  network  for  the  dependency \nmodel associated to the variables u I, xI, YI.  One of the most evident consequences \nof this independency model is  that only the previous state and the current input are \nrelevant  to determine  the  next-state.  This  one-step  memory property  is  analogue \nto the Markov  assumption in hidden  Markov models  (HMM).  In fact,  the Bayesian \nnetwork for  HMMs can be obtained by simply removing the Ut  nodes and arcs from \nthem  (see  top of Figure  Ib) . \n\n3  A  SUPERVISED  LEARNING ALGORITHM \nThe learning algorithm for  the proposed  architecture is derived from the maximum \nlikelihood principle.  The training data are a set of P  pairs of input/ output sequences \n(of length  Tp):  1)  =  {(uip(p),Yip(p));p  =  1 .. . P}.  Let  \u20acJ  denote  the  vector  of \nparameters obtained by  collecting all the parameters (Jj  and iJi of the  architecture. \nThe likelihood function  is  then given  by \n\np \n\nL(\u20acJ;  1)) = II p(Yip(p) I uip(p); \u20acJ). \n\n(6) \n\np=l \n\nThe output values  (used  here  as  targets)  may also be  specified  intermittently.  For \nexample,  in  sequence  classification  tasks ,  one  may  only  be  interested  in  the  out(cid:173)\nput YT  at the end  of each  sequence.  The modification of the likelihood to account \nfor  intermittent targets  is  straightforward.  According  to the  maximum likelihood \nprinciple,  the  optimal  parameters  are  obtained  by  maximizing  (6). \nIn  order  to \napply  EM  to  our  case  we  begin  by  noting  that  the  state  variables  Xt  are  not  ob(cid:173)\nserved.  Knowledge of the model's state trajectories  would  allow one to decompose \nthe  temporal  learning  problem  into  2n  static  learning subproblems.  Indeed,  if Xt \nwere  known,  the  probabilities (it  would  be  either  0  or  1 and  it  would  be  possible \nto train each subnetwork separately, without taking into account  any temporal de(cid:173)\npendency.  This  observation  allows  to  link  EM  learning  to  the  target  propagation \napproach discussed  in the introduction .  Note that if we  used  a Viterbi-like approxi(cid:173)\nmation (i.e., considering only the most likely path) , we  would indeed have  2n static \nlearning problems at each  epoch.  In  order  to we  derive  the learning  equations,  let \nus  define  the  complete  data  as  1)c  = {(uiP(p),yiP(p),xiP(p));p = 1 ... P}.  The \ncorresponding complete data l%-likelihood is \n\nIc(\u20acJ;1)c)  =  L...IOgP(YIP(P),ZlP(P) I u1P(p); \u20acJ). \n\n\"\"\" \n\nT \n\nT \n\nT \n\n(7) \n\np=l \n\nSince lc( \u20acJ;  1)c) depends  on  the hidden  state variables  it cannot  be maximized  di(cid:173)\nrectly.  The  MLE optimization is  then solved  by  introducing the auxiliary function \nQ(\u20acJ;  0) and iterating the following two,steps for  k  = 1, 2\n... :, \nr \nCompute Q(\u20acJ ; \u20acJ)  = E[lc(\u20acJ;  1)c)  1), \u20acJ] \n\nEstimation: \nMaximization:  Update the parameters as  0  t- arg max\u20acJ  Q( \u20acJ;  0) \n\n(8) \n\n\fAn Input Output HMM Architecture \n\n431 \n\nThe expectation of (7)  can  be expressed  as \n\nN \n\np  Tp  N \n\np=1  t=1  i=1 \n\nQ(0 ; 0) = L: L: L: (it!og P(Yt  I Xt  = i , Ut i 0) + L: hij,tlog<Pij,t \n\n(9) \nwhere  hij,t  = EIZitzj,t-l I uf, yf; 0J, denoting  Zit  for  an  indicator  variable = 1 if \nXt  =  i  and  0 otherwise.  The  hat  in  ( it  and  hij,t  means  that  these  variables  are \ncomputed  using  the  \"old\"  parameters  0 .  In  order  to compute  hij ,t  we  introduce \nthe  forward  probabilities  Qit  = P(YL Xt  = i ; uD  and  the  backward  probabilities \nf3it  = p(yf I Xt  = i , un, that  are  updated  as follows: \n\nj=1 \n\nf3it  =  fY(Yt;l1it) Lt <Pti(Ut+df3l,t+l \n\nQit  =  fY(Yt; l1it) Lt <pa(ut}Qt ,t-l . \n\nh \n\n-\nij ,t  -\n\nf3it Qj ,t-l<Pij (ut) \n\n\" \nwi QiT \n\n(10) \n\n(11) \n\nEach  iteration  of  the  EM  algorithm  requires  to  maximize  Q(0 ; 0).  We  first \nconsider  a  simplified  case,  in  which  the  inputs  are  quantized  (i.e.,  belonging \nto  a  finite  alphabet  {0\"1, \"\"  O\"K})  and  the  subnetworks  behave  like  lookup  ta(cid:173)\nbles  addressed  by  the  input  symbols  O\"t,  i.e.  we  interpret  each  parameter  as \nWi'k =  P(Xt  =  i I Xt-l  =  j , O\"t  =  k).  For  simplicity, we  restrict  the  analysis  to clas(cid:173)\nsification tasks and  we  suppose that targets  are specified  as  desired  final  states for \neach  sequence.  Furthermore,  no  output  subnetworks  are  used  in  this  particular \napplication of the algorithm.  In this case we  obtain the reestimation formulae: \n\nWijk  = \n\n\"  \"p\" \n\nw i ESj  wp=1 wt :Ut=k \n\n. \n\n.\n{j,t(j,t-l \n\u2022 \n\n,  +  T \n\"'T ' \n\n(12) \n\nIn general,  however,  if the subnetworks  have hidden  sigmoidal units, or use  a soft(cid:173)\nmax function  to constrain their outputs to sum to one,  the maximum of Q cannot \nbe found  analytically.  In  these  cases  we  can  resort  to  a  GEM  algorithm, that sim(cid:173)\nply  produces  an  increase  in  Q,  for  example  by  gradient  ascent.  In  this  case,  the \nderivatives of Q with  respect  to the parameters can  be  easily computed as follows. \nLet  Ojlt  be a  generic  weight in the state subnetwork  N j .  From equation  (9): \n\n8Q(0;0)  = L:L:L:hij,t_l_8<pij ,t \n<Pij,t  80jk \n\n80jk \n\nt \n\np \n\ni \n\n(13) \n\nwhere  the  partial derivatives  &:e~;t  can  be  computed using  backpropagation .  Sim(cid:173)\nilarly, denoting  with  t'Jik  a  generic  weight  of the  output  subnetwork  Oi,  we  have: \n\n8Q( 0; 0) \n\n8t'J \u00b7 \n,k \n\n87]a  t \n= L..JL..JL..J(i ,t~logfY(Yy;l1it) 8t'J .' \n,k \n\n'\" '\" '\" . \np \n\n8 \n7],t,t \n\nt \n\nt \n\n(14) \n\nwhere  ~;;:~t  are  also computed using  backpropagation.  Intuitively, the parameters \nare updated as if the estimation step of EM  had provided targets for  the outputs of \nthe 2n subnetworks, for each time t . Although GEM algorithms are also guaranteed \nto find  a  local  maximum of the  likelihood,  their  convergence  may be  significantly \nslower compared to EM.  In several experiments we  noticed that convergence can be \naccelerated  with stochastic gradient  ascent . \n\n\f432 \n\nYoshua  Bengio,  Paolo  Frasconi \n\n4  COMPARISONS \nIt appears  natural to find  similarities between  the  recurrent  architecture  described \nso far and standard HMMs (Levinson et al., 1983).  The architecture proposed in this \npaper differs  from  standard  HMMs  in two respects:  computing style  and learning. \nWith  IOHMMs,  sequences  are  processed  similarly to  recurrent  networks,  e.g.,  an \ninput  sequence  can  be  synchronously  transformed  into  an  output  sequence.  This \ncomputing style is real-time and predictions of the outputs are available as the input \nsequence is being processed.  This architecture thus allows one to implement all three \nfundamental  sequence  processing  tasks:  production,  prediction,  and  classification. \nFinally,  transition  probabilities  in  standard  HMMs  are  fixed,  i.e.  states  form  a \nhomogeneous  Markov  chain.  In  IOHMMs,  transition  probabilities  are  conditional \non the input and thus depend on time, resulting in an inhomogeneous  Markov chain. \nConsequently, the dynamics of the system (specified  by the transition probabilities) \nare  not fixed  but  are  adapted in time depending  on  the input sequence. \nThe  other  fundamental  difference  is  in  the  learning  procedure.  While  interesting \nfor  their  capabilities of modeling sequential  phenomena, a  major weakness  of stan(cid:173)\ndard  HMMs  is  their  poor  discrimination power  due  to  unsupervised  learning.  An \napproach  that has  been  found  useful  to improve discrimination in  HMMs  is  based \non  maximum mutual  information  (MMI)  training.  It  has  been  pointed  out  that \nsupervised learning and discriminant learning criteria like MMI are actually strictly \nrelated  (Bridle, 1989).  Although the parameter adjusting procedure we have defined \nis based on MLE, yf is used  as  desired output in response  to the input uf, resulting \nin  discriminant supervised  learning.  Finally, it is  worth  mentioning that a  number \nof hybrid approaches have been proposed to integrate connectionist approaches into \nthe  HMM  frame\\'Vork.  For  example in  (Bengio  et  al. ,  1992)  the  observations  used \nby  the  HMM  are  generated  by  a  feedforward  neural  network.  In  (Bourlard  and \nWellekens,  1990)  a feedforward  network is  used to estimate state probabilities, con(cid:173)\nditional  to  the  acoustic  sequence.  A  common feature  of these  algorithms and  the \none  proposed  in  this  paper  is  that  neural  networks  are  used  to extract  temporally \nlocal information whereas  a  Markovian system integrates  long-term constraints. \nWe  can  also  establish  a  link  between  IOHMMs  and  adaptive  mixtures  of experts \n(ME)  (Jacobs et al.,  1991).  Recently,  Cacciatore &  Nowlan  (1994)  have proposed a \nrecurrent  extension to the ME  architecture,  called  mixture  of controllers  (MC), in \nwhich the gating network has feedback  connections,  thus allowing to take temporal \ncontext into account .  Our IOHMM architecture can be interpreted as a special case \nof the  MC  architecture,  in  which  the  set  of state  subnetworks  play  the  role  of a \ngating network having a modular structure and second  order connections. \n\n5  REGULAR GRAMMAR INFERENCE \nIn  this  section  we  describe  an  application  of our  architecture  to  the  problem  of \ngrammatical inference.  In  this task  the learner  is  presented  a set  of labeled  strings \nand  is  requested  to  infer  a  set  of rules  that  define  a  formal  language.  It  can  be \nconsidered as a prototype for more complex language processing problems.  However, \neven  in  the  \"simplest\"  case,  i.e. \nregular  grammars,  the  task  can  be  proved  to \nbe  NP-complete  (Angluin  and  Smith,  1983) .  We  report  experimental  results  on \na  set  of regular  grammars  introduced  by  Tomita  (1982)  and  afterwards  used  by \nother researchers  to measure the accuracy of inference  methods based  on  recurrent \nnetworks  (Giles et al. , 1992;  Pollack,  1991;  Watrous and  Kuhn , 1992) . \nWe  used  a scalar output with supervision  on the final  output YT  that was  modeled \nas  a  Bernoulli  variable  fy (YT ; 7]T)  = 7]~T (1  -\n) l-YT,  with  YT  =  0  if the  string \nis  rejected  and  YT  =  1  if  it  is  accepted.  In  tbis  application  we  did  not  apply \n\n7] \n\n\fAn Input Output HMM Architecture \n\n433 \n\nTable  1:  Summary of experimental results on the seven  Tomita's grammars. \nGrammar \n\nConvergence \n\nAccuracies \n\nAverage  Worst  Best  W&K  Best \n1.000 \n1.000 \n.783 \n.609 \n.668 \n.462 \n.557 \n\n1.000 \n1.000 \n1.000 \n1.000 \n1.000 \n1.000 \n1.000 \n\n1.000 \n.834 \n.775 \n1.000 \n1.000 \n1.000 \n.815 \n\n1.000 \n.965 \n.867 \n1.000 \n1.000 \n1.000 \n.856 \n\nn* \n2 \n8 \n7 \n4 \n4 \n3 \n3 \n\n1 \n2 \n3 \n4 \n5 \n6 \n7 \n\nSizes \nFSA  min \n\n2 \n3 \n5 \n4 \n4 \n3 \n5 \n\n.600 \n.800 \n.150 \n.100 \n.100 \n.350 \n.450 \n\nexternal  inputs  to  the  output  networks.  This  corresponds  to  modeling  a  Moore \nfinite  state  machine.  Given  the  absence  of prior  knowledge  about  plausible state \npaths, we  used  an  ergodic transition graph (i.e., fully connected).In the experiments \nwe  measured  convergence  and  generalization  performance  using  different  sizes  for \nthe  recurrent  architecture.  For  each  setting  we  ran  20  trials  with  different  seeds \nfor  the initial weights.  We  considered  a  trial successful  if the  trained  network  was \nable to correctly  label  all  the  training strings.  The model  size  was  chosen  using  a \ncross-validation criterion  based  on  performance on  20  randomly generated  strings \nof length  T  ::;  12.  For  comparison,  in  Table  1  we  also  report  for  each  grammar \nthe  number of states  of the  minimal recognizing  FSA  (Tomita,  1982).  We  tested \nthe  trained  networks  on  a  corpus  of 213  - 1 binary strings  of length  T  ::;  12.  The \nfinal  results  are  summarized  in  Table  1.  The  column  \"Convergence\"  reports  the \nfraction of trials that succeeded  to separate the training set.  The next three columns \nreport averages and order statistics (worst and best trial) of the fraction of correctly \nclassified strings,  measured on the successful  trials.  For each  grammar these  results \nrefer  to the  model  size  n*  selected  by  cross-validation.  Generalization was  always \nperfect  on  grammars 1,4,5  and  6.  For  each  grammar, the  best  trial  also  attained \nperfect generalization.  These results compare very favorably to those obtained with \nsecond-order  networks  trained  by  gradient  descent,  when  using  the  learning  sets \nproposed  by  Tomita.  For  comparison, in  the last  column of Table  1 we  reproduce \nthe  results  reported  by  Watrous &  Kuhn  (1992)  in  the  best  of five  trials.  In most \nof the successful  trials  the  model  learned  an  actual  FSA  behavior  with  transition \nprobabilities asymptotically converging either  to 0 or to 1.  This renders  trivial the \nextraction of the corresponding  FSA . Indeed,  for  grammars 1,4,5, and 6,  we  found \nthat the trained  networks  behave exactly  like the minimal recognizing  FSA . \nA potential training problem is the presence of local maxima in the likelihood func(cid:173)\ntion.  For example, the number of converged  trials for  grammars 3,  4,  and 5 is quite \nsmall and the difficulty of discovering  the optimal solution might become  a serious \nrestriction for  tasks involving a large number of states.  In other experiments  (Ben(cid:173)\ngio &  Frasconi , 1994a) , we  noticed that restricting the connectivity of the transition \ngraph can significantly help to remove problems of convergence.  Of course,  this ap(cid:173)\nproach  can  be  effectively  exploited  only  if some  prior  knowledge  about  the  state \nspace is  available.  For example, applications of HMMs to speech  recognition always \nrely  on structured  topologies. \n\n6  CONCLUSIONS \nThere  are  still  a  number  of open  questions.  In  particular,  the effectiveness  of the \nmodel on tasks  involving large or  very  large state spaces needs  to be carefully eval(cid:173)\nuated.  In (Bengio &  Frasconi 1994b) we show that learning long term dependencies \nin  these models becomes  more difficult  as  we  increase  the connectivity of the state \n\n\f434 \n\nYoshua  Bengio,  Paolo  Frasconi \n\ntransition graph.  However,  because  transition probabilities of IOHMMs  change  at \neach t, they deal better with this problem of long-term dependencies  than standard \nHMMs.  Another interesting aspect to be investigated is the capability of the model \nto  successfully  perform  tasks  of sequence  production  or  prediction.  For  example, \ninteresting  tasks  that  could  also  be  approached  are  those  related  to  time  series \nmodeling and motor control learning. \n\nReferences \n\nAngluin, D.  and Smith, C.  (1983).  Inductive inference:  Theory and methods.  Com(cid:173)\n\nputing Surveys,  15(3):237-269. \n\nBengio, Y.  and  Frasconi, P.  (1994a) . Credit  assignment through time:  Alternatives \nto  backpropagation.  In  Cowan,  J.,  Tesauro,  G.,  and  Alspector,  J.,  editors, \nAdvances  in  Neural  Information  Processing Systems  6.  Morgan Kaufmann. \n\nBengio,  Y.  and  Frasconi,  P.  (1994b).  An  EM  Approach  to  Learning  Sequential \n\nBehavior.  Tech.  Rep.  RT-DSI/11-94, University of Florence. \n\nBengio, Y.,  De  Mori,  R.,  Flammia, G., and Kompe, R.  (1992).  Global optimization \nof a neural network-hidden markov model hybrid.  IEEE Transactions on Neural \nNetworks,  3(2):252-259. \n\nBengio,  Y.,  Simard, P.,  and  Frasconi,  P.  (1994) .  Learning  long-term dependencies \n\nwith gradient descent  is  difficult.  IEEE  Trans.  Neural  Networks,  5(2). \n\nBourlard,  H.  and Wellekens,  C.  (1990).  Links  between  hidden  markov models  and \nmultilayer perceptrons.  IEEE  Trans.  Pattern An.  Mach.  Intell.,  12:1167-1178. \n\nBridle,  J.  S.  (1989).  Training  stochastic  model  recognition  algorithms  as  net(cid:173)\n\nworks  can  lead  to maximum mutual information estimation of parameters.  In \nD.S.Touretzky, ed.,  NIPS2,  pages  211-217.  Morgan Kaufmann. \n\nCacciatore,  T.  W.  and  Nowlan,  S.  J.  (1994).  Mixtures  of  controllers  for  jump \nlinear  and  non-linear plants.  In  Cowan, J.  et.  al., editors,  Advances  in  Neural \nInformation  Processing Systems  6,  San Mateo,  CA.  Morgan Kaufmann. \n\nDempster, A.  P., Laird, N.  M.,  and Rubin, D.  B.  (1977).  Maximum-likelihood from \n\nincomplete data via the  EM  algorithm.  J.  Royal Stat.  Soc.  B,39:1-38. \n\nGiles, C.  L., Miller, C. B., Chen, D., Sun, G.  Z., Chen, H.  H., and Lee,  Y.  C.  (1992). \n\nLearning and extracting finite state automata with second-order recurrent  neu(cid:173)\nral  networks.  Neural  Computation,  4(3):393-405. \n\nJacobs,  R.  A.,  Jordan,  M.  I.,  Nowlan,  S.  J.,  and  Hinton,  G.  E.  (1991).  Adaptive \n\nmixture of local experts.  Neural  Computation,  3:79-87. \n\nLevinson,  S.  E.,  Rabiner,  L.  R.,  and  Sondhi,  M.  M.  (1983).  An  introduction  to \nthe application of the theory  of probabilistic functIons  of a  markov process  to \nautomatic speech recognition.  Bell System  Technical Journal, 64(4):1035-1074. \nMozer,  M.  C.  (1992).  The induction  of multiscale temporal structure.  In  Moody, \n\nJ. et.  al., eds,  NIPS 4 pages  275-282. Morgan  Kaufmann. \n\nPollack, J.  B.  (1991) .  The induction of dynamical recognizers.  Machine  Learning, \n\n7(2):196-227. \n\nTomita,  M.  (1982).  Dynamic construction of finite-state  automata from  examples \nusing hill-climbing.  Proc.  4th  Cog.  Science  Con!, pp.  105-108, Ann Arbor MI. \nWatrous,  R.  1. and  Kuhn,  G.  M.  (1992).  Induction of finite-state  languages using \n\nsecond-order  recurrent  networks.  Neural  Computation,  4(3):406-414 . \n\n\f", "award": [], "sourceid": 964, "authors": [{"given_name": "Yoshua", "family_name": "Bengio", "institution": null}, {"given_name": "Paolo", "family_name": "Frasconi", "institution": null}]}