{"title": "Connectionist Learning of Expert Preferences by Comparison Training", "book": "Advances in Neural Information Processing Systems", "page_first": 99, "page_last": 106, "abstract": null, "full_text": "99 \n\nConnectionist  Learning of Expert  Preferences  by \n\nComparison  Training \n\nGerald  Tesauro \n\nIBl\\f Thomas.1.  '''atson  Rcsearc11  Centcr \n\nPO  Box  704,  Yorktown  Heights,  NY  10598 USA \n\nAbstract \n\nA  new  training  paradigm,  caned  the  \"eomparison  pa.radigm,\"  is  introduced \nfor  tasks in which  a. network must  learn  to choose  a  prdcrred  pattern from  a \nset of n  alternatives,  based on  examplcs of Imma.n  expert  prderences.  In this \npa.radigm,  the inpu t  to  the network consists of t.wo  uf the  n  alterna tives,  and \nthe  trained  output is  the expert's judgement of which  pa.ttern is  better.  This \npara.digm is  applied  to  the lea,rning  of hackgammon,  a  difficult  board ga.me in \nwllieh  the  expert selects  a  move  from  a. set,  of legal  mm\u00b7es.  \\Vith  compa.rison \ntraining,  much  higher  levels  of performance  can  hc  a.chiew~d, with  networks \nthat  are  much  smaller,  and  with  coding  sehemes  t.hat  are  much  simpler  and \neasier  to  understand.  Furthermorf',  it  is  possible  to  set  up  the  network  so \ntha.t  it  always  produces  consisten t  rank-orderings . \n\n1. \n\nIntroduction \n\nThere is now widespread interest, in  tlH~ use of conncctillnist  networks fllr  real(cid:173)\nworld  practical  problem  solving.  The  principal  areas  of  applica.tion  which \nhave  been  studied  so  far  involvc  rela tiv('ly  low-level  signal  processing  and \npattern  recognition  t.asks.  However,  eOllllectionist  networks  might  also  he \nuseful  in  higher-level  tasks  whkh  a.re  curr('nt.l~\u00b7  tackled  hy  cxprrt  systems \na.nd  knowledge engineering approadl(,s [2].  In this pa per, 'vc considcr problem \ndomains in which tlte expert is givcn a s('t of 71.  alt.enulti,\u00b7es as input (71.  may be \neither sma.II  or large),  and  mnst  select.  t.}l<'  m()st  dC'sirtlhl('  ()T  most  prderable \nalternative.  This  type  of  task  occurs  rep('atccUy  throughout  the  domains \nof politics,  business,  eeonomics,  mcdicine,  a.nd  many  ot hers.  \\Vhcthcr  it is \nchoosing a fureign-policy option,  a.  w('apons cont,rador,  a  course of trea.tment \nfor  a  disease,  or  simply  what  to  have  for  dinner,  prohlems  requiring  choice \nare constan tly  being faced  and  801\\'(\\(1  by  111lma.n  experts. \n\nHow  might  a. learning  system  snch  as  a. cOlllH,ctionist  network  be  set  up  to \nlearn  to  ma.ke  such  choices  from  human  expert  cxampI~s?  The  immediately \nobvious  a,pproa,ch  is  to  train  the  network  to  produce  a  numerical  output \n\n\f100 \n\nTesauro \n\n\"score\"  for  ea,ch  input  alterna.tive.  To  make  a,  choice,  then,  une  would  have \nthe network  score  each  a.lterna,tive,  and select  t.he  alterna tive  with  the  high(cid:173)\nest score.  Since  the lea,rning system  learns from  examples,  it.  seems logical  to \ntrain  the  network  on  a  da.ta  base  of examples  in  which  a,  human  expert  has \nentered  a  numerical  score  for  ea,ch  possible  choice.  Howc\\'cr,  there  are  two \nmajor  problems  with  such  a,n  a,pproa,ch.  First, in  many  domains  in  which  n \nis  large,  it  would  be  tremendously  time-consuming  for  the  expert  to  create \na, data base in  which  each individual  a.lternative has  been painstaking  evalu(cid:173)\na,ted,  even  the  vast  number  of obviously  ba,d  alterna.tives  which  are  not even \nworth  considering.  (It  is  importa.nt  for  the  network  t,o  sec  examples  of ba,d \nalternatives,  otherwise it would  tend  t.o  produce  high  scores  for  everything.) \nl\\1:ore  importa.ntly, in ma,ny domains  human experts do  not  think in  terms of \nabsolute  scoring functions,  and it  would  thus  be extremely  difficult  to create \ntraining data containing  a,hsolute  scores,  because such  scoring  is  alien  to  the \nexpert's  wa,y  of thinking  a,hout  the  problem.  Inst.cad,  the  most  natural  way \nto  make  training  data is  simply  tu  record  the  expcrt in  adinn,  i.e.,  for  ea,eh \nproblem situation,  record each of the tlltcrna,th'cs hc htld  t.o  choose from,  a,nd \nrecord  which  one  he  actually  selected. \n\n. \n\nFor  these  reasons,  we  advoca.te  teaching  the  network  to  compare  pairs  of \nalterna.tives,  rather  than  scoring individual  aIterna.tivcs.  In ot.her  words,  the \ninput should  be  two  of the set  of n  alternatives,  and  thc output  should  be  a \n1  or  0  depending  on  which  of  the  two  alterna.tives  is  hetter.  From  a  set  of \nrecorded  huma,n  expert preferences,  one can  then  teach  thc  network  that  the \nexpert's choice  is  better  than  all  other  a.ltcrna tivcs. \n\nOne potential concern  raised  by  this  approach  is  tha t,  in  performance  mode \na,fter  the network is  trained, it might he neccssary  t.o  mtlke  n 2  comparisons  to \nselect  the best alterna.tive, whereas  only n  individual scor('s  tlrc  needed in the \nother  approa,ch.  However,  the  network  can  select  the  hcst  alterna.tive  with \nonly  n  compa.risons  by  going  through  the  list  of altern a tives  in  ordcr,  a.nd \ncompa,ring  the  current  alternative  wHh  thc  b('st  ait.erntltive  secn  so  fa.r.  If \nthe  current  alterna.tive is  better,  it  becomcs  the  ncw  1>('st  altcrnative,  and  if \nit is  worse,  it is  rejected.  Anot.hcr  poten tial  COIlcern  is  tha t  tI.  network  which \nonly  knows  how  t.o  compare  might  not  produce  a  consist('nt  rank-ordering, \ni.e.,  it  might  sa.y  that  alternativc  a  is  better  t.han  h,  b is  hdtN  t.han  c,  and \nc is  better  than  a,  and  then  one  do('s  not  know  which  alt('rnative  to  select. \nHowever,  we  shall see la.ter  that it is  possiblc  to  gllartlutc'c (,(lllsist('ncy  with  a \nconstrained  a.rchitecture  which  forccs  t.he  network  to  c:()mc~  IIp  wit.h  absolute \nnumerical  scores  for  individual  alterna tives. \n\nIn  the  following,  we  shall  exa.minc  the  applic~tI.ti()n  of  t.he  comparison  train(cid:173)\ning  pa.radigm  to  the  ga.me  of  backgammon,  as  considC'ra hIe  cxperience  ha.s \nalready  been  obtained  in  this  domain.  In  previous  papC'rs  [7,6]'  a  network \nwas  described  which lea.rned  to  pla.y  ba,ckgammon  from  an  expert data, base, \nusing  the so-called  \"back-proptl,ga.tion\"  learning  rule  [5J.  In  that system,  the \nnetwork  was  trained  to  score  individual  moyes.  In  other  words,  the  input \n\n\fConnectionist Learning of Expert Preferences \n\n101 \n\nconsists  of a  move  (drfined  by  the  initi::ll  position  bdnre  the  move  a.nd  the \nfinal  position  after  the  move),  and  the  desired  output  is  a.  real  number  indi(cid:173)\nca.ting  the  strength  of  the  move.  Henceforth  we  shall  refer  to  this  training \nparadigm  as  the  \"rcla.t.ive  score\"  par::ldigm. \n'Vhilc  this  approach  produced \nconsiderable  success,  it  had  a.  number  of serious  limitations. \n\\Ve  sha.ll  see \nthat  the  compa.rison  paradigm  solyes  one  of the  most  important  limita.tions \nof  t.he  previous  a.pproach,  with  the  rrsult  tha t  the  overall  performance  of \nUte  network  is  much  bet ter,  the  llum her  of  connedions  required  is  greatly \nreduced,  a.nd.  the  network's  input  coding  scheme  is  much  simpler  and  easier \nto  understa.nd. \n\n2.  Previous  backgammon networks \n\nIn  [7],  a  network  was  descrihed  which  learnrd  tl)  pla~\u00b7 fairly  good  ba.ckga.m(cid:173)\nmon  by  ba.ck-propa.gation  learning  of  a. large  expert  t.raining  set,  using  the \nrcla,tive  score  paradigm  describc(l  pr(,yiollsly.  After  tr::lining,  the  network \nwas  tested  both  by  measuring  its  prrformance  on  ::I  t('st  sct.  of positions  not \nused  in  training,  and  hy  a,ctual  game  pl::l)'  against  hunums  ::Ind  cOllventional \ncomputer  programs.  The  best  net.work  was  ablr  to  drfeat.  Sun  :Microsys(cid:173)\nterns'  Gammontool,  Ute  best  available  commercial  program,  hy a  sllbsta.ntial \nma.rgin,  but  it  was  still fa.r  from  human  expert-level  performance. \n\nThe  basic  conclusion  of [7]  was  that  it  was  possihle  to  achieve  decent  levels \nof performance  by  this  network  learning  procedure,  but  it,  was  not  a.n  ea.sy \nmatter,  and required suhst.antial hum::ln  intervention.  The choice  of a.  coding \nscheme for  the inpnt informa.t.ion,  for  rxamplr,  was fonn(l  to  be  an  extremely \nimportant issue.  The  hest  coding SclH'IrleS  contained  a. great  deal  of doma.in(cid:173)\nspecific  information.  The  best,  encoding  of the  \"raw\"  board  iuforma.tion  was \nin  terms of concepts that 111lman  exprrts use  to describe loc::II  tra.llsitions,  such \nas  \"slotting,\"  \"stripping,\"  et.c..  Also,  a  few  \"pre-colllputrd  features\"  were \nrequired  in  addition  t.o  the  raw  board  inform::lt.ion.  Thus  it  ,,\u00b7as  necessary \nto  be  a  domain  expert in  order  to  clpsign  a  suit,a hIe  ndwork  I.oding  sdteme, \na.nd  it  seemed  tha.t  the  only  way  t.o  discov('r  thr  hrst  cOlling  scheme  was \nby  painstaking  trial  and  error.  This  \"'::IS  som('\\dtat.  disappoiut,ing,  as  it  was \nhoped  tha.t  the  net\\vork lea rIling  l)foc('(illre  \\y()ul<l  alit lima t ically  produl.e  an \nexpert  ba.ckgammon  net.work  with  little  or  no  human  ('ffort. \n\n3.  Comparison  paradigul network set-up \n\n111  the  sta.nd.ard  pra.ctice  of  ha.c:k-propag::lt.inn,  a  compfHisoIt  paradigm  net(cid:173)\nwork  would  ha.ve  a.n  input layer,  (lIlC  ()r  morr  layrTs  of hidden  nnits,  and  a.n \noutput layer,  wit.h  full  connectivity hdwcrn  adjacell t  l::lyers.  The input.  layer \nwould represent, two final  board posit.ions  a  and  '\"  and  the out.put layer would \nha.ve  just  a  single  unit  t.o  represent  which  board  position  was  better.  The \n\n\f102 \n\nTesauro \n\nteacher signal for  the output  unit would  be  a, 1 if hoard  position  (l  was  better \nthan  b,  a,nd  a  0  if b was  better  tha.n  a. \n\nThe  proposed  compa,rison  pa.ra.digm  network  \"'ould  m;ercome  the  limita,t.ion \nof only  being  able  to  consider  individual  moves  in  isolation,  without  knO\\vl(cid:173)\nedge  of what  other  a.lt.erna,tives  arc  available.  In  addit.ion,  t.he  sophisticated \ncoding scheme that was developed  to encode transition informa tion would not \nbe needed,  since compa,risons  could  be based  solely  on  the final  board sta.tes. \nThe comparison  approa.ch offers  grca ter sensitivity in dist.inguishing  between \nclose  alternatives,  a,nd  as  stated  prc\"iously,  it corrcsp onds  more  closely  to \nthe  actual form  of human  expert  knowledge. \n\nThese  advantages  a,re  formida,ble,  hut.  t.here  are  some  import.ant.  problems \nwith  the  a.pproach  as  curren t.ly  descrihed.  One  technical  prohlem  is  tlta,t  the \nlearning  is  significantly  slower.  This  is  beca,use  271,  comparisons  per  t.ra.ining \nposition  a,re  prcsen ted  to  the  network,  where  11.  '\" 20,  wh('rC'tls  in  tllC  relative \nscore  approach,  only  about  3-4  moves  per  position  would  he  presented.  It \nwas  therefore  necessary  to  develop  a, number  of technical  tricks  to  increase \nthe  speed  of the simulator  code for  t.hi.s  specific  a.pplica tion  (to he  described \nin  a  future  publication). \n\nA  more fundamental problem  with  the a.pproa(\"h,  however, is  the issue of con(cid:173)\nsistency of network  comparisons .  Two  properties  arc  required  for  complete \nconsistency:  (1)  The  comparison  between  any  two  positions  mllst  be  unam(cid:173)\nbiguous,  i.e.,  if the  network says  that.  a is  bett('r  t.han  b when  a is  presented \non  the  left  a,nd  b  on  the  right,  it  !tact  hetter  say  that.  a  is  hetter  than  b  if \na  is  on  the  right  and  b is  on  the  left.  One  ca.n  show  tha t  t.his  requircs  t.he \nnetwork's  output  to  exaetly  invert  whenever  Ule  input.  hO(lrd  positions  are \nswapped.  (2)  The  compa,risons  must  he  transitire,  as  alluded  to  previously, \ni.e.,  if a  is  judged  better  than  h,  and  1J  is  judged  better  than  (',  the  network \nlta,d  better judge  a  to  be  better  than  c. \n\nSta.ndard  unconstrained  networks  hayc  no  gUll ra n tee  flf  sa t.isfying  eit.her  of \nthese  properties.  After  some  thought,  howe\\\"('I,  one  realizc's  that  the  output \ninversion  symmetry  can  be  enforcC'd  by  a  symmetry  rc'ltl tion  amongst  the \nweight.s  in  the  network,  and  that  the  transiti\"ity  ancl  rank-OHler  consist.ency \ncan be guaranteed by separability in  the  ardtitC'dure,  as illllstratC'd  in  Figurc \n1.  Here  we  see  tlla,t  t.his  network  really  consists  of t.wo  half-networks,  one  of \nwhich is only conceIllcd with t}w  evaluat.ion \"fhoard positi.on  (I,  and  the ot.her \nof which  is  concerned  only  wit.h  t.h~  C',\"alua t.iol\\  of hOH.T()  posit.inn  b.  (Duc  to \nthe indicated symmetry  relation,  nne neC'ds  only store oue half-network in  the \nsimulator  code.)  Each  half-network  mrly  have  on('  or  morc'  lay('rs  of hidden \nunits,  but  as  long  as  they  a,re  not  cross-coupled,  t.he  C'valuat.ion  of each  of \nthe  two  input  boa,rd  positions  is  hoilC'd  down  to  a  single  rC'al  number.  Since \nreal  numbers  always  ra,nk-order  consist.en tly,  t.he  ll('t\\,'ork's  compa,risons  a,re \nal ways  com;is ten t. \n\n\fConnectionist Learning of Expert Preferences \n\n103 \n\nfinal  position  (a) \n\nfinal  position  (b) \n\nFigure  1:  A  network design  for  cOlTlpari~()n trailling  wit h gnarallteed  consis(cid:173)\ntency  of comparisons.  \\Veight  groHps  han~ ~YlTlmet.ry rc1ati()ns  W 1 = W 2 \nand  W 3  ==  - W 4,  which  ensures  I hat  the  outPllt  cxact.Jy  in\\Trts upon  swap(cid:173)\nping  positions  in  the  input  arraJ'.  Separation  or  the  hidden  unils  cOllrlenses \nthe  evaluation  of each  final  hoard  position  into  a  single  real  IIIIJTlhcr,  thus \nensuring  transitivity. \n\nAn  importa.nt  a.dded  benefit  of  t.his  scheme  is  that  an  a.hsoillte  hoard  eval(cid:173)\nua.tion  fundion  is  obtained  in  each  half-network.  This  means  t.hat  the  net(cid:173)\nwurk,  to  the  extcnt  that  it.s  cvaluation  fllnct.ioll  is  ac-cnrat.e,  has  an  intrinsic \nunderstanding  of a. given  posit.ion,  as  opposed  to  HH'rely  heing  a hIe  to  de(cid:173)\ntect  features  which  correspond  t.o  good  moves.  As  has  heen  emphasil':cd  by \nBerliner  [1],  an  intrinsic  uudr.rst.a.llding  of the  position  is  crucial  for  play  at \nthe  highest  levels,  a.n<l  for  use  of  t,he  dOllhling  cnl)('.  Thnf-:,  this  a.pproach \ncan  serve  as  the  ba.sis  for  fut.ure  progress,  whNeas  thr  previous  approach  of \nscoring  moves  was  doomed  eventually  to  run  into  a  cirad  rud. \n\n4.  Results  of comparison  training \n\nThe  training  procedure for  the  comparison  paradigm  nC't.work  was  as  follows: \nNetworks \"\"'ere  set  up  with  289  input.  units  which enc(l(le  a  (ic'scription  of sin(cid:173)\ngle final  boa.rd  position,  varying nnmlH'rs (If hidden units,  and  a single output \nunit.  The  training  data was  taken  from  a  set  (If  40n  gamrs  ill  which  the  au(cid:173)\nthor  played  both  sides.  This  <ia ta  sd  con hlins  a  recording  of t.he  a.n t,hor's \npreferred  move for  each position,  and  no  ot.her  comments.  The engaged  posi(cid:173)\ntions in  the da.ta  set  weIe  selected  out, (discngflged  ra.cing  I)()sit.ions  were  not \nstudied)  and  divided  int.o  five  cat<'gories:  hearoff,  bearin,  opponent  bearoff, \nopponent  b~a.rin,  a.nd  a  default  ca t.egnry  ('overing  eVPlything  else.  In  each \n\n\f104 \n\nTesauro \n\nType of \n\nn.SP  net, \n\nCP  net \n\n_t~e_st_s~e~t _____ (~6_5_1-_1_2-_1~)~(_289-1)_ \n\nbcaroff \nhearin \nopp.  bea.roff\u00b7 \nopp.  bearin \nother \n\n.82 \n.54 \n.56 \n.60 \n.58 \n\n.83 \n.60 \n.54 \n.66 \n.65 \n\nTable  1:  Performance of neLs  of indicated  size on  respedin~ test.  sets, as mea(cid:173)\nsllred by  fraction of positions for  which  TIel  agrees wit h  lmTllrlll  expert choice \nof best  move.  HSP:  relative  score  paradigm,  CP:  comparison  paradigm. \n\ncategory,  200  positions  chosen  (I.t.  r(1UdOIU  were  set.  (lside  to  he  us('d  as  test(cid:173)\ning  <lat(l.;  tIte  remaining  da ta.  (a bon t.  1000  positions  in  ('(I eh  category  except \nt.he  dcfa.lllt  category,  for  which  a.bout  4000  positions  were  used)  was  used  to \ntra.iu  networks  which  spedalil':ed  in  each  category.  The  learning  algorithm \nllsed  was  st.a.nd(l.rd  back-propagation  with  IIWInelltllIn  a.nd  without  weight \ndecay. \n\nPerforma,nce  after  tr(lining  is  summaril':ed  in  Ta.hl('s  1  and  2.  Ta.hle  1  gives \nthe  performance  of  each  specialist  network  on  the  appropriate  set  of  test \npositions.  Results  for  the  comparison  par(ldigm  nebYllrk  are  shown  for  net(cid:173)\nworks  without hidden  units,  because it was fouud  that  the (lddition of hidden \nunits  did  not  improve  the  performance.  (This  is  discussed  in  the  following \nsection.)  \\Ve  contra.st.  these  results  with  result.s  of tr(lining  networks  in  the \nrela.tive  score  pa,radigm  on  the  same  tr(lilling  da.t(l  sds.  \"~e see  in  Ta hIe  1 \ntha.t  for  the  bearoff  and  oppon('nt.  hearofi' speeialists,  there  is  only  a  small \ncha.nge  in  perform(lnce  under  t.he  contpa.rison  p(lradigm.  Fllr  the  hea.rin  and \nopponcnt. bcarin specialists,  t,herr  is  an improv('ment. in IH'rformance of a.bout \n6 percenta.ge poin ts in each casco  For  t.his  pa.rtkular applica tion,  this is a. \"ery \nsubstantial improvement in perfoIIuance.  How('y<'I,  th(~ most,  import(lnt find(cid:173)\ning is  for  the  default  ca tegory,  which  is  much  l(lrgcr  and  mor('  difficult  than \nany  of  the  specialist  ea.t.egories.  Th('  d('f(lult  network's  prrformance  is  the \nkey factor  in determining  the syst('m's  IIvrr(lll  g(lme  prrfOrIll(lllCe.  'Vit.h  cum(cid:173)\npa.rison  t.rainillg,  we  find  an  improvc'Ineur.  in  perform(lll('e  from  58%  t.o  65%. \nGiven  the  sil':e  and  diffkllIt.y  of t,his  ca t('gory,  t his  ('(Ill  onl)- he  desnibcd  as  a. \nhuge improvement, in  performance,  and  is  all  t.h('  mllre  rem ark(l hIe  when  nne \nconsiders  th(lt.  t,he  comp(lrison  p(I.radigm  net has only  30()  w('ights,  (IS  opposed \nto  8000  weight,s  for  the  relative score  paradigm  net. \n\nNext,  a  combined  game-playing  s)-stC'ln  was  s('t.  up  llsing  t.hr  the  specialist. \nnets  for  all  engaged  posit.ions.  (The  G(llTlmontool  evaluat.ion  function  was \ncalled  for  ra.cing  posit.ions.)  Results  are  given  in  T(lble  2.  Ag(linst,  Gammon(cid:173)\ntool  itself,  the  pcrforma.nce  und('r  thr  comparison  p(lr(l(ligm  improves  from \n59%  to  64%.  Against  the  a.uthor  (and  tcachE'r),  the  pC'rforma.nce  improves \nfrom  a.n  estimat,ed  35%  (since  the  nsp  net.s  (Ire  so  hig  and  slow,  a.ccnra,te \n\n\fConnectionist Learning of Expert Preferences \n\n105 \n\nOpponent \nGa,mmon tool \nTesauIO \n\nRSP  nets \n.59  (500  games) \n.35  (l00 ga,mes) \n\nCP  nets \n.64  (2000  games) \n.42  (400  games) \n\nTable  2:  Game-playing  performance  of composite  network  systems  against \nGammontool and against the author, as measured by fractioTl  of games won, \nwithout  counting gammons or backgammons. \n\nstatistics  could  not  be obtained)  to about  42%. \n\nQualitatively, one  notices  a  subst.a,ntial  overall  improvement in  the  new  net(cid:173)\nwork's level of pla,y.  But what,  is most, striking is the  nct.work's  In)lst case be(cid:173)\nha,vior.  The previous  relative-score  network  ha.d  particularly  bad  worst-case \nbeha,vior:  about  once  every  other  ga,me,  the  network  would  make  an  atro(cid:173)\ncious  blunder  wltich  would  seriously  jeopardize  its  dlances  of winning  that \nga,me  [6].  An alarming fradion  of these  blunders ,,,,ere  seemingly  random and \ncould  not  be  logically  explained.  The  new  comparison  para,digm  network's \nwOIst-ca.se  behavior  is  vastly  improvcd  in  this  rega,rd.  The  frcquency  and \nseverity  of its  mistakes  are  significantly  reducE!d,  hut  more  imp(lrtantly,  its \nmista,kes  a.re  understandable.  (Some of the improvement in  this  respect  may \nbe dne  to  the  elimination  of the  noisy  teacher  signal  descrihed  in  [7].) \n\n5.  Conclusions \n\n\\Ve  have  seen  that,  in  the  domain  of backgammon,  the  int.roduct.ion  of the \ncomparison  training  pa.ra,digm  has  result.ed  in networks  whkh  perform  much \nbetter, with vastly reduced numbers Ilfweights,  a.ud with input. coding schemes \ntha.t  a.re  much  simpler  and  easier  t.o  understand.  It was  surprising  that snch \nhigh  performa.nce  could  be obtained  in  \"perceptrou\"  networks,  i.E!.,  networks \nwithou t  hidden  units.  This  reminds  us  t.hat.  one  should  not.  summarily  dis(cid:173)\nmiss perceptrons as  uninteresting or unworthy of study hecause  they arc only \nca.pable of lea.rning linearly separable funct.ions  [3].  A  su hst.an tial component \nof many  difficult  real-world  problems  may  lie  in  the liu('arly  separable  spec(cid:173)\ntrum,  and  thus it  makes  sense  to  try  perceptIons  at least  as  a  first  attempt. \nIt was  also  surprising  tha.t  the  use  (If hidden  units in  the  comparison- trained \nnetworks  does  not  improvc  the  pcrformanc('.  This  is  nn('xplained,  a.nd  is  t.he \nsubject of current resea.rch.  It is,  however,  not  without  precedent:  in  at least \none  other  real-world  application  [4],  it  has  heen  found  tha.t  networks  with \nhidden  units do  not  pcrform  any  bettE'r  than netwllrks  without  hidden  units. \n\n?vIore  generally,  one  might  conclude  t.ha.t,  in  training  a. neural  network  (or \nindeed  a.ny  learning  system)  from  human  expert.  E'xamples  in  a  cumplex  do(cid:173)\nmain,  there should  he a.  good match hetween  Ute  na t,ural  form  of the expert's \nknowledge  and  the  method  by  which  the  net.work  is  trained.  For domains  in \nwhich the expert must seh~ct a preferred  alt.crnative from  a set of alternatives, \n\n\f106 \n\nTesauro \n\nthe expert  naturally  thinks  in  terms  of comparisons  a.mongst  the  top few  a.l(cid:173)\nterna.tives,  and  the  compa.rison  paradigm  proposed  here  takes  advantage  of \nthat fact.  It would  be possihle in  principle  to  train  a  network  using  absolute \nevaluations,  hut  the  crea.tion  of sueh  a.  training  set  might  he  too  difficult  to \nunderta.ke  on  a  large  scale. \n\nIf the  above  discussion  is  coned,  then  the  comparisou  pa.ra.digm  should  be \nuseful  in  ot,her  applications  involving  expert  choice,  and  in  other  lea.rning \nsyst,ems  besides  connectionist  networks.  Typically  expert  systems  a.re  hand(cid:173)\ncrafted  by  knowledge  engineers,  ra.ther  than  learned from  human  expert  ex(cid:173)\namples;  however,  there has recently been some interest in sl1pervis(~d lea.rning \napproa.ches.  It will  be interesting  to see if the compa.rison  paradigm proves to \nbe  useful  when  supervised  lea.rning  procedures  are  applied  t,o  otller  domains \ninvolving  expert choice.  In  using  the compa.rison  paradigm, it.  will  be impor(cid:173)\ntant  to  ha.ve  some  way  to  gua.ra.nt,ee  that  the  syst,em's  comparisons  will  be \nunambiguous  and  t,ra.nsitive.  For feed-forward  networks, it was  shown in  this \npa.per  how  to  gua.rantee  this  using  symmetric,  separa.t.ed  nrtworks;  it should \nbe possible  to impose similar  constraints Oil  otll<'r learning systems  to  enforce \nconsistency. \n\nReferences \n\n[l}  H.  Berliner,  \"On the  construction or  c\\\"aluation  runctions  for  large  domains,\" \n\nProc.  of IleA T (1979) 53--55. \n\n[2]  S.  1.  Gallant,  \"Conncctionist  r.xpert  systems,\"  Comm.  l\\eM  31,  152 -- 169 \n\n(1988). \n\nP}  M.  Minsky  and  S.  Papcrt,  Pcrceptrons,  MIT Press,  Camhridge  ~fA (1969). \n\n[,1]  N.  Qian and  T.  J.  Sejnowski,  \"Predicting the'  secondary  slrucl,nre  of globular \n\nproteins  using  neural  nctwork  models,\"  .1.  Mol.  Bioi.  202,  R(jfi  8R,1 (1988). \n\n[5]  D.  E.  Rumelarl  and  J.  L.  ~lcClellanrl  (cds.),  Parallel J>i.<;frifllltecl  Processing: \nExplorations  in  I.Ire  Microsirucl.lIl'r  or  Cogn;t,;on,  Vt)ls.  I  and  2,  l\\f1T  Prrss, \nCambrioge  MA  (1986). \n\n[6}  G.  Tesauro,  \"Neural  network  ddral.s  creal.or  in  hack~annl1(ln match.\"  Univ. \nof ll1inoiR,  Center for  Complex  S~' strll1s Technical  Heporl.  CCSl1-8R-6  (1988). \n\n[71  G.  Tesauro  and  T.  J.  Sejnowski,  \"/\\  parallel  nei.work  that  ]rarns  to  play \n\nbackgammon,\"  A rtificialTntell;gmrcc,  in  press  (H)89) . \n\n\f", "award": [], "sourceid": 142, "authors": [{"given_name": "Gerald", "family_name": "Tesauro", "institution": null}]}