{"title": "Pattern Playback in the 90s", "book": "Advances in Neural Information Processing Systems", "page_first": 827, "page_last": 834, "abstract": null, "full_text": "Pattern Playback in the '90s \n\nMalcolm Slaney \n\nInterval Research Corporation \n\n180 l-C Page Mill Road, \n\nPalo Alto, CA 94304 \nmalcolm@interval.com \n\nAbstract \n\nDeciding  the  appropriate  representation  to  use  for  modeling  human \nauditory processing  is  a critical issue  in  auditory  science. While  engi(cid:173)\nneers have successfully performed many single-speaker tasks with LPC \nand  spectrogram  methods,  more  difficult  problems  will  need  a  richer \nrepresentation. This paper describes a powerful auditory representation \nknown as the correlogram and shows how this non-linear representation \ncan be converted back into  sound, with no  loss  of perceptually  impor(cid:173)\ntant information.  The correlogram  is  interesting because  it is  a  neuro(cid:173)\nphysiologically  plausible  representation  of  sound.  This  paper  shows \nimproved  methods  for  spectrogram  inversion  (conventional  pattern \nplayback), inversion of a cochlear model, and inversion of the  correlo(cid:173)\ngram representation. \n\n1  INTRODUCTIONl \n\nMy interest in auditory models and perceptual displays [2]  is motivated by the problem of \nsound  understanding,  especially  the  separation  of speech  from  noisy  backgrounds  and \ninterfering  speakers.  The  correlogram  and  related  representations  are  a  pattern  space \nwithin which sounds can be \"understood\" and \"separated\" [3][4]. I am therefore interested \nin resynthesizing  sounds  from  these representations  as  a  way  to  test and evaluate sound \nseparation algorithms, and as a way to apply sound separation to problems such as speech \nenhancement. The conversion of sound to a correlogram involves  the intermediate repre(cid:173)\nsentation  of  a  cochleagram,  as  shown  in  Figure  1,  so  cochlear-model  inversion  is \naddressed as one piece of the overall problem. \n\n1.  Much of this work was performed by Malcolm Slaney, Daniel Naar and Rich(cid:173)\nard F. Lyon while all three were employed at Apple Computer. The mathematical \ndetails of this work were presented at the  1994ICASSP[I]. \n\n\f828 \n\nMalcolm  Slaney \n\nWaveform \n\nj 1r--*T'T~\"\"\"I-~!\"'T!-~~-'-ooI-'1  ::::S:ec::ti :n:: h \n\nCochleagram \n\nCorrelogram \n\n.... t--'~~'--t.:lfga..~ l!!!Jnme \n\n------------~ \n\n------------~ \n\nFigure 1. Three stages in low-level auditory perception are shown here. Sound waves are con(cid:173)\nverted into a detailed representation with broad spectral bands, known as cochleagrams. The \ncorrelogram then summarizes the periodicities in the cochleagram with short-time autocorrela(cid:173)\ntion. The result is a perceptual movie synchronized to the acoustic signal. The two inversion \nproblems addressed in this work are indicated with arrows from  right to left \n\nThere are three factors which can be used to judge the quality of an auditory model: psy(cid:173)\nchoacoustic  comparisons,  neurophysiological  plausibility,  and does  it represent  the  per(cid:173)\nceptually  relevant  information?  First,  the  correlogram  has  been  shown  to  simply  and \naccurately predict human pitch perception [5]. The neurophysiological basis for the corre(cid:173)\nlogram has not been found, but there are neural circuits performing the same calculation in \nthe mustached bat's echolocation  system [6].  Finally,  from an information representation \npoint of view,  does  the  correlogram preserve the salient information? The results of this \npaper show that no information has been lost. Since the psychoacoustic, neurophysiologi(cid:173)\ncal, and information representation measures  are  all positive, perhaps the correlogram is \nthe basis of most auditory processing. \n\nThe  inversion techniques  described  here  are  important because  they  allow  us  to  readily \nevaluate the results  of sound separation models that \"zero out\"  unwanted portions of the \nsignal in the correlogram domain. This  work extends the convex projection approach of \nIrino [7]  and Yang [8]  by considering a different cochlear model, and by including the cor(cid:173)\nrelogram inversion. The convex projection approach is well suited to \"filling in\" missing \ninformation.  While  this  paper  only  describes  the  process  for  one  particular  auditory \nmodel, the techniques are equally useful for other models. \n\nThis paper describes three aspects of the problem: cochleagram inversion,  conversion of \nthe  correlogram into  spectrograms,  and spectrogram inversion. A number of reconstruc(cid:173)\ntion options are explored in this paper. Some are fast, while other techniques use time-con(cid:173)\nsuming iterations to produce reconstructions perceptually equivalent to the original sound. \nFast versions  of these algorithms  could  allow  us  to  separate  a  speaker's voice  from the \nbackground noise in real time. \n\n2  COCHLEAGRAM INVERSION \n\nFigure 2 shows  a block diagram of the cochlear model  [9]  that is  used in this  work. The \nbasis of the model is a bank of filters,  implemented as  a cascade of low-pass filters,  that \nsplits the input signal into broad spectral bands. The output from each filter in the bank is \ncalled a channel.  The energy in each channel is  detected and used to  adjust the channel \ngain,  implementing a simple model  of auditory sensitivity  adaptation, or automatic gain \ncontrol  (AGe). The half-wave rectifier  (HWR) detection  nonlinearity provides  a  wave(cid:173)\nform for each channel that roughly represents the instantaneous neural firing  rate at each \nposition along the cochlea. \n\nE \n~---+ \n> \n~ \n\nCochlear \nFilterbank \n\nDetector \n(HWR) \n\nAdaptation \n\nor \nAGC \n\nFigure 2. Three stages of the simple cochlear model used in this paper are shown above. \n\n\fPattern  Playback in the  '90s \n\n829 \n\nThe cochleagram is converted back into sound by reversing the three steps shown in Fig(cid:173)\nure 2. First the AGe is divided out, then the negative portions of each cochlear channel are \nrecovered by using the fact that each channel is spectrally limited. Finally, the cochlear fil(cid:173)\nters  are inverted by running the filters  backwards, and then correcting the resulting  spec(cid:173)\ntral slope. \n\nThe AGe stage in this cochlear model is controlled by its own output. It is a combination \nof a multiplicative gain and a simple first-order filter to track the history of the output sig(cid:173)\nnal. Since the controlling signal is directly available, the AGe can be inverted by tracking \nthe output history and then dividing instead of multiplying. The performance of this algo(cid:173)\nrithm is described by Naar [10] and will not be addressed here. It is worth noting that AGe \ninversion becomes more difficult as the level of the input signal is raised, resulting in more \ncompression in the forward path. \n\nThe next stage in the inversion process can be done in one of two ways. After AGC inver(cid:173)\nsion, both the positive values of the signal and the spectral extant of the signal are known. \nProjections onto convex sets  [11], in this case defined by the positive values of the detec(cid:173)\ntor output and the  spectral extant of the cochlear filters,  can be used to find  the original \nsignal. This is shown in the left half of Figure 3. Alternatively, the spectral projection filter \ncan be combined with the next stage of processing to make the algorithm more efficient. \nThe  increased efficiency  is due  to  better match  between  the  spectral  projection and the \ncochlear filterbank,  and due to the simplified computations  within each iteration. This  is \nshown in the right half of Figure 3. The result is an algorithm that produces nearly perfect \nresults with no iterations at all. \n\nFilter \nBank \n\nInversion \n\nSpectral \nProjection \n\nTemporal \nProjection \n\nE \n-go \n~Qi \n0> \nL... __  .....  CI)~ \n\nC  Other --110..  Inversion \nhannels-\n\nAGC \n\nInversion \n\n~ \n~ \n~ Q) \n~ \n8 \no \n\nFigure 3. There are two ways to use convex projections to recover the information lost \nby the detectors. The conventional approach is shown on the left. The right figure \nshows a more efficient approach where the spectral projection has been combined with \nthe filterbank inversion \n\nFinally, the multiple outputs from the cochlear filterbank are converted back into a single \nwaveform  by  correcting  the  phase  and  summing  all  channels.  In  the  ideal  case,  each \ncochlear channel contains a unique portion of the spectral energy, but with a bit of phase \ndelay and amplitude change. For example, if we run the signal through the same filter the \nspectral content does not change much but both the phase delay and amplitude change will \nbe doubled. More interestingly, if we run the signal through the filter backwards, the for(cid:173)\nward and backward phase changes cancel out. After this phase correction, we can sum all \nchannels and get back the Original waveform, with a bit of spectral coloration. The spec(cid:173)\ntral coloration or tilt can be fixed with a simple filter. A more efficient approach to correct \nthe  spectral  tilt  is  to  scale  each  channel  by  an  appropriate  weight  before  summing,  as \nshown in Figure 4. The result is a perfect reconstruction, over those frequencies where the \ncochlear filters are non-zero. \n\nFigure 5 shows results from the cochleagram inversion procedure. An impulse is shown on \nthe left,  before  and after  10 iterations of the  HWR inversion (using the  algorithm on the \nright half of Figure 3).  With  no  iterations  the result is  nearly  perfect,  except for  a bit of \nnoise near the center. The overall curvature of the baseline is due to the fact that informa-\n\n\f830 \n\n~ \n\nc:UI \n~'S \n.s5 \n.... 0-\niIO \n\nTime \n\nReversed \nFilter for \nInversion \n\nIIR or FIR \nFilter to \nCorrect \n\nTilt \n\nE \n-0 .... \n~ \n;:'0> \n0> \nCJ)~ \n\n~ \n\nc:,a \nas;:, \n\u20ac,e. \n0>;:, \nitO \n\nMalcolm  Slaney \n\nTime \n\nReversed \nFilter for \nInversion \n\nE \n-0 .... c:.g \n;:'0> \n0> \nCJ)~ \n\nFigure 4. Two approaches are shown here to invert the filterbank. The left diagram shows the \nnormal approach, the right figure shows a more efficient approach where the spectral-tilt filter \nis converted to a simple multiplication. \n\ntion near DC has been lost as it travels through the auditory system and there is no way to \nrecover it with the information that we have. A more interesting example is shown on the \nright.  Here the word \"tap\" 1 has been reconstructed,  with and without the AGC inversion. \nWith the AGe inversion the result is nearly identical to the original. The auditory system \nis very sensitive to onsets and quickly adapts to steady state sounds like vowels. It is inter(cid:173)\nesting to compare this to the reconstruction withoutAGC inversion. Without the AGC, the \nresult is  similar to  what the ear hears,  the onsets are more prominent and the vowels are \ndeemphasized. This is  shown in the right half of Figure 5. \n\nImpulse inversion \nwith no iterations \n\nImpulse iteration \nwith  10 iterations \n\n\"Tap\" reconstruction  \"Tap\" Reconstruction \nwith AGC Inversion  without AGC Inversion \n\nFigure 5. The cochlear reconstructions of an impulse and the word \"tap\" are shown here. The \nfirst and second reconstructions show an impulse reconstruction with and without iterations. \nThe third and fourth waveforms are the word \"tap\" with and without the AGe inversion. \n\n3  CORRELOGRAM INVERSION \n\nThe correlogram is an efficient way to capture the short-time periodicities in the auditory \nsignal.  Many  mechanical  measurements  of the  cochlea have  shown  that the  response  is \nhighly non-linear. As  the signal level changes there are large variations in the bandwidth \nand center frequency of the cochlear response. With these kinds  of changes, it is difficult \nto imagine a system that can make sense of the spectral profile. This is especially true for \ndecisions like pitch determination and sound separation. \n\nBut through all these changes in the cochlear filters,  the timing information in the signal is \npreserved. The spectral profile, as  measured by the cochlea, might change, but the rate of \nglottal pulses is preserved. Thus I believe the auditory system is based on a representation \nof sound that makes short-time periodicities apparent. One such representation is  the cor(cid:173)\nrelogram. The correlogram measures the temporal correlation within each channel, either \nusing  FFfs which are  most efficient in computer implementations,  or neural delay  lines \nmuch like those found in the binaural system of the owl. \n\n1. The syllable \"tap\", samples 14000 through 17000 of the \"trainldr5/fcdfll \nsxl06/sx106.adc\" utterance on the TIMIT Speech Database, is used in all voiced \nexamples in this paper. \n\n\fPattern  Playback in  the  '90s \n\n831 \n\nThe process of inverting the correlogram is simplified by noting that each autocorrelation \nis related by the Fourier transform to a power spectrum. By combining many power spec(cid:173)\ntrums into a picture, the result is a spectrogram. This process is  shown in Figure 6. In this \nway,  a separate spectrogram is  created for each channel. There are known techniques for \nconverting  a  spectrogram,  which  has  amplitude  information  but  no  phase  information, \nback into the original waveform. The process of converting from a spectrogram back into \na  waveform  is  described  in  Section  4.  The  correlogram  inversion  process  consists  of \ninverting many  spectrograms  to  form an estimate of a cochleagram. The cochleagram is \ninverted using the techniques described in Section 2. \n\nFrame 42 of Correlogram \n\nFrame 43 of Correlogram \n\nIFFTto get \nof spectrogram \n\nSpectrogram \n\nInversion  ., \n\nOne line of Cochleagram \n\nTime \n\nTime \n\nFigure 6.  Correlogram inversion is possible by noting that each row of the correlogram con(cid:173)\ntains the same information as a spectrogram of the same row of cochleagram output. By con(cid:173)\nverting the correlogram into many spectrograms, the spectrogram inversion techniques \ndescribed in Section 4 can be used. The lower horizontal stripe in the spectrogram is due to the \nnarrow passband of the cochlear channel. Half-wave rectification of the cochlear filter output \ncauses the upper horizontal stripes. \n\nOne important improvement to the basic method is possible due to the special characteris(cid:173)\ntics  of the correlogram. The essence  of the  spectrogram inversion problem is to recover \nthe phase information that has been thrown away. This is an iterative procedure and would \nbe costly ifit had to be performed on each channel. Fortunately, there is quite a bit of over(cid:173)\nlap between cochlear channels. Thus the phase recovered from one channel can be used to \ninitialize  the  spectrogram inversion for  the  next  channel.  A difficulty  with  spectrogram \ninversion is that the absolute phase is lost. By using the phase from one channel to initial(cid:173)\nize the next, a more consistent set of cochlear channel outputs is recovered. \n\n4  SPECTROGRAM INVERSION \n\nWhile spectrograms are not an accurate model of human perception, an implementation of \na correlogram includes the calculation of many spectrograms. Mathematically, an autocor(cid:173)\nrelation calculation is similar to a spectrogram or short-time power spectrum. One column \nof a conventional spectrogram is related to  an autocorrelation of a portion of the original \nwaveform ~y a Fourier transform (see Figure 6). Unfortunately, the final  representation of \nboth spectrograms and autocorrelations is missing the phase information. The main task of \na spectrogram inversion algorithm is to recover a consistent estimate ofthe missing phase. \nThis process is not magical, it can only recover a signal that has the same magnitude spec(cid:173)\ntrum as the original spectrogram. But the consistency constraint on the time evolution of \nthe signal power spectrum also constrains the time evolution of the spectral phase. \n\n\f832 \n\nMalcolm  Slaney \n\nThe basic procedure in spectrogram inversion [12]  consists  of iterating between the time \nand the  frequency  domains.  Starting from  the frequency  domain,  the  magnitude  but not \nthe phase is known. As an initial guess, any phase value can be used. The individual power \nspectra are inverse Fourier transformed and then summed to arrive at a single waveform. \nIf the original spectrogram used overlapping windows of data, the information from adja(cid:173)\ncent windows either constructively or destructively interferes to estimate a waveform.  A \nspectrogram of this  new  data is  calculated, and the phase is  now  retained. We  know  the \noriginal magnitude was correct. Thus we can estimate a better spectrogram by combining \nthe original magnitude information with the new phase information. It can be shown that \neach iteration will reduce the error. \n\nFigure 7  shows an outline of steps  that can be used to  improve the consistency of phase \nestimates during the first iteration. As  each portion of the waveform is  added to  the esti(cid:173)\nmated signal,  it is possible to  add a linear phase so  that each waveform lines up  with the \nproceedings segments. The algorithm described in the paragraph above assumes an initial \nphase of zero. A more likely phase guess is  to  choose a phase that is consistent with the \nexisting data. The result with no iterations is a waveform that is often closer to the original \nthan that calculated assuming zero initial phase and ten iterations. \n\nThe total computational cost is minimized by combining these improvements with the ini(cid:173)\ntial phase estimates from  adjacent channels of the  correlogram. Thus when inverting the \nfirst channel of the correlogram, a cross-correlation is used to pick the initial phase and a \nfew  more iterations insure a consistent result. After the first channel, the phase of the pro(cid:173)\nceeding channel is used to  initialize  the  spectrogram inversion and only a few  iterations \nare necessary to fine tune the waveform. \n\nReconstructions from \nsegments 1 through N \n\no \n\n400 \n\nCross Correlation \n\nRotated Segment N+ 1 \n\nhi\\;\\; I \n\n400 \n\no \n\nSegmentN+1 \n\nb;&/\\;  I \n\n400 \n\no \n\nNew Reconstruction \n\n300 \n\nMaximize fit by \nchOOSing peak \n\no \n\nFigure 7. A procedure for adjusting the phase of new segments when inverting a spectrogram is \nshown above. As each new segment (bottom left) is converted from a power spectrum into a \nwaveform, a linear phase is added to maximize the fit with the existing segments (top left.) The \namount of rotation is determined by a cross correlation (middle). Adding the new segment with \nthe proper rotation (top right) produces the new waveform (bottom right.) \n\n\fPattern  Playback in  the  '90s \n\n833 \n\n5  PUTTING IT TOGETHER \n\nThis paper has described two steps to convert a correlogram into a sound. These steps are \ndetailed below: \n\nI) \n\nFor each row of the correlogram: \na) \n\nb) \n\nc) \n\nConvert the autocorrelation data into power spectrum (Section \n3). \nUse  spectrogram inversion (Section 4)  to  convert the  spectro(cid:173)\ngrams into an estimate of cochlear channel output. \nAssemble the results of spectrogram inversion into an estimate \nof the cochleagram. \n\n2) \n\nInvert the cochleagram using the techniques described in Section 2. \n\nThis process is diagrammed in Figures  I and 6. \n\n6  RESULTS \n\nFigure  8  shows  the  results  of the  complete  reconstruction process  for  a  200Hz impulse \ntrain and the word \"tap.\" In both cases, no iterations were performed for either the spectro(cid:173)\ngram or filterbank inversion. More iterations reduce the spectral error, but do not make the \ngraphs look better or change the perceptual quality much. It is  worth noting that the \"tap\" \nreconstruction from  a correlogram looks  similar to the cochleagram reconstruction with(cid:173)\nout  the  AGC  (see  Figure  5.)  Reducing  the  level  of the  input  signal,  thus  reducing  the \namount of compression performed by  the AGC,  results  in  a  correlogram reconstruction \nsimilar to the original waveform. \n\n0.02 r---------. \n\n0.05f\"'\"'\"!~----\"\"\"\" \n\n0.01 \no \n-0.01 \n\n-0.02\"'--~~~~~ \n50  100  150  200 \n\no \n\n-0.05 \" ' - - - - - - -.... \n\n1000  2000  3000 \n\no \n\nFigure 8. Reconstructions from the correlogram representation of an impulse train and the word \n\"tap\" are shown above. Reducing the input signal level, thus minimizing the effect of errors \nwhen inverting the AGe, produces results identical to the original \"tap.\" \n\nIt is important to note that the algorithms described in this paper are designed to minimize \nthe  error in  the  mean-square  sense.  This  is  a  convenient  mathematical  definition,  but  it \ndoesn't always  correlate with human perception. A trivial example of this  is  possible by \ncomparing a waveform and a copy  of the  waveform delayed by  lOms.  Using  the  mean(cid:173)\nsquared  error,  the  numerical  error is  very  high  yet  the  two  waveforms  are  perceptually \nequivalent.  Despite  this,  the  results  of these  algorithms  based  on  mean-square  error do \nsound good. \n\n7  CONCLUSIONS \n\nThis paper has described several techniques that allow several stages of an auditory model \nto be  converted back into sound.  By converting each row of the correlogram into a spec(cid:173)\ntrogram, the spectrogram inversion techniques of Section 4 can be used. The special char(cid:173)\nacteristics of a correlogram described in Section 3 are  used to  make the calculation more \nefficient. Finally, the cochlear filterbank can be inverted to recover the original waveform. \nThe results are waveforms, perceptually identical to the original waveforms. \n\n\f834 \n\nMalcolm  Slaney \n\nThese techniques will be especially useful as  part of a sound separation system. I do not \nbelieve that our auditory system resynthesizes partial waveforms from the auditory scene. \nYet,  all  research  systems generate separated sounds  so  that we can  more easily perceive \ntheir success. More work is still needed to fine-tune these algorithm and to investigate the \nability to reconstruct sounds from partial correlograms. \n\nAcknowledgments \n\nI am grateful for the inspiration provided by Frank Cooper's work in the early  1950's on \npattern playback[13][14]. His work demonstrated that it was possible to convert a spectro(cid:173)\ngram, painted onto clear plastic, into sound. \nThis  work  in  this  paper  was  performed  with  Daniel Naar and Richard F.  Lyon.  We  are \ngrateful  for  the  help  we  have  received  from  Richard  Duda  (San  Jose  State),  Shihab \nShamma (U.  of Maryland),  Jim Boyles  (The  MathWorks)  and Michele  Covell  (Interval \nResearch). \n\nReferences \n\n[1] Malcolm Slaney, D. Naar, R. F. Lyon, \"Auditory model inversion for sound separa(cid:173)\ntion,\" Proc. of IEEE ICASSP, Volume II, pp. 77-80,  1994. \n[2]  M. Slaney and R. F. Lyon, \"On the importance of time-A temporal representation of \nsound,\" in Visual Representations of Speech Signals, eds. M. Cooke, S. Beet, and M. \nCrawford, J. Wiley and Sons, Sussex, England,  1993. \n[3] R. F. Lyon, \"A computational model of binaural localization and separation,\" Proc.  of \nIEEE ICASSP,  1148-1151, 1983. \n[4] M. Weintraub, \"The GRASP sound separation system,\" Proc.  of IEEE ICASSP, pp. \n18A.6.1-18A.6.4, 1984. \n[5] D. Hennes, \"Pitch analysis,\" in Visual Representations of Speech Signals, eds. M. \nCooke, S.  Beet, and M. Crawford, J. Wiley and Sons, Sussex, England, 1993. \n[6] N. Suga, \"Cortical computational maps for auditory imaging,\" Neural Networks, 3, 3-\n21, 1990. \n[7] T.  lrino, H. Kawahara, \"Signal reconstruction from modified auditory wavelet trans(cid:173)\nfonn,\" IEEE Trans.  on Signal Processing, 41,3549-3554, Dec. 1993. \n[8] x. Yang, K. Wang, and S.  Sharnma, \"Auditory representations of acoustic signals,\" \nIEEE Trans.  on Information Theory, 38, 824-839, 1992. \n[9] R. F. Lyon, \"A computational model of filtering, detection, and compression in the \ncochlea,\" Proc. of the IEEE ICASSP,  1282-1285,1982. \n[10] D.  Naar, \"Sound resynthesis from a correlogram,\" San Jose State University, Depart(cid:173)\nment of Electrical Engineering, Technical Report #3, May  1993. \n[11] R. W. Papoulis, \"A new algorithm in spectral analysis and band-limited extrapola(cid:173)\ntion,\" IEEE Trans.  Circuits Sys., vol. 22, 735,  1975. \n[12] D. Griffin and J. Lim, \"Signal estimation from modified short-time Fourier trans(cid:173)\nfonn,\" IEEE Trans.  on Acoustics, Speech, and Signal Processing, 32, 236-242,  1984. \n[13] F. S. Cooper, \"Some Instrumental Aids to Research on Speech,\" Report on the Fourth \nAnnual Round Table Meeting on Linguistics and Language Teaching, Georgetown Uni(cid:173)\nversity Press, 46-53, 1953. \n[14] F.  S. Cooper, \"Acoustics in human communications: Evolving ideas about the nature \nof speech,\" J.  Acoust. Soc. Am., 68(1),18-21, July 1980. \n\n\f", "award": [], "sourceid": 978, "authors": [{"given_name": "Malcolm", "family_name": "Slaney", "institution": null}]}