{"title": "HARMONET: A Neural Net for Harmonizing Chorales in the Style of J. S. Bach", "book": "Advances in Neural Information Processing Systems", "page_first": 267, "page_last": 274, "abstract": null, "full_text": "HARMONET: A Neural Net for Harmonizing \n\nChorales in the Style of l.S.Bach \n\nHermann Hild \nhhild@ira.uka.de \n\nJohannes Feulner \njohannes@ira.uka.de \n\nWolfram Menzel \nmenzel@ira.uka.de \n\nInstitut fur Logik, Komplexitat und Deduktionssysteme \n\nAm Fasanengarten 5 \nUniversitat Karlsruhe \n\nW-7500 Karlsruhe 1, Germany \n\nAbstract \n\nHARMONET, a system employing connectionist networks for music pro(cid:173)\ncessing, is presented. After being trained on some dozen Bach chorales \nusing error backpropagation, the system is capable of producing four-part \nchorales in the style of J .s.Bach, given a one-part melody. Our system \nsolves a musical real-world problem on a performance level appropriate \nfor musical practice. HARMONET's power is based on (a) a new coding \nscheme capturing musically relevant information and (b) the integration of \nbackpropagation and symbolic algorithms in a hierarchical system, com(cid:173)\nbining the advantages of both. \n\n1 \n\nINTRODUCTION \n\nNeural approaches to music processing have been previously proposed (Lischka, \n1989) and implemented (Mozer, 1991)(Todd, 1989). The promise neural networks \noffer is that they may shed some light on an aspect of human creativity that doesn't \nseem to be describable in terms of symbols and rules. Ultimately what music is (or \nisn't) lies in the eye (or ear) of the beholder. The great composers, such as Bach \nor Mozart, learned and obeyed quite a number of rules, e.g. the famous prohibition \nof parallel fifths. But these rules alone do not suffice to characterize a personal \nor even historic style. An easy test is to generate music at random, using only \n\n267 \n\n\f268 \n\nHild, Feulner, and Menzel \n\nA Chorale Melody \n\nBach's Chorale Harmoniza(cid:173)\ntion \n\nFigure 1: The beginning of the chorale melody \"Jesu, meine Zuversicht\" and its \nharmonization by J .S.Bach \n\nschoolbook rules as constraints. The result is \"error free\" but aesthetically offensive. \nTo overcome this gap between obeying rules and producing music adhering to an \naccepted aesthetic standard, we propose HARMONET, which integrates symbolic \nalgorithms and neural networks to compose four part chorales in the style of J .S . \nBach (1685 - 1750), given the one part melody. The neural nets concentrate on \nthe creative part of the task, being responsible for aesthetic conformance to the \nstandard set by Bach in nearly 400 examples. Original Bach Chorales are used \nas training data. Conventional algorithms do the bookkeeping tasks like observing \npitch ranges, or preventing parallel fifths. HARMONET's level of performance \napproaches that of improvising church organists, making it applicable to musical \npractice. \n\n2 TASK DEFINITION \n\nThe process of composing an accompaniment for a given chorale melody is called \nchorale harmonization. Typically, a chorale melody is a plain melody often \nharmonized to be sung by a choir. Correspondingly, the four voices of a chorale \nharmonization are called soprano (the melody part), alto, tenor and bass. Figure 1 \ndepicts an example of a chorale melody and its harmonization by J .S.Bach. For \ncenturies, music students have been routinely taught to solve the task of chorale \nharmonization. Many theories and rules about \"dos\" and \"don'ts\" have been de(cid:173)\nveloped. However, the task of HARMONET is to learn to harmonize chorales from \nexample. Neural nets are used to find stylisticly characteristic harmonic sequences \nand ornamentations. \n\n\fHARMONET: A Neural Net for Harmonizing Chorales \n\n269 \n\n3 SYSTEM OVERVIEW \n\nGiven a set of Bach chorales, our goal is to find an approximation j of the quite com(cid:173)\nplex function l f which maps chorale melodies into their harmonization as demon(cid:173)\nstrated by J .S.Bach on almost 400 examples. In the following sections we propose \na decomposition of f into manageable subfunctions. \n\n3.1 TASK DECOMPOSITION \n\nThe learning task is decomposed along two dimensions: \nDifferent levels of abstractions. The chord skeleton is obtained if eighth \nand sixteenth notes are viewed as omitable ornamentations. Furthermore, if the \nchords are conceived as harmonies with certain attributes such as \"inversion\" or \n\"characteristic dissonances\", the chorale is reducible to its harmonic skeleton, a \nthoroughbass-like representation (Figure 2). \n\nLocality in time. The accompaniment is divided into smaller parts, each of which \nis learned independently by looking at some local context, a window. Treating \nsmall parts independently certainly hurts global consistency. Some of the depen(cid:173)\ndencies lost can be regained if the current decision window additionally considers \nthe outcome of its predecessors (external feedback). Figure 3 shows two consecutive \nwindows cut out from the harmonic skeleton. \n\nTo harmonize a chorale, HARMONET starts by learning the harmonic skeleton, \nwhich then is refined to the chord skeleton and finally augmented with ornamenting \nquavers (Figure 4, left side). \n\n3.2 THE HARMONIC SKELETON \n\nChorales have a rich harmonic structure, which is mainly responsible for their \"mu(cid:173)\nsical appearance\". Thus generating a good harmonic skeleton is the most important \nof HARMONET's subtasks. HARMONET creates a harmonic sequence by sweep(cid:173)\ning through the chorale melody and determining a harmony for each quarter note, \nconsidering its local context and the previously found harmonies as input. \nAt each quarterbeat position t, the following information is extracted to form one \ntraining example: \n\n[ \\'-3 i'-2 ~~~1 ~j:.! 8'H 1 ! N;.:al! \n\n'~' \n\nI at \nI \nL \u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022 J. \u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022 J. \u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022 ~ \n\nI at \n\nI at \n\nThe target to be learned (the harmony H t at position t) is marked by the box. \nThe input consists of the harmonic context to the left (the external feedback H t - 3 , \nHt - 2 and Ht - 1 ) and the melodic context (pitches St-I! St and st+t). phrt contains \n\nITo be sure, f is not a function but a relation, since there are many ''legal\" accompa(cid:173)\n\nniments for one melody. For simplicity, we view f as a function. \n\n\f270 \n\nHild, Feulner, and Menzel \n\nJJ \n\nChord Skeleton \n\nIf J J J J (J J j \n\nHarmonic Skeleton \n\nFigure 2: The chord and the harmonic skeleton of the chorale from figure 1. \n\ninformation about the relative position of t to the beginning or end of a musical \nphrase. strt is a boolean value indicating whether St is a stressed quarter. A \nharmony H t has three components: Most importantly, the harmonic function relates \nthe key of the harmony to the key of the piece. The inversion indicates the bass \nnote of the harmony. The characteristic dissonances are notes which do not directly \nbelong to the harmony, thus giving it additional tension. \n\nThe coding of pitch is decisive for recognizing musically relevant regularities in \nthe training examples. This problem is discussed in many places (Shepard, 1982) \n(Mozer, 1991). We developed a new coding scheme guided by the harmonic neces(cid:173)\nsities of homophonic music pieces: A note s is represented as the set of harmonic \nfunctions that contain s, as shown below: \n\nFct. \nC \nD \nE \n\nT D S Tp Sp Dp DD DP TP d Vtp SS \n1 \n0 \n0 \n1 \n.. \n\n0 \n1 \n\n0 \n1 \n\n0 \n1 \n\n0 \n0 \n\n0 \n0 \n\n0 \n1 \n\n1 \n0 \n\n1 \n0 \n\n0 \n1 \n\n0 \n0 \n\nT, D, S, Tp etc. are standard musical abbreviations to denote harmonic functions. \nThe resulting representation is distributed with respect to pitch. However, it is local \nwith respect to harmonic functions. This allows the network to anticipate future \nharmonic developments even though there cannot be a lookahead for harmonies yet \nuncomposed. \nBesides the 12 input units for each of the pitches St-1, St, St+l, we need 12+5+3 = \n\n\fHARMONET: A Neural Net for Harmonizing Chorales \n\n271 \n\nftt \n\nu \n\n\" \nr. \"\" \nu \n\n2 I \n\n.. .. ,-j \n\nI \n\nI \n\nt \n\nI \n\n... \nTpr-..,.H t ? --\n\nT T \n\n~ \n\nt+l \n2 1 \n\nI \n\nI \n\nI \n\n.... \n-\n\nv;--\n-\nT T Tp DP3-+Wt+l? \n\nFigure 3: The harmonic skeleton broken into local windows. The harmony Ht , \ndetermined at quarterbeat position t, becomes part of the input of the window at \nposition t + 1. \n\n20 input units for each of the 3 components of the harmonies Ht -3, Ht -2 and Ht-l, \n9 units to code the phrase information phrt and 1 unit for the stress Strt. Thus our \nnet has a total of 3 * 12 + 3 * 20 + 9 + 1 = 106 input units and 20 output units. We \nused one hidden layer with 70 units. \n\nIn a more advanced version (Figure 4, right side), we use three nets (Nl, N2, N3) \nin parallel, each of which was trained on windows of different size. The harmonic \nfunction for which the majority of these three nets votes is passed to two subsequent \nnets (N4, N5) determining the chord inversion and characteristic dissonances of the \nharmony. Using windows of different sizes in parallel employs statistical information \nto solve the problem of chosing an appropriate window size. \n\n3.3 THE CHORD SKELETON \n\nThe task on this level is to find the two middle parts (alto and tenor) given the \nsoprano S of the chorale melody and the harmony H determined by the neural \nnets. Since H includes information about the chord inversion, the pitch of the bass \n(modulo its octave) is already given. The problem is tackled with a \"generate and \ntest\" approach: Symbolic algorithms select a \"best\" chord out of the set of all \nchords consistent with the given harmony H and common chorale constraints. \n\n3.4 QUAVER ORNAMENTATIONS \n\nIn the last subtask, another net is taught how to add ornamenting eighths to the \nchord skeleton. The output of this network is the set of eighth notes (if any) by \nwhich a particular chord C t can be augmented. The network's input describes the \nlocal context of Ct in terms of attributes such as the intervals between Ct and Ct +1 , \nvoice leading characteristics, or the presence of eighths in previous chords. \n\n\f272 \n\nHild, Feulner, and Menzel \n\nI \n\nDetermine Harmonies \n\nChorale Melody \n\n[If J J J J I \nIf J J J J I \n\nI \n\nT T TpDP3 \n\nI Expand Harmonies to Chords I \n\nI \n\nInsert Eighth Notes \n\nI \n\nI \n\n\" \n\" r i r r \n.J I.J J \nI I T \nI \n\n\" \nill I r\"\" \nI \n.J I.J J \n. \nI U I T \n\nIf J J J J (J J \n\nT T Tp \n\n? \n\n, \n\nHarmonic \nFunction \n\nInversion \n\nCharacteristic \nDi880nances \n\nH \n\nHarmonized Chorale \n\nFigure 4: Left side: Overall structure of HARMONET. Right side: A more special(cid:173)\nized architecture with parallel and sequential nets (see text). \n\n4 PERFORMANCE \n\nHARMONET was trained separately on two sets of Bach chorales, each containing \n20 chorales in major and minor keys, respectively. By passing the chorales through \na window as explained above, each set amounted to approx. 1000 training examples. \nAll nets were trained with the error backpropagation algorithm, needing 50 to 100 \nepochs to achieve reasonable convergence. \n\nFigures 5 and 6 show two harmonizations produced by HARMONET, given melodies \nwhich were not in the training set. An audience of music professionals judged the \nquality of these and other chorales produced by HARMONET to be on the level \non an improvising organist. HARMONET also compares well to non-neural ap(cid:173)\nproaches. In figure 6 HARMONET's accompaniment is shown on a chorale melody \nalso used in the Ph.D. thesis of (Ebcioglu, 1986) to demonstrate the expert system \n\"CHORAL\" . \n\n\fHARMONET: A Neural Net for Harmonizing Chorales \n\n273 \n\nChristus, der ist mein Leben \n\n:i \n\nIt, I J I J J J J I J J J I J J J J I J. J I J J J I J J J I J J J J I 2.11 \n\ne \n\n~ \n\n7 \n\n8 \n\n6 \n\n5 \n\n9 \n\n9 \n\n./ \n\n~ \n\n.. \n. \n\nf \n\nI \n\ni \n\nT \n\n\" 5 \n\"r I I \n\nI \n\nI \n. \nf \n\nT \n\nt \n\n9 \n\n./ \n\ni U r i \n\nr r i \nIT LSi r ur \n\nri \n\nI \n\nI \n\ni \n\nr \n\nr i U f \nro-, f1 \nI \nr \n\nI U I \n\nI \n\n6 \n\nI \n\n7 \n\n8 \n\n9 \n\ni wr r tri r r i iTt ~. r \nI 1 J 11 r1 \n. r \n\nr r I \n\nr ,-\n\nri \n\nJ \n\nro-\" \n\nI \n\nI \n\nI \n\nI \n\nI \n\nT \n\nFigure 5: A chorale in a major key harmonized by HARMONET. \n\nHappy Birthday to You \n\n\" \n\n-.J \n\nt I \n\n9 \n\n~ I \n\nI \n\n7.Dl \n\nI \n\n8 \n\nl \n\n5 J I 6 n \nLJ' U r I r \n\nI \n\nI \n\n[JI \n\nI \nJ J J J d JJ-J J J -J -J~ .Jl-J 11J J J \n\n--I \n\nI \n\nr I \n\nr r 1 \n\nI \nT S D7 \n\nTp DP \n\nI \nT3 S T \n\nI \nD T \n\n~ ;, DP,+ Tp D DD~+ D DP~+ Tp \n\nI \nD T \n\nFigure 6: \"Happy Birthday\" harmonized by HARMONET. \n\n\f274 \n\nHild, Feulner, and Menzel \n\n5 CONCLUSIONS \n\nThe music processing system HARMONET presented in this paper clearly shows \nthat musical real-world applications are well within the reach of connectionist ap(cid:173)\nproaches. We believe that HARMONET owes much of its success to a clean task \ndecomposition and a meaningful selection and representation of musically relevant \nfeatures. By using a hybrid approach we allow the networks to concentrate on musi(cid:173)\ncal essentials instead of on structural constraints which may be hard for a network to \nlearn but easy to code symbolically. The abstraction of chords to harmonies reduces \nthe problem space and resembles a musician's problem approach. The \"harmonic \nrepresentation\" of pitch shows the harmonic character of the given melody more \nexplicitly. \nWe have also experimented to replace the neural nets in HARMONET by other \nlearning techniques such as decision trees (ID3) or nearest neighbor classification. \nHowever, as also reported on other tasks (Dietterich et al., 1990), they were out(cid:173)\nperformed by the neural nets. \n\nHARMONET is not a general music processing system, its architecture is designed \nto solve a quite difficult but also quite specific task. However, due to HARMONET's \nneural learning component, only a comparatively small amount of musical expert \nknowledge was necessary to design the system, making it easier to build and more \nflexible than a pure rule based system. \n\nAcknowledgements \n\nWe thank Heinz Braun, Heiko Harms and Gudrun Socher for many fruitful discus(cid:173)\nsions and contributions to this research and our music lab. \n\nReferences \n\nJ .S.Bach (Ed.: Bernhard Friedrich Fischer) 389 Choralgesange fur vierstim(cid:173)\nmigen Chor. Edition Breitkopf, Nr. 3765. \nDietterich,T.G ., Hild,H., & Bakiri,G. A comparative study of ID3 and Back(cid:173)\npropagation for English Text-to-Speech Mapping. Proc. of the Seventh \nInternational Conference on Machine Learning (pp. 24-31). Kaufmann, 1990. \n\nEbcioglu,K. An Expert System for Harmonization of Chorales in the Style \nof J.S.Bach. Ph.D. Dissertation, Department ofC.S., State University of New York \nat Buffalo, New York, 1986. \n\nLischka,C. Understanding Music Cognition. GMD St.Augustin, FRG, 1989. \nMozer,M.C., Soukup,T. Connectionist Music Composition Based on \nMelodic and Stylistic Constraints. Advances in Neural Information Processing \n3 (NIPS 3), R .P. Lippmann, J. E. Moody, D.S. Touretzky (eds.), Kaufmann 1991. \nShepard, Roger N. Geometrical Approximations to the Structure of Musical \nPitch. Psychological Review, Vol. 89, Nr. 4, July 1982. \nTodd, Peter M. A Connectionist Approach To Algorithmic Composition. \nComputer Music Journal, Vol. 13, No.4, Winter 1989. \n\n\f", "award": [], "sourceid": 576, "authors": [{"given_name": "Hermann", "family_name": "Hild", "institution": null}, {"given_name": "Johannes", "family_name": "Feulner", "institution": null}, {"given_name": "Wolfram", "family_name": "Menzel", "institution": null}]}