{"title": "The Power of Selective Memory: Self-Bounded Learning of Prediction Suffix Trees", "book": "Advances in Neural Information Processing Systems", "page_first": 345, "page_last": 352, "abstract": null, "full_text": " The Power of Selective Memory:\nSelf-Bounded Learning of Prediction Suffix Trees\n\n\n\n Ofer Dekel Shai Shalev-Shwartz Yoram Singer\n School of Computer Science & Engineering\n The Hebrew University, Jerusalem 91904, Israel\n {oferd,shais,singer}@cs.huji.ac.il\n\n Abstract\n\n Prediction suffix trees (PST) provide a popular and effective tool for tasks\n such as compression, classification, and language modeling. In this pa-\n per we take a decision theoretic view of PSTs for the task of sequence\n prediction. Generalizing the notion of margin to PSTs, we present an on-\n line PST learning algorithm and derive a loss bound for it. The depth of\n the PST generated by this algorithm scales linearly with the length of the\n input. We then describe a self-bounded enhancement of our learning al-\n gorithm which automatically grows a bounded-depth PST. We also prove\n an analogous mistake-bound for the self-bounded algorithm. The result\n is an efficient algorithm that neither relies on a-priori assumptions on the\n shape or maximal depth of the target PST nor does it require any param-\n eters. To our knowledge, this is the first provably-correct PST learning\n algorithm which generates a bounded-depth PST while being competi-\n tive with any fixed PST determined in hindsight.\n\n\n\n1 Introduction\n\nPrediction suffix trees are elegant, effective, and well studied models for tasks such as\ncompression, temporal classification, and probabilistic modeling of sequences (see for in-\nstance [13, 11, 7, 10, 2]). Different scientific communities gave different names to variants\nof prediction suffix trees such as context tree weighting [13] and variable length Markov\nmodels [11, 2]. A PST receives an input sequence of symbols, one symbol at a time, and\npredicts the identity of the next symbol in the sequence based on the most recently ob-\nserved symbols. Techniques for finding a good prediction tree include online Bayesian\nmixtures [13], tree growing based on PAC-learning [11], and tree pruning based on struc-\ntural risk minimization [8]. All of these algorithms either assume an a-priori bound on the\nmaximal number of previous symbols which may be used to extend predictions or use a\npre-defined template-tree beyond which the learned tree cannot grow. Motivated by statis-\ntical modeling of biological sequences, Apostolico and Bejerano [1] showed that the bound\non the maximal depth can be removed by devising a smart modification of Ron et. al's al-\ngorithm [11] (and in fact many other variants), yielding an algorithm with time and space\nrequirements that are linear in the length of the input. However, when modeling very long\nsequences, both the a-priori bound and the linear space modification might impose serious\ncomputational problems.\n\n\f\nIn this paper we describe a variant of prediction trees for 0\nwhich we are able to devise a learning algorithm that grows\nbounded-depth trees, while remaining competitive with any\nfixed prediction tree chosen in hindsight. The resulting -3 -1\ntime and space requirements of our algorithm are bounded\nand scale polynomially with the complexity of the best pre-\ndiction tree. Thus, we are able to sidestep the pitfalls of pre- 1 4\nvious algorithms. The setting we employ is slightly more\ngeneral than context-based sequence modeling as we as-\nsume that we are provided with both an input stream and an -2 7\noutput stream. For concreteness, we assume that the input\nstream is a sequence of vectors x1, x2, . . . (xt Rn) and Figure 1: An illustration\nthe output stream is a sequence of symbols y1, y2, . . . over\n of the prediction process in-\na finite alphabet Y. We denote a sub-sequence yi, . . . , yj duced by a PST. The context\nof the output stream by j\n yi and the set of all possible se- in this example is: + + +\nquences by Y. We denote the length of a sequence s by\n|s|. Our goal is to correctly predict each symbol in the out-\nput stream y1, y2, . . .. On each time-step t we predict the symbol yt based on an arbitrarily\nlong context of previously observed output stream symbols, t-1\n y , and based on the current\n 1\ninput vector xt. For simplicity, we focus on the binary prediction case where |Y| = 2 and\nfor convenience we use Y = {-1, +1} (or {-, +} for short) as our output alphabet. Our\nalgorithms and analysis can be adapted to larger output alphabets using ideas from [5].\n\nThe hypotheses we use are confidence-rated and are of the form h : X Y R where the\nsign of h is the predicted symbol and the magnitude of h is the confidence in this prediction.\nEach hypothesis is parameterized by a triplet (w, T , g) where w Rn, T is a suffix-closed\nsubset of Y and g is a context function from T into R (T is suffix closed if s T it\nholds that all of the suffixes of s are also in T ). The prediction extended by a hypothesis\nh = (w, T ,g) for the t'th symbol is,\n h( t-1 t-1\n xt, y ) = 2-i/2 g\n 1 w xt + yt-i . (1)\n i: t-1\n y T\n t-i\n\nIn words, the prediction is the sum of an inner product between the current input vector\nxt with the weight vector w and the application of the function g to all the suffixes of the\noutput stream observed thus far that also belong to T . Since T is a suffix-closed set, it can\nbe described as a rooted tree whose nodes are the sequences constituting T . The children of\na node s T are all the sequences s T ( Y). Following the terminology of [11], we\nuse the term prediction suffix tree (PST) for T and refer to s T as a sequence and a node\ninterchangeably. We denote the length of the longest sequence in T by depth(T ). Given\ng, each node s T is associated with a value g(s). Note that in the prediction process, the\ncontribution of each context t-1\n yt-i is multiplied by a factor which is exponentially decreasing\nin the length of t-1\n yt-i . This type of demotion of long suffixes is common to most PST-\nbased approaches [13, 7, 10] and reflects the a-priori assumption that statistical correlations\ntend to decrease as the time between events increases. An illustration of a PST where\nT = { ,-,+,+\n 5 -,++,-++,+++}, with the associated prediction for y6 given the\ncontext y =\n 1 --+++ is shown in Fig. 1. The predicted value of y6 in the example is\nsign(w xt+2-1/2(-1)+2-14+2-3/27). Given T and g we define the extension\nof g to all strings over Y by setting g(s) = 0 for s T . Using this extension, Eq. (1) can\nbe simplified to,\n t-1\n h( t-1 t-1\n xt, y ) = 2-i/2 g\n 1 w xt + yt-i . (2)\n i=1\nWe use the online learning loss-bound model to analyze our algorithms. In the online\nmodel, learning takes place in rounds. On each round, an instance xt is presented to the\n\n\f\nonline algorithm, which in return predicts the next output symbol. The predicted symbol,\ndenoted ^\n y t-1\n t is defined to be the sign of ht(xt, y ). Then, the correct symbol y\n 1 t is revealed\nand with the new input-output pair (xt, yt) on hand, a new hypothesis ht+1 is generated\nwhich will be used to predict the next output symbol, yt+1. In our setting, the hypotheses\nht we generate are of the form given by Eq. (2). Most previous PST learning algorithms\nemployed probabilistic approaches for learning. In contrast, we use a decision theoretic\napproach by adapting the notion of margin to our setting. In the context of PSTs, this ap-\nproach was first suggested by Eskin in [6]. We define the margin attained by the hypothesis\nh t-1\n t to be ytht(xt, y ). Whenever the current symbol y\n 1 t and the output of the hypothesis\nagree in their sign, the margin is positive. We would like our online algorithm to correctly\npredict the output stream y1, . . . , yT with a sufficiently large margin of at least 1. This\nconstruction is common to many online and batch learning algorithms for classification\n[12, 4]. Specifically, we use the hinge loss as our margin-based loss function which serves\nas a proxy for the prediction error. Formally, the hinge loss attained on round t is defined as,\n t-1\n t = max 0, 1 - ytht xt,y . The hinge-loss equals zero when the margin exceeds\n 1\n1 and otherwise grows linearly as the margin gets smaller. The online algorithms discussed\nin this paper are designed to suffer small cumulative hinge-loss.\n\nOur algorithms are analyzed by comparing their cumulative hinge-losses and prediction\nerrors with those of any fixed hypothesis h = (w , T , g ) which can be chosen in hind-\nsight, after observing the entire input and output streams. In deriving our loss and mistake\nbounds we take into account the complexity of h . Informally, the larger T and the bigger\nthe coefficients of g (s), the more difficult it is to compete with h . The squared norm of\nthe context function g is defined as,\n\n g 2 = (g(s))2 . (3)\n\n sT\n\nThe complexity of a hypothesis h (and h in particular) is defined as the sum of 2\n w and\n g 2. Using the extension of g to Y we can evaluate g 2 by summing over all s Y .\nWe present two online algorithms for learning large-margin PSTs. The first incrementally\nconstructs a PST which grows linearly with the length of the input and output sequences,\nand thus can be arbitrarily large. While this construction is quite standard and similar\nmethods were employed by previous PST-learning algorithms, it provides us with an in-\nfrastructure for our second algorithm which grows bounded-depth PSTs. We derive an\nexplicit bound on the maximal depth of the PSTs generated by this algorithm. We prove\nthat both algorithms are competitive with any fixed PST constructed in hindsight. To our\nknowledge, this is the first provably correct construction of a PST-learning algorithm whose\nspace complexity does not depend on the length of the input-output sequences.\n\n\n2 Learning PSTs of Unbounded Depth\n\nHaving described the online prediction paradigm and the form of hypotheses used, we\nare left with the task of defining the initial hypothesis h1 and the hypothesis update rule.\nTo facilitate our presentation, we assume that all of the instances presented to the online\nalgorithm have a bounded Euclidean norm, namely, xt 1. First, we define the initial\nhypothesis to be h1 0. We do so by setting w1 = (0, . . . , 0), T1 = { } and g1() 0.\nAs a consequence, the first prediction always incurs a unit loss. Next, we define the updates\napplied to the weight vector wt and to the PST at the end of round t. The weight vector is\nupdated by 2\n wt+1 = wt + yttxt, where t = t/( xt + 3). Note that if the margin\nattained on this round is at least 1 then t = 0 and thus wt+1 = wt. This type of update\nis common to other online learning algorithms (e.g. [3]). We would like to note in passing\nthat the operation wt xt in Eq. (2) can be replaced with an inner product defined via a\nMercer kernel. To see this, note that wt can be rewritten explicitly as t-1 y\n i=1 ii xi and\n\n\f\n initialize: w1 = (0, . . . , 0), T1 = { }, g1(s) = 0 s Y , P0 = 0\n\n for t = 1, 2, . . . do\n Receive an instance xt s.t. xt 1\n Define: j = max{i : t-1\n yt-i Tt}\n Calculate: h t-1 t-1\n t\n\n Predict: ^\n y `xt,y1 = wt xt +\n t-1 Pj 2-i/2 g\n i=1 t `yt-i\n t = sign\n Receive y `ht`xt,y1 t-1\n t and suffer loss: t = max\n Set: 2 + 3 0,1-ytht`xt,y1 \n t = t/ ` xt and dt = t-1 s m\n el o\n if ( f-b\n t 1/2) then difi\n Set: ou cat\n t = 0, Pt = Pt-1, dt = 0, and continue to the next iteration nd i\n else o\n ed n\n v req\n ers\n Set: dt = max nj , l2log (2 + u\n 2 t) - 2 log2 \"pP2t-1 t t - Pt-1\"mo i i\n o red\n n\n Set: Pt = Pt-1 + 2t2-dt/2 for\n Update weight vector: wt+1 = wt + yttxt\n Update tree:\n T t-1\n t+1 = Tt {yt-i : 1 i dt}\n t-1\n g t-i : 1 i dt}\n t+1(s) = gt(s)+yt 2-|s|/2 t if s {y\n gt(s) otherwise\n\n\nFigure 2: The online algorithms for learning a PST. The code outside the boxes defines the\nbase algorithm for learning unbounded-depth PSTs. Including the pseudocode inside the\nboxes gives the self-bounded version.\n\n\n\ntherefore wt xt = y\n i ii xi xt. Using a kernel operator K simply amounts to replacing\nthe latter expression with y\n i iiK (xi, xt).\n\nThe update applied to the context function gt also depends on the scaling factor t. How-\never, gt is updated only on those strings which participated in the prediction of ^\n yt, namely\nstrings of the form t-1\n yt-i for 1\n t-1 t-1 i < t. Formally, for 1 i < t our update takes the form\ngt+1(yt-i ) = gt(yt-i ) + yt 2-i/2 t. For any other string s, gt+1(s) = gt(s). The pseudo-\ncode of our algorithm is given in Fig. 2. The following theorem states that the algorithm\nin Fig. 2 is 2-competitive with any fixed hypothesis h for which g is finite.\n\nTheorem 1. Let x1, . . . , xT be an input stream and let y1, . . . , yT be an output stream,\nwhere every xt Rn, xt 1 and every yt {-1, 1}. Let h = (w , T , g ) be an\narbitrary hypothesis such that g < and which attains the loss values ,...,\n 1 T on\nthe input-output streams. Let 1, . . . , T be the sequence of loss values attained by the\nunbounded-depth algorithm in Fig. 2 on the input-output streams. Then it holds that,\n\n T T\n 2 2\n t 4 w + g 2 + 2 ( t )2 .\n t=1 t=1\n\nIn particular, the above bounds the number of prediction mistakes made by the algorithm.\n\n\nProof. For every t = 1, . . . , T define 2 2\n t = wt - w - wt+1 - w and,\n ^ 2 2\n t = gt(s) - g (s) - gt+1(s) - g (s) . (4)\n sY sY \n\n\nNote that g 2\n t is finite for any value of t and that g 2 is finite due to our assumption,\ntherefore ^\n t is finite and well-defined. We prove the theorem by devising upper and lower\n\n\f\nbounds on ( \n t t + ^\n t), beginning with the upper bound. t t is a telescopic sum\nwhich collapses to 2 2\n w1 - w - wt+1 - w . Similarly,\n T\n ^ 2 2\n t = g1(s) - g (s) - gt+1(s) - g (s) . (5)\n t=1 sY sY \n\n\nOmitting negative terms and using the facts that w1 = (0, . . . , 0) and g1() 0, we get,\n T\n 2 2\n t + ^\n t w + (g (s))2 = w + g 2 . (6)\n t=1 sY \n\n\nHaving proven an upper bound on (\n t t + ^\n t), we turn to the lower bound. First, t can\nbe rewritten as 2\n t = wt -w - (wt+1 -wt)+(wt\n 2 -w ) 2 and by expansion of the\nright-hand term we get that t = - wt+1 -wt -2(wt+1 -wt)(wt -w ). Using the\nvalue of wt+1 as defined in the update rule of the algorithm (wt+1 = wt + yttxt) gives,\n\n 2\n t = - 2t xt - 2 yt t xt (wt - w ) . (7)\nNext, we use similar manipulations to rewrite ^\n t. Unifying the two sums that make up ^\n t\nin Eq. (4) and adding null terms of the form 0 = gt(s) - gt(s), we obtain,\n 2\n ^ 2\n t = gt(s)\n sY - g (s) - gt+1(s) - gt(s) + gt(s) - g (s)\n 2\n = sY - gt+1(s) - gt(s) - 2 gt+1(s) - gt(s) gt(s) - g (s) .\nLet dt = t - 1 as defined in Fig. 2. Using the fact that gt+1 differs from gt only on strings\nof the form t-1 t-1 t-1\n yt-i , where gt+1 yt-i = gt yt-i + yt2-i/2t, we can write ^\n t as,\n\n dt dt\n ^\n t-1 t-1\n t = -2-i 2t - 2 yt 2-i/2 t gt yt-i - g yt-i . (8)\n i=1 i=1\n\nSumming Eqs. (7-8) gives,\n\n 2 t-1\n t + ^\n t = -2t xt + dt 2-i 2-i/2 g\n i=1 - 2t yt wt xt + dti=1 t yt-i\n + 2 t-1\n t yt w xt + dt 2-i/2 g\n i=1 yt-i . (9)\n\nUsing dt 2-i\n i=1 1 with the definitions of ht and h from Eq. (2), we get that,\n 2 t-1 t-1\n t + ^\n t - 2t( xt + 1) - 2t yt ht xt,y + 2 . (10)\n 1 t yt h xt, y1\n\nDenote the right-hand side of Eq. (10) by t and recall that the loss is defined as max\n t-1 t-1 {0,1-\nytht(xt, y ) ) =\n 1 }. Therefore, if t > 0 then -ytht(xt,y1 t\n t-1 -1. Multiplying both sides\nof this equality by t gives -tytht(xt, y ) = \n 1 t( t - 1). Now note that this equality\nalso holds when t = 0 since then t = 0 and both sides of the equality simply equal\nzero. Similarly, we have that y t-1\n th (xt, y )\n 1 1 - t. Plugging these two inequalities into\nEq. (10) gives that,\n\n 2\n t - 2t( xt + 1) + 2t ( t - 1) + 2t (1 - t ) ,\nwhich in turn equals -2 2\n t ( xt + 1) + 2t t - 2t t. The lower bound on t still holds\nif we subtract from it the non-negative term (21/2t - 2-1/2 t)2, yielding,\n 2\n t -2t( xt + 1) + 2t t - 2t t - 22t - 2t t + ( t)2/2\n = -2 2\n t ( xt + 3) + 2t t - ( t)2/2 .\n\n\f\nUsing the definition of 2\n t and using the assumption that xt 1, we get,\n ( 2 (\n t )2 t t )2\n t - t t + 2t t - =\n 2 2\n xt + 3 - 2 2t/4 - ( t)2/2 . (11)\n\nSince Eq. (10) implies that t + ^\n t t, summing t + ^t over all values of t gives,\n T 1 T 1 T\n 2\n t + ^\n t (\n 4 t - 2 t )2 .\n t=1 t=1 t=1\n\nCombining the bound above with Eq. (6) gives the bound stated by the theorem. Finally, we\nobtain a mistake bound by noting that whenever a prediction mistake occurs, t 1.\n\nWe would like to note that the algorithm for learning unbounded-depth PSTs constructs a\nsequence of PSTs, T1, . . . , TT, such that depth(Tt) may equal t. Furthermore, the number\nof new nodes added to the tree on round t is on the order of t, resulting in Tt having\nO(t2) nodes. However, PST implementation tricks in [1] can be used to reduce the space\ncomplexity of the algorithm from quadratic to linear in t.\n\n\n3 Self-Bounded Learning of PSTs\n\nThe online learning algorithm presented in the previous section has one major drawback,\nthe PSTs it generates can keep growing with each online round. We now describe a mod-\nification to the algorithm which casts a limit on the depth of the PST that is learned. Our\ntechnique does not rely on arbitrary assumptions on the structure of the tree (e.g. maxi-\nmal tree depth) nor does it require any parameters. The algorithm determines the depth to\nwhich the PST should be updated automatically, and is therefore named the self-bounded\nalgorithm for PST learning. The self-bounded algorithm is obtained from the original un-\nbounded algorithm by adding the lines enclosed in boxes in Fig. 2.\n\nA new variable dt is calculated on every online iteration. On rounds where an update takes\nplace, the algorithm updates the PST up to depth dt, adding nodes if necessary. Below\nthis depth, no nodes are added and the context function is not modified. The definition\nof dt is slightly involved, however it enables us to prove that we remain competitive with\nany fixed hypothesis (Thm. 2) while maintaining a bounded-depth PST (Thm. 3). A point\nworth noting is that the criterion for performing updates has also changed. Before, the\nonline hypothesis was modified whenever t > 0. Now, an update occurs only when\n t > 1/2, tolerating small values of loss. Intuitively, this relaxed margin requirement is\nwhat enables us to avoid deepening the tree. The algorithm is allowed to predict with lower\nconfidence and in exchange the PST can be kept small. The trade-off between PST size\nand confidence of prediction is adjusted automatically, extending ideas from [9]. While the\nfollowing theorem provides a loss bound, this bound can be immediately used to bound the\nnumber of prediction mistakes made by the algorithm.\n\nTheorem 2. Let x1, . . . , xT be an input stream and let y1, . . . , yT be an output stream,\nwhere every xt Rn, xt 1 and every yt {-1, 1}. Let h = (w , T , g ) be an\narbitrary hypothesis such that g < and which attains the loss values ,...,\n 1 T on\nthe input-output streams. Let 1, . . . , T be the sequence of loss values attained by the self-\nbounded algorithm in Fig. 2 on the input-output streams. Then the sum of squared-losses\nattained on those rounds where t > 1/2 is bounded by,\n\n T 1/2 2\n 2 \n t (1 + 5) g + 2 w + 2 ( t )2 .\n t: t=1\n t > 1\n 2\n\n\f\nProof. We define t and ^\n t as in the proof of Thm. 1. First note that the inequality in\nEq. (9) in the proof of Thm. 1 still holds. Using the fact that dt 2-i\n i=1 1 with the\ndefinitions of ht and h from Eq. (2), Eq. (9) becomes,\n\n 2 t-1 t-1\n t + ^\n t - 2t( xt + 1) - 2t yt ht xt,y + 2\n 1 t yt h xt, y1 (12)\n - 2 t-1 t-1\n tyt 2-i/2 g\n i=d y\n t +1 t-i .\n\nUsing the Cauchy-Schwartz inequality we get that\n\n t-1 t-1 t-1\n 1/2 1\n 2 /2\n 2-i/2 g t-1 t-1\n yt 2-i g\n -i yt-i 2-dt/2 g .\n i=dt+1 i=dt+1 i=dt+1\n\nPlugging the above into Eq. (12) and using the definition of t from the proof of Thm. 1\ngives t + ^\n t t - 2t2-dt/2 g . Using the upper bound on t from Eq. (11) gives,\n t + ^\n t t t - ( t)2/2 - 2 t2-dt/2 g . (13)\nFor every 1 t T, define Lt = t \n i=1 i i and Pt = t\n i=1 i21-di/2, and let P0 =\nL0 = 0. Summing Eq. (13) over t and comparing to the upper bound in Eq. (6) we get,\n\n T\n L 2\n T g 2 + w + (1/2) ( t )2 + g PT . (14)\n t=1\n\nWe now use an inductive argument to prove that Pt Lt for all 0 t T. This\ninequality trivially holds for t = 0. Assume that P 2\n t-1 Lt-1. Expanding Pt we get that\n 2\n P 2\n t = Pt-1 + t21-dt/2 = P 2\n t- + P\n 1 t-1 22-dt/2 t + 22-dt 2\n t . (15)\n\nWe therefore need to show that the right-hand side of Eq. (15) is at most Lt. The definition\nof dt implies that 2-dt/2 is at most (P 2\n t- + \n 1 t t)1/2 - Pt-1 /(2t). Plugging this fact\ninto the right-hand side of Eq. (15) gives that P 2\n t cannot exceed P 2\n t- + \n 1 t t. Using the\ninductive assumption P 2\n t-1 Lt-1 we get that P 2t Lt-1 + t t = Lt and the induc-\ntive argument is proven. In particular, we have shown that PT LT. Combining this\ninequality with Eq. (14) we get that\n\n T\n 2\n L 2\n T - g LT - g 2 - w - (1/2) ( t )2 0 .\n t=1\n\nThe above equation is a quadratic inequality in LT from which it follows that Lt can\nbe at most as large as the positive root of this equation, namely,\n\n 1 T\n L 2\n T g + 5 g 2 + 4 w + 2 (\n 2 t )2 1/2 .\n t=1\n \nUsing the the fact that a2 + b2 (a + b) (a,b 0) we get that,\n 1 + 5 1 T\n LT g + w + (\n 2 2 t )2 1/2 . (16)\n t=1\n\nIf t 1/2 then t t = 0 and otherwise t t 2t/4. Therefore, the sum of 2t over the\nrounds for which t > 1/2 is less than 4 Lt, which yields the bound of the theorem.\n\n\f\nNote that if there exists a fixed hypothesis with g < which attains a margin of 1 on\nthe entire input sequence, then the bound of Thm. 2 reduces to a constant. Our next theorem\nstates that the algorithm indeed produces bounded-depth PSTs. Its proof is omitted due to\nthe lack of space.\n\nTheorem 3. Under the conditions of Thm. 2, let T1, . . . , TT be the sequence of PSTs gen-\nerated by the algorithm in Fig. 2. Then, for all 1 t T,\n 1 T\n depth(Tt) 9 + 2log 2 g + (\n 2 w + 2 t )2 1/2 + 1 .\n t=1\n\n\nThe bound on tree depth given in Thm. 3 becomes particularly interesting when there exists\nsome fixed hypothesis h for which (\n t t )2 is finite and independent of the total length of\nthe output sequence, denoted by T . In this case, Thm. 3 guarantees that the depth of the PST\ngenerated by the self-bounded algorithm is smaller than a constant which does not depend\non T . We also would like to emphasize that our algorithm is competitive even with a PST\nwhich is deeper than the PST constructed by the algorithm. This can be accomplished by\nallowing the algorithm's predictions to attain lower confidence than the predictions made\nby the fixed PST with which it is competing.\n\n\nAcknowledgments This work was supported by the Programme of the European Com-\nmunity, under the PASCAL Network of Excellence, IST-2002-506778 and by the Israeli\nScience Foundation grant number 522-04.\n\n\nReferences\n\n [1] G. Bejerano and A. Apostolico. Optimal amnesic probabilistic automata, or, how\n to learn and classify proteins in linear time and space. Journal of Computational\n Biology, 7(3/4):381393, 2000.\n [2] P. Buhlmann and A.J. Wyner. Variable length markov chains. The Annals of Statistics,\n 27(2):480513, 1999.\n [3] K. Crammer, O. Dekel, S. Shalev-Shwartz, and Y. Singer. Online passive aggressive\n algorithms. In Advances in Neural Information Processing Systems 16, 2003.\n [4] N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines.\n Cambridge University Press, 2000.\n [5] O. Dekel, J. Keshet, and Y. Singer. Large margin hierarchical classification. In Pro-\n ceedings of the Twenty-First International Conference on Machine Learning, 2004.\n [6] E. Eskin. Sparse Sequence Modeling with Applications to Computational Biology\n and Intrusion Detection. PhD thesis, Columbia University, 2002.\n [7] D.P. Helmbold and R.E. Schapire. Predicting nearly as well as the best pruning of a\n decision tree. Machine Learning, 27(1):5168, April 1997.\n [8] M. Kearns and Y. Mansour. A fast, bottom-up decision tree pruning algorithm with\n near-optimal generalization. In Proceedings of the Fourteenth International Confer-\n ence on Machine Learning, 1996.\n [9] P. Auer, N. Cesa-Bianchi and C. Gentile. Adaptive and self-confident on-line learning\n algorithms. Journal of Computer and System Sciences, 64(1):4875, 2002.\n[10] F.C. Pereira and Y. Singer. An efficient extension to mixture techniques for prediction\n and decision trees. Machine Learning, 36(3):183199, 1999.\n[11] D. Ron, Y. Singer, and N. Tishby. The power of amnesia: learning probabilistic\n automata with variable memory length. Machine Learning, 25(2):117150, 1996.\n[12] V.N. Vapnik. Statistical Learning Theory. Wiley, 1998.\n[13] F.M.J. Willems, Y.M. Shtarkov, and T.J. Tjalkens. The context tree weighting method:\n basic properties. IEEE Transactions on Information Theory, 41(3):653664, 1995.\n\n\f\n", "award": [], "sourceid": 2549, "authors": [{"given_name": "Ofer", "family_name": "Dekel", "institution": null}, {"given_name": "Shai", "family_name": "Shalev-shwartz", "institution": null}, {"given_name": "Yoram", "family_name": "Singer", "institution": null}]}