{"title": "Maximum Margin Semi-Supervised Learning for Structured Variables", "book": "Advances in Neural Information Processing Systems", "page_first": 33, "page_last": 40, "abstract": "", "full_text": "Maximum Margin Semi-Supervised\nLearning for Structured Variables\n\nY. Altun, D. McAllester\n\nM. Belkin\n\nTTI at Chicago\n\nChicago, IL 60637\n\naltun,mcallester@tti-c.org\n\nDepartment of Computer Science\n\nUniversity of Chicago\n\nChicago, IL 60637\n\nmisha@cs.uchicago.edu\n\nAbstract\n\nMany real-world classi\ufb01cation problems involve the prediction of\nmultiple inter-dependent variables forming some structural depen-\ndency. Recent progress in machine learning has mainly focused on\nsupervised classi\ufb01cation of such structured variables. In this paper,\nwe investigate structured classi\ufb01cation in a semi-supervised setting.\nWe present a discriminative approach that utilizes the intrinsic ge-\nometry of input patterns revealed by unlabeled data points and we\nderive a maximum-margin formulation of semi-supervised learning\nfor structured variables. Unlike transductive algorithms, our for-\nmulation naturally extends to new test points.\n\n1 Introduction\n\nDiscriminative methods, such as Boosting and Support Vector Machines have sig-\nni\ufb01cantly advanced the state of the art for classi\ufb01cation. However, traditionally\nthese methods do not exploit dependencies between class labels where more than\none label is predicted. Many real-world classi\ufb01cation problems, on the other hand,\ninvolve sequential or structural dependencies between multiple labels. For example\nlabeling the words in a sentence with their part-of-speech tags involves sequential\ndependency between part-of-speech tags; \ufb01nding the parse tree of a sentence in-\nvolves a structural dependency among the labels in the parse tree. Recently, there\nhas been a growing interest in generalizing kernel methods to predict structured\nand inter-dependent variables in a supervised learning setting, such as dual percep-\ntron [7], SVMs [2, 15, 14] and kernel logistic regression [1, 11]. These techniques\ncombine the e\ufb03ciency of dynamic programming methods with the advantages of\nthe state-of-the-art learning methods. In this paper, we investigate classi\ufb01cation of\nstructured objects in a semi-supervised setting.\n\nThe goal of semi-supervised learning is to leverage the learning process from a\nsmall sample of labeled inputs with a large sample of unlabeled data. This idea has\nrecently attracted a considerable amount of interest due to ubiquity of unlabeled\ndata. In many applications from data mining to speech recognition it is easy to\nproduce large amounts of unlabeled data, while labeling is often manual and expen-\nsive. This is also the case for many structured classi\ufb01cation problems. A variety\n\n\fof methods ranging from Naive Bayes [12], Cotraining [4], to Transductive SVM [9]\nto Cluster Kernels [6] and graph-based approaches [3] and references therein, have\nbeen proposed. The intuition behind many of these methods is that the classi\ufb01ca-\ntion/regression function should be smooth with respect to the geometry of the data,\ni. e. the labels of two inputs x and \u00afx are likely to be the same if x and \u00afx are similar.\nThis idea is often represented as the cluster assumption or the manifold assumption.\nThe unlabeled points reveal the intrinsic structure, which is then utilized by the\nclassi\ufb01cation algorithm. A discriminative approach to semi-supervised learning was\ndeveloped by Belkin, Sindhwani and Niyogi [3, 13], where the Laplacian operator\nassociated with unlabeled data is used as an additional penalty (regularizer) on the\nspace of functions in a Reproducing Kernel Hilbert Space. The additional regular-\nization from the unlabeled data can be represented as a new kernel \u2014 a \u201cgraph\nregularized\u201d kernel.\n\nIn this paper, building on [3, 13], we present a discriminative semi-supervised learn-\ning formulation for problems that involve structured and inter-dependent outputs\nand give experimental results on max-margin semi-supervised structured classi\ufb01-\ncation using graph-regularized kernels. The solution of the optimization problem\nthat utilizes both labeled and unlabeled data is a linear combination of the graph\nregularized kernel evaluated at the parts of the labeled inputs only, leading to a\nlarge reduction in the number of parameters.\nIt is important to note that our\nclassi\ufb01cation function is de\ufb01ned on all input points whereas some previous work is\nonly de\ufb01ned for the input points in the (labeled and unlabeled) training sample, as\nthey use standard graph kernels, which are restricted to in-sample data points by\nde\ufb01nition.\n\nThere is an the extensive literature on semi-supervised learning and the growing\nnumber of studies on learning structured and inter-dependent variables. Delaleau\net. al. [8] propose a semi-supervised learning method for standard classi\ufb01cation that\nextends to out-of-sample points. Brefeld et. al. [5] is one of the \ufb01rst studies investi-\ngating semi-supervised structured learning problem in a discriminative framework.\nThe most relevant previous work is the transductive structured learning proposed\nby La\ufb00erty et. al. [11].\n\n2 Supervised Learning for Structured Variables\n\nIn structured learning, the goal is to learn a mapping h : X \u2192 Y from structured\ninputs to structured response values, where the inputs and response values form a\ndependency structure. For each input x, there is a set of feasible outputs, Y(x) \u2286 Y.\nFor simplicity, let us assume that Y(x) is \ufb01nite for all x \u2208 X , which is the case in\nmany real world problems and in all our examples. We denote the set of feasible\ninput-output pairs by Z \u2286 X \u00d7 Y.\n\nIt is common to construct a discriminant function F : Z \u2192 < which maps the\nfeasible input-output pairs to a compatibility score of the pair. To make a prediction\nfor x, this score is maximized over the set of feasible outputs,\n\nh(x) = argmax\ny\u2208Y(x)\n\nF (x, y).\n\n(1)\n\nThe score of an hx, yi pair is computed from local fragments, or \u201cparts\u201d, of hx, yi.\nIn Markov random \ufb01elds, x is a graph, y is a labeling of the nodes of x and a\nlocal fragment (a part) of hx, yi is a clique in x and its labeling y. In parsing with\nprobabilistic context free grammars, a local fragment (a part) of hx, yi consist of\na branch of the tree y, where a branch is an internal node in y together with its\n\n\fchildren, plus all pairs of a leaf node in y with the word in x labeled by that node.\nNote that a given branch structure, such as NP \u2192 Det N, can occur more than\nonce in a given parse tree.\n\nIn general, we let P be a set of (all possible) parts. We assume a \u201ccounting function\u201d,\nc, such that for p \u2208 P and hx, yi \u2208 Z, c(p, hx, yi) gives the number of times that\nthe part p occurs in the pair hx, yi (the count of p in hx, yi). For a Mercer kernel\nk : P \u00d7 P \u2192 < on P, there is an associated RHKS Hk of functions f : P \u2192 <,\nwhere f measures the goodness of a part p. For any f \u2208 Hk, we de\ufb01ne a function\nFf on Z as\n\nFf (x, y) = X\n\np\u2208P\n\nc(p, hx, yi)f (p).\n\n(2)\n\nConsider a simple chain example. Let \u0393 be a set of possible observations and \u03a3\nbe a set of possible hidden states. We take the input x to be a sequence x1, . . . , x`\nwith xi \u2208 \u0393 and we take Y(x) to be the set of all sequences y1, . . . , y` with the same\nlength as x and with yi \u2208 \u03a3. We can take P to be the set of all pairs hs, \u00afsi plus all\npairs hs, ui with s, \u00afs \u2208 \u03a3 and u \u2208 \u0393. Often \u03a3 is taken to be a \ufb01nite set of \u201cstates\u201d\nand \u0393 = <d is a set of possible feature vectors. k(p, p0) is commonly de\ufb01ned as\n\nk(hs, \u00afsi, hs0, \u00afs0i) = \u03b4(s, s0)\u03b4(\u00afs, \u00afs0),\nk(hs, ui, hs0, u0i) = \u03b4(s, s0)ko(u, u0),\n\n(3)\n(4)\n\nwhere \u03b4(w, w0) denotes the Kronecker-\u03b4. Note that in this example there are two\ntypes of parts \u2014 pairs of hidden states and pairs of a hidden state and an observa-\ntion. Here we take k(p, p0) to be 0 if p and p0 are of di\ufb00erent types.\nIn the supervised learning scenario, we are given a sample S of ` pairs (hx1, y1i, . . .,\nhx`, y`i) drawn i. i. d.\nfrom an unknown but \ufb01xed probability distribution P on\nZ. The goal is to learn a function f on the local parts P with small expected loss\nEP [L(x, y, f )] where L is a prescribed loss function. This is commonly realized by\nlearning f that minimizes the regularized loss functional\n\nf \u2217 = argmin\nf \u2208Hk\n\n`\n\nX\n\ni=1\n\nL(xi, yi, f ) + \u03bbkf k2\nk,\n\n(5)\n\nwhere k.kk is the norm corresponding to Hk measuring the complexity of f . A vari-\nety of loss functions L have been considered in the literature. In kernel conditional\nrandom \ufb01elds (CRFs) [11], the loss function is given by\n\nL(x, y, f ) = \u2212Ff (x, y) + log X\n\n\u02c6y\u2208Y(x)\n\nexp(Ff (x, \u02c6y))\n\nIn structured Support Vector Machines (SVM), the loss function is given by\n\nL(x, y, f ) = max\n\u02c6y\u2208Y(x)\n\n\u2206(x, y, \u02c6y) + Ff (x, \u02c6y) \u2212 Ff (x, y),\n\n(6)\n\nwhere \u2206(x, y, \u02c6y) is some measure of distance between y and \u02c6y for a given observation\nx. A natural choice for \u2206 is to take \u2206(x, y, \u02c6y) to be the indicator 1[y6=\u02c6y] [2]. Another\nchoice is to take \u2206(x, y, \u02c6y) to be the size of the symmetric di\ufb00erence between the\nsets P(hx, yi) and P(hx, \u02c6yi) [14].\n\nLet P(x) \u2286 P be the set of parts having nonzero count in some pair hx, yi for\ny \u2208 Y(x). Let P(S) be the union of all sets P(xi) for xi in the sample. Then, we\nhave following straightforward variant of the Representer Theorem [10], which was\nalso presented in [11].\n\n\fDe\ufb01nition: A loss L is local if L(x, y, f ) is determined by the value of f on the\nset P(x), i.e., for f, g : P \u2192 < we have that if f (p) = g(p) for all p \u2208 P(x) then\nL(x, y, f ) = L(x, y, g).\nTheorem 1. For any local loss function L and sample S there exist weights \u03b1p for\np \u2208 P(S) such that f \u2217 as de\ufb01ned by (5) can be written as follows.\n\nf \u2217(p) = X\n\np0\u2208P(S)\n\n\u03b1p0 k(p0, p)\n\n(7)\n\nThus, even though the set of feasible outputs for x generally scales exponentially\nwith the size of output, the solution can be represented in terms of the parts of\nthe sample, which commonly scales polynomially. This is true for any loss function\nthat partitions into parts, which is the case for loss functions discussed above.\n\n3 A Semi-Supervised Learning Approach to Structured\n\nVariables\n\nIn semi-supervised learning, we are given a sample S consisting of l input-output\npairs {(x1, y1), . . . , (x`, y`)} drawn i. i. d.\nfrom the probability distribution P on\nZ and u unlabeled input patterns {x`+1, . . . , x`+u} drawn i. i. d from the marginal\ndistribution PX , where usually l < u. Let X (S) be the set {x1, . . . , x`+u} and let\nZ(S) be the set of all pairs hx, yi with x \u2208 X (S) and y \u2208 Y(x).\n\nIf the true classi\ufb01cation function is smooth wrt the underlying marginal distribution,\none can utilize unlabeled data points to favor functions that are smooth in this sense.\nBelkin et. al. [3] implement this assumption by introducing a new regularizer to the\nstandard RHKS optimization framework (as opposed to introducing a new kernel\nas discussed in Section 5)\n\nf \u2217 = argmin\nf \u2208Hk\n\n`\n\nX\n\ni=1\n\nL(xi, yi, f ) + \u03bb1||f ||2\n\nk + \u03bb2||f ||2\nkS\n\n,\n\n(8)\n\nwhere kS is a kernel representing the intrinsic measure of the marginal distribution.\nSindhwani et. al.[13] prove that the minimizer of (8) is in the span of a new kernel\nfunction (details below) evaluated at labeled data only. Here, we generalize this\nframework to structured variables and give a simpli\ufb01ed derivation of the new kernel.\n\nThe smoothness assumption in the structured setting states that f should be smooth\non the underlying density on the parts P, thus we enforce f to assign similar\ngoodness scores to two parts p and p0, if p and p0 are similar, for all parts of Z(S).\nLet P(S) be the union of all sets P(z) for z \u2208 Z(S) and let W be symmetric matrix\nwhere Wp,p0 represents the similarity of p and p0 for p, p0 \u2208 P(S).\n\nf \u2217 = argmin\nf \u2208Hk\n\n= argmin\n\nf \u2208Hk\n\n`\n\nX\n\ni=1\n\n`\n\nX\n\ni=1\n\nL(xi, yi, f ) + \u03bb1||f ||2\n\nk + \u03bb2 X\n\np,p0\u2208P(S)\n\nWp,p0 (f (p) \u2212 f (p0))2\n\nL(xi, yi, f ) + \u03bb1||f ||2\n\nk + \u03bb2f T Lf\n\n(9)\n\nHere W is a similarity matrix (like a nearest neighbor graph) and L is the Laplacian\nof W , L = D \u2212 W , where D is a diagonal matrix de\ufb01ned by Dp,p = Pp0 Wp,p0 . f\ndenotes the vector of f (p) for all p \u2208 P(S). Note that the last term depends only\non the value of f on the parts in the set P(S). Then, for any local loss L(x, y, f ),\n\n\fwe immediately have the following Representer Theorem for the semi-supervised\nstructured case where S includes the labeled and the unlabeled data.\n\n\u03b1(p) = X\nf \u2217\n\np0\u2208P(S)\n\n\u03b1p0 k(p0, p)\n\nSubstituting (10) into (9) leads to the following optimization problem\n\n\u03b1\u2217 = argmin\n\n\u03b1\n\n`\n\nX\n\ni=1\n\nL(xi, yi, f\u03b1) + \u03b1T Q\u03b1,\n\n(10)\n\n(11)\n\nwhere Q = \u03bb1K + \u03bb2KLK, K is the matrix of k(p, p0) for all p, p0 \u2208 P(S) and f\u03b1,\nas a vector in the space Hk, is a linear function of the vector \u03b1. Note that (11)\napplies to any local loss function and if L(x, y, f ) is convex in f , as in the case for\nlogistic or hinge loss, then (11) is convex in \u03b1.\n\nWe now have a loss function over labeled data regularized by the L2 norm (wrt\nthe inner product Q), for which we can re-evoke the Representer Theorem. Let S`\nbe the set of labeled inputs {x1, . . . , x`}, Z(S`) be the set of all pairs hx, yi with\nx \u2208 X (S`) and y \u2208 Y(x) and P(S`) be the set of al parts having nonzero count for\nsome pair in Z(S`). Let \u03b4p be a vector whose pth component is 1 and 0 elsewhere.\nUsing the standard orthogonality argument, let \u03b1\u2217 decompose into two: the vector\nin the span of \u03b3p = \u03b4pKQ\u22121 for all p \u2208 P(S`), and the vector in the orthogonal\ncomponent (under the inner product Q).\n\u03b1 = X\n\n\u03b2p\u03b3p + \u03b1\u22a5\n\np\u2208P(S`)\n\n\u03b1\u22a5 can only increase the quadratic term in the optimization problem. Notice that\nthe \ufb01rst term in (11) depends only on f\u03b1(p) for p \u2208 P(S`),\n\nf\u03b1(p) = \u03b4pK\u03b1 = (\u03b4pKQ\u22121)Q\u03b1 = \u03b3pQ\u03b1.\n\nSince \u03b3pQ\u03b1\u22a5 = 0, we conclude that the optimal solution to (11) is given by\n\n\u03b1\u2217 = X\n\np\u2208P(S`)\n\n\u03b2p\u03b3p = \u03b2KQ\u22121,\n\n(12)\n\nwhere \u03b2 is required to be sparse, such that only parts from the labeled data are\nnonzero. Plugging this into original equations we get\n\n\u02dck(p, p0) = kpQ\u22121kp0\nf\u03b2(p0) = X\n\np\u2208P(S`)\n\u03b2\u2217 = argmin\n\n\u03b2p\u02dck(p, p0)\n\nL(S`, f\u03b2) + \u03b2T \u02dcK\u03b2\n\n(13)\n\n(14)\n\n(15)\n\n\u03b2\n\nwhere kp is the vector of k(p, p0) for all p0 \u2208 P(S) and \u02dcK is the matrix of \u02dck(p, p0)\nfor all p, p0 in P(S`). \u02dck is the same as in [13].\nWe call \u02dck the graph-regularized kernel, in which unlabeled data points are used to\naugment the base kernel k wrt the standard graph kernel to take the underlying\ndensity on parts into account. This kernel is de\ufb01ned over the complete part space,\nwhere as standard graph kernels are restricted to P(S) only.\n\nGiven the graph-regularized kernel, the semi-supervised structured learning problem\nis reduced to supervised structured learning. Since in semi-supervised learning\nproblems, in general, labeled data points are far fewer than unlabeled data, the\ndimensionality of the optimization problems is greatly reduced by this reduction.\n\n\f4 Structured Max-Margin Learning\n\nWe now investigate optimizing the hinge loss as de\ufb01ned by (6) using graph-\nregularized kernel \u02dck. De\ufb01ning \u03b3x,y to be the vector where \u03b3x,y\np = c(p, hx, yi) is\nthe count of p in hx, yi, the linear discriminant can be written in matrix notation\nfor x \u2208 S` as\n\nFf\u03b2 (x, y) = \u03b2T \u02dcK\u03b3x,y.\n\nThen, the optimization problem for margin maximization is\n\nl\n\n\u03b2\u2217 = argmin\n\n\u03b2\n\nmin\n\n\u03be\n\n\u03bei \u2265 max\n\n\u02c6y\u2208Y(xi)\n\ni=1\n\n\u03bei + \u03b2T \u02dcK\u03b2\n\nX\n4(\u02c6y, yi) \u2212 \u03b2T \u02dcK (cid:16)\u03b3xi,yi\n\n\u2212 \u03b3xi,\u02c6y(cid:17)\n\n\u2200i \u2264 l.\n\nThis gives a convex quadratic program over the vectors indexed by P(S), a poly-\nnomial size problem in terms of the size of the structures. Following [2], we replace\nthe convex constraints by linear constraints for all y \u2208 Y(x) and using Lagrangian\nduality techniques, we get the following dual Quadratic program:\n\n\u03b8\u2217 = argmin\n\n\u03b8T dR \u03b8 \u2212 \u2206T \u03b8\n\n(16)\n\n\u03b8\n\n\u03b8(xi,y) \u2265 0, X\n\ny\u2208Y(x)\n\n\u03b8(xi,y) = 1,\n\n\u2200y \u2208 Y(xi),\n\n\u2200i \u2264 l,\n\nwhere \u2206 is a vector of 4(y, \u02c6y) for all y \u2208 Y(x) of all labeled observations x, d\u03b3 is a\nmatrix whose (xi, y)th column d\u03b3., (xi, y) = \u03b3xi,yi\n\u2212 \u03b3xi,y and dR = d\u03b3T \u02dcKd\u03b3. Due\nto the sparse structure of the constraint matrix, even though this is an exponential\nsized QP, the algorithm proposed in [2] is proven to solve (16) to \u03b7 proximity in\npolynomial time in P(Sl) and 1\n\n\u03b7 [15].\n\n5 Semi-Supervised vs Transductive Learning\n\nSince one major contribution of this paper is learning a classi\ufb01er for structured ob-\njects that is de\ufb01ned over the complete part space P, we now examine the di\ufb00erences\nof semi-supervised and transductive learning in more detail. The most common ap-\nproach to realize the smoothness assumption is to construct a data dependent kernel\nkS derived from the graph Laplacian on a nearest neighbor graph on the labeled and\nunlabeled input patterns in the sample S. Thus, kS is not de\ufb01ned on observations\nthat are out of the sample. Given kS, one can construct a function \u02dcf \u2217 on S as\n\n\u02dcf \u2217 = argmin\nf \u2208HkS\n\n`\n\nX\n\ni=1\n\nL(xi, yi, f ) + \u03bb||f ||2\nkS\n\n.\n\n(17)\n\nIt is well known that kernels can be combined linearly to yield new kernels. This\nobservation in the transductive setting leads to the following optimization problem,\nwhen the kernel of the optimization problem is taken to be a linear combination of\na graph kernel kS and a standard kernel k restricted to P(S).\n\n\u00aff \u2217 =\n\nargmin\n\nf \u2208H(\u00b51 k+\u00b52 kS )\n\n`\n\nX\n\ni=1\n\nL(xi, yi, f ) + \u03bb||f ||2\n\n(\u00b51k+\u00b52kS )\n\n(18)\n\nA structured semi-supervised algorithm based on (18) has been evaluated in [11].\nThe kernel is (18) is the weighted mean of k and kS, whereas the graph-regularized\n\n\fkernel, resulting from weighted mean of two regularizers, is the harmonic mean of k\nand kS [16]. An important distinction between \u00aff \u2217 and f \u2217 in (8), the optimization\nperformed in this paper, is that \u00aff \u2217 is only de\ufb01ned on P(S) (only on observations\nin the training data) while f \u2217 is de\ufb01ned on all of P and can be used for novel (out\nof sample) inputs x. We note that in general P is in\ufb01nite. Out-of-sample extension\nis already a serious limitation for transductive learning, but it is even more severe\nin the structured case where parts of P can be composed of multiple observation\ntokens.\n\n6 Experiments\n\nSimilarity Graph: We build the similarity matrix W over P(S) using K-nearest\nneighborhood relationship. Wp,p0 is 0 if p and p0 are not in the K-nearest neighbor-\nhood of each other or if p and p0 are of di\ufb00erent types. Otherwise, the similarity is\ngiven by a heat kernel. In our applications, the structure is a simple chain, therefore\nthe cliques involved single observation label pairs,\n\nWp,p0 = \u03b4(y(up), y(u0\n\np0 ))e\n\nkup\u2212u0\n\np0 k2\n\nt\n\n,\n\n(19)\n\nwhere up denotes the observation part of p and y(u) denotes the labeling of u 1. In\ncases where k(p, p0) = Wp,p0 = 0 for p, p0 of di\ufb00erent types, as in our experiments, the\nGram matrix K and the Laplacian L can be presented as block diagonal matrices,\nwhich signi\ufb01cantly reduces the computational complexity, the computation of Q\u22121\nin particular.\n\nApplications: We performed experiments using a simple chain model for pitch\naccent (PA) prediction and OCR. In PA prediction, Y(x) = {0, 1}T with T = |x|\nand xt \u2208 <31, \u2200t. In OCR, xt \u2208 {0, 1}128 and |\u03a3| = 15.\n\nPA\n\nSVM\n\nSTR\n\nU:0\n65.92\n\n65.81\n\n-\n\n-\n\nU:0\n70.34\n\nU:80\n68.83\n69.94\n70.28\n70.72\n\nU:80 U:200\n71.27\n73.68\n73.11\n72.00\n76.37\n74.92\n75.66\n77.45\n\nWe ran experiments com-\nparing\nsemi-supervised\nstructured (referred as STR)\nand unstructured (referred\nas SVM) max-margin opti-\nmization. For both SVM and\nSTR, we used RBF kernel as\nthe base kernel ko in (4) and\na 5-nearest neighbor graph\nto construct the Laplacian.\nWe chose the width of the RBF kernel by cross-validation on SVM and used the\nsame value for STR. Following [3], we \ufb01xed \u03bb1 : \u03bb2 ratio at 1 : 9. We report\nthe average results of experiments with 5 random selection of labeled sequences\nin Table 1 and 2, with number of labeled sequences 4 on the left side of Table\n1, 40 on the right side, and 10 in Table 2. We varied the number of unlabeled\nsequences and reported the per-label accuracy of test sequences (on top of each\ncell) and of unlabeled sequences (bottom) (when U > 0). The results in pitch\naccent prediction shows the advantage of a sequence model over a non-structured\n\nTable 1: Per-label accuracy for Pitch Accent.\n\n72.15\n\n-\n\n-\n\n1For more complicated parts, di\ufb00erent measures can apply. For example, in sequence\nclassi\ufb01cation, if the classi\ufb01er is evaluated wrt the correctly classi\ufb01ed individual labels in\nthe sequence, W can be s. t. Wp,p0 = Pu\u2208p,u0\u2208p0 \u03b4(y(u), y(u0))\u02dcs(u, u0) where \u02dcs denotes\nsome similarity measure such as the heat kernel.\nIf the evaluation is over segments of\nthe sequence, the similarity can be Wp,p0 = \u03b4(y(p), y0(p0)) Pu\u2208p,u0\u2208p0 \u02dcs(u, u0) where y(p)\ndenotes all the label nodes in the part p.\n\n\fmodel, where STR consistently performs better than SVM. We also observe the\nusefulness of unlabeled data both in the structured and unstructured models, where\nas U increases, so does the accuracy. The improvement from unlabeled data and\nfrom structured classi\ufb01cation can be considered as additive. The small di\ufb00erence\nbetween the accuracy of in-sample unlabeled data and the test data indicates the\nnatural extension of our framework to new data points.\n\nIn OCR, on the other hand, STR does not improve\nover SVM. Even though unlabeled data improves ac-\ncuracy, performing sequence classi\ufb01cation is not help-\nful due to the sparsity of structural information. Since\n|\u03a3| = 15 and there are only 10 labeled sequences with\naverage length 8.3, the statistics of label-label depen-\ndency is quite noisy.\n\n7 Conclusions\n\nOCR\n\nSVM\n\nSTR\n\nU:0\n43.62\n\n-\n\n49.25\n\n-\n\nU:412\n49.96\n47.56\n49.91\n49.65\n\nTable 2: OCR\n\nWe presented a discriminative approach to semi-supervised learning of structured\nand inter-dependent response variables.\nIn this framework, we derived a maxi-\nmum margin formulation and presented experiments for a simple chain model. Our\napproach naturally extends to the classi\ufb01cation of unobserved structured inputs\nand this is supported by our empirical results which showed similar accuracy on\nin-sample unlabeled data and out-of-sample test data.\n\nReferences\n\n[1] Y. Altun, T. Hofmann, and A. Smola. Gaussian process classi\ufb01cation for segmenting\n\nand annotating sequences. In ICML, 2004.\n\n[2] Y. Altun, I. Tsochantaridis, and T. Hofmann. Hidden markov support vector ma-\n\nchines. In ICML, 2003.\n\n[3] M. Belkin, P. Niyogi, and V. Sindhwani. Manifold regularization: a geometric frame-\n\nwork for learning from examples. Technical Report 06, UChicago CS, 2004.\n\n[4] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-\n\ntraining. In COLT, 1998.\n\n[5] U. Brefeld, C. B\u00a8uscher, and T. Sche\ufb00er. Multi-view discriminative sequential learning.\n\nIn (ECML), 2005.\n\n[6] O. Chappelle, J. Weston, and B. Scholkopf. Cluster kernels for semi-supervised learn-\n\ning. In (NIPS), 2002.\n\n[7] M. Collins and N.l Du\ufb00y. Convolution kernels for natural language. In (NIPS), 2001.\n[8] Olivier Delalleau, Yoshua Bengio, and Nicolas Le Roux. E\ufb03cient non-parametric\n\nfunction induction in semi-supervised learning. In Proceedings of AISTAT, 2005.\n\n[9] Thorsten Joachims. Transductive inference for text classi\ufb01cation using support vector\n\nmachines. In (ICML), pages 200\u2013209, 1999.\n\n[10] G. Kimeldorf and G. Wahba. Some results on tchebychean spline functions. Journal\n\nof Mathematics Analysis and Applications, 33:82\u201395, 1971.\n\n[11] John La\ufb00erty, Yan Liu, and Xiaojin Zhu. Kernel conditional random \ufb01elds: Repre-\n\nsentation, clique selection, and semi-supervised learning. In (ICML), 2004.\n\n[12] K. Nigam, A. K. McCallum, S. Thrun, and T. M. Mitchell. Learning to classify text\nfrom labeled and unlabeled documents. In Proceedings of AAAI-98, pages 792\u2013799,\nMadison, US, 1998.\n\n[13] V. Sindhwani, P. Niyogi, and M. Belkin. Beyond the point cloud: from transductive\n\nto semi-supervised learning. In (ICML), 2005.\n\n[14] B. Taskar, C. Guestrin, and D. Koller. Max-margin markov networks. In NIPS, 2004.\n[15] I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun. Support vector machine\n\nlearning for interdependent and structured output spaces. In (ICML), 2004.\n\n[16] T. Zhang. personal communication.\n\n\f", "award": [], "sourceid": 2850, "authors": [{"given_name": "Y.", "family_name": "Altun", "institution": null}, {"given_name": "D.", "family_name": "McAllester", "institution": null}, {"given_name": "M.", "family_name": "Belkin", "institution": null}]}