{"title": "Learning from Data of Variable Quality", "book": "Advances in Neural Information Processing Systems", "page_first": 219, "page_last": 226, "abstract": "", "full_text": "Learning from Data of Variable Quality\n\nKoby Crammer, Michael Kearns, Jennifer Wortman\n\nComputer and Information Science\n\nUniversity of Pennsylvania\n\nPhiladelphia, PA 19103\n\n{crammer,mkearns,wortmanj}@cis.upenn.edu\n\nAbstract\n\nWe initiate the study of learning from multiple sources of limited data,\neach of which may be corrupted at a different rate. We develop a com-\nplete theory of which data sources should be used for two fundamental\nproblems: estimating the bias of a coin, and learning a classi\ufb01er in the\npresence of label noise. In both cases, ef\ufb01cient algorithms are provided\nfor computing the optimal subset of data.\n\n1\n\nIntroduction\n\nIn many natural machine learning settings, one is not only faced with data that may be cor-\nrupted or de\ufb01cient in some way (classi\ufb01cation noise or other label errors, missing attributes,\nand so on), but with data that is not uniformly corrupted. In other words, we might be pre-\nsented with data of variable quality \u2014 perhaps some small amount of entirely \u201cclean\u201d data,\nanother amount of slightly corrupted data, yet more that is signi\ufb01cantly corrupted, and so\non. Furthermore, in such circumstances we may often know at least an upper bound on the\nrate and type of corruption in each pile of data. An extreme example is the recent interest in\nsettings where one has a very limited set of correctly labeled examples, and an effectively\nunlimited set of entirely unlabeled examples, as naturally arises in problems such as clas-\nsifying web pages [1]. Another general category of problems that falls within our interest\nis when multiple piles of data are drawn from processes that differ perhaps slightly and in\nvarying amounts from the process we wish to estimate. For example, we might wish to\nestimate a conditional distribution P (X|Y = y) but have only a small number of observa-\ntions in which Y = y, but a larger number of observations in which Y = y\u2032 for values of\ny\u2032 \u201cnear\u201d to y. In such circumstances it might make sense to base our model on a larger\nnumber of observations, at least for those y\u2032 closest to y.\nWhile there is a large body of learning theory both for uncorrupted data and for data that\nis uniformly corrupted in some way [2, 3], there is no general framework and theory for\nlearning from data of variable quality. In this paper we introduce such a framework, and\ndevelop its theory, for two basic problems: estimating a bias from corrupted coins, and\nlearning a classi\ufb01er in the presence of varying amounts of label noise. For the corrupted\ncoins case we provide an upper bound on the error that is expressed as a trade-off between\nweighted approximation errors and larger amounts of data. This bound provides a building\nblock for the classi\ufb01cation noise setting, in which we are able to give a bound on the\ngeneralization error of empirical risk minimization that speci\ufb01es the optimal subset of the\n\n\fdata to use. Both bounds can computed by simple and ef\ufb01cient algorithms. We illustrate\nboth problems and our algorithms with numerical simulations.\n\n2 Estimating the Bias from Corrupted Coins\n\nWe begin by considering perhaps the simplest possible instance of the general class of\nproblems in which we are interested \u2014 namely, the problem of estimating the unknown\nbias of a coin. In this version of the variable quality model, we will have access to different\namounts of data from \u201ccorrupted\u201d coins whose bias differs from the one we wish to esti-\nmate. We use our solution for this simple problem as a building block for the classi\ufb01cation\nnoise setting in Section 3.\n\n2.1 Problem Description\n\nSuppose we wish to estimate the bias \u03b2 of a coin given K piles of training observations\nN1, ..., NK. Each pile Ni contains ni outcomes of \ufb02ips of a coin with bias \u03b2i, where the\nonly information we are provided is that \u03b2i \u2208 [\u03b2 \u2212 \u01ebi, \u03b2 + \u01ebi], and 0 \u2264 \u01eb1 \u2264 \u01eb2 \u2264 ... \u2264 \u01ebK.\nWe refer to the \u01ebi as bounds on the approximation errors of the corrupted coins. We denote\nby hi the number of heads observed in the ith pile. Our immediate goal is to determine\nwhich piles should be considered in order to obtain the best estimate of the true bias \u03b2.\nWe consider estimates for \u03b2 obtained by merging some subset of the data into a single\nuni\ufb01ed pile, and computing the maximum likelihood estimate for \u03b2, which is simply the\nfraction of times heads appears as an outcome in the uni\ufb01ed pile. Although one can con-\nsider using any subset of the data, it can be proved (and is intuitively obvious) that an\noptimal estimate (in the sense that will be de\ufb01ned shortly) always uses a pre\ufb01x of the data,\ni.e. all data from the piles indexed 1 to k for some k \u2264 K, and possibly a subset of the data\nfrom pile k + 1. In fact, it will be shown that only complete piles need to be considered.\nTherefore, from this point on we restrict ourselves to estimates of this form, and identify\nthem by the maximal index k of the piles used. The associated estimate is then simply\n\nWe denote the expectation of this estimate by\n\n\u02c6\u03b2k =\n\nh1 + . . . + hk\nn1 + . . . + nk\n\n.\n\n\u00af\u03b2k = Eh \u02c6\u03b2ki =\n\nn1\u03b21 + . . . + nk\u03b2k\n\nn1 + . . . + nk\n\n.\n\nTo simplify the presentation we denote by ni,j the number of outcomes in piles Ni, . . . , Nj,\n\nthat is, ni,j =Pj\n\nm=i nm .\n\nWe now bound the deviation of the estimate \u02c6\u03b2k from the true bias of the coin \u03b2 using the\nexpectation \u00af\u03b2k:\n\n|\u03b2 \u2212 \u02c6\u03b2k| = |\u03b2 \u2212 \u00af\u03b2k + \u00af\u03b2k \u2212 \u02c6\u03b2k|\n\u2264 |\u03b2 \u2212 \u00af\u03b2k| + | \u00af\u03b2k \u2212 \u02c6\u03b2k|\n\n\u2264\n\nk\n\nXi=1\n\nni\nn1,k\n\n\u01ebi + | \u00af\u03b2k \u2212 \u02c6\u03b2k|\n\nThe \ufb01rst inequality follows from the triangle inequality and the second from our assump-\ntions. Using the Hoeffding inequality we can bound the second term and \ufb01nd that with high\nprobability for an appropriate choice of \u03b4 we have\n\n|\u03b2 \u2212 \u02c6\u03b2k| \u2264\n\nni\nn1,k\n\n\u01ebi +s log(2K/\u03b4)\n\n2n1,k\n\n.\n\nk\n\nXi=1\n\n(1)\n\n\fTo summarize, we have proved the following theorem.\n\nTheorem 1 Let \u02c6\u03b2k be the estimate obtained by using only the data from the \ufb01rst k piles.\nThen for any \u03b4 > 0, with probability \u2265 1 \u2212 \u03b4 we have\n\nsimultaneously for all k = 1, . . . , K.\n\n(cid:12)(cid:12)(cid:12)\n\n\u03b2 \u2212 \u02c6\u03b2k(cid:12)(cid:12)(cid:12)\n\n\u2264\n\nni\nn1,k\n\n\u01ebi +s log(2K/\u03b4)\n\n2n1,k\n\nk\n\nXi=1\n\nTwo remarks are in place here. First, the theorem is data-independent since it does not take\ninto account the actual outcomes of the experiments h1, . . . , hK. Second, the two terms in\nthe bound re\ufb02ect the well-known trade-off between bias (approximation error) and variance\n(estimation error). The \ufb01rst term bounds the approximation error of replacing the true coin\n\u03b2 with the average \u00af\u03b2k. The second term corresponds to the estimation error which arises\nas a result of our \ufb01nite sample size.\n\nThis theorem implies a natural algorithm to choose the number of piles k\u2217 as is the mini-\nmizer of the bound over the number of piles used:\n\nk\u2217 = argmin\n\nk\u2208{1,...,K}( k\nXi=1\n\nni\nn1,k\n\n\u01ebi +s log(2K/\u03b4)\n\n2n1,k ) .\n\nTo conclude this section we argue that our choice of using a pre\ufb01x of piles is optimal. First,\nnote that by adding a new pile with a corruption level \u01eb smaller then the current corruption\nlevel, we can always reduce the bounds. Thus it is optimal to use pre\ufb01x of the piles and\nnot to ignore piles with low corruption levels. Second, we need to show that if we decide\nto use a pile, it will be optimal to use all of it. Note that we can choose to view each coin\ntoss as a separate pile with a single observation, thus yielding n1,K piles of size 1. The\nfollowing technical lemma states that under this view of singleton piles, once we decide to\nadd a pile with some corruption level, it will be optimal to use all singleton piles with the\nsame corruption level. The proof of this lemma is omitted due to lack of space.\n\nLemma 1 Assume that all the piles are of size ni = 1 and that \u01ebk \u2264 \u01ebp+k = \u01ebp+k+1. Then\nthe following two inequalities cannot hold simultaneously:\n\nni\nn1,k\n\nk\n\nXi=1\n\nni\n\nn1,k+p+1\n\nk+p+1\n\nXi=1\n\n2n1,k\n\n\u01ebi +s log(2n1,K /\u03b4)\n\u01ebi +s log(2n1,K /\u03b4)\n\n2n1,k+p+1\n\n>\n\n\u2265\n\nni\n\nn1,k+p\n\nni\n\nn1,k+p\n\nk+p\n\nk+p\n\nXi=1\nXi=1\n\n2n1,k+p\n\n\u01ebi +s log(2n1,K /\u03b4)\n\u01ebi +s log(2n1,K /\u03b4)\n\n2n1,k+p\n\n.\n\nIn other words, if the bound on |\u03b2 \u2212 \u02c6\u03b2k+p| is smaller than the bound on |\u03b2 \u2212 \u02c6\u03b2k|, then\nthe bound on |\u03b2 \u2212 \u02c6\u03b2k+p+1| must be smaller than both unless \u01ebk+p+1 > \u01ebk+p. Thus if the\npth and p+1th samples are from the same original pile (and \u01ebk+p+1 = \u01ebk+p), then once we\ndecide to use samples through p, we will always want to include sample p + 1. It follows\nthat we must only consider using complete piles of data.\n\n2.2 Corrupted Coins Simulations\n\nThe theory developed so far can be nicely illustrated via some simple simulations. We\nbrie\ufb02y describe just one such experiment in which there were K = 8 piles. The tar-\nget coin was fair: \u03b2 = 0.5. The approximation errors of the corrupted coins were\n\n\fActual Error\nBound\nSingeltons Bound\n\ne4 + B(4, k)\n\ne6\n\ne5\n\ne4\n\ne3\n\ni\u2217\nk\n\n= 2\n\ne2\n\ne1\n\nr\no\nr\nr\n\nE\n\n1\n\n0.8\n\n0.6\n\n0.4\n\n0.2\n\n0\n\n2\n\n4\n\nError Bound\nActual Error\nAchieved Error\n\n6\n\nNumber of Piles Used\n\n8\n\n10\n\n12\n\n1\n\n0.9\n\n0.8\n\n0.7\n\n0.6\n\n0.5\n\n0.4\n\n0.3\n\n0.2\n\n0.1\n\nr\no\nr\nr\n\nE\n\n101\n\n102\nNumber of Examples Used\n\n103\n\n104\n\nFigure 1: Left: Illustration of the actual error and our error bounds for estimating the bias\nof a coin. The error bars show one standard deviation. Center: Illustration of the interval\nconstruction. Right: Illustration of actual error of a 20 dimensional classi\ufb01cation problem\nand the error bounds found using our methods.\n\n~\u01eb = (0.001, 0.01, 0.02, 0.03, 0.04, 0.2, 0.3, 0.5), and number of outcomes in the corre-\nsponding piles were ~n = (10, 50, 100, 500, 1500, 2000, 3000, 10000). The following pro-\ncess was repeated 1, 000 times. We set the probability of the ith coin to be \u03b2i = \u03b2 + \u01ebi\nand sampled ni times from it. We then used all possible pre\ufb01xes 1, . . . , k of piles to es-\ntimate \u03b2. For each k, we computed the bound for the estimate using piles 1, . . . , k using\nthe theory developed in the previous section. To illustrate Lemma 1 we also computed the\nbound using partial piles. This bound is slightly higher than the suggested bound since we\nuse effectively more piles (n1,K instead of K). As the lemma predicts, it is not valuable to\nuse subsets of piles. Simulations with other values of K, ~\u01eb and ~n yield similar qualitative\nbehavior. We note that a strength of the theory developed is its generality, as it provides\nbounds for any model parameters.\n\nThe leftmost panel of Figure 1 summarizes the simulation results. Empirically, the best\nestimate of the target coin is using the \ufb01rst four piles, while our algorithm suggests using\nthe \ufb01rst \ufb01ve piles. However, the empirical difference in quality between the two estimates\nis negligible, so the theory has given near-optimal guidance in this case. We note that while\nour bounds have essentially the right shape (which is what matters for the computation\nof k\u2217), numerically they are quite loose compared to the true behavior. There are various\nlimits to the numerical precision we should expect without increasing the complexity of the\ntheory \u2014 for example, the precision is limited by accuracy of constants in the Hoeffding\ninequality and the use of the union bound.\n\n3 Classi\ufb01cation with Label Noise\n\nWe next explore the problem of classi\ufb01cation in the presence of multiple data sets with\nvarying amounts of label noise. The setting is as follows. We assume there is a \ufb01xed and\nunknown binary function f : X \u2192 {0, 1} and a \ufb01xed and unknown distribution P on the\ninputs X to f. We are presented again with K piles of data, N1, ..., NK. Now each pile\nNi contains ni labeled examples (x, y) that are generated from the target function f with\nlabel noise at rate \u03b7i, where 0 \u2264 \u03b71 < \u03b72 < ... < \u03b7K. In other words, for each example\n(x, y) in pile Ni, y = f (x) with probability 1 \u2212 \u03b7i and y = \u00acf (x) with probability \u03b7i.\nThe goal is to decide which piles of data to use in order to choose a function h from a set\nof hypothesis functions H with minimal generalization (true) error e(h) with respect to f\nand P . As before, for any pre\ufb01x of piles N1, . . . , Nk, we examine the most basic estimator\nbased on this data, namely the hypothesis minimizing the observed or training error:\n\n\u02c6hk = argmin\n\n{\u02c6ek(h)}\n\nh\u2208H\n\n\fwhere \u02c6ek(h) is the fraction of times h(x) 6= y over all (x, y) \u2208 N1 \u222a \u00b7 \u00b7 \u00b7 \u222a Nk. Thus we\nexamine the standard empirical risk minimization framework [2]. Generalizing from the\nbiased coin setting, we are interested in three primary questions: what can we say about\nthe deviation |e(\u02c6hk) \u2212 \u02c6e(\u02c6hk)|, which is the gap between the true and observed error of the\nestimator \u02c6hk; what is the optimal value of k; and how can we compute the corresponding\nbounds?\n\nWe note that the classi\ufb01cation noise setting can naturally be viewed as a special case of\na more general and challenging \u201cagnostic\u201d classi\ufb01cation setting that we discuss brie\ufb02y in\nSection 4. Here we provide a more specialized solution that exploits particular properties\nof class label noise.\nWe begin by observing that for any \ufb01xed function h, the question of how \u02c6ek(h) is related\nto e(h) bears great similarity to the biased coin setting. More precisely, the expected clas-\nsi\ufb01cation error of h on pile Ni only is\n\n(1 \u2212 \u03b7i)e(h) + \u03b7i(1 \u2212 e(h)) = e(h) + \u03b7i(1 \u2212 2e(h)) .\n\nThus if we set\n\n\u03b2 = e(h),\n\n\u01ebi = \u03b7i |1 \u2212 2e(h)|\n\n(2)\nand if we were only concerned with making the best use of the data in estimating e(h), we\ncould attempt to apply the theory developed in Section 2 using the reduction above. There\nare two distinct and obvious dif\ufb01culties. The \ufb01rst dif\ufb01culty is that even restricting attention\nto estimating e(h) for a \ufb01xed h, the values for \u01ebi above (and thus the bounds computed\nby the methods of Section 2) depend on e(h), which is exactly the unknown quantity we\nwould like to estimate. The second dif\ufb01culty is that in order to bound the performance of\nempirical error minimization within H, we must say something about the probability of\nany h \u2208 H being selected. We address each of these dif\ufb01culties in turn.\n\n3.1 Computing the Error Bound Matrix\n\nFor now we assume that {e(h) : h \u2208 H} is a \ufb01nite set containing M values e1 < . . . < eM .\nThis assumption clearly holds if |H| is \ufb01nite, and can be removed entirely by discretizing\nthe values in {e(h) : h \u2208 H}. For convenience we assume that for all levels ei there exists\na function h \u2208 H such that e(h) = ei. This assumption can also be removed (details of\nboth omitted due to space considerations). We de\ufb01ne a matrix B of estimation errors as\nfollows. Each row i of B represents one possible value of e(h) = ei, while each column\nk represents the use of only piles N1, . . . , Nk of noisy labeled examples of the target f.\nThe entry B(i, k) will contain a bound on |e(h) \u2212 \u02c6ek(h)| that is valid simultaneously for all\nh \u2208 H with e(h) = ei. In other words, for any such h, with high probability \u02c6ek(h) falls in\nthe range [ei \u2212 B(i, k), ei + B(i, k)]. It is crucial to note that we do not need to know which\nfunctions h \u2208 H satisfy e(h) = ei in order to either compute or use the bound B(i, k), as\nwe shall see shortly. Rather, it is enough to know that for each h \u2208 H, some row of B will\nprovide estimation error bounds for each k.\nThe values in B can be now be calculated using the settings provided by Eq. (2) and the\nbound in Eq. (1). However, since Eq. (1) applies to the case of a single biased coin and here\nwe have many (essentially one for each function at a given generalization error ei), we must\nmodify it slightly. We can (pessimistically) bound the VC dimension of all functions with\nerror rate e(h) = ei by the VC dimension d of the entire class H. Formally, we replace the\nsquare root term in Eq. (1) with the following expression, which is a simple application of\nVC theory [2, 3]:\n\nO s 1\n\nn1,k (cid:18)d log(cid:16) n1,k\n\nd (cid:17) + log(cid:18) KM\n\n\u03b4 (cid:19)(cid:19)! .\n\n(3)\n\n\fWe note that in cases where we have more information on the structure of the generalization\nerrors in H, an accordingly modi\ufb01ed equation can be used, which may yield considerably\nimproved bounds. For example, in the statistical physics theory of learning curves[4] it is\ncommon to posit knowledge of the density or number of functions in H at a given gener-\nalization error ei. In such a case we could clearly substitute the VC dimension d by the\n(potentially much smaller) VC dimension di of just this subclass.\nIn a moment we describe how the matrix B can be used to choose the number k of piles\nto use, and to compute a bound on the generalization error of \u02c6hk. We \ufb01rst formalize the\ndevelopment above as an intermediate result.\n\nLemma 2 Suppose H is a set of binary functions with VC dimension d. Let M be the\nnumber of noise levels and K be the number of piles. Then for all \u03b4 > 0, with probability\nat least 1 \u2212 \u03b4, for all i \u2208 {1, . . . , M }, for all h \u2208 H with e(h) = ei, and for all k \u2208\n{1, . . . , K} we have\n\nThe matrix B can be computed in time linear in its size O(KM ).\n\n|e(h) \u2212 \u02c6ek(h)| \u2264 B(i, k) .\n\n3.2 Putting It All Together\n\nBy Lemma 2, the matrix B gives, for each possible generalization error ei and each k, an\nupper bound on the deviation between observed and true errors for functions of true error\nei when using piles N1, . . . , Nk. It is thus natural to try to use column k of B to bound the\nerror of \u02c6hk, the function minimizing the observed error on these piles.\nSuppose we \ufb01x the number of piles used to be k. The observed error of any function\nwith true generalization error ei must, with high probability, lie in the interval Ii,k =\n[ei \u2212 B(i, k), ei + B(i, k)]. By simultaneously considering these intervals for all values of\nei, we can put a bound on the generalization error of the best function in the hypothesis\nclass. This process is best illustrated by an example.\n\nConsider a hypothesis space in which the generalization error of the available functions can\ntake on the discrete values 0, 0.1, 0.2, 0.3, 0.4, and 0.5. Suppose the matrix B has been\ncalculated as above and the kth column is (0.16, 0.05, 0.08, 0.14, 0.07, 0.1). We know, for\nexample, that all functions with true generalization error e2 = 0.1 will show an error in\nthe range I2,k = [0.05, 0.15], and that all functions with true generalization error e4 = 0.3\nwill show an error in the range I4,k = [0.16, 0.44]. The center panel of Figure 1 illustrates\nthe span of each interval.\nExamining this diagram, it becomes clear that the function \u02c6hk minimizing the error on\nN1 \u222a \u00b7 \u00b7 \u00b7 \u222a Nk could not possibly be a function with true error e4 or higher as long as\nH contains at least one function with true error e2 since the observed error of the latter\nwould necessarily be lower (with high probability). Likewise, it would not be possible\nfor a function with true error e5 or e6 to be chosen. However, a function with true error e3\ncould produce a lower observed error than one with true error e1 or e2 (since e3 \u2212B(3, k) <\ne2 + B(2, k) and e3 \u2212 B(3, k) < e1 + B(1, k)), and thus could be chosen as \u02c6hk. Therefore,\nthe smallest bound we can place on the true error of \u02c6hk in this example is e3 = 0.2.\nIn general, we know that \u02c6hk will have true error corresponding to the midpoint of an a inter-\nval which overlaps with the interval with the least upper bound (I2,k in this example). This\nleads to an intuitive procedure for calculating a bound on the true error of \u02c6hk. First, we de-\nk = argmini{ei + B(i, k)}. Consider\ntermine the interval with the smallest upper bound, i\u2217\nthe set of intervals which overlap with i\u2217\nk, k)}.\nIt is possible for the smallest observed error to come from a function corresponding to any\n\nk, namely Jk = {i : ei \u2212B(i, k) \u2264 ei\u2217\n\nk +B(i\u2217\n\n\fof the intervals in Jk. Thus, a bound on the true error of \u02c6hk can be obtained by taking the\nmaximum e(h) value for any function in Jk, i.e. C(k)\nOur overall algorithm for bounding e(\u02c6hk) and choosing k\u2217 can thus be summarized:\n\n= maxi\u2208Jk {ei}.\n\ndef\n\n1. Compute the matrix B as described in Section 3.1 .\n2. Compute the vector C described above.\n3. Output k\u2217 = argmink{C(k)}.\n\nWe have established the following theorem.\n\nTheorem 2 Suppose H is a set of binary functions with VC dimension d. Let M be the\nnumber of noise levels and K be the number of piles. For all k = 1, ..., K, let \u02c6hk =\nargminh{\u02c6ek(h)} be the function in H with the lowest empirical error evaluated using the\n\ufb01rst k piles of data. Then for all \u03b4 > 0, with probability at least 1 \u2212 \u03b4,\n\ne(\u02c6hk) \u2264 C(k)\n\nThe suggested choice of k is thus k\u2217 = argmink {C(k)}.\n\n3.3 Classi\ufb01cation Noise Simulations\n\nIn order to illustrate the methodology described in this section, simulations were run on a\nclassi\ufb01cation problem in which samples ~x \u2208 {0, 1}20 were chosen uniformly at random,\n\ni=1 xi > 10.\n\nand the target function f (~x) was 1 if and only ifP20\n\nClassi\ufb01cation models were created for k = 1, ..., K by training using the \ufb01rst k piles of data\nusing logistic regression with a learning rate of 0.0005 for a maximum of 5, 000 iterations.\nThe generalization error for each model was determined by testing on a noise-free sample\nof 500 examples drawn from the same uniform distribution. Bounds were calculated using\nthe algorithm described above with functions binned into 101 evenly spaced error values\n~e = (0, 0.01, 0.02, ..., 1) with \u03b4 = 0.001.\nThe right panel of Figure 1 shows an example of the bounds found with K = 12 piles,\nnoise levels ~\u03b7 = (0.001, 0.002, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5), and\nsample sizes ~n = (20, 150, 300, 400, 500, 600, 700, 1000, 1500, 2000, 3000, 5000). The\nalgorithm described above correctly predicts that the eighth pile should be chosen as the\ncutoff, yielding an optimal error value of 0.018. It is interesting to note that although the\nerror bounds shown are signi\ufb01cantly higher than the actual error, the shapes of the curves\nare similar. This phenomena is common to many uniform convergence bounds.\n\nFurther experimentation has shown that the algorithm described here works well in general\nwhen there are small piles of low noise data and large piles of high noise data. Its predic-\ntions are more useful in higher dimensional space, since it is relatively easy to get good\npredictions without much available data in lower dimensions.\n\n4 Further Research\n\nIn research subsequent to the results presented here [5], we examine a considerably more\ngeneral \u201cagnostic\u201d classi\ufb01cation setting [6]. As before, we assume there is a \ufb01xed and\nunknown binary function f : X \u2192 {0, 1} and a \ufb01xed and unknown distribution P on the\ninputs X to f. We are presented again with K piles of data, N1, ..., NK. Now each pile Ni\ncontains ni labeled examples (x, y) that are generated from an unknown function hi such\nthat e(hi) = e(hi, f ) = PrP [hi(x) 6= f (x)] \u2264 \u01ebi for given values \u01eb1 \u2264 . . . \u2264 \u01ebK. Thus\n\n\fwe are provided piles of labeled examples of unknown functions \u201cnearby\u201d the unknown\ntarget f, where \u201cnearby\u201d is quanti\ufb01ed by the sequence of \u01ebi.\nIn forthcoming work [5] we show that with high probability, for any k \u2264 K\n\ne(\u02c6hk, f ) \u2264 min\nh\u2208H\n\n{e(f, h)}+2\n\nk\n\nn1,k(cid:19) \u01ebi+O s 1\n\nXi=1(cid:18) ni\n\nn1,k (cid:18)d log(cid:16) n1,k\n\nd (cid:17) + log(cid:18) K\n\n\u03b4 (cid:19)(cid:19)!\n\nThis result again allows us to express the optimal number of piles as a trade-off between\nweighted approximation errors and increasing sample size. We suspect the result can be\nextended to a wider class of loss functions that just classi\ufb01cation.\n\nReferences\n[1] A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings\n\nof the Eleventh Annual Conference on Computational Learning Theory, pages 92\u2013100, 1998.\n\n[2] V. N. Vapnik. Statistical Learning Theory. Wiley, 1998.\n[3] M. J. Kearns and U. V. Vazirani. An Introduction to Computational Learning Theory. MIT Press,\n\n1994.\n\n[4] D. Haussler, M. Kearns, H.S. Seung, and N. Tishby. Rigorous learning curve bounds from\nstatistical mechanics. In Proceedings of the Seventh Annual ACM Conference on Computational\nLearning Theory, pages 76\u201387, 1994.\n\n[5] K. Crammer, M. Kearns, and J. Wortman. Forthcoming. 2006.\n[6] M. Kearns, R. Schapire, and L. Sellie. Towards ef\ufb01cient agnostic learning. Machine Learning,\n\n17:115\u2013141, 1994.\n\n\f", "award": [], "sourceid": 2920, "authors": [{"given_name": "Koby", "family_name": "Crammer", "institution": null}, {"given_name": "Michael", "family_name": "Kearns", "institution": null}, {"given_name": "Jennifer", "family_name": "Wortman", "institution": null}]}