{"title": "Searching for Character Models", "book": "Advances in Neural Information Processing Systems", "page_first": 331, "page_last": 338, "abstract": "", "full_text": "Searching for Character Models\n\nDepartment of Computer Science\n\nDepartment of Computer Science\n\nJaety Edwards\n\nUC Berkeley\n\nBerkeley, CA 94720\n\nDavid Forsyth\n\nUC Berkeley\n\nBerkeley, CA 94720\n\njaety@cs.berkeley.edu\n\ndaf@cs.berkeley.edu\n\nAbstract\n\nWe introduce a method to automatically improve character models for a\nhandwritten script without the use of transcriptions and using a minimum\nof document speci\ufb01c training data. We show that we can use searches for\nthe words in a dictionary to identify portions of the document whose\ntranscriptions are unambiguous. Using templates extracted from those\nregions, we retrain our character prediction model to drastically improve\nour search retrieval performance for words in the document.\n\n1 Introduction\n\nAn active area of research in machine transcription of handwritten documents is reducing\nthe amount and expense of supervised data required to train prediction models. Traditional\nOCR techniques require a large sample of hand segmented letter glyphs for training. This\nper character segmentation is expensive and often impractical to acquire, particularly if the\ncorpora in question contain documents in many different scripts.\n\nNumerous authors have presented methods for reducing the expense of training data by\nremoving the need to segment individual characters. Both Kopec et al [3] and LeCun et al\n[5] have presented models that take as input images of lines of text with their ASCII tran-\nscriptions. Training with these datasets is made possible by explicitly modelling possible\nsegmentations in addition to having a model for character templates.\n\nIn their research on \u201cwordspotting\u201d, Lavrenko et al [4] demonstrate that images of entire\nwords can be highly discriminative, even when the individual characters composing the\nword are locally ambiguous. This implies that images of many suf\ufb01ciently long words\nshould have unambiguous transcriptions, even when the character models are poorly tuned.\nIn our previous work, [2], the discriminatory power of whole words allowed us to achieve\nstrong search results with a model trained on a single example per character.\n\nThe above results have shown that A) one can learn new template models given images of\ntext lines and their associated transcriptions, [3, 5] without needing an explicit segmentation\nand that B) entire words can often be identi\ufb01ed unambiguously, even when the models for\nindividual characters are poorly tuned. [2, 4]. The \ufb01rst of these two points implies that\ngiven a transcription, we can learn new character models. The second implies that for at\nleast some parts of a document, we should be able to provide that transcription \u201cfor free\u201d,\nby matching against a dictionary of known words.\n\n\fs\n1\n\u2212d\n\ns\n2\ndi\n\ns\n3\nix\n\ns\n4\nxe\n\ns\n5\ner\n\ns\n6\nri\n\ns\n7\nis\n\ns\n8\ns\u2212\n\nFigure 1: A line, and the states that generate it. Each state st is de\ufb01ned by its left and\nright characters ctl and ctr (eg \u201cx\u201d and \u201ce\u201d for s4). In the image, a state spans half of each\nof these two characters, starting just past the center of the left character and extending to\nthe center of the right character, i.e. the right half of the \u201cx\u201d and the left half of the \u201ce\u201d\nin s4. The relative positions of the two characters is given by a displacement vector dt\n(superimposed on the image as white lines). Associating states with intracharacter spaces\ninstead of with individual characters allows for the bounding boxes of characters to overlap\nwhile maintaining the independence properties of the Markov chain.\nIn this work we combine these two observations in order to improve character models\nwithout the need for a document speci\ufb01c transcription. We provide a generic dictionary of\nwords in the target language. We then identify \u201chigh con\ufb01dence\u201d regions of a document.\nThese are image regions for which exactly one word from our dictionary scores highly\nunder our model. Given a set of high con\ufb01dence regions, we effectively have a training\ncorpus of text images with associated transcriptions. In these regions, we infer a segmen-\ntation and extract new character examples. Finally, we use these new exemplars to learn\nan improved character prediction model. As in [2], our document in this work is a 12th\ncentury manuscript of Terence\u2019s Comedies obtained from Oxford\u2019s Bodleian library [1].\n\n2 The Model\n\nHidden Markov Models are a natural and widely used method for modeling images of text.\nIn their simplest incarnation, a hidden state represents a character and the evidence variable\nis some feature vector calculated at points along the line. If all characters were known to\nbe of a single \ufb01xed width, this model would suf\ufb01ce. The probability of a line under this\nmodel is given as\n\np(line) = p(c1|\u03b1) Y\n\np(ct|ct\u22121)p(im[w\u2217(t\u22121):w\u2217t]|ct)\n\n(1)\n\nt>1\n\nwhere ct represents the tth character on the line, \u03b1 represents the start state, w is the width\nof a character, and im[w(t\u22121)+1:wt] represents the column of pixels beginning at column\nw \u2217 (t \u2212 1) + 1 of the image and ending at column w \u2217 t, (i.e. the set of pixels spanned by\nc)\n\nUnfortunately, character\u2019s widths do vary quite substantially and so we must extend the\nmodel to accommodate different possible segmentations. A generalized HMM allows us to\ndo this. In this model a hidden state is allowed to emit a variable length series of evidence\nvariables. We introduce an explicit distribution over the possible widths of a character.\nLetting dt be the displacement vector associated with the tth character, and ctx refer to the\nx location of the left edge of a character on the line, the probability of a line under this\nrevised model is\n\np(line) = p(c1|\u03b1) Y\n\np(ct|ct\u22121)p(dt|ct)p(im[ctx+1:ctx+d]|dt, ct)\n\n(2)\n\nt>1\n\nThis is the model we used in [2]. It performs far better than using an assumption of \ufb01xed\nwidths, but it still imposes unrealistic constraints on the relative positions of characters. In\n\n\fparticular, the portion of the ink generated by the current character is assumed to be inde-\npendent of the preceding character. In other words, the model assumes that the bounding\nboxes of characters do not overlap. This constraint is obviously unrealistic. Characters\nroutinely overlap in our documents. \u201cf\u201ds, for instance, form ligatures with most follow-\ning characters. In previous work, we treated this overlap as noise, hurting our ability to\ncorrectly localize templates. Under this model, local errors of alignment would also of-\nten propagate globally, adversely affecting the segmentation of the whole line. For search,\nthis noisy segmentation still provides acceptable results. In this work, however, we need\nto extract new templates, and thus correct localization and segmentation of templates is\ncrucial.\n\nIn our current work, we have relaxed this constraint, allowing characters to partially over-\nlap. We achieve this by changing hidden states to represent character bigrams instead of\nsingle characters (Figure 1). In the image, a state now spans the pixels from just past the\ncenter of the left character to the pixel containing the center of the right character. We\nadjust our notation somewhat to re\ufb02ect this change, letting st now represent the tth hid-\nden state and ctl and ctr be the left and right characters associated with s. dt is now the\ndisplacement vector between the centers of ctl and ctr.\nThe probability of a line under this, our actual, model is\n\np(line) = p(s1|\u03b1) Y\n\np(st|st\u22121)p(dt|ctl, ctr)p(im[stx+1:stx+dt]|ctl, ctr, dt)\n\n(3)\n\nt>1\n\nThis model allows overlap of bounding boxes, but it does still make the assumption that\nthe bounding box of the current character does not extend past the center of the previous\ncharacter. This assumption does not fully re\ufb02ect reality either. In Figure 1, for example,\nthe left descender of the x extends back further than the center of the preceding character.\nIt does, however, accurately re\ufb02ect the constraints within the heart of the line (excluding\nascenders and descenders). In practice, it has proven to generate very accurate segmenta-\ntions. Moreover, the errors we do encounter no longer tend to affect the entire line, since\nthe model has more \ufb02exibility with which to readjust back to the correct segmentation.\n\n2.1 Model Parameters\n\nOur transition distribution between states is simply a 3-gram character model. We train this\nmodel using a collection of ASCII Latin documents collected from the web. This set does\nnot include the transcriptions of our documents.\n\nConditioned on displacement vector, the emission model for generating an image chunk\ngiven a state is a mixture of gaussians. We associate with each character a set of image\nwindows extracted from various locations in the document. We initialize these sets with\none example a piece from our hand cut set (Figure 2). We adjust the probability of an image\ngiven the state to include the distribution over blocks by expanding the last term of Equation\n3 to re\ufb02ect this mixture. Letting bck represent the kth exemplar in the set associated with\ncharacter c, the conditional probability of an image region spanning the columns from x to\nx\u2032 is given as\n\np(imx:x\u2032|ctl, ctr, dt) = X\n\np(imx:x\u2032|bctli, bctrj, dt)\n\n(4)\n\ni,j\n\nIn principle, the displacement vectors should now be associated with an individual block,\nnot a character. This is especially true when we have both upper and lower case letters.\nHowever, our model does not seem particularly sensitive to this displacement distribution\nand so in practice, we have a single, fairly loose, displacement distribution per character.\n\nGiven a displacement vector, we can generate the maximum likelihood template image\nunder our model by compositing the correct halves of the left and right blocks. Reshaping\n\n\fthe image window into a vector, the likelihood of an image window is then modeled as\na gaussian, using the corresponding pixels in the template as the means, and assuming\na diagonal covariance matrix. The covariance matrix largely serves to mask out empty\nregions of a character\u2019s bounding box, so that we do not pay a penalty when the overlap of\ntwo characters\u2019 bounding boxes contains only whitespace.\n\n2.2 Ef\ufb01ciency Considerations\n\nThe number of possible different templates for a state is O(|B| \u00d7 |B| \u00d7 |D|), where |B| is\nthe number of different possible blocks and |D| is the number of candidate displacement\nvectors. To make inference in this model computationally feasible, we \ufb01rst restrict the\ndomain of d. For a given pair of blocks bl and br, we consider only displacement vectors\nwithin some small x distance from a mean displacement mbl,br, and we have a uniform\ndistribution within this region. m is initialized from the known size of our single hand cut\ntemplate. In the current work, we do not relearn the m. These are held \ufb01xed and assumed\nto be the same for all blocks associated with the same letter.\n\nEven when restricting the number of d\u2019s under consideration as discussed above, it is com-\nputationally infeasible to consider every possible location and pair of blocks. We therefore\nprune our candidate locations by looking at the likelihood of blocks in isolation and only\nconsidering locations where there is a local optimum in the response function and whose\nvalue is better than a given threshold. In this case our threshold for a given location is that\nL(block) < .7L(background) (where L(x) represents the negative log likelihood of x).\nIn other words, a location has to look at least marginally more like a given block than it\nlooks like the background.\n\nAfter pruning locations in this manner, we are left with a discrete set of \u201csites,\u201d where we\nde\ufb01ne a site as the tuple (block type, x location, y location). We can enumerate the set of\npossible states by looking at every pair of sites whose displacement vector has a non-zero\nprobability.\n\n2.3 Inference In The Model\n\nThe statespace de\ufb01ned above is a directed acyclic graph, anchored at the left edge and\nright edges of a line of text. A path through this lattice de\ufb01nes both a transcription and\na segmentation of the line into individual characters. Inference in this model is relatively\nstraightforward because of our constraint that each character may overlap only one pre-\nceding and one following character, and our restriction of displacement vectors to a small\ndiscrete range. The \ufb01rst restriction means that we need only consider binary relations be-\ntween templates. The second preserves the independence relationships of an HMM. A\ngiven state st is independent of the rest of the line given the values of all other states within\ndmax of either edge of st (where dmax is the legal displacement vector with the longest\nx component.) We can therefore easily calculate the best path or explicitly calculate the\nposterior of a node by traversing the state graph in topological order, sorted from left to\nright. The literature on Weighted Finite State Transducers ([6], [5]) is a good resource for\nef\ufb01cient algorithms on these types of statespace graph.\n\n3 Learning Better Character Templates\n\nWe initialize our algorithm with a set of handcut templates, exactly 1 per character, (Figure\n2), and our goal is to construct more accurate character models automatically from unsu-\npervised data. As noted above, we can easily calculate the posterior of a given site under\nour model. (Recall that a site is a particular character template at a given (x,y) location in\nthe line.) The traditional EM approach to estimating new templates would be to use these\n\n\fFigure 2: Original Training Data These 22 glyphs are our only document speci\ufb01c training\ndata. We use the model based on these characters to extract the new examples shown below\n\nFigure 3: Examples of extracted templates We extract new templates from high con\ufb01dence\nregions. From these, we choose a subset to incorporate into the model as new exemplars.\nTemplates are chosen iteratively to best cover the space of training examples. Notice that\nfor \u201cq\u201d and \u201ca\u201d, we have extracted capital letters, of which there were no examples in\nour original set of glyphs. This happens when the combination of constraints from the\ndictionary the surrounding glyphs make a \u201cq\u201d or \u201ca\u201d the only possible explanation for\nthis region, even though its local likelihood is poor.\n\nsites as training examples, weighted by their posteriors. Unfortunately, the constraints im-\nposed by 3 and even 4-gram character models seem to be insuf\ufb01cient. The posteriors of\nsites are not discriminative enough to get learning off the ground.\n\nThe key to successfully learning new templates lies is the observation from our previous\nwork [2], that even when the posteriors of individual characters are not discriminative, one\ncan still achieve very good search results with the same model. The search word in effect\nserves as its own language model, only allowing paths through the state graph that actually\ncontain it, and the longer the word the more it constrains the model. Whole words impose\nmuch tighter constraints than a 2 or 3-gram character model, and it is only with this added\npower that we can successfully learn new character templates.\n\nWe de\ufb01ne the score for a search as the negative log likelihood of the best path containing\nthat word. With suf\ufb01ciently long words, it becomes increasingly unlikely that a spurious\npath will achieve a high score. Moreover, if we are given a large dictionary of words and\nno alternative word explains a region of ink nearly as well as the best scoring word, then\nwe can be extremely con\ufb01dent that this is a true transcription of that piece of ink.\n\nStarting with a weak character model, we do not expect to \ufb01nd many of these \u201chigh con\ufb01-\ndence\u201d regions, but with a large enough document, we should expect to \ufb01nd some. From\nthese regions, we can extract new, reliable templates with which to improve our character\nmodels. The most valuable of these new templates will be those that are signi\ufb01cantly dif-\nferent from any in our current set. For example, in Figure 3, note that our system identi\ufb01es\ncapital Q\u2019s, even though our only input template was lower case. It identi\ufb01es this ink as\na Q in much the same way that a person solves a crossword puzzle. We can easily infer\nthe missing character in the string \u201cobv-ous\u201d because the other letters constrain us to one\npossible solution. Similarly, if other character templates in a word match well, then we can\nunambiguously identify the other, more ambiguous ones. In our Latin case, \u201cQuid\u201d is the\nonly likely explanation for \u201c-uid\u201d.\n\n3.1 Extracting New Templates and Updating The Model\n\nWithin a high con\ufb01dence region we have both a transcription and a localization of template\ncenters. It remains only to cut out new templates. We accomplish this by creating a template\nimage for the column of pixels from the corresponding block templates and then assigning\nimage pixels to the nearest template character (measured by Euclidean distance).\n\nGiven a set of templates extracted from high con\ufb01dence regions, we choose a subset of\n\n\fl\n\ne\nd\no\nM\n\n \nr\ne\nd\nn\nU\n \ne\nr\no\nc\nS\n\nworse\n3400\n3350\n\n3300\nbest\n\nConfidence Margins\n\nFigure 4: Each line segment in the lower \ufb01gure represents a proposed location for a word\nfrom our dictionary. It\u2019s vertical height is the score of that location under our model. A\nlower score represents a better \ufb01t. The dotted line is the score of our model\u2019s best possible\npath. Three correct words, \u201cnec\u201d, \u201cquin\u201d and \u201cdari\u201d, are actually on the best path. We\nde\ufb01ne the con\ufb01dence margin of a location as the difference in score between the best\n\ufb01tting word from our dictionary and the next best.\n\nFigure 5: Extracting Templates For a region with suf\ufb01ciently high con\ufb01dence margin, we\nconstruct the maximum likelihood template from our current exemplars. left, and we assign\npixels from the original image to a template based on its distance to the nearest pixel in\nthe template image, extracting new glyph exemplars right. These new glyphs become the\nexemplars for our next round of training.\n\ntemplates that best explain the remaining examples. We do this in a greedy fashion by\nchoosing the example whose likelihood is lowest under our current model and adding it to\nour set. Currently, we threshold the number of new templates for the sake of ef\ufb01ciency. Fi-\nnally, given the new set of templates, we can add them to the model and rerun our searches,\npotentially identifying new high con\ufb01dence regions.\n\n4 Results\n\nOur algorithm iteratively improves the character model by gathering new training data from\nhigh con\ufb01dence regions. Figure 3 shows that this method \ufb01nds new templates signi\ufb01cantly\ndifferent from the originals. In this document, our set of examples after one round appears\nto cover the space of character images well, at least those in lower case. Our templates are\nnot perfect. The \u201ca\u201d, for instance, has become associated with at least one block that is in\nfact an \u201co\u201d. These mistakes are uncommon, particularly if we restrict ourselves to longer\nwords. Those that do occur introduce a tolerable level noise into our model. They make\ncertain regions of the document more ambiguous locally, but that local ambiguity can be\novercome with the context provided by surrounding characters and a language model.\nImproved Character Models We evaluate the method more quantitatively by testing the\nimpact of the new templates on the quality of searches performed against the document.\nTo search for a given word, we rank lines by the ratio of the maximum likelihood tran-\nscription/segmentation that contains the search word to the likelihood of the best possible\nsegmentation/transcription under our model. The lowest possible search score is 1, happen-\ning when the search word is actually a substring of the maximum likelihood transcription.\nHigher scores mean that the word is increasingly unlikely under our model. In Figure 7, the\n\ufb01gure on the left shows the improvement in ranking of the lines that truly contain selected\nsearch words. The odd rows (in red) are search results using only the original 22 glyphs,\n\n\f20\n40\n60\n80\n\n2700\n\n2650\n\n2600\n\n100\n\n200\n\n300\n\n400\n\n500\n\n600\n\ndotted (wrong):\nsolid (correct):\n1920\n1900\n1880\n1860\n1840\ndotted (wrong):\nsolid (correct):\n\nnupta\nnuptiis\n\ninquam\n\n(v|u)ideo\nvidet\n\niam\n\nnupta\nnuptiis\n\ninquam\n\npost\npostquam\n\n(v|u)ideo\nvidet\n\n \n\n1\nd\nn\nR\n\n \n\n2\nd\nn\nR\n\nFigure 6: Search Results with (Rnd 1) initial templates only and with (Rnd 2) templates\nextracted from high con\ufb01dence regions. We show results that have a score within 5% of the\nbest path. Solid Lines are the results for the correct word. Dotted lines represent other\nsearch results, where we have made a few larger in order to show those words that are\nthe closest competitors to the true word. Many alternative searches, like the highlighted\n\u201cpost\u201d are actually portions of the correct larger words. These restrict our selection of\ncon\ufb01dence regions, but do not impinge on search quality.\nEach correct word has signi\ufb01cantly improved after one round of template reestimation.\n\u201ciam\u201d has been correctly identi\ufb01ed, and is a new high con\ufb01dence region. Both \u201cnuptiis\u201d\nand \u201cpostquam\u201d are now the highest likelihood words for their region barring smaller\nsubsequences, and \u201cvidet\u201d has narrowed the gap between its competitor \u201cvideo\u201d.\n\nwhile the even rows (in green) use an additional 332 glyphs extracted from high con\ufb01dence\nregions. Search results are markedly improved in the second model. The word \u201cest\u201d, for\ninstance, only had 15 of 24 of the correct lines in the top 100 under the original model,\nwhile under the learned model all 24 are not only present but also more highly ranked.\nImproved Search Figure 6 shows the improved performance of our re\ufb01tted model for\na single line. Most words have greatly improved relative to their next best alternative.\n\u201cpostquam\u201d and \u201ciam\u201d were not even considered by the original model and now are nearly\noptimal. The right of Figure 7 shows the average precision/recall curve under each model\nfor 21 words with more than 4 occurrences in the dataset. Precision is the percentage\nof lines truly containing a word in the top n search results, and recall is the percentage\nof all lines containing the word returned in the top n results. The learned model clearly\ndominates. The new model also greatly improves performance for rare words. For 320\nwords ocurring just once in the dataset, 50% are correctly returned as the top ranked result\nunder the original model. Under the learned model, this number jumps to 78%.\n\n5 Conclusions and Future Work\n\nIn most fonts, characters are quite ambiguous locally. An \u201cn\u201d looks like a \u201cu\u201d, looks like\n\u201cii\u201d, etc. This ambiguity is the major hurdle to the unsupervised learning of character\ntemplates. Language models help, but the standard n-gram models provide insuf\ufb01cient\nconstraints, giving posteriors for character sites too uninformative to get EM off the ground.\n\n\fSelected Words, Top 100 Returned Lines\nest\n(15,24)/24\nnescio\n( 1, 1)/ 1\npostquam\n( 0, 2)/ 2\nquod\n(14,14)/14\nmoram\n( 0, 2)/ 2\nnon\n( 8, 8)/ 8\nquid\n( 9, 9)/ 9\n\n10 20 30 40 50 60 70 80 90100\n\ni\n\ni\n\nn\no\ns\nc\ne\nr\nP\n\n0.75\n0.7\n0.65\n0.6\n0.55\n0.5\n0.45\n0.4\n0.35\n\nAggregate Precision/Recall Curve\n\nOriginal Model\nRefit Model\n\n0.2\n\n0.4\n\n0.6\n\n0.8\n\n1\n\nRecall\n\nFigure 7: The \ufb01gure on the left shows the those lines with the top 100 scores that actually\ncontain the speci\ufb01ed word. The \ufb01rst of each set of two rows (in red) is the results from\nRound 1. The second (in green) is the results for Round 2. Almost all search words in our\ncorpus show a signi\ufb01cant improvement. The numbers to the right (x/y) mean that out of\ny lines that actually contained the search word in our document, x of them made it into\nthe top ten. On the right are average precision/recall curves for 21 high frequency words\nunder the model with our original templates (Rnd 1) and after re\ufb01tting with new extracted\ntemplates (Rnd 2). Extracting new templates vastly improves our search quality\n\nAn entire word is much different. Given a dictionary, we expect many word images to have\na single likely transcription even if many characters are locally ambiguous. We show that\nwe can identify these high con\ufb01dence regions even with a poorly tuned character model. By\nextracting new templates only from these regions of the document, we overcome the noise\nproblem and signi\ufb01cantly improve our character models. We demonstrate this improvement\nfor the task of search where the re\ufb01tted models have drastically better search responses than\nwith the original. Our method is indifferent to the form of the actual character emission\nmodel. There is a rich literature in character prediction from isolated image windows, and\nwe expect that incorporating more powerful character models should provide even greater\nreturns and help us in learning less regular scripts.\n\nFinding high con\ufb01dence regions to extract good training examples is a broadly applica-\nble concept. We believe this work should extend to other problems, most notably speech\nrecognition. Looked at more abstractly, our use of language model in this work is actu-\nally encoding spatial constraints. The probability of a character given an image window\ndepends not only on the identify of surrounding characters but also on their spatial con-\n\ufb01guration. Integrating context into recognition problems is an area of intense research in\nthe computer vision community, and we are investigating extending the idea of con\ufb01dence\nregions to more general object recognition problems.\n\nReferences\n\n[1] Early Manuscripts at Oxford University. Bodleian library ms. auct. f. 2.13. http://image.ox.ac.uk/.\n[2] J. Edwards, Y.W. Teh, D. Forsyth, R. Bock, M. Maire, and G. Vesom. Making latin manuscripts\n\nsearchable using ghmm\u2019s. In NIPS 17, pages 385\u2013392. 2005.\n\n[3] G. Kopec and M. Lomelin. Document-speci\ufb01c character template estimation. In Proceedings,\n\nDocument Image Recognition III, SPIE, 1996.\n\n[4] V. Lavrenko, T. Rath, and R. Manmatha. Holistic word recognition for handwritten historical\n\ndocuments. In dial, pages 278\u2013287, 2004.\n\n[5] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document\n\nrecognition. Proceedings of the IEEE, 86(11):2278\u20132324, 1998.\n\n[6] M. Mohri, F. Pereira, and M. Riley. Weighted \ufb01nite state transducers in speech recognition. ISCA\n\nITRW Automatic Speech Recognition, pages 97\u2013106, 2000.\n\n\f", "award": [], "sourceid": 2919, "authors": [{"given_name": "Jaety", "family_name": "Edwards", "institution": null}, {"given_name": "David", "family_name": "Forsyth", "institution": null}]}