{"title": "Learning Hierarchical Semantic Image Manipulation through Structured Representations", "book": "Advances in Neural Information Processing Systems", "page_first": 2708, "page_last": 2718, "abstract": "Understanding, reasoning, and manipulating semantic concepts of images have been a fundamental research problem for decades. Previous work mainly focused on direct manipulation of natural image manifold through color strokes, key-points, textures, and holes-to-fill. In this work, we present a novel hierarchical framework for semantic image manipulation. Key to our hierarchical framework is that we employ structured semantic layout as our intermediate representations for manipulation. Initialized with coarse-level bounding boxes, our layout generator first creates pixel-wise semantic layout capturing the object shape, object-object interactions, and object-scene relations. Then our image generator fills in the pixel-level textures guided by the semantic layout. Such framework allows a user to manipulate images at object-level by adding, removing, and moving one bounding box at a time. Experimental evaluations demonstrate the advantages of the hierarchical manipulation framework over existing image generation and context hole-filing models, both qualitatively and quantitatively. Benefits of the hierarchical framework are further demonstrated in applications such as semantic object manipulation, interactive image editing, and data-driven image manipulation.", "full_text": "Learning Hierarchical Semantic Image Manipulation\n\nthrough Structured Representations\n\nXinchen Yan\u2020\n\nThomas Huang\u2020\n\nHonglak Lee\u2021,\u2020\n\nSeunghoon Hong\u2020\n\n\u2020University of Michigan\n\u2020{hongseu,xcyan,thomaseh,honglak}@umich.edu\n\n\u2021Google Brain\n\n\u2021honglak@google.com\n\nAbstract\n\nUnderstanding, reasoning, and manipulating semantic concepts of images have\nbeen a fundamental research problem for decades. Previous work mainly focused\non direct manipulation on natural image manifold through color strokes, key-\npoints, textures, and holes-to-\ufb01ll. In this work, we present a novel hierarchical\nframework for semantic image manipulation. Key to our hierarchical framework is\nthat we employ structured semantic layout as our intermediate representation for\nmanipulation. Initialized with coarse-level bounding boxes, our structure generator\n\ufb01rst creates pixel-wise semantic layout capturing the object shape, object-object\ninteractions, and object-scene relations. Then our image generator \ufb01lls in the\npixel-level textures guided by the semantic layout. Such framework allows a\nuser to manipulate images at object-level by adding, removing, and moving one\nbounding box at a time. Experimental evaluations demonstrate the advantages\nof the hierarchical manipulation framework over existing image generation and\ncontext hole-\ufb01ling models, both qualitatively and quantitatively. Bene\ufb01ts of the\nhierarchical framework are further demonstrated in applications such as semantic\nobject manipulation, interactive image editing, and data-driven image manipulation.\n\nIntroduction\n\n1\nLearning to perceive, reason and manipulate images has been one of the core research problems in\ncomputer vision, machine learning and graphics for decades [1, 7, 8, 9]. Recently the problem has\nbeen actively studied in interactive image editing using deep neural networks, where the goal is to\nmanipulate an image according to the various types of user-controls, such as color strokes [20, 32],\nkey-points [19, 32], textures [26], and holes-to-\ufb01ll (in-painting) [17]. While these interactive image\nediting approaches have made good advances in synthesizing high-quality manipulation results, they\nare limited to direct manipulation on natural image manifold.\nThe main focus of this paper is to achieve semantic-level manipulation of images.\nInstead of\nmanipulating images on natural image manifold, we consider semantic label map as an interface for\nmanipulation. By editing the label map, users are able to specify the desired images at semantic-level,\nsuch as the location, object class, and object shape. Recently, approaches based on image-to-\nimage translation [3, 11, 24] have demonstrated promising results on semantic image manipulation.\nHowever, the existing works mostly focused on learning a style transformation function from label\nmaps to pixels, while manipulation of structure of the labels remains fully responsible to users. The\nrequirement on direct control over pixel-wise labels makes the manipulation task still challenging\nsince it requires a precise and considerable amount of user inputs to specify the structure of the\nobjects and scene. Although the problem can be partly addressed by template-based manipulation\ninterface (e.g. adding the objects from the pre-de\ufb01ned sets of template masks [24], blind pasting\nof the object mask is problematic since the structure of the object should be determined adaptively\ndepending on the surrounding context.\nIn this work, we tackle the task of semantic image manipulation as a hierarchical generative process.\nWe start our image manipulation task from a coarse-level abstraction of the scene: a set of semantic\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: Overall pipeline of the proposed semantic manipulation framework.\n\nbounding boxes which provide both semantic (what) and spatial (where) information of the scene in\nan object-level. Such representation is natural and \ufb02exible that enables users to manipulate the scene\nlayout by adding, removing, and moving each bounding box. To facilitate the image manipulation\nfrom coarse-level semantic bounding boxes, we introduce a hierarchical generation model that\npredicts the image in multiple abstraction levels. Our model consists of two parts: layout and image\ngenerators. Speci\ufb01cally, our structure generator \ufb01rst infers the \ufb01ne-grained semantic label maps\nfrom the coarse object bounding boxes, which produces structure (shape) of the manipulated object\naligned with the context. Given the predicted layout, our image generator infers the style (color and\ntextures) of the object considering the perceptual consistency to the surroundings. This way, when\nadding, removing, and moving semantic bounding boxes, our model can generate an appropriate\nimage seamlessly integrated into the surrounding image.\nWe present two applications of the proposed method on interactive and data-driven image editing.\nIn the experiment on interactive image editing, we show that our method allows users to easily\nmanipulate images using object bounding boxes, while the structure and style of the generated objects\nare adaptively determined depending on the context and naturally blended into the scene. Also, we\nshow that our simple object-level manipulation interface can be naturally extended to data-driven\nimage manipulation, by automatically sampling the object boxes and generating the novel images.\nThe bene\ufb01ts of the hierarchical image manipulation are three-fold. First, it supports richer manip-\nulation tasks such as adding, moving or removing objects through object-level control while the\n\ufb01ne-grained object structures are inferred by the model. Second, when conditioned on coarse and\n\ufb01ne-grained semantic representations, the proposed model produces better image manipulation results\ncompared to models learned without structural control. Finally, we demonstrate the effectiveness of\nthe proposed idea on interactive and automatic image manipulation.\n2 Related Work\nDeep visual manipulation. Visual manipulation is a task of synthesizing the new image by ma-\nnipulating parts of a reference image. Thanks to the emergence of generative adversarial networks\n(GANs) [6] and perceptual features discovered from deep convolutional neural networks [13, 21, 22],\nneural image manipulation [5, 14, 17, 20, 30, 32] has gained increasing popularity in recent years.\nZhu et al. [32] presented an novel image editing framework with a direct constraint capturing real\nimage statistics using deep convolutional generative adversarial networks [18]. Li et al. [14] and Yeh\net al. [30] investigated semantic face image editing and completion using convolution encoder-decoder\narchitecture, jointly trained with a pixel-wise reconstruction constraint, a perceptual (adversarial)\nconstraint, and a semantic structure constraint. In addition, Pathak et al. [17] and Denton et al.\n[5] studied context-conditioned image generation that performs pixel-level hole-\ufb01lling given the\nsurrounding regions using deep neural networks, trained with adversarial and reconstruction loss.\nContrary to the previous works that manipulate images based on low-level visual controls such\nas visual context [5, 14, 17, 30] or color strokes [20, 32], our model allows semantic control over\nmanipulation process through labeled bounding box and the inferred semantic layout.\nStructure-conditional image generation. Starting from the pixel-wise semantic structure, recent\nbreakthroughs approached the structure-conditional image generation through direct image-to-image\ntranslation [3, 11, 16, 33]. Isola et al. [11] employed a convolutional encoder-decoder network\nwith conditional adversarial objective to learn label-to-pixel mapping. Later approaches improved\nthe generation quality by incoorporating perceptual loss from a pre-trained classi\ufb01er [3] or feature-\n\n2\n\n\fmatching loss from multi-scale discriminators [24]. In particular, Wang et al. [24] demonstrated a\nhigh-resolution image synthesis results and its application to semantic manipulation by controlling the\npixel-wise labels. Contrary to the previous works focusing on learning a direct mapping from label\nto image, our model learns hierarchical mapping from coarse bounding box to image by inferring\nintermediate label maps. Our work is closely related to Hong et al. [10], which generates an image\nfrom a text description through multiple levels of abstraction consisting of bounding boxes, semantic\nlayouts, and \ufb01nally pixels. Contrary to Hong et al. [10], however, our work focuses on manipulation\nof parts of an image, which requires incorporation of both semantic and visual context of surrounding\nregions in the hierarchical generation process in such a way that structure and style of the generated\nobject are aligned with the other parts of an image.\n3 Hierarchical Image Manipulation\nGiven an input image I in \u2208 RH\u00d7W\u00d73, our goal is to synthesize the new image I out by manipulating\nits underlying semantic structure. Let M in \u2208 RH\u00d7W\u00d7C denotes a semantic label map of the image\nde\ufb01ned over C categories, which is either given as ground-truth (in training time) or inferred by the\npre-trained visual recognition model (in testing time). Then our goal is to synthesize the new image\nby manipulating M in, which allows the semantically-guided manipulation of an image.\nThe key idea of this paper is to introduce an object bounding box B as an abstracted interface to\nmanipulate the semantic label map. Speci\ufb01cally, we de\ufb01ne a controllable bounding box B = {b, c}\nas a combination of box corners b \u2208 R4 and a class label c = {0, . . . , C}, which represents the\nlocation, size and category of an object. By adding the new box or modifying the parameters of\nexisting boxes, a user can manipulate the image through adding, moving or deleting the objects1.\nGiven an object-level speci\ufb01cation by B, the image manipulation is then posed as a hierarchical\ngeneration task from a coarse bounding box to pixel-level predictions of structure and style.\nFigure 1 illustrates the overall pipeline of the proposed algorithm. When the new bounding box B\nis given, our algorithm \ufb01rst extracts the local observations of label map M \u2208 RS\u00d7S\u00d7C and image\nI \u2208 RS\u00d7S\u00d73 by cropping squared windows of size S \u00d7 S centered around B. Then conditioned on\nM, I and B, our model generates the manipulated image by the following procedures:\n\n\u2022 Given a bounding box B and the semantic label map M, the structure generator predicts the\n\nmanipulated semantic label map by \u02c6M = GM (M, B) (Section 3.1)\n\n\u2022 Given the manipulated label map \u02c6M and image I, the image generator predicts the manipu-\n\nlated image \u02c6I by \u02c6I = GI ( \u02c6M , I) (Section 3.2)\n\nAfter generating the manipulated image patch \u02c6I, we place it back to the original image to \ufb01nish the\nmanipulation task. The manipulation of multiple objects is performed iteratively by applying the\nabove procedures for each box. In the following, we explain the manipulation pipeline on a single\nbox B.\n3.1 Structure generator\nThe goal of the structure generator is to infer the latent structure of the region speci\ufb01ed by B = {b, c}\nin the form of pixel-wise class labels \u02c6M \u2208 RS\u00d7S\u00d7C. The outputs of the structure generator should\nre\ufb02ect the class-speci\ufb01c structure of the object de\ufb01ned by B as well as interactions of the generated\nobject with the surrounding context (e.g. a person riding a motorcycle). To consider both conditions\nin the generation process, the structure generator incorporates the label map M and the bounding box\nB as inputs and performs a conditional generation by \u02c6M = GM (M, B).\nFigure 2 illustrates the overall architecture of the structure generator. The structure generator takes in\nthe masked layout \u00afM \u2208 RS\u00d7S\u00d7C and the binary mask \u00afB \u2208 RS\u00d7S\u00d71, where \u00afMijc = 1 and \u00afBij = 1\nfor all pixels (i, j) inside the box for class c. Given these inputs, the model predicts the manipulated\noutcome using two decoding pathways.\nOur design principle is motivated by generative layered image modeling [19, 23, 27, 28, 29], which\ngenerates foreground (i.e. object) and background (i.e. context) using separate output streams. In our\nmodel, the foreground output stream produces the predictions on binary object mask \u02c6Mobj \u2208 RS\u00d7S\u00d71,\nwhich de\ufb01nes the object shape tightly bounded by object box B. The background output stream\nproduces per-pixel label maps \u02c6Mctx \u2208 RS\u00d7S\u00d7C inside B.\n\n1We used c = 0 to indicate a deletion operation, where the model \ufb01lls the labels with surroundings.\n\n3\n\n\fFigure 2: Architecture of the structure generator. Given a masked layout \u00afM and a binary mask\n\u00afB encoding the class and location of the object, respectively, the model produces the manipulated\nlayout \u02c6M by the outputs from the two-stream decoder corresponding to the binary mask of object\nand semantic label map of entire region inside the box.\nThe objective for our structure generator is then given by\n\nLlayout = Ladv( \u02c6Mobj, M\u2217\n\nobj) + \u03bbobjLrecon( \u02c6Mobj, M\u2217\n\nobj) + \u03bbctxLrecon( \u02c6Mctx, \u00afM ),\n\n(1)\nobj is the ground-truth binary object mask on B and Lrecon(\u00b7,\u00b7) is the reconstruction loss\nobj) is the conditional adversarial loss de\ufb01ned on object mask\n\nwhere M\u2217\nusing cross-entropy. Ladv( \u02c6Mobj, M\u2217\nensuring the perceptual quality of predicted object shape, which is given by\n\nLadv( \u02c6Mobj, M\u2217\n\nobj) = EM\u2217\n\nobj\n\n(cid:2)log(DM (M\u2217\n\nobj, \u00afM ))(cid:3) + E \u02c6Mobj\n\n(cid:104)\n\n(cid:105)\n\n1 \u2212 log(DM ( \u02c6Mobj, \u00afM ))\n\n,\n\n(2)\n\nwhere DM (\u00b7,\u00b7) is a conditional discriminator.\nDuring inference, we construct the manipulated layout \u02c6M using \u02c6Mobj and \u02c6Mctx. Contrary to the prior\nworks on layered image model, we selectively choose outputs from either foreground or background\nstreams depending on the manipulation operation de\ufb01ned by B. The output \u02c6M is given by\n\n\u02c6M =\n\nif c = 0 (deletion)\n( \u02c6Mobj1c) + (1 \u2212 ( \u02c6Mobj1c)) (cid:12) M otherwise (addition)\n\n\u02c6Mctx\n\n,\n\n(3)\n\n(cid:26)\n\nImage generator\n\nwhere c is the class label of the bounding box B and 1c \u2208 {0, 1}1\u00d7C is the one-hot encoded vector\nof the class c.\n3.2\nGiven an image I and the manipulated layout \u02c6M obtained by the structure generator, the image\ngenerator outputs a pixel-level prediction of the contents inside the regions de\ufb01ned by B. To make\nthe prediction being semantically meaningful and perceptually plausible, the output from the image\ngenerator should re\ufb02ect the semantic structure de\ufb01ned by the layout while being coherent in its style\n(e.g. color and texture) with the surrounding image. We design the conditional image generator\nGI such that \u02c6I = GI (I, \u02c6M ), where I, \u02c6I \u2208 RS\u00d7S\u00d73 represent the local image before and after\nmanipulation with respect to bounding box B.\nFigure 3 illustrates the overall architecture of the image generator. The model takes the masked\nimage \u00afI whose pixels inside the box are \ufb01lled with 0 and the manipulated layout \u02c6M as inputs, and\nproduces the manipulated image \u02c6I as output. We design the image generator GI to have a two-stream\nconvolutional encoder and a single-stream convolutional decoder connected by an intermediate\nfeature map F . As we see in the \ufb01gure, the convolutional encoder has two separate downsampling\nstreams, which we referred to as image encoder fimage( \u00afI) and layout encoder flayout( \u02c6M ), respectively.\nThe intermediate feature F is obtained by an element-wise feature interaction layer gated by the\nbinary mask BF on object location.\n\nF = flayout( \u02c6M ) (cid:12) BF + fimage( \u00afI) (cid:12) (1 \u2212 BF ).\n\n(4)\nFinally, the convolutional decoder gimage(\u00b7) takes the fused feature F as input and produces the\nmanipulated image through \u02c6I = gimage(F ). Note that we use the ground-truth layout M during model\ntraining but the predicted layout \u02c6M is used at the inference time during testing.\n\n4\n\nMasked LayoutForeground branchBackground branch\fFigure 3: Architecture of the image generator. Given a masked image \u00afI and the semantic layout \u02c6M,\nthe model encodes visual style and semantic structure of the object using separate encoding pathways\nand produces the manipulated image.\nWe de\ufb01ne the following loss for the image generation task.\n\n(5)\n\nLimage = Ladv( \u02c6I, I) + \u03bbfeatureLfeature( \u02c6I, I).\nThe \ufb01rst term Ladv( \u02c6I, I) is the conditional adversarial loss de\ufb01ned by\n\n(cid:104)\n\n(cid:105)\n\n(cid:104)\n\n+ E \u02c6I\n\n(cid:105)\n\n(cid:88)\n\nLadv( \u02c6I, I) = EI\n\n(6)\nThe second term Lfeature( \u02c6I, I) is the feature matching loss [24]. Speci\ufb01cally, we compute the distance\nbetween the real and manipulated images using the intermediate features from the discriminator by\n\nlog(DI (I, \u02c6M ))\n\n1 \u2212 log(DI ( \u02c6I, \u02c6M ))\n\n.\n\nLfeature( \u02c6I, I) = E\n\nI, \u02c6I\n\n(cid:107)D(i)(I, \u02c6M ) \u2212 D(i)( \u02c6I, \u02c6M )(cid:107)2\nF ,\n\n(7)\n\ni=1\n\nImplementation Details\n\nwhere D(i) is the outputs of ith layer in discriminator, (cid:107) \u00b7 (cid:107)F is the Frobenius norm.\nDiscussions. The proposed model encodes both image and layout using two-stream encoding\npathways. With only image encoder, it simply performs image in-painting [17], which attempts to \ufb01ll\nthe hole with patterns coherent with the surrounding region. On the other hand, our model with only\nlayout encoder becomes similar to image-to-image translation models [3, 11, 24], which translates\nthe pixel-wise semantic label maps to the RGB pixel values. Intuitively, by combining information\nfrom both encoders, the model learns to manipulate images that re\ufb02ect the underlying image structure\nde\ufb01ned by the label map with appearance patterns naturally blending into the surrounding context,\nwhich is semantically meaningful and perceptually plausible.\n4 Experiments\n4.1\nDatasets. We conduct both quantitative and qualitative evaluations on the Cityscape dataset [4], a\nsemantic understanding benchmark of European urban street scenes containing 5,000 high-resolution\nimages with \ufb01ne-grained annotations including instance-wise map and semantic map from 30\ncategories. Among 30 semantic categories, we treat 10 of them including person, rider, car,\ntruck, bus, caravan, trailer, train, motorcycle, bicycle as our foreground object classes\nwhile leaving the rest as background classes for image editing purpose. For evaluation, we measure\nthe generation performance on each of the foreground object bounding box in 500 validation images.\nTo further demonstrate the image manipulation results on more complex scene, we also conduct\nqualitative experiments on bedroom images from ADE20K dataset [31]. Among 49 object categories\nfrequently appearing in a bedroom, we select 31 movable ones as foreground objects for manipulation.\nTraining. As one challenge, collecting ground-truth examples before and after image manipulation\nis expensive and time-consuming. Instead, we simulate the addition (c \u2208 {1, ..., C}) and deletion\n(c = 0) operations by sampling boxes from the object and random image regions, respectively. For\ntraining, we employ an Adam optimizer [12] with learning rate 0.0002, \u03b21 = 0.5, \u03b22 = 0.999 and\nlinearly decrease the learning rate after the \ufb01rst 100-epochs for training. The hyper-parameters \u03bbobj,\n\u03bbctx, \u03bbfeature are set to 10. Our PyTorch implementation will be open-sourced.\n\n5\n\nImage EncoderLayout EncoderPredicted Image\fSingleStream-Image\nSingleStream-Layout\nSingleStream-Concat\n\nTwoStream\n\nTwoStream-Pred\n\nLayout\n\n-\nGT\nGT\nGT\n\nPredicted\n\nSSIM\n0.285\n0.291\n0.331\n0.336\n0.299\n\nSegmentation (%) Human eval. (%)\n\n59.6\n71.5\n76.7\n78.5\n77.9\n\n12.3\n9.2\n23.2\n33.6\n20.7\n\nTable 1: Comparisons between variants of the proposed method. The last two rows (TwoStream and\nTwoStream-Pred) correspond to our full model using ground-truth and predicted layout, respectively.\n\nFigure 4: Qualitative comparisons to the baselines in Table 1. The \ufb01rst two columns show the\nmasked image and ground-truth layout used as input to the models (except TwoStream-Pred). The\nmanipulated objects are indicated by blue arrows. Best viewed in color.\n\nEvaluation metrics. We employ three metrics to measure the perceptual and conditional generation\nquality. We use Structural Similarity Index (SSIM) [25] to evaluate the similarity of the ground-\ntruth and predicted images based on low-level visual statistics. To measure the quality of layout-\nconditional image generation, we apply a pre-trained semantic segmentation model DeepLab v3 [2]\nto the generated images, and measure the consistency between the input layout and the predicted\nsegmentation labels in terms of pixel-wise accuracy (layout \u2192 image \u2192 layout). Finally, we conduct\nuser study using Mechanical Turk (AMT) to evaluate the perceptual generation quality. We present\nthe manipulation results of different methods together with the input image and the class label of the\nbounding box and ask users to choose the best method based on how natural the manipulated images\nare. We collect the results for 1,000 examples, each of which is evaluated by 5 different Turkers. We\nreport the performance of each method based on the ratio that method ranked as the best in AMT.\n4.2 Quantitative Evaluation\nAblation study. To analyze the effectiveness of the proposed method, we conduct an ablation\nstudy on several variants of our method. We \ufb01rst consider three baselines of our image generator:\nconditioned only on image context (SingleStream-Image), conditioned only on semantic layout\n(SingleStream-Layout), or conditioned on both by concatenation but using a single encoding\npathway (SingleStream-Concat). We also compare our full model using a ground-truth and the\npredicted layouts (TwoStream and TwoStream-Pred). Table 1 summarize the results.\nAs shown in Table 1, conditioning the generation with only image or layout leads to either\npoor class-conditional generation (SingleStream-Image) or less perceptually plausible results\n(SingleStream-Layout).\nIt is because the former misses the semantic layout encoding criti-\ncal information on which object to generate, while the later misses color and textures of the im-\nage that makes the generation results visually consistent with its surroundings. Combining both\n(SingleStream-Concat), the generation quality improves in all metrics, which shows comple-\nmentary bene\ufb01ts of both conditions in image manipulation. In addition, a comparison between\nSingleStream-Concat and TwoStream shows that modeling the image and layout conditions\nusing separate encoding pathways further improves the generation quality. Finally, replacing the\nlayout condition from the ground-truth (TwoStream) to the predicted one (TwoStream-Pred) leads\n\n6\n\nContext LayoutSingleStreamImageSingleStreamLayoutSingleStreamConcatTwoStreamTwoStreamPred\fContext Encoder [17]\nContext Encoder++\n\nPix2PixHD [24]\n\nTwoStream\n\nLayout\n\n-\nGT\nGT\nGT\n\nSSIM\n0.265\n0.269\n0.288\n0.336\n\nTable 2: Quantitative comparison to other methods.\n\nSegmentation (%) Human eval. (%)\n\n26.4\n35.7\n79.6\n78.5\n\n1.1\n1.4\n18.0\n79.5\n\nFigure 5: Generation results on various locations in an image.\n\nFigure 6: Generation results in diverse contexts.\n\nto a small degree of degradation in perceptual quality partly due to the prediction errors in layout\ngeneration. However, clear improvement of TwoStream-Pred over SingleStream-Image shows\nthe effectiveness of layout prediction in image generation.\nFigure 4 shows the qualitative comparisons of the baselines. Among all variants, our TwoStream\nmodel tends to exhibit most recognizable and coherent appearance with the surrounding environment.\nInterestingly, our model with predicted layout TwoStream-Pred generates objects different from\nthe ground-truth layout but still match the bounding box condition. such as a person walking in the\ndifferent direction (the \ufb01rst row) and objects placed in different order (the second row).\nComparison to other methods. We also compare against a few existing work on context hole-\n\ufb01ling and structure-conditioned image generation. First, we consider the recent work on high-\nresolution pixel-to-pixel translation [24] (referred as Pix2PixHD in Table 2). Compared to\nour SingleStream-Layout model, Pix2PixHD model generates an entire image from seman-\ntic layout. Second, we consider the work for context-driven image in-painting [17] (referred as\nContextEncoder) as another baseline. Similar to our SingleStream-Image, ContextEncoder\ndoes not have access to the semantic layout during training and testing. For fair comparisons, we\nextended ContextEncoder so that it takes segmentation layout as additional input. We refer this\nmodel as ContextEncoder++ in Table 2. As we see in the Table 2, our two-stream model still\nachieves the best SSIM and competitive segmentation pixel-wise accuracy. The segmentation accu-\nracy of Pix2PixHD is higher than ours since it generates higher resolution images and object textures\nnon-relevant to the input image, but our method still achieves perceptually plausible results consistent\nwith the surrounding image. Note that the motivations of Pix2PixHD and our model are different, as\nPix2PixHD performs image generation from scratch while our focus is local image manipulation.\n4.3 Qualitative Analysis\nSemantic object manipulation. To demonstrate how our hierarchical model manipulates structure\nand style of an object depending on the context, we conduct qualitative analysis in various settings.\nIn Figure 5, we present the manipulation results by moving the same bounding box of a car to\ndifferent locations in an image. As we see in the \ufb01gure, our model generates a car with diverse but\n\n7\n\nABCDEABCDEocclusiongeometric context\freasonably-looking shape and appearance when we move its bounding box from one side of the road\nto another. Interestingly, the shape, orientation, and appearance of the car also change according to\nthe scene layout and shadow in the surrounding regions. Figure 6 illustrates generation results in\nmore diverse contexts. It shows that our model generates appropriate structure and appearance of the\nobject considering the context (e.g. occlusion with other objects, interaction with the scene, etc). In\naddition to generating object matching the surroundings, we can also easily extend our framework to\nallow users to directly control object style, which we demonstrate below.\nExtension to style manipulation Although we focused on addition and deletion of objects as ma-\nnipulation tasks, we can easily extend our framework to add control over object styles by conditioning\nthe image generator with additional style vector s by GI ( \u02c6M , I, s). To demonstrate this capability, we\nde\ufb01ne the style vector as a mean color of the object, and synthesized objects while changing the input\ncolor (Figure 7). The results show that our model successfully synthesizes various objects with the\nspeci\ufb01ed color while retaining other parts of the images unchanged. Modeling more complicated\nstyles (e.g. texture) can be achieved by learning a style-encoder s = E(X) where X is an object\ntemplate that user can select, but we leave it as a future work.\n\nFigure 7: Controlling object color with style vector (left-upper corners indicate conditions).\n\nInteractive and data-driven image editing. As one of the key applications, we perform interactive\nimage manipulation by adding, removing, and moving object bounding boxes. Figure 8 illustrates the\nresults. It shows that our method generates reasonable semantic layouts and images that smoothly\naugment content of the original image. In addition to interactive manipulation, we can also automate\nthe manipulation process by sampling bounding boxes in an image in a data-driven manner. To\ndemonstrate this idea, we present an application of data-driven image manipulation in Figure 9. In this\ndemo, we implement box sampling using a simple non-parametric approach; Given a query image, we\n\ufb01rst compute its nearest neighbor from the training set based on low-level similarity. Then we compute\nthe geometric transformation between a query and a training image using SIFT Flow [15], and move\nobject bounding boxes from one scene to another based on the scene-level transformation. As shown\nin the \ufb01gure, the proposed methods reasonably sample boxes from appropriate locations. However,\ndirectly placing the object (or its mask) may lead to unnatural manipulation due to mismatches\nin scene con\ufb01guration (e.g. occlusion and orientation). Our hierarchical model generates objects\nadaptive to the new context.\n\nFigure 8: Examples of manipulation of multiple objects in images. The line style indicates\nmanipulation operation (solid: addition, doted: deletion) and the color indicates the object class.\n\n8\n\nOriginalManipulated\fFigure 9: Example of data-driven image manipulation. We manipulate the target image by transferring\nbounding boxes from source image (blue boxes).\nResults on indoor scene dataset.\nIn addition to Cityscape dataset, we conduct qualitative experi-\nments on bedroom images using ADE20K datasets [31]. Figure 10 illustrates the interactive image\nmanipulation results. Since objects in the indoor images involve much more diverse categories and\nappearances, generating appropriate object shapes and textures aligned with other components in a\nscene is much more challenging than the street images. We observe that the generated objects by the\nproposed method usually are looking consistent the surrounding context.\n\nFigure 10: Examples of image manipulation results on indoor images.\n\n5 Conclusions\nIn this paper, we presented a hierarchical framework for semantic image manipulation. We \ufb01rst learn\nto generate the pixel-wise semantic label maps given the initial object bounding boxes. Then we learn\nto generate the manipulated image from the predicted label maps. Such framework allows the user to\nmanipulate images at object-level by adding, removing, and moving an object bounding box at a time.\nExperimental evaluations demonstrate the advantages of the hierarchical manipulation framework\nover existing image generation and context hole-\ufb01ling models, both qualitatively and quantitatively.\nWe further demonstrate its practical bene\ufb01ts in semantic object manipulation, interactive image\nediting and data-driven image editing.\n\n9\n\nSource ImageTarget ImageManipulated ImageOriginalManipulatedOriginalManipulatedPicturePillowCurtainCushionChairCabinetDresserMirrorSconceTVBlindsBedTableWindowDoorCarpetVaseArmchairPlantFlowerBookShelfFlowerpotSofaPicturePillowCurtainCushionChairCabinetDresserMirrorSconceTVBlindsLampArmchairVaseFlower\fAcknowledgement This work was supported in part by ONR N00014-13-1-0762, NSF CAREER\nIIS-1453651, DARPA Explainable AI (XAI) program #313498, Sloan Research Fellowship, and\nAdobe Research Fellowship and Google PhD Fellowship to X. Yan.\n\nReferences\n[1] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. Patchmatch: A randomized\ncorrespondence algorithm for structural image editing. ACM Transactions on Graphics, 28(3):\n24, 2009.\n\n[2] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous\n\nseparable convolution for semantic image segmentation. arXiv:1802.02611, 2018.\n\n[3] Q. Chen and V. Koltun. Photographic image synthesis with cascaded re\ufb01nement networks. In\n\nICCV, 2017.\n\n[4] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth,\nand B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.\n\n[5] E. Denton, S. Gross, and R. Fergus. Semi-supervised learning with context-conditional genera-\n\ntive adversarial networks. arXiv preprint arXiv:1611.06430, 2016.\n\n[6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and\n\nY. Bengio. Generative adversarial nets. In NIPS, 2014.\n\n[7] A. Gupta, A. A. Efros, and M. Hebert. Blocks world revisited: Image understanding using\n\nqualitative geometry and mechanics. In ECCV, 2010.\n\n[8] D. Hoiem, A. A. Efros, and M. Hebert. Geometric context from a single image. In ICCV, 2005.\n\n[9] D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. International Journal of\n\nComputer Vision, 80(1):3\u201315, 2008.\n\n[10] S. Hong, D. Yang, J. Choi, and H. Lee. Inferring semantic layout for hierarchical text-to-image\n\nsynthesis. In CVPR, 2018.\n\n[11] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional\n\nadversarial networks. In CVPR, 2017.\n\n[12] D. Kingma and J. Ba. Adam: A method for stochastic optimization.\n\narXiv:1412.6980, 2014.\n\narXiv preprint\n\n[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classi\ufb01cation with deep convolutional\n\nneural networks. In NIPS, 2012.\n\n[14] Y. Li, S. Liu, J. Yang, and M.-H. Yang. Generative face completion. In CVPR, 2017.\n\n[15] C. Liu, J. Yuen, and A. Torralba. Sift \ufb02ow: Dense correspondence across scenes and its\n\napplications. In Dense Image Correspondences for Computer Vision, pages 15\u201349. 2016.\n\n[16] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. In\n\nNIPS, 2017.\n\n[17] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature\n\nlearning by inpainting. In CVPR, 2016.\n\n[18] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolu-\n\ntional generative adversarial networks. In ICLR, 2016.\n\n[19] S. E. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and H. Lee. Learning what and where to\n\ndraw. In NIPS, 2016.\n\n[20] P. Sangkloy, J. Lu, C. Fang, F. Yu, and J. Hays. Scribbler: Controlling deep image synthesis\n\nwith sketch and color. In CVPR, 2017.\n\n10\n\n\f[21] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image\n\nrecognition. arXiv preprint arXiv:1409.1556, 2014.\n\n[22] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke,\n\nA. Rabinovich, et al. Going deeper with convolutions. CVPR, 2015.\n\n[23] C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In NIPS,\n\n2016.\n\n[24] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image\n\nsynthesis and semantic manipulation with conditional gans. In ICCV, 2017.\n\n[25] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error\nvisibility to structural similarity. IEEE transactions on Image Processing, 13(4):600\u2013612, 2004.\n\n[26] W. Xian, P. Sangkloy, J. Lu, C. Fang, F. Yu, and J. Hays. Texturegan: Controlling deep image\n\nsynthesis with texture patches. In CVPR, 2018.\n\n[27] X. Yan, J. Yang, K. Sohn, and H. Lee. Attribute2image: Conditional image generation from\n\nvisual attributes. In ECCV, 2016.\n\n[28] J. Yang, A. Kannan, D. Batra, and D. Parikh. Lr-gan: Layered recursive generative adversarial\n\nnetworks for image generation. In ICLR, 2017.\n\n[29] Y. Yang, S. Hallman, D. Ramanan, and C. C. Fowlkes. Layered object models for image\nsegmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(9):1731\u2013\n1743, 2012.\n\n[30] R. A. Yeh, C. Chen, T. Y. Lim, A. G. Schwing, M. Hasegawa-Johnson, and M. N. Do. Semantic\n\nimage inpainting with deep generative models. In CVPR, pages 5485\u20135493, 2017.\n\n[31] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through\n\nade20k dataset. In CVPR, 2017.\n\n[32] J.-Y. Zhu, P. Kr\u00e4henb\u00fchl, E. Shechtman, and A. A. Efros. Generative visual manipulation on\n\nthe natural image manifold. In ECCV, 2016.\n\n[33] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using\n\ncycle-consistent adversarial networks. In ICCV, 2017.\n\n11\n\n\f", "award": [], "sourceid": 1399, "authors": [{"given_name": "Seunghoon", "family_name": "Hong", "institution": "University of Michigan"}, {"given_name": "Xinchen", "family_name": "Yan", "institution": "University of Michigan"}, {"given_name": "Thomas", "family_name": "Huang", "institution": "University of Michigan"}, {"given_name": "Honglak", "family_name": "Lee", "institution": "Google / U. Michigan"}]}