{"title": "Recovery of Jointly Sparse Signals from Few Random Projections", "book": "Advances in Neural Information Processing Systems", "page_first": 1433, "page_last": 1440, "abstract": "", "full_text": "Recovery of Jointly Sparse Signals\n\nfrom Few Random Projections\n\nMichael B. Wakin\nECE Department\nRice University\n\nwakin@rice.edu\n\nMarco F. Duarte\nECE Department\nRice University\n\nShriram Sarvotham\n\nECE Department\nRice University\n\nduarte@rice.edu\n\nshri@rice.edu\n\nDror Baron\n\nECE Department\nRice University\n\ndrorb@rice.edu\n\nRichard G. Baraniuk\n\nECE Department\nRice University\n\nrichb@rice.edu\n\nAbstract\n\nCompressed sensing is an emerging \ufb01eld based on the revelation that a small group\nof linear projections of a sparse signal contains enough information for reconstruc-\ntion. In this paper we introduce a new theory for distributed compressed sensing\n(DCS) that enables new distributed coding algorithms for multi-signal ensembles\nthat exploit both intra- and inter-signal correlation structures. The DCS theory rests\non a new concept that we term the joint sparsity of a signal ensemble. We study\nthree simple models for jointly sparse signals, propose algorithms for joint recov-\nery of multiple signals from incoherent projections, and characterize theoretically\nand empirically the number of measurements per sensor required for accurate re-\nconstruction. In some sense DCS is a framework for distributed compression of\nsources with memory, which has remained a challenging problem in information\ntheory for some time. DCS is immediately applicable to a range of problems in\nsensor networks and arrays.\n\nIntroduction\n\n1\nDistributed communication, sensing, and computing [13, 17] are emerging \ufb01elds with nu-\nmerous promising applications. In a typical setup, large groups of cheap and individu-\nally unreliable nodes may collaborate to perform a variety of data processing tasks such\nas sensing, data collection, classi\ufb01cation, modeling, tracking, and so on. As individual\nnodes in such a network are often battery-operated, power consumption is a limiting fac-\ntor, and the reduction of communication costs is crucial.\nIn such a setting, distributed\nsource coding [8, 13, 14, 17] may allow the sensors to save on communication costs. In the\nSlepian-Wolf framework for lossless distributed coding [8, 14], the availability of corre-\nlated side information at the decoder enables the source encoder to communicate losslessly\nat the conditional entropy rate, rather than the individual entropy. Because sensor networks\nand arrays rely on data that often exhibit strong spatial correlations [13, 17], distributed\ncompression can reduce the communication costs substantially, thus enhancing battery life.\nUnfortunately, distributed compression schemes for sources with memory are not yet ma-\nture [8, 13, 14, 17].\n\n\fWe propose a new approach for distributed coding of correlated sources whose signal cor-\nrelations take the form of a sparse structure. Our approach is based on another emerging\n\ufb01eld known as compressed sensing (CS) [4, 9]. CS builds upon the groundbreaking work\nof Cand`es et al. [4] and Donoho [9], who showed that signals that are sparse relative to a\nknown basis can be recovered from a small number of nonadaptive linear projections onto\na second basis that is incoherent with the \ufb01rst. (A random basis provides such incoherence\nwith high probability. Hence CS with random projections is universal \u2014 the signals can\nbe reconstructed if they are sparse relative to any known basis.) The implications of CS\nfor signal acquisition and compression are very promising. With no a priori knowledge of\na signal\u2019s structure, a sensor node could simultaneously acquire and compress that signal,\npreserving the critical information that is extracted only later at a fusion center.\nIn our framework for distributed compressed sensing (DCS), this advantage is particularly\ncompelling. In a typical DCS scenario, a number of sensors measure signals that are each\nindividually sparse in some basis and also correlated from sensor to sensor. Each sensor\nindependently encodes its signal by projecting it onto another, incoherent basis (such as a\nrandom one) and then transmits just a few of the resulting coef\ufb01cients to a single collection\npoint. Under the right conditions, a decoder at the collection point can reconstruct each of\nthe signals precisely. The DCS theory rests on a concept that we term the joint sparsity of a\nsignal ensemble. We study in detail three simple models for jointly sparse signals, propose\ntractable algorithms for joint recovery of signal ensembles from incoherent projections, and\ncharacterize theoretically and empirically the number of measurements per sensor required\nfor reconstruction. While the sensors operate entirely without collaboration, joint decoding\ncan recover signals using far fewer measurements per sensor than would be required for\nseparable CS recovery. This paper presents our speci\ufb01c results for one of the three models;\nthe other two are highlighted in our papers [1, 2, 11].\n\n2 Sparse Signal Recovery from Incoherent Projections\nIn the traditional CS setting, we consider a single signal x \u2208 RN , which we assume to be\nsparse in a known orthonormal basis or frame \u03a8 = [\u03c81, \u03c82, . . . , \u03c8N ]. That is, x = \u03a8\u03b8\nfor some \u03b8, where k\u03b8k0 = K holds.1 The signal x is observed indirectly via an M \u00d7\nN measurement matrix \u03a6, where M < N. We let y = \u03a6x be the observation vector,\nconsisting of the M inner products of the measurement vectors against the signal. The M\nrows of \u03a6 are the measurement vectors, against which the signal is projected. These rows\nare chosen to be incoherent with \u03a8 \u2014 that is, they each have non-sparse expansions in\nthe basis \u03a8 [4, 9]. In general, \u03a6 meets the necessary criteria when its entries are drawn\nrandomly, for example independent and identically distributed (i.i.d.) Gaussian.\nAlthough the equation y = \u03a6x is underdetermined, it is possible to recover x from y under\ncertain conditions. In general, due to the incoherence between \u03a6 and \u03a8, \u03b8 can be recovered\nby solving the `0 optimization problem\n\nb\u03b8 = arg min k\u03b8k0\n\ns.t. y = \u03a6\u03a8\u03b8.\n\nIn principle, remarkably few random measurements are required to recover a K-sparse\nsignal via `0 minimization. Clearly, more than K measurements must be taken to avoid\nambiguity; in theory, K + 1 random measurements will suf\ufb01ce [2]. Unfortunately, solving\nthis `0 optimization problem appears to be NP-hard [6], requiring a combinatorial enumer-\n\nation of the(cid:0)N\n\nK(cid:1) possible sparse subspaces for \u03b8.\n\nThe amazing revelation that supports the CS theory is that a much simpler problem yields\nan equivalent solution (thanks again to the incoherence of the bases): we need only solve\n\n1The `0 \u201cnorm\u201d k\u03b8k0 merely counts the number of nonzero entries in the vector \u03b8. CS theory also\napplies to signals for which k\u03b8kp \u2264 K, where 0 < p \u2264 1; such extensions for DCS are a topic of\nongoing research.\n\n\ffor the `1-sparsest vector \u03b8 that agrees with the observed coef\ufb01cients y [4, 9]\n\nb\u03b8 = arg min k\u03b8k1\n\ns.t. y = \u03a6\u03a8\u03b8.\n\nThis optimization problem, known also as Basis Pursuit (BP) [7], is signi\ufb01cantly more\ntractable and can be solved with traditional linear programming techniques. There is no\nfree lunch, however; more than K + 1 measurements will be required in order to recover\nsparse signals. In general, there exists a constant oversampling factor c = c(K, N ) such\nthat cK measurements suf\ufb01ce to recover x with very high probability [4, 9]. Commonly\nquoted as c = O(log(N )), we have found that c \u2248 log2(1 + N/K) provides a useful\nrule-of-thumb [2]. At the expense of slightly more measurements, greedy algorithms have\nalso been developed to recover x from y. One example, known as Orthogonal Matching\nPursuit (OMP) [15], requires c \u2248 2 ln(N ). We exploit both BP and greedy algorithms for\nrecovering jointly sparse signals.\n\nJoint Sparsity Models\n\n3\nIn this section, we generalize the notion of a signal being sparse in some basis to the\nnotion of an ensemble of signals being jointly sparse. We consider three different joint\nsparsity models (JSMs) that apply in different situations. In most cases, each signal is itself\nsparse, and so we could use the CS framework from above to encode and decode each one\nseparately. However, there also exists a framework wherein a joint representation for the\nensemble uses fewer total vectors.\nWe use the following notation for our signal ensembles and measurement model. Denote\nthe signals in the ensemble by xj, j \u2208 {1, 2, . . . , J}, and assume that each signal xj \u2208 RN .\nWe assume that there exists a known sparse basis \u03a8 for RN in which the xj can be sparsely\nrepresented. Denote by \u03a6j the measurement matrix for signal j; \u03a6j is Mj \u00d7 N and, in\ngeneral, the entries of \u03a6j are different for each j. Thus, yj = \u03a6jxj consists of Mj < N\nincoherent measurements of xj.\nJSM-1: Sparse common component + innovations. In this model, all signals share a\ncommon sparse component while each individual signal contains a sparse innovation com-\nponent; that is,\n\nxj = zC + zj,\n\nj \u2208 {1, 2, . . . , J}\n\nwith\n\nzC = \u03a8\u03b8C, k\u03b8Ck0 = K\n\nand\n\nzj = \u03a8\u03b8j, k\u03b8jk0 = Kj.\n\nThus, the signal zC is common to all of the xj and has sparsity K in basis \u03a8. The signals\nzj are the unique portions of the xj and have sparsity Kj in the same basis. A practical\nsituation well-modeled by JSM-1 is a group of sensors measuring temperatures at a number\nof outdoor locations throughout the day. The temperature readings xj have both temporal\n(intra-signal) and spatial (inter-signal) correlations. Global factors, such as the sun and\nprevailing winds, could have an effect zC that is both common to all sensors and structured\nenough to permit sparse representation. More local factors, such as shade, water, or ani-\nmals, could contribute localized innovations zj that are also structured (and hence sparse).\nSimilar scenarios could be imagined for a network of sensors recording other phenomena\nthat change smoothly in time and in space and thus are highly correlated.\nJSM-2: Common sparse supports. In this model, all signals are constructed from the\nsame sparse set of basis vectors, but with different coef\ufb01cients; that is,\n\nxj = \u03a8\u03b8j,\n\nj \u2208 {1, 2, . . . , J},\n\nwhere each \u03b8j is supported only on the same \u2126 \u2282 {1, 2, . . . , N } with |\u2126| = K. Hence,\nall signals have `0 sparsity of K, and all are constructed from the same K basis elements,\nbut with arbitrarily different coef\ufb01cients. A practical situation well-modeled by JSM-2\nis where multiple sensors acquire the same signal but with phase shifts and attenuations\n\n\fcaused by signal propagation. In many cases it is critical to recover each one of the sensed\nsignals, such as in many acoustic localization and array processing algorithms. Another\nuseful application for JSM-2 is MIMO communication [16].\nJSM-3: Nonsparse common + sparse innovations. This model extends JSM-1 so that\nthe common component need no longer be sparse in any basis; that is,\n\nxj = zC + zj,\n\nj \u2208 {1, 2, . . . , J}\n\nwith\n\nzC = \u03a8\u03b8C\n\nand\n\nzj = \u03a8\u03b8j, k\u03b8jk0 = Kj,\n\nbut zC is not necessarily sparse in the basis \u03a8. We also consider the case where the supports\nof the innovations are shared for all signals, which extends JSM-2. A practical situation\nwell-modeled by JSM-3 is where several sources are recorded by different sensors together\nwith a background signal that is not sparse in any basis. Consider, for example, a computer\nvision-based veri\ufb01cation system in a device production plant. Cameras acquire snapshots\nof components in the production line; a computer system then checks for failures in the\ndevices for quality control purposes. While each image could be extremely complicated,\nthe ensemble of images will be highly correlated, since each camera is observing the same\ndevice with minor (sparse) variations. JSM-3 could also be useful in some non-distributed\nscenarios. For example, it motivates the compression of data such as video, where the\ninnovations or differences between video frames may be sparse, even though a single frame\nmay not be very sparse. In general, JSM-3 may be invoked for ensembles with signi\ufb01cant\ninter-signal correlations but insigni\ufb01cant intra-signal correlations.\n\n4 Recovery of Jointly Sparse Signals\nIn a setting where a network or array of sensors may encounter a collection of jointly\nsparse signals, and where a centralized reconstruction algorithm is feasible, the number\nof incoherent measurements required by each sensor can be reduced. For each JSM, we\npropose algorithms for joint signal recovery from incoherent projections and characterize\ntheoretically and empirically the number of measurements per sensor required for accurate\nreconstruction. We focus in particular on JSM-3 in this paper but also overview our results\nfor JSMs 1 and 2, which are discussed in further detail in our papers [1, 2, 11].\n\nJSM-1: Sparse common component + innovations\n\n4.1\nFor this model (see also [1, 2]), we have proposed an analytical framework inspired by the\nprinciples of information theory. This allows us to characterize the measurement rates Mj\nrequired to jointly reconstruct the signals xj. The measurement rates relate directly to the\nsignals\u2019 conditional sparsities, in parallel with the Slepian-Wolf theory. More speci\ufb01cally,\nwe have formalized the following intuition. Consider the simple case of J = 2 signals. By\nemploying the CS machinery, we might expect that (i) (K + K1)c coef\ufb01cients suf\ufb01ce to\nreconstruct x1, (ii) (K +K2)c coef\ufb01cients suf\ufb01ce to reconstruct x2, yet only (iii) (K +K1+\nK2)c coef\ufb01cients should suf\ufb01ce to reconstruct both x1 and x2, since we have K + K1 + K2\nnonzero elements in x1 and x2. In addition, given the (K + K1)c measurements for x1\nas side information, and assuming that the partitioning of x1 into zC and z1 is known,\ncK2 measurements that describe z2 should allow reconstruction of x2. Formalizing these\narguments allows us to establish theoretical lower bounds on the required measurement\nrates at each sensor; Fig.1(a) shows such a bound for the case of J = 2 signals.\nWe have also established upper bounds on the required measurement rates Mj by proposing\na speci\ufb01c algorithm for reconstruction [1]. The algorithm uses carefully designed measure-\nment matrices \u03a6j (in which some rows are identical and some differ) so that the resulting\nmeasurements can be combined to allow step-by-step recovery of the sparse components.\nThe theoretical rates Mj are below those required for separable CS recovery of each signal\nxj (see Fig. 1(a)). We also proposed a reconstruction technique based on a single exe-\ncution of a linear program, which seeks the sparsest components [zC; z1;\n. . . zJ ] that\n\n\f1\n\n0.9\n\n0.8\n\n0.7\n\n0.6\n\n2\n\nR\n\n0.5\n\n0.4\n\n0.3\n\n0.2\n\n0.1\n\nSimulation\n\nConverse\nAnticipated\nAchievable\nSeparate\n\nn = 50, k = 5\n\n32\n\n16\n\n8\n\n4\n\n2\n\n1\n\n2\n\n4\n\n8\n\n16\n\n32\n\n1\n\n0.9\n\n0.8\n\n0.7\n\n0.6\n\n0.5\n\n0.4\n\n0.3\n\n0.2\n\n0.1\n\nn\no\n\ni\nt\nc\nu\nr\nt\ns\nn\no\nc\ne\nr\n \nt\nc\na\nx\ne\n\n \nf\n\no\n\n \n.\n\nb\no\nr\nP\n\n0\n0\n\n0.2\n\n0.4\n\n0.6\n\n0.8\n\n1\n\n0\n0\n\n5\n\n10\n\n15\n\n20\n\n25\n\n30\n\nR\n1\n\n(b)\n\nNumber of measurements per sensor\n\n(a)\nFigure 1: (a)Converseboundsandachievablemeasurementratesfor J = 2 signalswithcommon\nsparsecomponent andsparseinnovations (JSM-1). We\ufb01xsignallengths N = 1000 andsparsities\nK = 200, K1 = K2 = 50. Themeasurementrates Rj := Mj/N re\ufb02ectthenumberofmeasure-\nmentsnormalizedbythesignallength. Bluecurvesindicateourtheoreticalandanticipatedconverse\nbounds;redindicatesaprovablyachievableregion,andpinkdenotestheratesrequiredforseparable\nCSsignalreconstruction. (b)Reconstructingasignalensemblewithcommonsparsesupports(JSM-\n2). We plot the probability of perfect reconstruction via DCS-SOMP (solid lines) and independent\nCSreconstruction(dashedlines)asafunctionofthenumberofmeasurementspersignal M andthe\nnumber of signals J. We \ufb01x the signal length to N = 50 and the sparsity to K = 5. An oracle\nencoderthatknowsthepositionsofthelargecoef\ufb01cientswoulduse 5 measurementspersignal.\n\naccount for the observed measurements. Numerical simulations support such an approach\n(see Fig.1(a)). Future work will extend JSM-1 to `p-compressible signals, 0 < p \u2264 1.\n\n4.2\n\nJSM-2: Common sparse supports\n\nUnder the JSM-2 signal ensemble model (see also [2, 11]), independent recovery of each\nsignal via `1 minimization would require cK measurements per signal. However, algo-\nrithms inspired by conventional greedy pursuit algorithms (such as OMP [15]) can sub-\nstantially reduce this number.\nIn the single-signal case, OMP iteratively constructs the\nsparse support set \u2126; decisions are based on inner products between the columns of \u03a6\u03a8\nand a residual. In the multi-signal case, there are more clues available for determining the\nelements of \u2126.\nTo establish a theoretical justi\ufb01cation for our approach, we \ufb01rst proposed a simple One-\nStep Greedy Algorithm (OSGA) [11] that combines all of the measurements and seeks the\nlargest correlations with the columns of the \u03a6j\u03a8. We established that, assuming that \u03a6j\nhas i.i.d. Gaussian entries and that the nonzero coef\ufb01cients in the \u03b8j are i.i.d. Gaussian, then\nwith M \u2265 1 measurements per signal, OSGA recovers \u2126 with probability approaching 1\nas J \u2192 \u221e. Moreover, with M \u2265 K measurements per signal, OSGA recovers all xj with\nprobability approaching 1 as J \u2192 \u221e. This meets the theoretical lower bound for Mj.\nIn practice, OSGA can be improved using an iterative greedy algorithm. We proposed a\nsimple variant of Simultaneous Orthogonal Matching Pursuit (SOMP) [16] that we term\nDCS-SOMP [11]. For this algorithm, Fig. 1(b) plots the performance as the number of\nsensors varies from J = 1 to 32. We \ufb01x the signal lengths at N = 50 and the sparsity of\neach signal to K = 5. With DCS-SOMP, for perfect reconstruction of all signals the aver-\nage number of measurements per signal decreases as a function of J. The trend suggests\nthat, for very large J, close to K measurements per signal should suf\ufb01ce. On the contrary,\nwith independent CS reconstruction, for perfect reconstruction of all signals the number of\nmeasurements per sensor increases as a function of J. This surprise is due to the fact that\neach signal will experience an independent probability p \u2264 1 of successful reconstruction;\ntherefore the overall probability of complete success is pJ . Consequently, each sensor must\ncompensate by making additional measurements.\n\n\fJSM-3: Nonsparse common + sparse innovations\n\n4.3\nThe JSM-3 signal ensemble model provides a particularly compelling motivation for joint\nrecovery. Under this model, no individual signal xj is sparse, and so separate signal recov-\nery would require fully N measurements per signal. As in the other JSMs, however, the\ncommonality among the signals makes it possible to substantially reduce this number.\nOur recovery algorithms are based on the observation that if the common component zC\nwere known, then each innovation zj could be estimated using the standard single-signal\nCS machinery on the adjusted measurements yj \u2212\u03a6jzC = \u03a6jzj. While zC is not known in\nadvance, it can be estimated from the measurements. In fact, across all J sensors, a total of\n\nPj Mj random projections of zC are observed (each corrupted by a contribution from one\nnumber of measurements is suf\ufb01ciently large (Pj Mj (cid:29) N), zC can be estimated using\n\nof the zj). Since zC is not sparse, it cannot be recovered via CS techniques, but when the\n\nstandard tools from linear algebra. A key requirement for such a method to succeed in\nrecovering zC is that each \u03a6j be different, so that their rows combine to span all of RN . In\nthe limit, zC can be recovered while still allowing each sensor to operate at the minimum\nmeasurement rate dictated by the {zj}. A prototype algorithm, which we name Transpose\nEstimation of Common Component (TECC), is listed below, where we assume that each\nmeasurement matrix \u03a6j has i.i.d. N (0, \u03c32\n\nj ) entries.\n\nTECC Algorithm for JSM-3\n\nMj \u03c32\nj\n\n2. Estimate measurements generated by innovations: Using the previous estimate, sub-\ntract the contribution of the common part on the measurements and generate estimates\n\n1. Estimate common component: De\ufb01ne the matrix b\u03a6 as the concatenation of the regu-\n\u03a6j, that is, b\u03a6 = [b\u03a61,b\u03a62, . . . ,b\u03a6J ].\nlarized individual measurement matrices b\u03a6j = 1\nCalculate the estimate of the common component as czC = 1\nfor the measurements caused by the innovations for each signal: byj = yj \u2212 \u03a6jczC.\nobtain estimates of the innovations bzj from the estimated innovation measurements byj.\n4. Obtain signal estimates: Sum the above estimates, letting bxj = czC + bzj.\n\n3. Reconstruct innovations: Using a standard single-signal CS reconstruction algorithm,\n\nThe following theorem shows that asymptotically, by using the TECC algorithm, each\nsensor need only measure at the rate dictated by the sparsity Kj.\n\nJ b\u03a6T y.\n\nTheorem 1 [2] Assume that the nonzero expansion coef\ufb01cients of the sparse innovations\nzj are i.i.d. Gaussian random variables and that their locations are uniformly distributed\non {1, 2, ..., N }. Then the following statements hold:\n1. Let the measurement matrices \u03a6j contain i.i.d. N (0, \u03c32\n\nj ) entries with Mj \u2265 Kj +\n1. Then each signal xj can be recovered using the TECC algorithm with probability\napproaching 1 as J \u2192 \u221e.\n\n2. Let \u03a6j be a measurement matrix with Mj \u2264 Kj for some j \u2208 {1, 2, ..., J}. Then with\nprobability 1, the signal xj cannot be uniquely recovered by any algorithm for any J.\n\nFor large J, the measurement rates permitted by Statement 1 are the lowest possible for any\nreconstruction strategy on JSM-3 signals, even neglecting the presence of the nonsparse\ncomponent. Thus, Theorem 1 provides a tight achievable and converse for JSM-3 signals.\nThe CS technique employed in Theorem 1 involves combinatorial searches for estimating\nthe innovation components. More ef\ufb01cient techniques could also be employed (including\nseveral proposed for CS in the presence of noise [3, 5, 7, 10, 12]).\nWhile Theorem 1 suggests the theoretical gains from joint recovery as J \u2192 \u221e, practical\ngains can also be realized with a moderate number of sensors. For example, suppose in\n\nthe TECC algorithm that the initial estimate czC is not accurate enough to enable correct\n\n\fidenti\ufb01cation of the sparse innovation supports {\u2126j}. In such a case, it may still be possible\n\nfor a rough approximation of the innovations {zj} to help re\ufb01ne the estimate czC. This in\n\nturn could help to re\ufb01ne the estimates of the innovations. Since each component helps to\nestimate the others, we propose an iterative algorithm for JSM-3 recovery. The Alternating\nCommon and Innovation Estimation (ACIE) algorithm exploits the observation that once\nthe basis vectors comprising the innovation zj have been identi\ufb01ed in the index set \u2126j,\ntheir effect on the measurements yj can be removed to aid in estimating zC.\n\nACIE Algorithm for JSM-3\n\n1. Initialize: Set b\u2126j = \u2205 for each j. Set the iteration counter ` = 1.\nbe the Mj \u00d7 |b\u2126j| submatrix obtained\n2. Estimate common component: Let \u03a6j,b\u2126j\nby sampling the columns b\u2126j from \u03a6j and construct an Mj \u00d7 (Mj \u2212 |b\u2126j|) matrix\nQj = [qj,1 . . . qj,Mj \u2212|b\u2126j |] having orthonormal columns that span the orthogonal com-\nplement of colspan(\u03a6j,b\u2126j\n). Remove the projection of the measurements into the afore-\nmentioned span to obtain measurements caused exclusively by vectors not in b\u2126j, letting\nJ(cid:3)T\nj yj and e\u03a6j = QT\n2 . . . eyT\neyj = QT\nand modi\ufb01ed holographic basis e\u03a6 = he\u03a6T\n1 e\u03a6T\nmeasurements caused by the common part of the signal, setting fzC = e\u03a6\u2020eY , where\n3. Estimate innovation supports: For each signal j, subtract fzC from the measurements,\nbyj = yj \u2212 \u03a6jfzC, and estimate the sparse support of each innovation b\u2126j.\n\nj \u03a6j. Use the modi\ufb01ed measurements eY = (cid:2)eyT\n1 eyT\n\n4. Iterate: If ` < L, a preset number of iterations, then increment ` and return to Step 2.\n\nA\u2020 = (AT A)\u22121AT denotes the pseudoinverse of matrix A.\n\nJiT\n2 . . . e\u03a6T\n\nto re\ufb01ne the estimate of the\n\nOtherwise proceed to Step 5.\n\nj,b\u2126j\n\n= \u03a6\u2020\n\n(yj \u2212 \u03a6jfzC), where b\u03b8j,b\u2126j\n\n5. Estimate innovation coef\ufb01cients: For each signal j, estimate the coef\ufb01cients for the\nis a sampled version of\n\nindices in b\u2126j, setting b\u03b8j,b\u2126j\nthe innovation\u2019s sparse coef\ufb01cient vector estimate b\u03b8j.\n6. Reconstruct signals: Estimate each signal as bxj = fzC + bzj = fzC + \u03a6jb\u03b8j.\nIn the case where the innovation support estimate is correct (b\u2126j = \u2126j), the measurements\neyj will describe only the common component zC. If this is true for every signal j and the\nnumber of remaining measurementsPj Mj\u2212KJ \u2265 N, then zC can be perfectly recovered\n\nin Step 2. Because it may be dif\ufb01cult to correctly obtain all \u2126j in the \ufb01rst iteration, we \ufb01nd\nit preferable to run the algorithm for several iterations.\nFig. 2(a) shows that, for suf\ufb01ciently large J, we can recover all of the signals with signi\ufb01-\ncantly fewer than N measurements per signal. We note the following behavior in the graph.\nFirst, as J grows, it becomes more dif\ufb01cult to perfectly reconstruct all J signals. We be-\nlieve this is inevitable, because even if zC were known without error, then perfect ensemble\nrecovery would require the successful execution of J independent runs of OMP. Second,\nfor small J, the probability of success can decrease at high values of M. We believe this is\n\ndue to the fact that initial errors in estimating zC may tend to be somewhat sparse (since czC\n\nroughly becomes an average of the signals {xj}), and these sparse errors can mislead the\nsubsequent OMP processes. For more moderate M, it seems that the errors in estimating\nzC (though greater) tend to be less sparse. We expect that a more sophisticated algorithm\ncould alleviate such a problem, and we note that the problem is also mitigated at higher J.\nFig. 2(b) shows that when the sparse innovations share common supports we see an even\ngreater savings. As a point of reference, a traditional approach to signal encoding would\nrequire 1600 total measurements to reconstruct these J = 32 nonsparse signals of length\nN = 50. Our approach requires only about 10 per sensor for a total of 320 measurements.\n\n\f1\n\n0.9\n\n0.8\n\n0.7\n\n0.6\n\n0.5\n\n0.4\n\n0.3\n\n0.2\n\n0.1\n\nn\no\n\ni\nt\nc\nu\nr\nt\ns\nn\no\nc\ne\nR\n\n \nt\nc\na\nx\nE\n\n \nf\n\no\n\n \ny\nt\ni\nl\ni\n\nb\na\nb\no\nr\nP\n\n1\n\n0.9\n\n0.8\n\n0.7\n\n0.6\n\n0.5\n\n0.4\n\n0.3\n\n0.2\n\n0.1\n\nn\no\n\ni\nt\nc\nu\nr\nt\ns\nn\no\nc\ne\nR\n\n \nt\nc\na\nx\nE\n\n \nf\n\no\n\n \ny\nt\ni\nl\ni\n\nb\na\nb\no\nr\nP\n\n 8\n16\n32\n\n 8\n16\n32\n\n0\n0\n\n5\n\n10\n\n15\n\n20\n\n0\n0\n\n50\n\n10\n40\nNumber of Measurements per Signal, M\n\n30\n\n20\n\n(b)\n\n(a)\nFigure 2: Reconstructing a signal ensemble with nonsparse common component and sparse inno-\nvations(JSM-3)usingACIE.(a)ReconstructionusingOMPindependentlyoneachsignalinStep3\noftheACIEalgorithm(innovationshavearbitrarysupports). (b)ReconstructionusingDCS-SOMP\njointly on all signals in Step 3 of the ACIE algorithm (innovations have identical supports). Signal\nlength N = 50,sparsity K = 5. ThecommonstructureexploitedbyDCS-SOMPenablesdramatic\nsavingsinthenumberofmeasurements. Weaverageover1000simulationruns.\n\nNumber of Measurements per Signal, M\n\n25\n\n30\n\n35\n\nAcknowledgments: Thanks to Emmanuel Cand`es, Hyeokho Choi, and Joel Tropp for in-\nformative and inspiring conversations.\n\nReferences\n\n[1] D. Baron, M. F. Duarte, S. Sarvotham, M. B. Wakin, and R. G. Baraniuk. An information-\ntheoretic approach to distributed compressed sensing. In Allerton Conf. Comm., Control, Com-\nput., Sept. 2005.\n\n[2] D. Baron, M. B. Wakin, M. F. Duarte, S. Sarvotham, and R. G. Baraniuk. Distributed com-\n\npressed sensing. 2005. Preprint. Available at www.dsp.rice.edu/cs.\n\n[3] E. Cand`es, J. Romberg, and T. Tao. Stable signal recovery from incomplete and inaccurate\n\nmeasurements. Comm. Pure Applied Mathematics, 2005. To appear.\n\n[4] E. Cand`es and T. Tao. Near optimal signal recovery from random projections and universal\n\nencoding strategies. 2004. Preprint.\n\n[5] E. Cand`es and T. Tao. The Dantzig selector: Statistical estimation when p is much larger than\n\nn. 2005. Preprint.\n\n[6] E. Cand`es and T. Tao. Error correction via linear programming. 2005. Preprint.\n[7] S. Chen, D. Donoho, and M. Saunders. Atomic decomposition by basis pursuit. SIAM Journal\n\non Scienti\ufb01c Computing, 20(1):33\u201361, 1998.\n\n[8] T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, New York, 1991.\n[9] D. Donoho. Compressed sensing. 2004. Preprint.\n[10] D. Donoho and Y. Tsaig. Extensions of compressed sensing. 2004. Preprint.\n[11] M. F. Duarte, S. Sarvotham, D. Baron, M. B. Wakin, and R. G. Baraniuk. Distributed com-\npressed sensing of jointly sparse signals. In Asilomar Conf. Signals, Sys., Comput., Nov. 2005.\n[12] J. Haupt and R. Nowak. Signal reconstruction from noisy random projections. 2005. Preprint.\n[13] S. Pradhan and K. Ramchandran. Distributed source coding using syndromes (DISCUS): De-\n\nsign and construction. IEEE Trans. Inform. Theory, 49:626\u2013643, March 2003.\n\n[14] D. Slepian and J. K. Wolf. Noiseless coding of correlated information sources. IEEE Trans.\n\nInform. Theory, 19:471\u2013480, July 1973.\n\n[15] J. Tropp and A. C. Gilbert. Signal recovery from partial information via orthogonal matching\n\npursuit. 2005. Preprint.\n\n[16] J. Tropp, A. C. Gilbert, and M. J. Strauss. Simulataneous sparse approximation via greedy\n\npursuit. In IEEE 2005 Int. Conf. Acoustics, Speech, Signal Processing, March 2005.\n\n[17] Z. Xiong, A. Liveris, and S. Cheng. Distributed source coding for sensor networks. IEEE Signal\n\nProc. Mag., 21:80\u201394, September 2004.\n\n\f", "award": [], "sourceid": 2754, "authors": [{"given_name": "Michael", "family_name": "Wakin", "institution": null}, {"given_name": "Marco", "family_name": "Duarte", "institution": null}, {"given_name": "Shriram", "family_name": "Sarvotham", "institution": null}, {"given_name": "Dror", "family_name": "Baron", "institution": null}, {"given_name": "Richard", "family_name": "Baraniuk", "institution": null}]}