“Nonparametric Local Smoothing” is not image registration
BMC Research Notes volume 5, Article number: 610 (2012)
Image registration is one of the most important and universally useful computational tasks in biomedical image analysis. A recent article by Xing & Qiu (IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(10):2081–2092, 2011) is based on an inappropriately narrow conceptualization of the image registration problem as the task of making two images look alike, which disregards whether the established spatial correspondence is plausible. The authors propose a new algorithm, Nonparametric Local Smoothing (NLS) for image registration, but use image similarities alone as a measure of registration performance, although these measures do not relate reliably to the realism of the correspondence map.
Using data obtained from its authors, we show experimentally that the method proposed by Xing & Qiu is not an effective registration algorithm. While it optimizes image similarity, it does not compute accurate, interpretable transformations. Even judged by image similarity alone, the proposed method is consistently outperformed by a simple pixel permutation algorithm, which is known by design not to compute valid registrations.
This study has demonstrated that the NLS algorithm proposed recently for image registration, and published in one of the most respected journals in computer science, is not, in fact, an effective registration method at all. Our results also emphasize the general need to apply registration evaluation criteria that are sensitive to whether correspondences are accurate and mappings between images are physically interpretable. These goals cannot be achieved by simply reporting image similarities.
Image registration is one of the most commonly encountered and important problems in biomedical image analysis[1–7]. It is “a process for determining the correspondence of features between images”, “the determination of a one-to-one mapping or transformation between the coordinates in one space and those in another”, and with the objective to “bring the modalities involved into spatial alignment”. Registration ”geometrically aligns two images”, thus “determining the spatial alignment between images”.
What is not the goal of registration, however, is to make one (“moving”) image appear maximally like another (“fixed” or “reference”) image. If this were the case, nothing would be gained by the process of registration, as we are given a priori the fixed image. Instead, the primary result of registration is the correspondence established between the images, and maximization of image similarity is a by-product of the accuracy of this correspondence. A useful geometric transformation between images must allow interpretation of their differences and, potentially, statistics in a well-defined transformation space. Performance measures of image registration, therefore, must consider the specific properties of correspondences in addition to the numerical similarity achieved between the images, especially when the transformation itself is the object of further analysis, as is the case for the widely used Tensor-Based or Deformation-Based Morphometry methods.
Nonparametric Local Smoothing (NLS)
In their recent paper, “Intensity-based image registration by Nonparametric Local Smoothing,” Xing & Qiu state, incorrectly, that “the major goal of image registration is to find a geometrical transformation T(x,y)=(T1(x,y),T2(x,y)) such that Z M (T(x,y)) is as close to Z R (x,y) as possible.” (Herein, Z R is a two-dimensional fixed image and Z M the moving image being registered to Z R via transformation T(x,y).)
Solving this type of optimization problem is common in image registration, but it is not the goal in and of itself. Instead, the fundamental goal of image registration is to find a transformation T such that the difference ||T−Ttrue||between estimated and true transformation is minimized. This, however, cannot be determined based on image similarity, even when, as Xing & Qiu constrain, the moving image is “a geometrically altered version of [the fixed image].”
This misconception of the purpose of image registration, however, is the foundation of their experiments that rely on image similarity alone to quantify registration quality. But image similarity is not a valid measure of registration accuracy, thus leaving unsupported the conclusion that “Nonparametric Local Smoothing” (NLS) performs effective image registration.
In this communication we provide direct and specific evidence that the experimental design and reasoning employed by Xing & Qiu are flawed and that their NLS algorithm is not suitable for image registration at all by experimentally substantiating two crucial observations. Firstly, we show that a deformation field computed by the NLS method is largely arbitrary and substantially different from the ground truth. Second, to explain why the NLS algorithm appeared to achieve superior registrations, we show that all evaluation criteria used by Xing & Qiu are unsuitable to validate image registration, as a simple permutation-based algorithm outperforms the NLS method without achieving any meaningful alignment at all.
The NLS method does not create interpretable transformations
We obtained the identical images used in Ref. and shown in Figures two, six, eight, and ten therein. From the images of the “Ball” example, we reconstructed the ground truth transformation between them as follows. We determined that the top-left area, 156×196 pixels, of the fixed image had been copied, shifted by 50 pixels, and pasted into the top-right image corner (Figure1). The ground truth deformation field, u, is thus partitioned into four distinct regions: “A,” duplicated content, two equally correct mappings, u=(0,0)and u=(50,0); “B,” u=(50,0). “C,” content pasted over, transformation undefined; “D,” unmoved, u=(0,0).
We obtained the actual deformation field computed by the NLS algorithm and visualized it in Figure2 by color-coding and overlaying the x and y components of the deformation vector at each pixel onto the fixed image. For convenience, the 156×196 pixel region that was shifted is also marked by a white box. The actual deformation field compares with the ground truth as follows:
The boundary that separates the shifted from the stationary region at the bottom is curved and at a substantial distance from the true boundary.
In Region “C,” there is no ground truth, but the actual deformation is dominated by an area with a horizontal shift (which wraps around the image edge, via an undocumented periodic boundary condition), and a second large area with a dominant vertical shift. The boundary between them is curved and its location arbitrary.
Pixels with zero deformation are scattered throughout regions “A,” “B,” and “C” and even inside the principal object (ball).
These observations confirm our contention that the deformation field computed by the NLS method is largely arbitrary and cannot be relied upon for interpretation. A further significant observation is that the reformatted and difference images based on this deformation field and shown in Ref. (Figures four(a) and five(a) therein) reveal no hint as to where the computed deformation is accurate with respect to the true deformation and where it is not.
Image similarity fails as a measure of registration quality
To explain why the failure of NLS was not apparent from the results presented in Ref. we performed “registrations” on the exact same image pairs used therein. To each image pair, we applied a rank-order permutation algorithm, “CURT”, the “Completely Useless Registration Tool.” In short, CURT sorts the pixels in each image in order of increasing intensity and maps each fixed image pixel to the moving image pixel at the equivalent index in the sort order. Thus, image similarity is maximized, but no meaningful spatial transformation is actually computed. Indeed, CURT was conceived specifically to demonstrate the inadequacy of image similarity for evaluating image registration.
The quantitative measures of image similarity achieved by CURT, as well as those achieved by NLS, are listed in Table1: root residual mean squares (RRMS), cross correlation (CC), and entropy of image difference (EID). For all measures and all examples, CURT clearly outperformed NLS. Also, for all examples, the reformatted moving images are visually indistinguishable from the fixed images (see Additional file1: Figure S1).
Thus, by the (flawed) reasoning employed in Ref., CURT would have to be declared the far superior registration algorithm. Yet it is obvious from CURT’s design that it does not compute any valid spatial correspondence, i.e., it is not an effective registration algorithm. This demonstrates that image similarity cannot be used by itself to quantify image registration performance.
The “Nonparametric Local Smoothing” algorithm is not an effective registration technique. Correcting the research record on this matter is particularly important because the method was published in perhaps the most respected journal in computer sciencea. We made every effort to publish our results in that same journal but were unsuccessful (see Additional file2).
Analysis of a deformation field computed by NLS revealed that it is largely arbitrary, not based in reality, and thus not suitable for the purposes of interpretation, quantification, and measurement. These issues are particularly salient in the case of high-dimensional registration where the number of parameters can match or even exceed the number of pixels in the image. In such cases, transformation regularity is fundamental to not only yielding a well-posed algorithm but also producing interpretable results. That is why the idea has persisted from early to more current research in (especially dense, high-dimensional) image registration.
Although indeed continuous transformation models cannot represent discontinuous motion, we found that the NLS algorithm is unable to correctly recover such motion that actually is continuous. This is a significant shortcoming because motion involving actual, macroscopic physical objects must be at least locally continuous. In addition, while NLS is able to represent discontinuous motion in general, it is unable to determine the correct discontinuous motion (see Figures1 and2).
Our results demonstrate fundamental flaws in the NLS algorithm, resulting ultimately in its complete and utter failure to compute meaningful registrations. By using the exact data used by the algorithm’s authors, we have ensured that our evaluation satisfies the exact same assumptions made in the original paper. One particular such assumption is constancy of image intensities, as implied by the requirement M(T1(x,y),T2(x,y))=R(x,y) for the true transformation (T1,T2)(although one of the authors’ very own examples, “Satellite,” violates this assumption; see Additional file3: Figure S2).
But intensity constancy is not even sufficient for this particular algorithm to work. In actual fact, the method requires that if, then, i.e., it requires that identity of image intensity (or here: of local patch texture) uniquely encodes the true correspondence between any fixed image pixel (or patch) and its location in the moving image. In other words, NLS implicitly assumes intensity (or patch) uniqueness, but this is simply not realistic in any conceivable application scenario. If it were realistic, then the “CURT” algorithm would compute valid registrations under these conditions as well, and as we have also demonstrated herein it would have to be considered superior to NLS.
Xing & Qiu published an algorithm incorrectly assuming that maximizing image similarity is sufficient for image registration. The fact that accurate registration may improve image similarity metrics does not mean that optimizing similarity metrics necessarily leads to accurate registration which, with Xing and Qiu’s assistance, we have demonstrated experimentally herein.
It is tempting to blame the unconstrained and discontinuous transformation models of both NLS and CURT for the failure of image similarity to reflect the accuracy of spatial alignment. But even in the presence of an appropriate transformation model, “the desired optimum when registering images using voxel similarity measures is frequently not the global optimum, but is one of the local optima”. This is the well-known issue of capture range – intensity-based registration typically converges to the “correct” solution only when initialized with a transformation within a certain neighborhood of the correct alignment. Thus, even with a perfectly appropriate and well-constrained transformation model, it is only within the capture range of the correct transformation that we can reasonably assume that improved image similarity corresponds to improved alignment.
In summary, it is imperative to understand that the goal of image registration is to establish spatial correspondence between images, often at a given specific scale, but not merely to make them look alike. Thus, as Crum et al. noted, registration “validation tests the ability of registration to establish correspondence,” whereas image similarity is “uninformative about the magnitude of errors of correspondence.” The usefulness of the NLS algorithm for image registration was originally supported by experiments that ignored correspondence. But with correspondence now properly considered herein, we must conclude that NLS simply does not perform registration, which is a fatal flaw for a method specifically advertised as a registration algorithm.
aRanked number one out of 108 journals in the “COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE” category of the 2010 Journal Citation Reports.
Completely Useless Registration Tool
Entropy of Image Difference
Nonparametric Local Smoothing
Hawkes DJ: Algorithms for radiological image registration and their clinical application. J Anat. 1998, 193 (Pt 3): 347-361.
Maintz JBA, Viergever MA: A survey of medical image registration. Med Image Anal. 1998, 2: 1-36.
Lester H, Arridge SR: A survey of non-linear medical image registration. Pattern Recognit. 1999, 32: 129-149. 10.1016/S0031-3203(98)00095-8.
Hill DLG, Batchelor PG, Holden M, Hawkes DJ: Medical image registration. Phys Med Biol. 2001, 46: R1-R45. 10.1088/0031-9155/46/3/201.
Pluim JPW, Maintz JBA, Viergever MA: Mutual-information-based registration of medical images: a survey. IEEE Trans Med Imag. 2003, 22 (8): 986-1004. 10.1109/TMI.2003.815867.
Crum WR, Hartkens T, Hill DLG: Non-rigid image registration: theory and practice. Br J Radiol. 2004, 77: S140-S153. 10.1259/bjr/25329214.
Gholipour A, Kehtarnavaz N, Briggs R, Devous M, Gopinath K: Brain functional localization: a survey of image registration techniques. IEEE Trans Med Imag. 2007, 26 (4): 427-451.
Maurer CR Jr, Fitzpatrick JM, Wang MY, Galloway RL Jr, Maciunas RJ, Allen GS: Registration of head volume images using implantable fiducial markers. IEEE Trans Med Imag. 1997, 16 (4): 447-462. 10.1109/42.611354.
Zitová B, Flusser J: Image registration methods: a survey. Image Vision Comput. 2003, 21 (11): 977-1000. 10.1016/S0262-8856(03)00137-9.
Ashburner J, Hutton C, Frackowiak R, Johnsrude I, Price C, Friston K: Identifying global anatomical differences: deformation-based morphometry. Hum Brain Map. 1998, 6 (5–6): 348-357.
Xing C, Qiu P: Intensity-based image registration by nonparametric local smoothing. IEEE Trans Pattern Anal Machine Intell. 2011, 33 (10): 2081-2092.
Rohlfing T: Image similarity and tissue overlaps as surrogates for image registration accuracy: widely used but unreliable. IEEE Trans Med Imag. 2012, 31 (2): 153-163.
Horn BKP, Schunck BG: Determining optical flow. Artif Intell. 1981, 17: 185-203. 10.1016/0004-3702(81)90024-2.
Yeo BT, Sabuncu MR, Desikan R, Fischl B, Golland P: Effects of registration regularization and atlas sharpness on segmentation accuracy. Med Image Anal. 2008, 12 (5): 603-615. 10.1016/j.media.2008.06.005.
Crum WR, Griffin LD, Hill DLG, Hawkes DJ: Zen and the art of medical image registration: correspondence, homology, and quality. NeuroImage. 2003, 20 (3): 1425-1437. 10.1016/j.neuroimage.2003.07.014.
C Xing kindly provided the images used in Ref. as well as the deformation field generated by the NLS method for the “Ball” example.
The “moving” and “fixed” images for the “Ball,” “Bird,” “Satellite”, and “MRI” examples were previously published in: Chen Xing and Peihua Qiu, “Intensity-Based Image Registration by Nonparametric Local Smoothing,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 10, pp. 2081–2092, Oct. 2011, doi: 10.1109/TPAMI.2011.26. Ⓒ2011 IEEE. Reprinted, with permission, from IEEE Transactions on Pattern Analysis and Machine Intelligence (see Additional file4).
Torsten Rohlfing was supported by the National Institute of Biomedical Imaging and Bioengineering (NIBIB), Grant No. EB008381, and National Institute on Alcohol Abuse and Alcoholism (NIAAA), Grants No. AA005965, AA012388, AA013521, AA017347, AA017168, and ARRA funding under Grant No. AA005965-27S1. Brian Avants acknowledges ARRA funding from the National Library of Medicine (NLM) via award HHSN276201000492p. NIBIB, NIAAA, and NLM did not have any influence on study design; collection, analysis, and interpretation of data; the writing of the manuscript; and on the decision to submit the manuscript for publication.
The authors declare that they have no competing interests.
Both authors jointly conceived this study. TR obtained the image data used in the criticized paper, implemented the CURT algorithm, and ran it on the test data. BA implemented the ANTs registration software, and ran it on the test data. Both authors jointly wrote the text of this article and approved the final submitted manuscript.
Electronic supplementary material
Additional file 1:Supplemental Figure1 – Results of “CURT” algorithm. Results of “CURT” algorithm applied to images previously published in: C. Xing and P. Qiu, “Intensity-Based Image Registration by Nonparametric Local Smoothing,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.33, no.10, pp. 2081–2092, Oct. 2011, doi: 10.1109/TPAMI.2011.26. Ⓒ2011 IEEE. Reprinted, with permission, from IEEE Transactions on Pattern Analysis and Machine Intelligence. The “moving” and “fixed” images were kindly provided by C. Xing. (PDF 75 KB)
Additional file 2:Correspondence with IEEE-TPAMI. This PDF document contains, in this order, 1. our Comment originally submitted to IEEE-TPAMI, 2. the notice of immediate rejection, 3. our request for reconsideration with detailed list of procedural and technical flaws in the editorial decision, 4. the final rejection notice, and 5. our comments on the final rejection notice (these were not submitted to TPAMI but are included here for clarification). (PDF 532 KB)
Additional file 3:Supplemental Figure2 – “Satellite” Example. Image intensity difference, not registration, carries information in Xing & Qiu’s “Satellite” example (Image source: http://webmodis.iis.u-tokyo.ac.jp/Landsat/). Top row: input satellite images. Bottom row: difference image and transparent red overlay of difference onto fixed image. Correcting changes in image intensity via spatial transformations (beyond affine alignment to match fields of view) has no basis in reality and violates the authors’ own stated assumption of intensity constancy. Input satellite images were previously published in: C. Xing and P.Qiu, “Intensity-Based Image Registration by Nonparametric Local Smoothing,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.33, no.10, pp. 2081–2092, Oct. 2011, doi: 10.1109/TPAMI.2011.26. Ⓒ2011 IEEE. Reprinted, with permission, from IEEE Transactions on Pattern Analysis and Machine Intelligence. (PDF 57 KB)
Authors’ original submitted files for images
About this article
Cite this article
Rohlfing, T., Avants, B. “Nonparametric Local Smoothing” is not image registration. BMC Res Notes 5, 610 (2012). https://doi.org/10.1186/1756-0500-5-610
- Image registration