Optimized cross-modality guided contrast enhancement for liver tumor segmentation

— Radiologists currently use CT images with intravenous contrast infusion, in order to detect lesions and vessels in the liver. This step is quite time consuming when it is done manually. There are many algorithms developed for segmentation which are based on edge or region characteristics. These are dependent on the quality of the image. The contrast of a CT or MRI image plays an important role in identifying region of interest i.e. lesion(s). Traditional enhancement methods suffer from limitations such as saturation, over-enhancement, and uneven contrast spatial distribution that may result from the uncontrolled CE process. One way to overcome such limitations is to combine the contrast enhancement approach with a quality control scheme. One way to overcome such limitations is to combine the contrast enhancement approach with a quality control scheme. Inspired by the guided filtering approach and the simplicity of context-aware histogram-based image quality enhancement, propose in this paper a cross-modality guided histogram specification technique to improve the contrast of liver CT images using MRI images as guiding input data. The proposed method is based on two concepts, namely guided image enhancement and image quality control through an optimization scheme. The proposed OPTimized Guided Contrast Enhancement (OPTGCE) scheme exploits both contextual information from the guidance image and structural information from the input image. Tumor segmentation algorithm is applied on the enhanced images to analyze the performance of the proposed method in facilitating tumor segmentation. The qualitative and quantitative analysis using metrics including entropy, MCCEE, and MIGLCM shows the superiority of the proposed method in comparison with the existing methods that do not include guidance mechanism


I. INTRODUCTION
Liver cancer is considered one of the major causes of death in humans [1]. Early detection of tumors is essential for increasing the survival chances of patients. Recent advancements in medical imaging modalities have enabled the acquisition of high-resolution CT datasets, and thus, allowing physicians to identify both small and large tumors by manual visual inspection. Owing to the large number of images in medical datasets, it is difficult to manually analyze all images, and useful diagnostic information may be overlooked. Moreover, the diagnoses are mainly based on the physician's subjective evaluation and are dependent on the physician's experience. Therefore, computer assisted diagnosis (CAD) and computer assisted surgery have become one of the major research subjects.
Image contrast and quality enhancement or image detail improvement algorithms are necessary to highlight the image details in different medical image processing applications, particularly non-contrast CT scans. A common purpose in these applications is to enhance the contrast and the quality of like image while preserving the edge information [2]. Indeed, like degradations may have a crucial impression on the quality of the image; and consequently, it affects the human interpretation as well as the performances of computer assisted methods in medical imaging Different techniques are proposed in literature to repair the damaged images and improve their quality, improve their contrast and brightness ( [3,4,5], Jha and Chouhan, 2014). In fact, contrast enhancement of dark medical images, like noncontrast CT images, is an important and crucial step in image processing, particularly when reimaging is not possible [6,7,8]. In general, we can distinguish two main categories of contrast enhancement methods: spatial and frequency domain methods. Spatial domain algorithms are considered for a direct processing of image's pixels. Histogram equalization (HE) methods are one of the most image enhancement techniques used in spatial domain [9]. Nevertheless, HE does not always give acceptable performances since it could produce a contrast lost for less frequent gray levels and an over-enhancement for frequent ones [10]. Singular Value Decomposition method is also considered for image equalization [11]. This method preserves the general shape of the histogram and significantly reduce the loss of information contained in the histogram.
Traditional enhancement methods suffer from limitations such as saturation, over-enhancement, and uneven contrast spatial distribution, that may result from the uncontrolled CE process. One way to overcome such limitations is to combine the contrast enhancement approach with a quality control scheme. Inspired by the guided filtering approach and the simplicity of context-aware histogram-based image quality enhancement, we propose in this paper a cross-modality guided histogram specification technique to improve the contrast of liver CT images using MRI images as guiding input data. Furthermore, optimization is incorporated to prevent the saturation artifacts inherent to histogram-based methods. A similar idea was proposed for enhancing natural images in [12]. It consists of mapping the histogram of the input image to that of a reference image combined with an optimization technique to preserve the structures of the input image. In this work, we propose a similar approach for medical images using cross-modal information. The new CE approach is hence based on two concepts, namely, cross modality-guided medical image enhancement to improve the global contrast, and quality control to preserve the local structures during enhancement. Here, we formulate the crossmodal CE as an optimization problem, where the gradient of structural similarity index measure (SSIM) is used for local structure preservation and minimizing artifacts introduced during enhancement [13]. Later, the role of CE is analyzed in facilitating tumor segmentation. The overall processing scheme is evaluated on a real dataset containing CT and MRI of the human liver with segmentation ground truth.
The main contributions of this paper are: • The 2D histogram specification-based CE process is formulated as an optimization problem and extended to multimodal medical imaging data for the first time.
• SSIM gradient is incorporated in the optimized crossmodality guided 2D-HS framework to preserve structural fidelity of the enhanced image with the original image while applying enhancement.
• To achieves a nice balance between retaining structural similarity with input image (by integrating SSIM gradient) and enhancing contrast by employing 2D entropy. The suggested combination of cross-modal guidance and quality control enhances the CT image exploiting contextual information, as opposed to context-unaware schemes.
• A new goal-oriented performance evaluation of the proposed approach is done utilizing objective quality metrics and through segmentation results applied on real multi-modal liver data. Comparison with single enhancement techniques validate the superior performance of the proposed method.
The rest of the paper is organized as follows. Section II provides a brief review of relevant contrast enhancement methods. Section III describes the proposed Optimized guided CE method. Experimental results of CE are discussed in section IV. The results of applying segmentation on the enhanced images are described in section V, followed by conclusion in section VI.

II. RELATED WORKS
Several algorithms are proposed in the literature to enhance the contrast of different types of images. Global histogram equalization (GHE) algorithm is one of the most used techniques for contrast enhancement. The principle of GHE is based on distributing the pixel values of an input image over its dynamic intensity range according to its linear cumulative histogram [9]. In fact, the whole image is divided into equidistant levels defining different boundaries. In order to obtain a flat histogram, the boundaries limiting the initial levels are modified by intercalating a number of new boundaries between the old ones. The new boundaries are calculated using a recursive relation [15] Nevertheless, GHE algorithm conducts in some cases to an excessive contrast enhancement [10]. To overcome these problems, different HE algorithms are proposed, such as Brightness preserving bi-histogram equalization (BBHE) [16], dualistic sub-image histogram equalization (DSIHE) [17], brightness preserving dynamic histogram equalization (BPDHE) [18], minimum mean brightness error bi-histogram equalization [19,20] and recursively separated and weighted HE algorithm, which is a combination of BBHE and DSIHE [10]. Like improved algorithms split a histogram in different sub-blocks and perform HE on each individual partition. Tan et al. proposed a nonlinear local HE method to enhance the contrast of CT brain images [21]. The main idea of this algorithm is to decompose the input image into different subblocks. The distribution of the maximum levels in the histogram are excluded for each sub-block and therefore redistributed to the other gray levels with a threshold limitation. A gray level reallocation function is finally defined. The major drawback of this algorithm is the manual setting of the threshold used for contrast limitation. Huang and Yeh proposed a histogram equalization based image enhancement method including two main modules. The first histogram separation module is used to separate the histogram in smallscale detail. The second intensity transformation module is considered for a further contrast enhancement of the image with complete brightness preservation for each generated subhistogram [22]. Sundaram et al. described a technique to improve the contrast of mammogram images. This technique included both histogram modifications and local contrast enhancement techniques [23]. Al-Juboori presented a method to enhance the contrast of mammographic image. This method combined both contrast limited adaptive histogram equalization and retinex techniques [24]. Ganesan et al. presented a seed dependent adaptive region growing approach for contrast enhancement of CT images [25].
SVD techniques are also considered to improve the contrast of dark image and also to overcome the limits of HE methods [26,27]. In fact, the singular value matrix contains the illumination information in the image, so that the conversion of singular values will directly change the illumination of the image, and other information present in the image will be as same as before. In order to preserve the edge information from possible degradation, Bhandari et al. proposed the DWT-SVD algorithm for satellite image enhancement [28]. In this algorithm, authors applied SVD method only on LL sub-band image computed using DWT. A new LL sub-band is obtained by multiplying the initial singular value matrix by a correction factor. This factor is equal to the ratio of highest singular value of the created normalized matrix, having mean zero and variance of one, over a normalized input image. Modified LL sub-band image is combined with unprocessed LH, HL, and HH sub-bands to generate the enhanced image. The performances of this technique are compared with GHE, BPDHE, and SVD techniques, and results showed the higher performances of DWT-SVD over the others. Bhandari et al. proposed an improved algorithm for satellite images enhancement based on artificial bee colony using DWT-SVD [29]. The method employed the artificial bee Colony method to learn the parameters of the adaptive thresholding function needed for optimum enhancement. Atta and Abdel-Kader presented an improved method using SVD and DWT for contrast enhancement [30]. Indeed, authors considered the singular matrices of both input image and processed image using GHE to compute the enhanced singular value matrix.
The technique provides enhanced images without the addition of artefacts and modification in mean brightness of the original image.
Yang et al. reported a technique using both wavelet and Haar transforms for the contrast enhancement of medical images [8]. Wan and Shi proposed a contrast enhancement method based on Daubechies wavelets and exact histogram equalization [31]. Kaur and Singh proposed an algorithm for contrast improvement of cephalometric images. This algorithm is based on adaptive HE with gray level information histogram using wavelet [32]. Cheng and Huang presented a method for image and video contrast enhancement using Bezier curve. The parametric Bezier curve is used to illustrate the mapping curve for the intensity transformation [33]. Lee et al. proposed a dominant brightness level analysis and adaptive intensity transformation based contrast enhancement method. This technique performs brightness-adaptive intensity transfer functions using the LL sub-band image and therefore transforms intensity values according to the transfer function [34].

III PROPOSED SYSTEM
The main idea of the proposed method is to adapt the guided image filtering concept to the cross-modality context for medical image enhancement. The idea behind employing guided image processing in the medical imaging context is to exploit the diversity and the complementary information conveyed by the different modalities in order to highlight salient features for facilitating the medical diagnosis. The proposed method OPTGCE operates according to this strategy. It applies HS-based CE to the low-contrast CT image based on the second-order distribution of an image of a complementary modality, that is MRI.

Fig. 1. Block diagram of the proposed method
One strategy to prevent the CE from side effects is to control the enhancement by using a local similarity measure between the input and enhanced image or some stopping criteria. Here, we perform the optimization using a measure that is directly related to the structural information in the image and carries contrast information. Furthermore, the extent of contrast is quantified through the two-dimensional entropy. The flowchart of the proposed technique is shown in Fig. 1. The three essential components of the proposed method, namely the 2D histogram specification-based CE, the structural gradient-based similarity measure, and 2D entropy are described below

A) 2D HISTOGRAM SPECIFICATION
2D Histogram Specification methods can improve the contrast in an image by increasing the pixel-value differences among the neighbouring pixels. This has the disadvantage of not taking into account the strong spatial correlation of pixels and exploiting it in order to avoid side effects associated with histogram approaches. These limitations have led to the use of higher-order statistics of pixel values and characteristics to develop more efficient methods.
Let f(m,n) denotes the input image signal at pixel (m,n), the associated 2D histogram is defined as below: (1) Here, i and j represent the pixel values and (m, n) and (p, q) represent the image coordinates, K is the total number of grey levels, and 0 ≤ i, j ≤ K − 1, The transition probability of grey-levels, i.e. the 2D normalized histogram, is derived from the GLCM as follows: The 2D-histogram is then used in the pixel grey-level mapping process using the histogram specification method as described below. This mapping process is based on the two-dimensional The expression of the 2D-CDF of the guidance image is computed similarly and is represented as Hg. Once the 2D-CDF of both images is computed, the transformation T allowing the mapping between the input signal and the desired signal is obtained as follows: The algorithm searches for T(c,d)= [ ,T(c,d)2], the target pixel value, where T(c,d)1,T(c,d)2 indicate pixel values corresponding to c and d. The second term in the above expression selects a closer pixel pair if difference of first term among candidate pixel pairs are very small. At this point, a target pixel value pair is calculated for each input pixel value pair. Now, each pixel is paired with every pixel in its neighborhood, therefore, a relaxed solution is presented to obtain the output pixel value f(m,n). Each adjacent pixel in the neighborhood casts a vote for target pixel value of f(m,n). The value that gives the minimum sum of absolute difference of votes is taken as target pixel value. Practically, it is the median of pixel values voted by adjacent pixels. Using 2D CDF manipulation, target pixel value pair is calculated for each input pixel value pair.

From Eq. 5, it can be inferred that transformation of each value in the original image [ ] to a new value in the enhanced
image [ ] also depends on its neighboring element. Therefore, unlike the 1D histogram specification which only considers individual pixel values for calculating the CDFs and ultimately mapping these values, this approach also exploits the contextual information among the pixels. Next, we look at the SSIM gradient approach.

B) GRADIENT BASED STRUCTURAL SIMILARITY MEASURE
The idea is to apply global HS to a low-contrast image driven by an SSIM-based measure to control the enhancement through structural similarity changes between the original image and its enhanced variant. SSIM is a well-established measure to calculate the extent of similarity between two images [13]. Considering one image as a reference, the index provides the quality of the image under analysis in comparison with a reference. SSIM index is calculated between corresponding local blocks in images [A] and [B], after which the average of the values is taken to obtain a single value of SSIM as the overall similarity index. Let us assume that ax and bx represent corresponding blocks in both images; µax and µbx represent the mean intensity values of ax and bx and the standard deviations are given by and . C1 and C2 are small numbers greater than 0 to ensure the denominator is not zero. The SSIM between the two blocks ax and bx is then expressed as: Few terms in Eq. 6 are described mathematically a where w is 11 × 11 Gaussian kernel and * indicates convolution. Eq. 6 could be regarded as expression for SSIM index map, SSIMmap calculated via element wise addition and multiplication using parameters expressed in Eq. 7. Then, at all points, SSIMmap indicates local similarity between images [A] and [B]. The global SSIM index for the overall images can then be expressed a where Z denotes the number of pixels in either image.
. For the local SSIM measures in Eq. 6, we define the following terms for compactness.
. Here, 2D-HS is applied to enhance CT images by exploiting the better quality of MR images. When applied in the framework of optimization, the SSIM gradient refines the enhancement process incrementally.
The integration of SSIM ultimately preserves the overall morphology of the original image with minimal information loss during enhancement. Here, we denote the input image as [f] and the image whose structural similarity is being compared with Calculating the derivative of Eq. 10 with reference to [fe] gives the SSIM gradient expression as follows: + where α1, α2, β1 and β2 have been described in Eq. 9a and 9b. The estimated increase in SSIM at iteration t is mathematically described as Based on the behavior of SSIM(t) at several iterations, 1 SSIM(t) can be modeled by αrs t [15]. The final value of SSIM (after several iterations) can be expressed as: Proposed experiments show that SSIM value changes faster in earlier iterations, therefore the algorithm is executed three times to calculate the quantities in Eq. 13. Replacing SSIMf value by 1 (the ideal value) and substituting the above values in Eq. 13, the approximated upper bound on α can be calculated as: Proposed method to measure the contrast enhanced at each iteration by applying 2D-HS. Therefor2D entropy is used to control the level of enhancement. Here, we have used 2D entropy to formulate this criterion a The change in entropy of the enhanced image gained with every iteration is calculated as follows . At a specific point in the optimization process, when 1E becomes negligible (close to zero) or when the 1E value starts oscillating, the enhancement process is stopped Fig.4 Enhanced Output

IV. PERFORMANCE EVALUATION
We tested our method on 5 patients' data constituting 10 CT-MR image pairs (containing tumors). The images from different volumes are of different spatial sizes (such as 512 × 512, 360×240) with pixel values in the range [0, 255]. In medical image processing tasks such as segmentation and enhancement, the processing is often restricted to a particular organ and the nearby organs are removed from the medical image. The liver area in the images is therefore separated and processing is applied only to this region.

V. TOWARDS AN OPTIMAL SEGMENTATION PRESERVING LOCAL STRUCTURES
The results of applying gradient-driven SRG algorithm on enhanced as well as input images are demonstrated in Fig. 5. In general, application of the CE methods improve the contrast of the input image, which ultimately enables SRG to locate tumor contours favorably. However, OPTGCE well preserves uniformity in the structure of tumors in the enhanced image together with yielding sharp tumor edges. Therefore, Seeded Region Growing (SRG) algorithm is better able to locate the tumor contours in the OPTGCE-enhanced images. This property enables OPTGCE to outperform other CE methods in facilitating tumor segmentation.

V1. CONCLUSION
This paper proposes an optimization-based guided contrast enhancement approach OPTGCE for low contrast CT images. The proposed technique adopts a context-aware 2D histogrambased scheme of exploiting information in the better perceptual quality guidance image for global contrast enhancement, while local image structures are enhanced through SSIM based measure in an optimization framework. This combination effectively improves the contrast while minimizing the artifacts associated with typical histogrambased enhancement methods to preserve the morphological information of the image during enhancement. The qualitative and quantitative analysis using metrics including entropy, MCCEE, and MIGLCM shows the superiority of the proposed method in comparison with the existing methods that do not include guidance mechanism. Finally, a tumor segmentation algorithm is applied on the enhanced images to analyze the performance of the proposed method in facilitating tumor segmentation. The comparison with the ground truth and quantitative assessment using Hausdorff distance, dice, and PPV metrics validate the superior performance of OPTGCE. With the availability of more data, goal-oriented contrast enhancement can be implemented using deep neural networks to facilitate tumor segmentation in different organs.