Literature Survey on Super Resolution and its Challenges Murali Krishna AtmakuriAsst.
Prof,Dept. of ECERVR&JCCE,[email protected] Kumar KattaAsst. Prof,Dept. of ECERVR&JCCE,Gunturanilecedept@gmail.
comDr.V.V.K.D.
V Prasadprofessor&Head, Dept. of ECEGEC,[email protected] In the recent trends, image processing field became an interesting area for researchers due to vast advancements over the few decades. Now-a-days one of the most recent and important trend of image processing is Super Resolution. Super resolution utilizes the features of reconstruction; reconstruction is a kind of producing high spatial image from one or more low resolution images. Super resolution combines the non-redundant information of low-resolution images to develop high-resolution images. This article envisages the recent advances in super-resolution techniques and provides its advantages and disadvantages.
This article also explains the challenges of super resolution and the scope further research studies.Index Terms- Image Resolution, Super Resolution, Interpolation, Wavelet Transform, Learning, Reconstruction.INTRODUCTIONVision is one of the most valuable of five senses, so images not only play the important role but also used to make decisions based on human perception. In order to improve human perception, it is important to use high resolution images 1-6. High resolution images are used in various kinds of applications, some of the examples are as follows: Military and Civilian 7-10.
The recent trends in image and video sensing have been intensified by the expectations of the user on visual quality of captured-data 11, 12. High quality visual captured-data can be obtained with the help of high resolution cameras. The limitation of using high resolution cameras are: Expensive, Need high power, Need high memory size and limited band width even though the high resolution cameras used, sometimes it is not possible to obtain high resolution image 13. To overcome above limitations, super resolution would be an effective solution.
In many of the digital image processing applications, high resolution images are used. High image resolution gives more details about the image 14-18. Super resolution became an interesting area for researcher since more the resolution gives more data about image 19-22. Resolution is generally determined by pixel density 23.Figure 1 presents the generic image acquisition process, where diverse factors affect the image quality like: Over the air (OTA), Charge-couple device (CCD), Pre-processors and Environment. Optical Blur is a non-symmetric design of the lens and an aperture before or behind the optic center of the lens lead to image distortions. Motion blur results either due to rapid movement or long exposure of camera.
Noise in an image is an undesirable by-product which corrupts the original information.One way of producing a high resolution (HR) image, is by installing a high resolution sensor. But it is not practicable to do so. It results in increase of a cost as well as increase in power consumption. A simple example of this is satellite imaging system or a medical imaging system, where it is infeasible to use a high resolution sensor.
So, to come over these drawbacks, post processing is required to develop a better resolved image that holds more information. One of the promising approaches for this is signal processing techniques to obtain HR image from multiple low resolution (LR) images. Now-a-days such approach is more active in research area, and is called super resolution or Resolution Enhancement 21.
IMAGE sUPER RESOLUTIONSuper resolution can be achieved either by processing multiple low resolved images as input and generating a high detail containing a single super resolved image as output or enhancing the details in a single low resolved image and generating a high resolved image for analysis. In SR from multiple LR images, it is a construction of HR image from several LR images, thereby increasing the high frequency components. The basic idea is to combine non-repetitive information contained by multiple LR images. Fig. 1. Generic image acquisition systemRestoration for noise and blur removalRegistration or Motion estimationInterpolation onto a high resolution gridInput: LR IMAGES Output: SR IMAGES PAGE * MERGEFORMAT 2Fig.
2. Basic super resolution reconstruction stagesThe main advantage of SR approach is that, a HR image can be obtained even with the existing LR imaging with lower cost and less power consumption.Usually, a super resolution method consists of the following basic processing steps: (1) Registration, (2) Interpolation and (3) De-blurring or noise removal.Image registration is the process of overlapping more than one images of the same scene which has been taken from different angles by the sensors. In registration two or more images are align geometrically to obtain the information through image fusion or change detection.Interpolation is a process of estimating the intermediate pixels between the pixel values.
When any image is converted from LR to HR, intermediate gaps are introduced and these values have to be estimated and filled with interpolation process.As the process of interpolation introduces some artifacts the resultant image will be blurred or noisy. Through different filters and techniques noise will be removed and finally a super resolved image is generated.Approaches to Super resolutionSuper-resolution techniques can be classified as (1) Frequency domain approach and (2) Spatial domain approach. Frequency domain approachTransform the LR image into frequency domain by applying Discrete Fourier Transform (DFT) and combine them according to the relationship between the aliased DFT coefficients of the observed LR images and that of the unknown high-resolution image.
The combined data are then transformed back to the spatial domain where the new image could have a higher resolution than that of the input images 3. The principle of frequency domain approach is follows: i) what is the shift property of the Fourier transform, ii) The aliasing between the continuous Fourier transform (CFT) of an original HR image and the discrete Fourier transform (DFT) of combined LR images, iii) the presumption that an original HR image is band limited. Frequency domain approach has some advantages to enhance the details by extrapolating the high frequency information presented in LR images, and it has lesser computational complexity. But the drawback is that it is incapable of handling the real-world applications.Spatial Domain ApproachThe frequency domain approach has few disadvantages like it limits the inter-frame motion to be translational, Also it is very difficult in frequency domain to use the prior knowledge. As the main problem is ill-posed image in SR, prior knowledge is required to overcome this.
The main benefit of spatial domain is the support for unbind motion between frames and prior knowledge availability for solving the problems. Some of the methods are interpolation, iterative back projection and projection onto convex.3.2.1. InterpolationInterpolation is the process of transferring image from one resolution to another without losing image quality. Image interpolation can be used in several image processing applicatios for image zooming, enhancement of image, resizing and many more.
Most common interpolation techniques are nearest neighbor, bilinear and cubic convolution. An image is a two dimensional signal represented as brightness vs spatial coordinates. An analog image can be transformed into digital domain by sampling and quantization process. The basic element of an image is a pixel. When we increase the resolution of image from low to high, it is called up-sampling or up-scaling while reverse is called down sampling or down scaling.
Interpolation is of three types: (i) Bi-linear Interpolation: Bi-linear interpolated point is filled with weighted average of it’s four closest pixel’s. Bi-linear interpolation is recommended for continuous data like elevation and raw slope values. (ii) Bi-cubic Interpolation: Bi-cubic interpolation is recommended for smoothing continuous data, but this incurs a processing performance overhead and (iii) Nearest Neighbor Interpolation: In this method, nearest value is copied for interpolation and this technique has less computational complexity. Nearest neighbor interpolation is recommended for categorical data such as land use classification.3.2.
2. Iterative Back Projection (IBP)In IBP approach, HR image is estimated by back projecting the difference between the simulated LR image and captured LR on interpolated image. This iterative process of SR does iterations until the minimization of the cost function is achieved.3.2.3. Classical Multi-Image Super ResolutionIn the classical multi-image SR, a set of LR images of the same scene is taken. If enough LR images were available then the equation is determined and a SR image is reconstructed.
The assumption here is that the two or more LR images should contain distinguishable features. This approach would give poor results, if distinguishable features in LR images are less.3.2.
4. Example Based Super ResolutionIn Example- Based approach, the same rule is applied. This approach is useful when only single LR image is available. In this approach, the image has small patches that redundantly reappear, both within the scale as well as across the scale.
Each LR patch in an image is replaced by its corresponding HR patch to generate the SR image. Here assumption is that, the image should have enough HR patches for the correspondence LR patches.3.2.5. Learning Based Super ResolutionIt is a concept of machine learning, where the machine is trained to classify LR images and its corresponding HR patches. In this approach, both LR and HR patches are divided into different classes. Hence, the number of comparisons will be decreased, as it has to compare LR with only HR patches.
For an edge-area of the LR image, routine example-based image SR algorithm can be preferred to implement the local and fine SR. For the flat regions of the low-resolution, only interpolation algorithm can be applied for super-resolution. The performance of learning based super-resolution depends on HR patches retrieved from the training data of an input LR patch.Table 1. Comparison among various super resolution approaches 21.
Categorization Description DisadvantagesInterpolation Based Different interpolation techniques can be used Over-smooth jagged artifactsReconstruction Based Reconstruction constraint and image prior Ringing artifacts, imposing additional priorLearning Based Learning high frequency details from the trained data. High frequency artifacts.Challenging issues of Super ResolutionIn practice developing super resolution image, there are many challenging issues. Some of the challenging issues are mentioned below: Image RegistrationImage registration is a common problem because of ill-posed images. Image registration becomes more and more difficult when the input LR image is having very high aliasing effects. The registration error increases with decrease in the resolution of input images. These registration error affects the quality of an image resolution more than that of interpolation 21.
Computational EfficiencyReal time application is always requires good efficiency. As there are large numbers of unknowns in reconstructing super resolution images, computational complexity of matrix increases.RobustnessSuper resolution techniques are defense-less to motion errors, inaccurate blur models, noise, moving objects, motion blur etc. These effects are not easy to estimate which are not acceptable in many applications.ConclusionThis paper provides literature review about various techniques used to achieve super resolution image with the help of single image or multiple low resolution images.
In this paper, interpolation based, reconstruction based and learning based techniques for super resolution are studied. The future scope of the super resolution is to develop new methods by extending or integrating the existing methods to address their challenges. Finally images with super resolution can provide more details as compared to low resolution images with low cost.ReferencesM. S. Alam, J. G. Bognar, R.
C. Hardie, and B. J.
Yasuda. Infrared im-age registration and high-resolution reconstruction using multiple translationally shifted aliased video frames. IEEE Transactions on Instrumentation and Measurement, 49(5):915-923, 2000.S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1167-1183, 2002.
O. Bowen and C. S. Bouganis.
Real-time image super resolution using an FPGA. In International Conference on Field Programmable Logic and Applications, pages 89-94, 2008.A. Buades, B. Coll, and J. M. More.
A non-local algorithm for image denoising. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 60-65, 2005.J. Chung, E. Haber, and J. Nagy. Numerical methods for coupled super-resolution. Inverse Problems, 22(4):1261-1272, 2006.
D. Datsenko and M. Elad. Example-based single document image super-resolution: a global MAP approach with outlier rejection.
Multidimen-sional System and Signal Processing, 18(2-3):103-121, 2007.D. L.
Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):1289-1306, 2006.D.
L. Donoho. For most large underdetermined systems of linear equations, the minimal `1-norm near-solution approximates the spars-est near-solution.
Communications on Pure and Applied Mathematics, 59(7):907-934, 2006.X. Jia F. Li and D. Fraser.
Universal HMT based super resolution for remote sensing images. In IEEE International Conference on Image Processing, pages 333-336, 2008.S. Farsiu, D. Robinson, M. Elad, and P.
Milanfar. Fast and robust multi-frame super-resolution. IEEE Transaction on Image Processing, 13(10):1327-1344, 2004.P. H. Hennings-Yeomans, S.
Baker, and B. V. K. V. Kumar. Simul-taneous super-resolution and feature extraction for recognition of low-resolution faces.
In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1-8, 2008.K. Jia and S. Gong.
Generalized face super-resolution. IEEE Transactions on Image Processing, 17(6):873-886, 2008.A. Levin, Y. Weiss, F. Durand, and W. Freeman. Understanding and evaluating blind deconvolution algorithms.
In Proceedings of IEEE Com-puter Society Conference on Computer Vision and Pattern Recognition, pages 1964-1971, 2009.Frank Lin, Clinton B. Fookes, VinodChandran, and SridhaSridharan. Investigation into optical low super-resolution for surveillance applica-tions. In The Austrilian Pattern Recognition Society Worshop on Digital Image Computing, 2005.K.
Malczewski and R. Stasinski. Toeplitz-based iterative image fusion scheme for MRI. In IEEE International Conference on Image Processing, pages 341-344, 2008.B.
Narayanan, R. C. Hardie, K. E. Barner, and M. Shao.
A computa-tionally efficient super-resolution algorithm for video processing using partition filters. IEEE Transactions on Circuits and Systems for Video Technology, 17(5):621-634, 2007.N. Nguyen and P.
Milanfar. An efficient wavelet-based algorithm for image super-resolution. In Proceedings of International Conference on Image Processing, volume 2, pages 351-354, 2000.N.
Nguyen, P. Milanfar, and G. H. Golub. A computationally e±cient image superresolution algorithm. IEEE Transactions on Image Process-ing, 10(5):573-583, 2001.M. Protter and M.
Elad. Super resolution with probabilistic motion estimation. IEEE Transactions on Image Processing, 18(8):1899-1904, 2009.P. Hardeep, B. Prashant, Swadas, M. Joshi, A Survey on Techniques and Challenges in Image Super Resolution Reconstruction.
International Journal of Computer Science and Mobile Computing, 2013.T. sai and Huang. A survey on super-resolution imaging. Springer, 2010.R. SudheerBabu, K.
E. S. Murthy. A Survey on the Methods of Super-Resolution Image Reconstruction. International Journal of Computer Applications.
15(2), 2011. A. J. Shah, B. S. Gupta. Image Super Resolution – A Survey.
IEEE, 2012.
