Chair of
Multimedia Communications and Signal Processing
Prof. Dr.-Ing. André Kaup
The content of the English page is outdated, please use the updated German version of our page at the moment: lms.tf.fau.de

Error Concealment of Image Data

Field of activity: Video Signal Processing and Transmission
Research topic: Video Coding and Transmission
Staff: Dr.-Ing. Jürgen Seiler
M.Sc. Nils Genser

If images or video sequences are transmitted over wireless channels or the internet, the risk of transmission errors is ubiquitous. This results in the problem that individual regions cannot be decoded and displayed correctly. But it is possible to estimate these lost areas from the other correctly received regions. To achieve this, we developed the Selective Extrapolation. This algorithm is able to reconstruct arbitrary image contents and can be applied to images as well as video sequences.

For estimating the lost signal parts, Selective Extrapolation generates a model of signal, defined over the complete considere area. For this, the model results from an iterative superposition of orthogonal basis functions. Prior to the model generation, a set of basis functions has to be selected. During our research we discovered, that the functions of the discrete Fourier-transform form a good basis function set. In this case, the algorithm is called Frequency Selective Extrapolation (FSE). In general, other basis function sets can be used as well. For generating the model, the known signal is projected onto all basis functions in every iteration. Resulting from the projections, one basis function is selected to be added to the model. This is the one, that maximizes the decrement of the approximation error energy. These steps are repeated for a predefined number of times. As the basis functions cover the known as well as unknown areas, the resulting model continues the signal into the distorted areas.

For a detailed description of Selective Extrapolation, please refer to 2008-10 and 2005-20.


Examples for concealment of distorted image data

Original image
Distorted image
Concealed image
Lena original image Lena distorted image Lena concealed image
Peppers original image Peppers distorted image Peppers concealed image


To demonstrate the abilies of Selective Extrapolation we provide a MATLAB-Example for error concealment by 2D Selective Extrapolation. This program is free of charge for scientific and personal use. Any commercial use is prohibited.

Source code (frequency_selective_extrapolation.m)
Example image (lena_y.tif)


Error concealment of corrupted video data

If video sequences have to be concealed instead of imaged, correctly received previous frames can be used for model generation in addition to the correctly received regions from the actual frame. With that one obtains a three-dimensional data volume. For this a model is generated using three-dimensional basis functions. As the model generation uses correctly received regions from the actual frame as well as from previous frames, a very high quality of the concealed sequence can be achieved. In the case that Fourier basis functions are used for model generation, the algorithm is called 3D Frequency Selective Extrapolation (3D-FSE). A detailed description of 3D-FSE can be found in 2007-30.

The concealment quality can be further improved if the motion of the sequence is compensated prior to the model generation. In doing so, the different layers of the volume sonsist similar image contents which leads to a more precise model generation. This extension is called Motion Compensated Frequency Seletive Extrapolation (MC-FSE). For a detailed description of Selective Extrapolation, please refer to 2008-25.

Below, some example sequenzes to prove the abilities of 3D-FSE and MC-FSE:

 
Sequence "Discovery City"
  
Sequence "Discovery Orient"
 
 
AVI
YUV
 
AVI
YUV
Original Discovery City original Discovery City original   Discovery Orient original Discovery Orient original
Distorted Discovery City distorted Discovery City distorted  Discovery Orient distorted Discovery Orient distorted
Concealed by 3D-FSE Discovery City concealed by 3D-FSE Discovery City concealed by 3D-FSE  Discovery Orient concealed by 3D-FSE Discovery Orient concealed by 3D-FSE
Concealed by MC-FSE Discovery City concealed by MC-FSE Discovery City concealed by MC-FSE  Discovery Orient concealed by MC-FSE Discovery Orient concealed by MC-FSE




Image restauration by Selective Extrapolation

Besides concealment of distortions resulting from transmission errors, Selective Extrapolation can also be used for image restauration. In doing so, defects or disturbing objects can be removed from images. To achieve this, the regions to be extrapolated are marked manually in a first step. For this a binary mask with zero at the regions to replace is generated. Then, the image is divided into blocks and the 2D Selective Extrapolation is applied to all blocks that contain regions to be extrapolated.

Two examples for image restauration by Selective Extrapolation:

Beach 
 Chicago



We provide program fse_restauration which was used for creating these images free of charge for scientific and personal use. Any commercial use is prohibited. For compiling the source code, openmp and fftw is necessary. In addition to the command line program we provide a small graphical user interface. This can be used to adjust the individual parameters more easily and the GUI further supports more file formats as the command line program only supports ppm-files. The program was developed and tested using OpenSUSE11. The usage of the program is on your own risk.

Source code (fse_restauration.c)
Binary for OpenSUSE 11 (fse_restauration)
GUI (fse_restauration_gui.py)


Brief manunal for FSE-Restauration

In addition to the image to be processed, the program needs a mask that indicates all samples to be extrapolated. This mask can be created with an arbitrary image processing program. The image and the mask e.g. look like this:

Beispielbild Wall Beispielmaske Wall 

 

These two images are selected either using the command line program or the GUI. Further an output-file has to be defined. For the extrapolation, additional parameters can be adjusted. These parameters are:


All parameters can also be adjusted by using fse_restauration_gui.

fse_restauration_gui


For a detailed discussion of the individual parameters, please refer to the corresponding publications on Selective Extrapolation. For the example above we can obtain the following image:

Beispielbild WallThe parameters for this image are:
    25000 iterations
    FFT size 128x128
    Block size 32x32 pixel
    Support area 32 pixel wide
    Decay factor 0.9
    Orthogonality deficiency compensation 0.8
    Already concealed areas weighted by 0.4

Publications

2016-45
CRIS
J. Seiler, W. Schnurrer, S. Schöll, A. Kaup
   [bib]

Optimized Processing Order for 3D Hole Filling in Video Sequences Using Frequency Selective Extrapolation
accepted for Picture Coding Symposium (PCS), Nuremberg, Germany, Dec. 2016
2015-34
CRIS
J. Seiler, M. Jonscher, M. Schöberl, A. Kaup
   [link]   [doi]   [bib]

Resampling Images to a Regular Grid from a Non-Regular Subset of Pixel Positions Using Frequency Selective Reconstruction
IEEE Transactions on Image Processing (IEEE TIP) Vol. 24, Num. 11, Pages: 4540-4555, Nov. 2015
2014-25
CRIS
J. Koloda, J. Seiler, A. Kaup, V. Sánchez, A.M. Peinado
   [doi]   [bib]

An error-based recursive filling ordering for image error concealment
accepted for IEEE Int. Conf. on Image Processing (ICIP), Pages: 2517-2521, Paris, France, Oct. 2014
2014-10
CRIS
J. Koloda, J. Seiler, A. Kaup, V. Sánchez, A.M. Peinado
   [doi]   [bib]

Frequency Selective Extrapolation with Residual Filtering for Image Error Concealment
IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Pages: 1976-1980, Florence, Italy, May 2014
2013-21
CRIS
J. Seiler, M. Schöberl, A. Kaup
   [bib]

Spatio-Temporal Error Concealment in Video by Denoised Temporal Extrapolation Refinement
IEEE Int. Conf. on Image Processing (ICIP), Melbourne, Australia, Sep. 2013
2013-18
CRIS
W. Schnurrer, J. Seiler, A. Kaup
   [link]   [doi]   [bib]

Improving Block-Based Compensated Wavelet Lifting by Reconstructing Unconnected Pixels
International Symposium on Signals, Circuits and Systems (ISSCS), Pages: 1-4, Iasi, Romania, Jul. 2013
2011-63
CRIS
J. Seiler
   [link]   [bib]

Signal Extrapolation Using Sparse Representations and its Applications in Video Communication
Verlag Dr. Hut, München, Dec. 2011
2011-43
CRIS
J. Seiler, A. Kaup
   [bib]

Optimized and Parallelized Processing Order for Improved Frequency Selective Signal Extrapolation
European Signal Processing Conf. (EUSIPCO), Pages: 269-273, Barcelona, Spain, Aug. 2011
2011-14
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

A Fast Algorithm for Selective Signal Extrapolation with Arbitrary Basis Functions
EURASIP Journal on Advances in Signal Processing (JASP) Vol. 2011, 2011
2011-4
CRIS
J. Seiler, A. Kaup
   [link]   [doi]   [bib]

Motion Compensated Three-Dimensional Frequency Selective Extrapolation for Improved Error Concealment in Video Communication
Journal of Visual Communication and Image Representation (VCIR) Vol. 22, Pages: 213-225, Mar. 2011
2010-56
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Complex-Valued Frequency Selective Extrapolation for Fast Image and Video Signal Extrapolation
IEEE Signal Processing Letters (IEEE SPL) Vol. 17, Num. 11, Pages: 949 - 952, Nov. 2010
2010-25
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Content-Adaptive Motion Compensated Frequency Selective Extrapolation for Error Concealment in Video Communication
IEEE International Conference on Image Processing (ICIP 2010), Pages: 469-472, Hong Kong, China, Sep. 2010
2009-29
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Multiple Selection Extrapolation for Spatial Error Concealment
IEEE International Workshop on Multimedia Signal Processing (MMSP), Rio de Janeiro, Brazil, Oct. 2009
2008-27
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Adaptive Joint Spatio-Temporal Error Concealment for Video Communication
IEEE International Workshop on Multimedia Signal Processing (MMSP 2008), Pages: 229-234, Cairns, Australia, Oct. 2008
2008-25
CRIS
J. Seiler, A. Kaup
   [link]   [bib]

Motion Compensated Frequency Selective Extrapolation for Error Concealment in Video Coding
16th European Signal Processing Conference (EUSIPCO), Lausanne, Switzerland, Aug. 2008
2008-10
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Fast Orthogonality Deficiency Compensation for Improved Frequency Selective Image Extrapolation
IEEE International Conference on Acoustics, Speech, and Signal Processing, Pages: 781-784, Las Vegas, Nevada, Mar. 2008
2008-3
CRIS
K. Meisinger
   [bib]

Selective Signal Extrapolation and its Application in Image and Video Communications, PhD thesis
Friedrich-Alexander-Universität Erlangen-Nürnberg, University Erlangen-Nuremberg, Germany, Jan. 2008
2007-51 A. Kaup, K. Meisinger
   [bib]

3D Signalextrapolation
DE 10 2007 002 545 A1, Jul. 2007
2007-35
CRIS
J. Seiler, K. Meisinger, A. Kaup
   [link]   [bib]

Orthogonality Deficiency Compensation for Improved Frequency Selective Image Extrapolation
Picture Coding Symposium, Lisbon, Portugal, Nov. 2007
2007-30
CRIS
K. Meisinger, A. Kaup
   [doi]   [bib]

Spatiotemporal Selective Extrapolation for 3-D Signals and its Applications in Video Communications
IEEE Transactions on Image Processing (IEEE TIP) Vol. 16, Num. 9, Pages: 2348-2360, Sep. 2007
2007-29
CRIS
K. Meisinger, T. Aach, A. Kaup
   [doi]   [bib]

Spatio-Temporal Defect Pixel Interpolation Using 3-D Frequency Selective Extrapolation
IEEE International Conference on Image Processing, Vol. IV, Pages: 149-152, San Antonio, Texas, Sep. 2007
2006-36
CRIS
K. Meisinger, S. Martin, A. Kaup
   [doi]   [bib]

Spatio-Temporal Concealment in H.264/AVC Video Coding by 3-D Selective Extrapolation
8th Int. Workshop on Multimedia Signal Processing (MMSP), Pages: 291-295, Victoria, BC, Canada, Oct. 2006
2006-25
CRIS
K. Meisinger, A. Kaup
   [doi]   [bib]

2D Frequency Selective Extrapolation for Spatial Error Concealment in H.264/AVC Video Coding
IEEE International Conference on Image Processing (ICIP), Vol. -, Pages: 2233-2236, Atlanta, Oct. 2006
2006-24
CRIS
K. Meisinger, S. Martin, A. Kaup
   [link]   [bib]

Spatio-Temporal Selective Extrapolation for 3-D Signals Applied to Concealment in Video Communications
14th European Signal Processing Conference (EUSIPCO), Sep. 2006
2005-35
CRIS
K. Meisinger, T. Tröger, M. Zeller, A. Kaup
   [bib]

Automatische TV-Logo-Entfernung durch statistische modellbasierte Detektion und frequenzselektive Kompensation
Elektronische Medien: 11. Dortmunder Fernsehseminar, ITG-Fachbericht 188, Pages: 211-214, Dortmund, Deutschland, Sep. 2005
2005-32
CRIS
K. Meisinger, T. Tröger, M. Zeller, A. Kaup
   [link]   [bib]

Automatic TV logo removal using statistical based logo detection and frequency selective inpainting
13th European Signal Processing Conference (EUSIPCO), Antalya, Turkey, Sep. 2005
2005-20
CRIS
A. Kaup, K. Meisinger, T. Aach
   [doi]   [bib]

Frequency Selective Signal Extrapolation with Applications to Error Concealment in Image Communication
AEÜ - International Journal of Electronics and Communication Vol. 59, Num. 3, Pages: 147-156, 2005
2004-26
CRIS
K. Meisinger, J. Garbas, A. Kaup
   [bib]

Error Control and Concealment of JPEG2000 Coded Image Data in Error Prone Environments
Picture Coding Symposium (PCS), San Francisco, Dec. 2004
2004-13
CRIS
K. Meisinger, A. Kaup
   [doi]   [bib]

Minimizing a Weighted Error Criterion for Spatial Error Concealment of Missing Image Data
IEEE International Conference on Image Processing (ICIP), Pages: 813-816, Singapore, Oct. 2004
2004-8
CRIS
K. Meisinger, A. Kaup
   [doi]   [bib]

Spatial Error Concealment of Corrupted Image Data using Frequency Selective Extrapolation
Intern. Conf. on Acoustics, Speech and Signal Processing, Vol. III, Pages: 209-212, Montreal, Canada, May 2004
2003-22
CRIS
K. Meisinger, A. Kaup
   [bib]

Örtliche Fehlerverschleierung von gestört empfangenen Bilddaten durch frequenzselektive Extrapolation
Elektronische Medien: 10. Dortmunder Fernsehseminar, ITG-Fachbericht 179, Pages: 189-194, Sep. 2003