Higher Order SVD Analysis for Dynamic Texture Synthesis

Roberto Costantini, Luciano Sbaiz, Sabine Süsstrunk

Abstract

Videos representing flames, water, smoke, etc. are often defined as dynamic textures: "textures" because they are characterized by redundant repetition of a pattern and "dynamic" because this repetition is also in time and not only in space. Dynamic textures have been modeled as linear dynamic systems by unfolding the video frames into column vectors and describing their trajectory as time evolves. After the projection of the vectors onto a lower dimensional space by a Singular Value Decomposition (SVD), the trajectory is modeled using system identification techniques. Synthesis is obtained by driving the system with random noise. In this paper, we show that the standard SVD can be replaced by a Higher Order SVD (HOSVD), originally known as Tucker decomposition. HOSVD decomposes the dynamic texture as a multidimensional signal (tensor) without unfolding the video frames on column vectors. This is a more natural and flexible decomposition, since it permits to perform dimension reduction in spatial, temporal, and chromatic domain, while standard SVD allows for temporal reduction only. We show that for a comparable synthesis quality, the HOSVD approach requires on average five times less parameters than the standard SVD approach. The analysis part is more expensive, but the synthesis has the same cost as existing algorithms. Our technique is thus well suited to dynamic texture synthesis on devices limited by memory and computational power, such as PDAs or mobile phones.

Reference

Code and Data

Here you can find the Matlab code and data to reproduce all images and tables from the paper. If you use this code in your research and publications, please also put a reference to this paper. Thank you!
Matlab code + Data (47 Mb)

Tested Configurations

6.5.1.199709 (R13) Service Pack 1 (Windows)

Comments and remarks

roberto.costantini@epfl.ch

Videos

Clip	Thumbnail	Synthesis Result
Flame		3D-YCbCr SVD (2.5x) 3D-YCbCr Tucker 4D-RGB SVD (3.2x) 4D-RGB Tucker
Pond		3D-YCbCr SVD (3.9x) 3D-YCbCr Tucker 4D-RGB SVD (3.3x) 4D-RGB Tucker
River		3D-YCbCr SVD (6.9x) 3D-YCbCr Tucker 4D-RGB SVD (5.2x) 4D-RGB Tucker
Grass		3D-YCbCr SVD (5.9x) 3D-YCbCr Tucker 4D-RGB SVD (4.8x) 4D-RGB Tucker
Sparkle		3D-YCbCr SVD (3.3x) 3D-YCbCr Tucker 4D-RGB SVD (2.6x) 4D-RGB Tucker
Waterfall 1		3D-YCbCr SVD (4.4x) 3D-YCbCr Tucker 4D-RGB SVD (3.9x) 4D-RGB Tucker
Waterfall 2		3D-YCbCr SVD (3.9x) 3D-YCbCr Tucker 4D-RGB SVD (1.3x) 4D-RGB Tucker

Acknowledgements

We have used some of the videos available at http://www.cc.gatech.edu/cpl/projects/graphcuttextures/. Here, we thank the authors of the paper

Vivek Kwatra , Arno Schödl , Irfan Essa , Greg Turk and Aaron Bobick, "Graphcut Textures: Image and Video Synthesis Using Graph Cuts", SIGGRAPH 2003, pp. 277-286

for maintaining this excellent webpage that permits to compare different methods for texture synthesis. Moreover, we would like to thank the laboratories and all the researchers cited in the credits for these videos.

This project is supported by the Swiss National Science Foundation (SNSF) under grant number 21-067012.01. The authors
thank Cristian Carmeli and Joos Vandewalle for fruitful discussions on the use of tensor decomposition.