Information theory is an outstanding framework to measure uncertainty, dependence and relevance in data and systems. It has several desirable properties for real world applications: it naturally deals with multivariate data, it can handle heterogeneous data types, and the measures can be interpreted in physical units. However, it has not been adopted by a wider audience because obtaining information from multidimensional data is a challenging problem due to the curse of dimensionality. Here we propose an indirect way of computing information based on a multivariate Gaussianization transform. Our proposal mitigates the difficulty of multivariate density estimation by reducing it to a composition of tractable (marginal) operations and simple linear transformations, which can be interpreted as a particular deep neural network. We introduce specific Gaussianization-based methodologies to estimate total correlation, entropy, mutual information and Kullback-Leibler divergence. We compare them to recent estimators showing the accuracy on synthetic data generated from different multivariate distributions. We made the tools and datasets publicly available to provide a test-bed to analyze future methodologies. Results show that our proposal is superior to previous estimators particularly in high-dimensional scenarios; and that it leads to interesting insights in neuroscience, geoscience, computer vision, and machine learning.

Software

RBIG Python toolbox

Includes tools to compute Information Theory measures used in the current paper [RBIG4IT2020]

Demo in Google Colab

This demo provides an interactive example of how to use the RBIG Python toolbox to compute information measures like entropy and mutual information. It’s an easy-to-follow resource for those interested in testing the methods on their data.

RBIG Matlab toolbox

Includes tools to compute Information Theory measures and scripts to generate the synthetic data for the experiments in the current paper [RBIG4IT2020]

Paper Summary

The measures that can be computed using RBIG defined in this paper are the ones in the following figure + the Kulback-Leibler divergence. The main point is that RBIG allows to get acurated estimations of these measures even in multidimensional datasets.

Extended results

Here extra results for the paper are shown. Mainly figures for results on synthetic data that would taken too much space in the original paper.

Total Correlation

The total correlation is a measure of multivariate dependence among several variables. In this section, we show how the RBIG methodology allows for precise estimation of total correlation, even for datasets with non-Gaussian distributions or high dimensionality.

Entropy

Entropy is a fundamental concept in information theory, representing the amount of uncertainty in a dataset. Using RBIG, we show that entropy can be computed more efficiently compared to classical estimators, particularly in datasets with complex dependencies.

KLD

Kullback-Leibler divergence measures how one probability distribution diverges from a second, reference distribution. RBIG offers a more accurate way to compute KLD in high-dimensional settings, improving performance in tasks like anomaly detection and model comparison.

Mutual Information

Mutual information quantifies the amount of information obtained about one random variable through another. The RBIG methodology demonstrates superior performance in estimating mutual information, especially in datasets with nonlinear dependencies, making it a valuable tool for feature selection and data analysis.

The image representations along the retina-cortex pathway are analyzed in terms of their ability to capture information about the visual scenes. The considered series of representations includes: (1) the LMS retinal images, (2) their von-Kries adapted version, (3) the opponent images at LGN, (4) their nonlinear version after Weber-like saturation, (5) the cortical local-frequency representation filtered by achromatic and chromatic CSFs, and (6) the cortical representation after divisive normalization.

Assuming a single-step transform from the retinal input to each of these representations, and sensors of the same Signal-to-Noise quality in each representation, our estimations of transmitted information show that: (a) progressively deeper representations are better in terms of the amount of captured information, (b) the transmitted information up to the cortical representation follows the probability of natural scenes over the chromatic and achromatic dimensions of the stimulus space, (c) the contribution of spatial transforms to capture visual information is substantially greater (67%) than the contribution of chromatic transforms (33%), and (d) nonlinearities of the responses contribute substantially to the transmitted information (about 28%) but less than the linear transforms (72%).

The parameters of the model have psychophysical origin and they were not statistically optimized in any way. The information estimates were computed using our Gaussianization transform (Laparra et al. IEEE TNN 11) over our database of colorimetrically-calibrated images (Laparra et al. Neur.Comp. 12), but equivalent results are obtained using other estimates (e.g. offset corrected Kozachenko-Leonenko [Marin et al. IEEE PAMI 13]) or other databases (e.g. Foster et al [Vis.Res.15,16]).

The Matlab toolbox for linear spatio-temporal Vision Models

VistaLab is a Matlab toolbox that provides the linear building-blocks to create spatio-temporal vision models and the tools to control the spatio-temporal properties of video sequences. These building blocks include the spatio-temporal receptive fields of LGN, V1, and MT cells, and the spatial and spatio-temporal Contrast Sensitivity Functions (CSFs). Additionally, VistaLab allows accurate spatio-temporal sampling, spatio-temporal Fourier domain visualization, and generation of video sequences with controlled texture and speed. Tools for video sequence generation include noise, random dots, and rigid-body animations with Lambertian reflectance.

The perception and video synthesis tools enable accurate illustrations of the visibility of achromatic spatio-temporal patterns. Linear filters in VistaLab provide rough approximations of pattern visibility, which can be enhanced with non-linear models available in related toolboxes.

The standard tools in VistaLab (and ColorLab) are essential for building more sophisticated vision models, available on the VistaModels dedicated site.

Table of Contents

• Retina and Lateral Geniculate Nucleus (LGN)
• Primary Visual Cortex (V1)
• Middle Temporal (MT) region
• Spatio-temporal Contrast Sensitivities
• Controlled spatio-temporal stimuli
• Extensions of VistaLab

Retina and Lateral Geniculate Nucleus (LGN)

Most of the Retinal Ganglion Cells and cells in the LGN can be modelled with center-surround receptive fields with monophasic or biphasic temporal response. VistaLab comes with a configurable implemenation of such receptive fields according to the general expressions in [Cai, Freeman, DeAngelis, J. Neurophysiol. 97]. Using these units it is easy to generate artificial retinas with arbitrary sampling [Martinez-Garcia et al. 16, Martinez-Garcia et al. 17].

The examples below show (a) the receptive field of a representative neuron in the spatiotemporal and in the 3D Fourier domain, and (b) the response of a population of such neurons to a natural movie assuming uniform retinal sampling and spatial invariance of the receptive field. VistaLab allows explicit implementation of each sensor response using the scalar product by the corresponding receptive field to get rid of the uniform sampling and the convolution assumptions.

Primary Visual Cortex (V1)

Simple cells in the V1 cortex can be modelled with Gabor-like receptive fields tuned to certain spatial and temporal frequencies. VistaLab comes with a configurable implemenation of such receptive fields according to the general expressions in [Daugman JOSA A 89, Simoncelli & Heeger Vis. Res. 98]. Using these units it is easy to generate artificial cortex with arbitrary sampling [Martinez-Garcia et al. 17].

The examples below show six representative neurons tuned to the same spatial frequencies (7 cpd) but different temporal frequencies 2, 7, and 10 Hz, both positive and negative. Eventhough there is no conclusive tuning to two-dimensional speed due to the aperture problem [Heeger JOSA 87], in the direction perpendicular to the grating, these are tuned to 0.3, 1 and 1.5 degrees/sec respectively (both positive and negative). Figures show: (a) the receptive fields in the spatiotemporal and in the 3D Fourier domain, and (b) the response of a population of such neurons to a natural movie assuming uniform retinal sampling and spatial invariance of the receptive field. VistaLab allows explicit implementation of each sensor response using the scalar product by the corresponding receptive field to get rid of the uniform sampling and the convolution assumptions.

Middle Temporal (MT) region

Cells in the MT cortex receive projections from V1 cells aligned in a plane in the spatio-temporal Fourier domain. Therefore, they are narrow-band in speed tuning. VistaLab comes with a configurable implemenation of such receptive fields according to the general expressions in [Simoncelli & Heeger Vis. Res. 98]. Using these units and a spatio-temporal window it is easy to generate artificial MT cortex with arbitrary sampling [Martinez-Garcia et al. 17].

The examples below shows six representative sets of neurons tuned to tuned to speeds of 0.3, 1 and 1.5 degrees/sec respectively (both positive and negative). In this case while Figures show: (a) the receptive fields in 3D Fourier domain, the kind of features these cells are optimally tuned to, and (b) the response of a population of such neurons to a natural movie assuming uniform retinal sampling and spatial invariance of the receptive field. VistaLab allows explicit implementation of each sensor response using the scalar product by the corresponding receptive field to get rid of the uniform sampling and the convolution assumptions.

Spatio-temporal Contrast Sensitivities

VistaLab comes with different Contrast Sensitivity Functions (CSFs): (a) the spatial-achromatic CSF from the OSA Standard Spatial Observer Watson & Malo IEEE ICIP 02, (b) the spatial-chromatic, Red-Green and Yellow-Blue CSFs of K. Mullen [Vis. Res. 85], with approrpiate scaling Gutierrez et al. 12, and (c) the achromatic spatio-temporal CSFs of D. Kelly [JOSA 79], and S. Daly (with object tracking speed compensation) [SPIE 98].

Controlled spatio-temporal stimuli

The movies below illustrate the abilities of VistaLab for accurate motion control.

• First row: includes sequences of the motion of a lambertian rigid body evolving in a gravitatory field with inelastic restrictions recorded from different points of view, this example allows arbitrary locations of the illumination and camera. In this case the actual motion in 3D world and the optical flow (motion in the retinal plane) are known.

• Second row: includes an example of random dots moving according to arbitrary optical flow fields.

• Third row: shows how static pictures can be animated using spatially uniform flows of arbitrary speed leading to interesting shape-from-motion effects in the case of noise patterns.

• Fourth row: shows different movies of the same periodic pattern moving at progressively increasing speeds. Aliasing introduces speed reversal at the expected place, as demonstrated by the Fourier diagrams below.

Extensions of VistaLab

VistaLab only addresses the linear part of the neural mechanisms that mediate the preattentive perception of spatio-temporal patterns. However, it doesnt combine these mechanisms to compute motion (optical flow), it doesnt include the nonlinear interactions between the linear mechanisms, and it doesnt include color.

These issues can be addressed with other toolboxes, namely VistaVideoCoding, BioMultiLayer_L_NL_color in VistaModels, and Colorlab.

Key Capabilities

• Spatio-temporal Modeling: Build models for LGN, V1, and MT neural responses.
• Contrast Sensitivity: Apply achromatic and chromatic CSFs to video and images.
• Video Synthesis: Create controlled video sequences with specific spatio-temporal properties.
• Fourier Domain Tools: Visualize spatio-temporal frequency response of neural models.

The Toolboxes in the VistaModels site are organized in three categories of different nature: (a) Empirical-mechanistic Models, tuned to reproduce basic phenomena of color and texture perception, (b) Principled Models, derived from information theoretic arguments, and (c) Engineering-motivated Models, developed to address applied problems in image and video processing.

The algorithms in VistaModels require the standard building blocks provided in the (more basic) toolboxes VistaLab and ColorLab. However, the necessary functions from these more basic toolboxes are included in the packages listed below for the user convenience.

Table of contents

• (A) Empirical-mechanistic Models
  • 1995 - 2008: Linear opponent color channels, local-DCT and Divisive Normalization
  • 2009 - 2010: Linear opponent color channels, Orthogonal Wavelet and Divisive Normalization
  • 2013 - 2018: Multi-Layer network with nonlinear opponent color, Overcomplete Wavelet and Divisive Normalization
  • 2019 - 2021: Convolutional and differentiable implementations
  • Psychophysical test-bed for model tuning and comparison
  • Model Comparison
• (B) Principled Models
  • Efficient coding in mechanistic models
  • Statistically-based linear receptive fields
  • Statistically-based nonlinearities
• (C) Engineering-motivated Models
  • Perceptually-weighted motion estimation: VistaVideoCoding
  • Image Coding: VistaCoRe
  • Image and Video Quality: VistaQualityTools

(A) Empirical-mechanistic Models

Cascades of linear transforms and nonlinear saturations are ubiquitous in neuroscience and artificial intelligence ever since the [McCulloch-Pitts model]. More recently this has been exemplified in subtractive and divisive models of cortical interaction [Wilson & Cowan, Kybernetik 73; Carandini and Heeger, Nature Rev. Neurosci. 12].

Over the years, we have developed progressively better versions of such cascades to be applicable to color images and video sequences. These parametric models were empirically tuned to give a rough description of different color and texture perception phenomena (see the psychophysical test-bed below for model tuning and comparison).

See a visual example of the effect of the local spatial-frequency transforms and the divisive normalization below (illustration of the 2018 model)

1995 - 2008: Linear opponent color channels, local-DCT and Divisive Normalization

This model is invertible and was originally tuned to reproduce contrast response curves obtained from contrast incremental thresholds [Pons PhD Thesis, 1997]. It was applied to reproduce subjective distortion opinion [Im.Vis.Comp.97, Displays 99] and to improve the perceptual quality of JPEG and MPEG through (a) transform coding of the achromatic channel [Elect.Lett.95, Elect.Lett.99, Im.Vis.Comp.00, IEEE TIP 01, Patt.Recog.03, IEEE TNN 05, IEEE TIP 06a, JMLR08], (b) the color channels [RPSP12], and (c) by improving the motion estimation [LNCS97, Elect.Lett.98, Elect.Lett.00a, Elect.Lett.00b, IEEE TIP 01].

• Download the Toolbox!: V1_model_DCT_DN_color.zip (74MB)

2009 - 2010: Linear opponent color channels, Orthogonal Wavelet and Divisive Normalization

Even though we developed our own Matlab code for some specific overcomplete wavelets in the mid 90’s [MSc Thesis 95, J.Mod.Opt.97], it took some time until we applied the Divisive Normalization interaction to Simoncelli’s wavelets in MatlabPyrTools (which are substantially more efficient). The model was fitted to reproduce subjective image distortion opinion [JOSA A 10] following exhaustive grid search as in [IEEE ICIP 02]. This model (which relies on the orthogonal wavelets of the MatlabPyrTools) was found to have excellent redundancy reduction properties [LNCS10, Neur.Comp.10].

• Download the Toolbox!: V1_model_wavelet_DN_color.zip (14MB)

2013 - 2018: Multi-Layer network with nonlinear opponent color, Overcomplete Wavelet and Divisive Normalization

Even though we developed a comprehensive color vision toolbox in the early 2000’s (see ColorLab ), it took some time until we included a fully adaptive chromatic front-end before the spatial processing models based on overcomplete wavelets. Note that the older toolboxes rely on (too rough) linear RGB to YUV transforms. This multi-layer model (or biologically-plausible deep network) performs the following chain of perceptually meaningful operations [PLoS 18].

The parameters of the different layers were fitted in different ways: while the 2nd and 3rd layers (contrast and CSF+masking) were determined using Maximum Differentiation [Malo and Simoncelli SPIE 15], layers 1st and 4th (chromatic front-end and wavelet layer) were fitted to reproduce subjective image distortion data [PLoS 18], and then fine-tuned to reproduce classical masking [Front. Neurosci. 19].

• Download the Toolbox!: BioMultiLayer_L_NL_color.zip (49MB)

2019 - 2021: Convolutional and differentiable implementations

The matrix formulation developed in [PLoS 18, Front. Neurosci. 19] and implemented in BioMultiLayer_L_NL_color is elegant but not applicable to large images nor appropriate to be included in python deep-learning schemes since it is implemented in Matlab. Recently we worked to solve these issues and confirm the choices of the chromatic part. This led to the deep Percepnet [IEEE ICIP 20], and to the convolutional version the above MultiLayer L+NL cascade [J.Vision, Proc. VSS 2021]. While Percepnet has the advantage of being implemented in python and hence ready for automatic differentiation (state-of-the-art in image quality), it has the disadvantage of being based on a restricted version of Divisive Normalization (no explicit interactions in space/scale) [ICLR 17]. On the other hand, the BioMultiLayer_L_NL_color_convolutional has a more general and interpretable version of the Divisive Normalization (in includes full range of interactions in space/scale/orientation). Moreover, the color adaptation choices and the scaling of the achromatic and chromatic channels has been confirmed by positive psychophysical and statistical behaviors [J. Neurophysiol.19, J. Math.Neurosci.20]. However, derivatives are implemented in matlab, so it is not ready to be included in deep-learning schemes right away. There is a lot of room for improvement of its parameters!.

• Download the Toolbox!: BioMultiLayer_L_NL_color_convolutional.zip (76MB)

• Visit Github!: Perceptnet

• Statistical and Psychophysical support for the chromatic choices (I/II): The small scale model in this paper uses the same chromatic choices as the BioMultiLayer_L_NL models. Code and Data for small scale recurrent Wilson-Cowan network [J.Neurophysiol. 2020]

• Statistical and Psychophysical support for the chromatic choices (II/II): The small scale model in this paper uses the same chromatic choices as the BioMultiLayer_L_NL models. Code and Data for small scale Div. Norm. [J.Math.Neurosci. 2020]

Psychophysical test-bed for model tuning and comparison

The figure below (computed using VISTALAB and ColorLab) illustrates distinctive features of early vision: (a) the bandwidth of the achromatic and the chromatic channels is markedly different, (b) the response to contrast is a saturating nonlinearity, its slope (sensitivity) depends on the frequency and the response attenuates as a function of the properties of the background (note how the test is more salient -highlighted in green- on top of a very different background while it is masked -highlighted red- on top of similar backgrounds), and (c) the visibility of i.i.d. noise seen on top of a natural image is not uniform: e.g. visibility is smaller in high contrast regions.

These quite visible facts can be used to tune the parameters of the mechanistic models considered above. One could play with the parameters by hand until the response curves qualitatively reproduce what one actually sees. We suggested this idea to improve model fit in natural image databases [Front.Neurosci.18] and (for the first time!) here is data and code to perform such tune-it-yourself experiments: experiments_VistaModels.zip (400MB).

File is huge because it contains thousands of tests to compute detailed contrast response curves and distortion measures on the TD database. Moreover, it also has the corresponding responses of the three mechanistic models!.

Results below suggest that models are equivalent but the most recent displays better behavior (on top of having more plausible receptive fields [PLoS 18]). More importantly, while the results on Image Quality are way better than the popular Structural Similarity Index SSIM (see VistaQualityTools ) there is still a lot of room for improvement through these tune-it-yourself experiments!.

Model Comparison

(B) Principled Models

Efficient coding in mechanistic models

We have shown that models including point-wise Weber-like saturation for brightness lead to decreasing signal-to-noise ratio as a function of the luminance [J.Opt.95]. Moreover, taking into account more general cascades of linear+nonlinear layers (e.g. local-frequency transforms and divisive normalization after Weber-brightness) we have seen that the efficiency of such systems (in terms of redundancy reduction) decreases with luminance and contrast, which is consistent with the distribution of natural images in local frequency domains [PLoS 18]. We have seen that the discrimination ability of Local-DCT+Div.Norm. models is bigger in the more populated regions of the frequency-amplitude domain [Im.Vis.Comp.97]. Additionally, we have seen that the mutual information between the coefficients of the image representation progressively reduces from the retina to the normalized representation, both in the local-DCT + DN case [IEEE TIP 06] and in the Orthogonal wavelet+DN case Neur.Comp.10.

The above body of results means that the Mechanistic Models considered above display remarkable adaptation to the natural image statistics.

In the same line, in collaboration with NYU (Balle and Simoncelli) we have optimized the described linear+nonlinear architectures for optimal autoencoding. By including both the linear and the nonlinear parts in the optimization we get unprecedented rate-distortion performances (see paper and code here [ICLR 17]), way better than our previous image coders based on V1 models with fixed linear stages (See the VistaCoRe Toolbox).

Statistically-based linear receptive fields

Statistical goals such as decorrelation (Principal Component Analysis, PCA), and Independent Component Analysis (ICA) many time lead to sensible linear receptive fields when trained with natural scenes. For instance, spatio-spectral PCA leads to compact representations to disentangle reflectance and spectral illumination from retinal irradiance and lead to spatial-frequency sensors with smooth spectral response [IEEE TGRS 13] (see VistaSpatioSpectral). In collaboration with Helsinki University (Gutman and Hyvarinen) we explored ICA-related techniques. Complex ICA led to local and oriented receptive fields in phase quadrature [LNCS11] (download the Complex ICA Toolbox). Higher Order Canonical Correlation Analysis (HOCCA) combines the sparsity goal with optimal correspondence between identified features in domain adaptation problems leading to biologically plausible spatiochromatic receptive fields which adapt to changes in the illumination (PLoS 14, see the HOCCA Toolbox).

This analysis of ICA methods concluded with a refutation of a classical result in cortical organization based on Topographica ICA: in fact (as opposed to Hyvarinen & Hoyer Vis. Res. 2001) it does not lead to orientation domains [PLoS 17]. See code and results to analyze TICA receptive fields.

Statistically-based nonlinearities

Instead of optimizing the mechanistic models for efficient coding we tried a stronger approach to test the Efficient Coding Hypothesis: use pure data-driven techniques instead of assuming models which already have the right functional form. We developed a family of invertible techniques for manifold unfolding and for manifold Gaussianization.

The unfolding techniques identify nonlinear sensors that follow curved manifolds. These include Sequential Principal Curves Analysis SPCA and sequels: Principal Polynomial Analysis PPA and Dimensionality Reduction based on Regression, DDR.

The Gaussianization technique (Rotation-Based Iterative Gaussianization, RBIG) does not identify sensors but it allows to compute the PDF. Therefore it is useful to define discrimination regions according to information maximization or error minimization. See the kind of predictions made by these unfolding techniques (SPCA [Network 06, NeCo12, Front. Human Neurosci.15, ArXiv 16, https://arxiv.org/pdf/1606.00856.pdf], and PPA-DRR [SPIE13, Int.J.Neur.Syst.14, IEEE Sel.Top.Sig.Proc.15]) and by the Gaussianization technique [Talk at LeCun Lab NYU 13, IEEE TNN 11].

Closely related to optimal discrimination (or optimal metric) for error minimization is the concept of Fisher Information. Our lab has a tradition in the study of Riemannian metrics induced by nonlinear perception systems [J. Malo PhD 99, Displ.99]. Over the years, the ideas about the geometrical transforms induced by the system and their effect on information processing have evolved from distance computation to the consideration of the transformation of neural noise [Displ.99, Patt.Recog.03, IEEE TIP 06, JOSA A 10, SPIE 15, NIPS 17, PLoS 18].

(C) Engineering-motivated

Perceptually-weighted motion estimation: VistaVideoCoding

What can be predicted is not worth transmitting!. This simple idea is the core of predictive coding used in most successful video coders (e.g. MPEG). In predictive coding motion information is the key to predict future-from-past. MPEG-like coders first compute the optical flow (or displacement field) and encode the prediction error in a transformed domain which (not surprisingly!) is similar to the V1 mechanistic models described above.

In this video-coding context we improved motion estimation by connecting the optical flow computation with the perceptual relevance of the prediction error: we proposed to improve the resolution of the motion estimate only if the prediction error was hard to encode for our improved V1 models [LNCS97, Electr.Lett.98, J.Vis.01]. This gave rise to smoother motion flows more appropriate for motion-based segmentation [Electr.Lett.00a], and to better video coders [Electr.Lett.00b, IEEE TIP 01].

• Download the motion estimation and video coding toolbox! VistaVideoCoding.

Image Coding: VistaCoRe

Image compression requires vision models that rank visual features according to their perceptual relevance so that extra bits can be allocated to encode the subjectively important aspects of the image.

The vision model based on DCT and Divisive Normalization considered above leads to better decoded images at the same compression ratio than JPEG and variants based on simpler models of masking.

See the VistaCoRe (Coding and Restoration Toolbox), and the references [Eletr.Lett95, Eletr.Lett99, Im.Vis.Comp.00 Patt.Recog.03, IEEE TNN 05, IEEE TIP 06a, IEEE TIP 06b, JMLR08].

Image and Video Quality: VistaQualityTools

Computing perceptual distances between images requires vision models that identify relevant and negligible visual features. Distortions in features that will be neglected by the observers should induce no effect in the distance. And the other way around for visually relevant features. The different models can be quantitatively compared by their accuracy in reproducing the opinion of viewers in subjectively rated databases.

The three vision models considered above (based on DCTs, orthonormal wavelets, and overcomplete wavelets) have been used to propose distortion metrics that overperform SSIM. See VistaQualityTools, and the references [Im.Vis.Comp.97, Displays99, Patt.Recog.03, IEEE Trans.Im.Proc.06] for the DCT metric, [JOSA 10, Neur.Comp.10] for the orthogonal wavelet metric, and [PLoS 18, Frontiers Neurosci.18] for the metric based on overcomplete wavelets.

The Wilson-Cowan interaction of wavelet-like visual neurons is analyzed in total correlation terms for the first time. Theoretical and empirical results show that a psychophysically-tuned interaction achieves the biggest efficiency in the most frequent region of the image space. This an original confirmation of the Efficient Coding Hypothesis and suggests that neural field models can be an alternative to Divisive Normalization in image compression.