See also the VistaLab and VistaModels toolboxes.

Information theory is an outstanding framework to measure uncertainty, dependence and relevance in data and systems. It has several desirable properties for real world applications: it naturally deals with multivariate data, it can handle heterogeneous data types, and the measures can be interpreted in physical units. However, it has not been adopted by a wider audience because obtaining information from multidimensional data is a challenging problem due to the curse of dimensionality. Here we propose an indirect way of computing information based on a multivariate Gaussianization transform. Our proposal mitigates the difficulty of multivariate density estimation by reducing it to a composition of tractable (marginal) operations and simple linear transformations, which can be interpreted as a particular deep neural network. We introduce specific Gaussianization-based methodologies to estimate total correlation, entropy, mutual information and Kullback-Leibler divergence. We compare them to recent estimators showing the accuracy on synthetic data generated from different multivariate distributions. We made the tools and datasets publicly available to provide a test-bed to analyze future methodologies. Results show that our proposal is superior to previous estimators particularly in high-dimensional scenarios; and that it leads to interesting insights in neuroscience, geoscience, computer vision, and machine learning.

Software

RBIG Python toolbox

Includes tools to compute Information Theory measures used in the current paper [RBIG4IT2020]

Demo in Google Colab

This demo provides an interactive example of how to use the RBIG Python toolbox to compute information measures like entropy and mutual information. It’s an easy-to-follow resource for those interested in testing the methods on their data.

RBIG Matlab toolbox

Includes tools to compute Information Theory measures and scripts to generate the synthetic data for the experiments in the current paper [RBIG4IT2020]

Paper Summary

The measures that can be computed using RBIG defined in this paper are the ones in the following figure + the Kulback-Leibler divergence. The main point is that RBIG allows to get acurated estimations of these measures even in multidimensional datasets.

Extended results

Here extra results for the paper are shown. Mainly figures for results on synthetic data that would taken too much space in the original paper.

Total Correlation

The total correlation is a measure of multivariate dependence among several variables. In this section, we show how the RBIG methodology allows for precise estimation of total correlation, even for datasets with non-Gaussian distributions or high dimensionality.

Entropy

Entropy is a fundamental concept in information theory, representing the amount of uncertainty in a dataset. Using RBIG, we show that entropy can be computed more efficiently compared to classical estimators, particularly in datasets with complex dependencies.

KLD

Kullback-Leibler divergence measures how one probability distribution diverges from a second, reference distribution. RBIG offers a more accurate way to compute KLD in high-dimensional settings, improving performance in tasks like anomaly detection and model comparison.

Mutual Information

Mutual information quantifies the amount of information obtained about one random variable through another. The RBIG methodology demonstrates superior performance in estimating mutual information, especially in datasets with nonlinear dependencies, making it a valuable tool for feature selection and data analysis.

The image representations along the retina-cortex pathway are analyzed in terms of their ability to capture information about the visual scenes. The considered series of representations includes: (1) the LMS retinal images, (2) their von-Kries adapted version, (3) the opponent images at LGN, (4) their nonlinear version after Weber-like saturation, (5) the cortical local-frequency representation filtered by achromatic and chromatic CSFs, and (6) the cortical representation after divisive normalization.

Assuming a single-step transform from the retinal input to each of these representations, and sensors of the same Signal-to-Noise quality in each representation, our estimations of transmitted information show that: (a) progressively deeper representations are better in terms of the amount of captured information, (b) the transmitted information up to the cortical representation follows the probability of natural scenes over the chromatic and achromatic dimensions of the stimulus space, (c) the contribution of spatial transforms to capture visual information is substantially greater (67%) than the contribution of chromatic transforms (33%), and (d) nonlinearities of the responses contribute substantially to the transmitted information (about 28%) but less than the linear transforms (72%).

The parameters of the model have psychophysical origin and they were not statistically optimized in any way. The information estimates were computed using our Gaussianization transform (Laparra et al. IEEE TNN 11) over our database of colorimetrically-calibrated images (Laparra et al. Neur.Comp. 12), but equivalent results are obtained using other estimates (e.g. offset corrected Kozachenko-Leonenko [Marin et al. IEEE PAMI 13]) or other databases (e.g. Foster et al [Vis.Res.15,16]).

The Wilson-Cowan interaction of wavelet-like visual neurons is analyzed in total correlation terms for the first time. Theoretical and empirical results show that a psychophysically-tuned interaction achieves the biggest efficiency in the most frequent region of the image space. This an original confirmation of the Efficient Coding Hypothesis and suggests that neural field models can be an alternative to Divisive Normalization in image compression.