Methods related to the Human Camera

Let the brain do image coding for you: it is better than JPEG2000

The long term Human Camera goal could be seen as a new Brain Machine Interface (BMI) for image transmission in which the nontrivial feature extraction and dimensionality reduction stage is done by a human brain instead of by the conventional compression algorithm. The (input) signal should be reconstructed (from the neural signal) at the receiver.

Here we present an oversimplified, but illustrative, example of the decoding of the cortical signal to show some of the computational issues of the modeling and BMI problems, its relations with image compression and enhancement, and to show that we already are working with the technologies that allow the inference of the stimulus from the neural response.

Simplified (Forward) Model to start playing with

The considered example uses (i) a nonlinear response based on divisive normalization and fitted with psychophysics (Carandini et al. 2012, Malo et al. 2006, Malo et al. 2010), (ii) different distortion sources that modify the nonlinear signal as for instance, random neural noise or additional pooling stages not considered in the assumed model, (iii) dimensionality reduction, i.e. missing responses in the recorded signal. Of course, this model will be augmented in different ways (e.g. including irregular spatial sampling, the diversity in shapes of the receptive fields (Martinez et al. 2014), the adaptive interaction in the nonlinear stage (Schwartz et al. 2009), or correlated noise patterns.

Multi information numbers (in bits) between four coefficients of the different representation show the redundancy reduction along the path, which is consistent with the efficient encoding hypothesis [Olshausen96, Schwartz01], and is consistent with our previous results [Malo06b, Malo10].

Reconstructions from the (stochastic) responses of the (partially or totally) unknown model

Next figure shows the reconstruction using different inversion techniques: linear (similar to [stanley99]), Kernel (similar to [Miyawaki08]), and analytic, as in [Malo06b]. The example explores the effect of different distortion sources (noise and elements not considered in the model) on top of a representation of reduced dimensionality (we are missing 61% of the neurons -those tuned to high frequencies-). The good properties of analytic inversion could be used to complement (as prior knowledge) the current regression techniques.

Estimated stimuli from the neural signal under different distortion sources (dimensionality reduction, different neural noise and additional pooling stages not considered in the forward model). In this numerical experiment linear reconstruction is more sensitive to distortion, which is alleviated by nonlinear regression, which still displays significant artifacts. The analytical inverse seems to reconstruct better but also amplifies the noise. The knowledge extracted from the structure of the inverse should be included in the conventional methods.