First words ex-cathedra

Jesús Malo (San Francisco, Starbucks at 390 Stockton St., February 2015)

Circa 2015, applications for full professorship in Spanish universities (cathedra) involved writing an essay to describe your career and personal views on science. Here is what I wrote to get the condition of Accredited University Professor from the official National Evaluation Agency…

Now (after the positive outcome in July 2015), I upload the version with uncensored pictures, full text, and over 150 hyperlinks!. These are my first words ex-cathedra (even though my salary, as well as the salary of over 2500 colleagues in the same situation, will remain the same for a while unless we do something):

Table of Contents

1. Why a physicist would ever care about Human Vision?

Think again: human vision is cool!

The leit-motif of my research and teaching activity is the study of visual information processing in the human brain. This is a biological and subjective problem: not very appealing adjectives for a big-bang theory guy. Nevertheless, the aspects of this problem that may be of interest for physicists determined the direction of my scientific career.

Despite the overuse of the word multidisciplinary, you have to consider that Visual Perception is a truly multidisciplinary problem. On the one hand, the input signal certainly involves plain Physics such as light emission and scattering in every-day scenes (classical Radiometry) and image formation in biological systems (classical Physiological Optics). However, on the other hand, the analysis of such input signal is a problem for Neuroscience: examples of the latter include the study of (natural) neural networks for image understanding. Human Vision is not at all limited to the laws of image formation, that basically date back to Newton classical Optics, but also include the formulation of laws that determine the organization of the sensors that make sense of these signals. And this is a quite different issue!. Regarding this analysis part, a theory that explains the visual cortex phenomena requires concepts coming from Statistics and Information Theory, or in nowadays jargon, Machine Learning. A particularly interesting feature of this problem is the fact that, as opposed to other science problems, the relation between maths and application (here Maths and Neuroscience) is not one-directional: in this case the system to be understood is actually a computing machine that may also inspire original mathematical approaches. Finally, the models coming from Theoretical Neuroscience may be applied in Electrical Engineering and Computer Science.

From a personal (and hence arguable) point of view, the Human Vision problem is interesting for a physicist not for the aspects related to classical Optics (fundamentally solved long ago), but for the study of the visual brain. Vision is not in the (well known) eye of the beholder, but in his/her (highly unknown) brain. The visual brain is a natural system with complex dynamics (the jargon physicists love), quantitative theories for partial explanations are very recent, and many of them are still under discussion. The study of Vision combines experiments, mathematical theories and technological applications, and this combination is the core of how the physicists approach the problems. It doesn’t matter that the experimental methods come from the Psychology, the Optometry or the Neurophysiology (all of them use the so called Psycho-Physics) or that the applications are in Image Processing and Computer Vision: the study of the Human Visual System is certainly quite appropriate for a physicist.

The fascination for the surprising behavior of the visual system is what determined my scientific exploration: over the last 20 years I made some contributions (or managed to introduce some colored noise 😉 in most of the disciplines cited above.

2. Chronological summary of my career

While Khun and Marx were kind of wrong, Sinatra was right: I did it my way!

Selecting a multidisciplinary problem implies having a wide range of collaborators over the years. The topics and the collaborators to address them are the parts of the scientific career that one can actually choose. Thomas Kuhn (or even Karl Marx) would certainly say that economic constraints sometimes impose their own choices. In my case, even though money sometimes determined the order in which I visited different aspects of the problem (e.g., applications before foundations), economic constraints didn’t imply modifications in the selected direction since I was fortunate enough to get steady funds along these two decades (more details on economic constraints below).

Constraints are usually harder in the teaching part since it is determined by the duties of the department where you happen to develop your research. Nevertheless, with some dedication, this part can also be modulated. Similarly to the research side (where I started at the Optics Department in the Physics School, but then I looked for collaborators in Maths, Electrical Engineering, and Computer Science), in the teaching side I decided to give lectures in PhD and Master programs out of my department (beyond the department-related duties). This was a way to convey the knowledge acquired in research activities to a broader audience.

Below is the list of multidisciplinary collaborators I found (or looked for) over time. Note that in the formative years and right after getting my first permanent position, I focused on applications (e.g., image coding) to maximize funding probabilities. More recently, particularly after my second Spanish NSF Project as PI, I turned to the fundamental issues (the theory and the consideration of a higher abstraction level, as for instance in the current Explora Project - my 4th as PI), yet still paying attention to technology transfer:

As a summary, despite the troubles of a truly multidisciplinary topic, the path has been (kind of) coherent and successful. At this point, I have to thank Prof. Jose María Artigas for his lectures on Physics of Vision: a small course for the physics students which (unfortunately!) is no longer available in my university. In those lessons, he told us about something completely different. As fresh and educative as the Monty Python for physics students.

3. My research contributions

Colored noise in vision sciences and some thoughts on the h-index

  • Experiments in vision science
  • Theory: empirical models in vision science
  • Theory: principled models in vision science (computational visual neuroscience)
  • Theory: statistical learning
  • Applications in image processing
  • Preliminary conclusions
  • Impact of the above: h-index or just colored noise?

Experiments in Vision Science (7 JCR publications)

I made experimental contributions in three aspects: Physiological Optics, Psychophysics, and Image Statistics. (i) In the field of Physiological Optics, we measured the optical transfer function of the lens+cornea system in-vivo Opth.Phys.Opt.97. This work received the European Vistakon Research Award 94’. (ii) In Psychophysics, we proposed simplified methods to measure the Contrast Sensitivity Function in all the frequency domain J.Opt.94, and a fast and accurate method to measure the parameters of multi-stage linear+nonlinear vision models Proc.SPIE15. Finally, (iii) in Image Statistics we gathered spatially and spectrally calibrated image samples to determine the properties of these signals and their variation under changes in illumination, contrast, and motion Im.Vis.Comp.00, Neur.Comp.12, IEEE-TGRS14, PLoS-ONE14, Rem.Sens.Im.Proc.11, Front.Neurosci.15.

Theory: empirical models in Vision Science (8 JCR publications)

We proposed mathematical descriptions of different visual dimensions: Texture, Color, and Motion. (i) We used wavelet representations to propose nonstationary Texture Vision models J.Mod.Opt.97, MScThesis95. (ii) We developed Color Vision models with illumination invariance that allow the reproduction of chromatic anomalies, adaptation, and aftereffects Vis.Res.97, J.Opt.96, J. Opt.98, JOSA04, Neur.Comp.12. (iii) Motion Vision models Alheteia08 focus on optical flow computation in perceptually relevant moving regions J.Vis.01, PhDThesis99, and explain the static motion aftereffect Front.Neurosci.15.

All these psychophysical and physiological models have a parallel linear+nonlinear structure where receptive fields and surround-dependent normalization play an important role.

Theory: principled models in Vision Science (12 JCR publications)

This category refers to the proposition of organization laws of sensory systems that explain empirical phenomena, showing how neural function is adapted to the statistics of visual stimuli. (i) We worked on the derivation of the linear properties of the sensors, finding that spatio-chromatic sensitivity, receptive field changes, and phase properties arise from optimal solutions to the adaptation problem under noise constraints and manifold matching PLoS-ONE14, IEEE-TGRS13, from statistical independence requirements LNCS11, NeuroImag.Meeting11, and from optimal estimation of object reflectance IEEE-TGRS14. (ii) We also derived the non-linear behavior of visual sensors like chromatic, texture, and motion sensors, linking non-linearities to optimal information transmission and/or error minimization in noisy systems Network06, Neur.Comp.12, Front.Neurosci.15, J.Opt.95, Im.Vis.Comp.00, LNCS00, Patt.Recog.03, Neur.Comp.10, LNCS10, NeuroImag.Meeting11.

Theory: Statistical Learning (7 JCR publications)

In theoretical neuroscience the derivation of properties of biological sensors from the regularities visual scenes requires novel tools for statistical learning. In this field, we developed new techniques for unsupervised manifold learning, feature extraction (or symmetry detection in datasets), dimensionality reduction, probability density estimation, multi-information estimation, distance learning, and automatic adaptation from optimal dataset matching. Given my interest in applicability in Vision Science problems, I focused on techniques that can be explicitly represented in the image domain to be compared with receptive fields of visual neurons, as opposed to the usual practice in the Machine Learning community. Techniques include Rotation-based Iterative Gaussianization -RBIG- IEEE TNN 11, Sequential Principal Curves Analysis -SPCA- Network06, Neur.Comp.12, Front. Neurosci.15, Principal Polynomial Analysis -PPA- Int.J.Neur.Syst.14, Dimensionality Reduction based on Regression -DRR- IEEE JSTSP15, and Graph Matching for Adaptation IEEE TGRS13.

Applications: Image Processing (24 JCR publications + 1 patent)

We proposed original image processing techniques using both perception models and image statistics including (i) improvements of JPEG standard for image coding through nonlinear texture vision models Electr.Lett.95, Electr.Lett.99, IEEE TNN05, IEEE TIP06a, JMLR08, RPSP12, Patent08, (ii) improvements of MPEG standard for video coding with new perceptual quantization scheme and new motion estimation focused on perceptually relevant optical flow LNCS97, Electr.Lett.98, Electr.Lett.00a, Electr.Lett.00b, IEEE TIP01, Redund.Reduct.99, (iii) new image restoration techniques based on nonlinear contrast perception models and the image statistics in local frequency domains IEEE TIP 06b, JMLR10; (iv) new approaches to color constancy either based on relative chromatic descriptors Vis.Res.97, J.Opt.96, statistically-based chromatic adaptation models Neur.Comp.12, PLoS-ONE14, or Bayesian estimation of surface reflectance IEEE-TGRS14; (v) new subjective image and video distortion measures using nonlinear perception models Im.Vis.Comp.97, Disp.99, IEEE ICIP02, JOSA10, Proc.SPIE15; and (vi) image classification and knowledge extraction (or regression) based on our feature extraction techniques IEEE-TNN11, IEEE-TGRS13,Int.J.Neur.Syst.14, IEEE-JSTSP15. See code for image and video processing applications here.

Preliminary Conclusions

  • The visual brain is astonishingly well adapted to the natural visual world. This sentence shouldn’t be surprising for any teenager that heard about Charles Darwin. The cool thing in that conclusion was preparing accurate image data, developing the appropriate mathematical tools to derive the behavior described by computational models as seen in psychophysical illustrations. By putting all this together in a single piece of code you realize that the statement is true.

  • Appropriate (mathematical) formulation of visual phenomena is the only way to understand the problem and to derive applications. This statement is not very original either, given the famous Galileo Galilei quote [the book of nature is written in mathematical language]. However, in this multidisciplinary world, a special effort has to be done to translate physiological facts into models that work on, lets say, actual video sequences. By doing so, you transcend the specific details of a set of experiments, and think about all the additional problems faced (and solved) by the visual brain. Numerical simulations are useful to put a specific physiological behavior in perspective. Moreover, well-formulated models allow us to explore new experimental questions through the appropriate stimuli. Not to speak about the straightforward use in image processing and computer vision…

  • Nonlinear techniques are fancy, but it is amazing the percentage of reality that we can explain with linear models. Another old-fashion statement for eigenvector lovers. Besides, linear algebra is easy! For instance, a simple rotation (the Principal Component Analysis of Karl Pearson) applied to small patches of natural sequences, explains the major features of the receptive fields of LGN-V1 visual neurons. This includes opponent color coding, neurons tuned to spatial texture, and motion-sensitive neurons. Different kinds of simple affine transforms (linear scaling and translations) explain basic sensitivity to color, texture, and motion as well as the basic trends of adaptation. Amazing!

  • We roughly understand low-level visual information processing in the brain. However, there is still a long way to understand how we derive abstract concepts from low-level primitives. Not a surprising statement either if you saw the appropriate documentary or heard about David Marr. Despite all the knowledge about color, spatial texture, motion, and depth information processing in LGN, V1, and MT, little is known about how these pieces are put together in other parts of the brain (e.g. IT). What are the organization laws of these higher abstraction mechanisms? What about their relations to language? What about our ability to synthesize images (draw) from a written description?.

An educated teenager that heard about Darwin, Galileo, Pearson, and Marr (evolution, mathematical modeling, eigenvectors, decorrelation, and vision) could be disappointed by the simplicity of these conclusions. However, note that my claims can be louder now than 20 years ago because of the time spent in accumulating evidence (and writing this piece of code). I hope that the next 20 years are fruitful enough to make these conclusions stronger or (even better!) to change some of them.

Impact of Colored Noise in science libraries

As I told Eero Simoncelli once, while few people make a big impact on the scientific community, what others (including myself) do can be seen as injecting colored noise in the science libraries and the internet. Nevertheless, as argued below, that is not a major problem, but even something worth funding.

Some thoughts on JCR publications and the h-index

For ordinary (not-Nobel-laureate) people, research is mainly a personal learning experience. Such process starts with some childish initial curiosity and ends with refereed publication. It involves putting the question in context, saying something coherent about it, and convincing critical reviewers about the accuracy of such statement (no matter it is ground-breaking or not). Given the quality control imposed by peer review (particularly in high impact journals) the publishing-in-fine-journals exercise is one of the most comprehensive learning procedures ever developed. Even though the publications of the average scientist remain unknown or never make a global difference (i.e. low h-index), the rigor of the learning process in JCR-journals ensures this person has the deepest understanding of the issues. And this has a local impact in the dissemination of knowledge to others, either (local) students or (local) industries. A cohort of average scientists well trained through the publication process have to be there, ready to understand, confirm, disseminate and apply what (the few) original scientist happen to discover. In my view, that is the justification of devoting public money to fund average scientific research (or random colored noise generation ;-). Note that samples from colored noise do not distribute as a sphere, but collectively they point to a certain direction, hopefully the right one!.

For those of you who do not share this personal learning view, and love rankings better, here is the impact of my research (by July 2015, i.e. automatically outdated) according to my Google Scholar profile: my Hirsch index was 19, the total number of citations to my work was 876, so I was the 3rd most-cited scientist in the world in the (Google Scholar ;-) category of Image Statistics, the 28th one in Human Vision, the 87th in Visual Perception, and the 263rd in Vision. So what?

To me, the undeniable peak of my scientific career happened when a mild morning of February 2001, I left my office at the NASA Ames Research Center and drove my Toyota through the rocket wind tunnels to attend a talk at a nearby town in Silicon Valley on the vision abilities of HAL-9000, the famous computer of Stanley Kubrick’s 2001: A Space Odyssey. The combination of NASA, 2001, and HAL-9000 together really felt like big science. Particularly compared to my Spanish postdoc salary and housing prices of the dot-com bubble. Since that glorious morning, I felt like Dr. Dave Bowman for a second; everything else has been a steady decline.

4. My teaching activities: like Richard Dawkins in a Republican Convention

Why an optometrist (or engineer) would ever care about Maths (or Science)?

My teaching activity at the university spans over 19 years (only one less than my research activity). This means that I had to teach while obtaining my PhD. This undesirable situation happened since at that time (mid 90s) getting a PhD grant was restricted to students of professors having public funds (which was not the case of my advisor). Therefore, I stayed at the university only because (i) I won the European Vistakon Research Award (which I used to pay my PhD research for one year), and (ii) a new degree on Optometry and Vision Science was established at my university and it generated several openings for junior assistant professors.

The quality of my teaching over these two decades has been rated by my students according to the regulations in my university (in a scale of 5) as 3.6 ± 0.3, i.e. they gave me a positive rating with a small variance over the years.

My teaching activity has been modulated by (1) my interest in Vision Science, and (2) by having most of my docent duties associated to the Degree and Master on Optometry and Vision Science. The correspondence between these two factors has been positive since it gave coherence to the research and teaching activities. However, the problem with Optometry students is that they imagine themselves as Medical Doctors (and you know that Evidence-Based Medicine is a recent field!). As a result, these students are not quite prepared for the practice of quantitative science (is there any non-quantitative science anyway?). This problem represented, (i) a challenge to convey the quantitative message to students with non-quantitative interests, and (ii) an incentive to diversify my teaching activity looking for students not scared by scalar products. The challenge posed by the non-quantitative students lead to the development of Matlab tools such as COLORLAB, BasicVideoTools, and VirtualNeuroLabs, and new docent methodologies [ProyDocente02] to convey the quantitative credo to students afraid of Maths. In this quantitative effort I found a lot of help and support from M.J. Luque and P. Capilla (respectively ;-) Sometimes I really feel as hopeless as Richard Dawkins at a Republican convention. But having a lot of fun, though!. On second thoughts, my Optometry undergrads are not that bad: deficient education is always a problem of the teachers, not the students. Please excuse the trivial comparison with the creationists!.

In order to diversify the audience, I also lectured in PhD and Master programs with Excellence distinction out of my Optics department, as for instance at the Applied Maths and Computer Science departments of my university, at the Institute of Applied Ophtalmo-Biology (Univ. Valladolid), and at the Institut de Robòtica i Informàtica Industrial (UPC). You can find slides, lecture notes and computer material for PhD courses here.

Finally, I have to mention my best (or more patient) students: those who dared to be advised by me in their PhD years: Irene Epifanio, Juan Gutiérrez, and Valero Laparra. They got doctorate degrees with a number of JCR publications, best PhD and Master Thesis awards in PhD programs with European Excellence distinction, etc… Nevertheless, the best is what I learned from them: thank you all, it was a lot of fun!.

5. Economic constraints of science in Spain

Why a positive evaluation for professorship does not imply an actual position in Spain?

Saying that “I did it my way despite the economic constraints” was an obvious literary license (for the evaluation committee). The truth is that my generation has been extremely lucky since Spain experienced an unprecedented window of opportunities for young scientists in the late 90s and early 2000s. In this short time window the economic effort started in the 80s (after we got democracy) to build a European-like science system, led to a mature public research system in the 90s (private sector didn’t go that fast). Favorable economic environment in the 90s and European funds steadily fueled this system and average scientific production in Spain achieved world-class level for the first time in history. In this situation it is easier to do it your way. It is fair to acknowledge that scientific freedom is a by-product of favorable conditions.

May be Marx wasn’t that wrong after all. Particularly considering how the situation has changed since the 2008 crisis. Sadly, the favorable time window may be closing in Spain (and in other places in southern Europe). Conservative governments in Spain do not see basic research as an investment for the future, but as a luxury you can disregard (Nature, Dec. 2011).

This short-sighted policy affects both young and senior scientists. Massive budget cuts reduce the possibility to get PhD students, and those who finally complete their PhD have small chances here. Postdocs are scarce and, for some years now, new associate professor positions are extremely rare. In the same vein, no new full professor position has been created since 2011, and retirement-related positions are only covered at a 50% rate. Before the crisis, the official Accreditation for University Professor (after a thorough independent review) used to be equivalent to getting an actual Professorship since there were no major funding problems. Now those days are over. The careers of accredited scholars (otherwise professors) are indefinitely truncated.

An association of Accredited University Professors (website in Spanish) was created to demand solutions for this unfair blocked-career situation (see the manifest in English). Major worker unions CCOO and UGT support our demands. As in scientific research (see the colored noise concept) it is the collective action what defines the direction to go. Please sign up!

For further details on each of these sections, including my research contributions and teaching philosophy, I have included over 150 hyperlinks throughout the text, providing access to my full publications, tools, and additional resources.