Redundancy reduction and multi-information.

We consider a population of sensory neurons that transforms natural image patches into a set of neural activities or . We always use to denote responses to linear filters, and for the output of divisive normalization or radial factorization. The goal of redundancy reduction is to remove statistical dependencies between the single coefficients of or .

Redundancy is quantified by the information theoretic measure called multi-information(1)which measures how much the representation differs from having independent components. More precisely, the multi-information is the Kullback-Leibler divergence between the joint distribution and the product of its marginals or, equivalently, the difference between the sum of the marginal entropies and the joint entropy. In case of this equals the better known mutual information. If the different entries of are independent, then its joint distribution equals the product of the single marginals or–-equivalently–-the joint entropy equals the sum of the marginal entropies. Thus, the multi-information is zero if and only if the different dimensions of the random vector are independent, and positive otherwise. In summary, the multi-information measures all kinds of statistical dependencies among the single coefficients of a random vector. In the Methods Section, we describe how we estimate the multi-information for the various signals considered here.

Divisive normalization.

From all existing divisive normalization models considered previously in the literature, ours is most closely related to the one used by Schwartz and Simoncelli [9]. It consists of two main components: a linear filtering step and a rescaling step based on the Euclidean norm of the filter responses(2)While the linear filters capture the receptive field properties, the rescaling step captures the nonlinear interactions between the single neurons. Most divisive normalization models use filters that resemble the receptive fields of complex cells [9], [15], [16]. Therefore, we use filters obtained from training an Independent Subspace Analysis (ISA) on a large collection of randomly sampled image patches [15], [16, see also Methods]. ISA can be seen as a redundancy reduction transform whose outputs are computed by the complex cell energy model [17], [18]. For this study, the algorithm has the advantage that it not only yields complex cell-like filter shapes, but also ensures that single filter responses are decorrelated and already optimized for statistical independence. This ensures that the redundancies removed by divisive normalization and radial factorization are the ones that cannot be removed by the choice of linear filters [7], [19].

Several divisive normalization models exist in the literature. They differ, for instance, by whether a unit is contained in its own normalization pool, or in the exact form of the rescaling function also known as Naka-Rushton function. From the viewpoint of redundancy reduction, the former distinction between models is irrelevant because the influence of a single unit on its normalization pool can always be removed by the elementwise invertible transformation which does not change the redundancies between the responses [20] (the multi-information is invariant with respect to elementwise invertible transformations). Sometimes, a more general form of the Naka-Rushton function is found in the literature which uses different types of exponents(3)The divisive normalization model considered in this study (equation (2)) differs from this more general version by the type of the norm used for rescaling the single responses: Where equation (3) uses the -norm we use the Euclidean norm. Because radial factorization is defined for the more general -norm (see Methods), all analyses in this paper could be carried out for this more general transform. However, we instead chose to use the Euclidean norm for simplicity and to make our model more comparable to the ones most commonly used in redundancy reduction studies of divisive normalization [9], [20]–[22].

Also note that the Naka-Rushton function is often defined as the th power of equation (3). However, the form of equation (3) is more common in redundancy reduction studies in order to maintain the sign of . We mention the consequences of this choice in the discussion.

Radial factorization.

Radial factorization is an optimal radial rescaling for redundancy reduction. We will now briefly introduce radial factorization starting from divisive normalization. For more mathematical details see the Methods Section.

On a population level, the rescaling step of divisive normalization is a nonlinear mapping that changes the Euclidean radius of the filter response population. This can be seen by decomposing divisive normalization into two multiplicative terms(4)The second term normalizes the response vector to length one while the Naka-Rushton function in the first term determines the new radius. Since the rescaling depends only on the norm, the new radius does not depend on any specific direction of .

The redundancy between the coefficients of is determined by three factors: The statistics of natural image patches which—together with the choice of filters —determine the statistics of , and the radial transformation . If we allow the radial transformation to be a general invertible transform on the Euclidean norm, we can now ask how the different model components can be chosen in order to minimize the redundancy in .

A substantial part of the redundancies in natural images are second order correlations, which can be removed by linear filters during whitening [6]. Whitening does not completely determine the filters since the data can always be rotated afterwards and still stay decorrelated. Higher order decorrelation algorithms like independent component analysis use this rotational degree of freedom to decrease higher order dependencies in the filter responses [3]. However, there is no set of filters that could remove all statistical dependencies from natural images [6], [7], because whitened natural images exhibit an approximately spherical but non-Gaussian joint distribution [7], [21], [23], [24]. Since spherical symmetry is invariant under rotation and because the only spherically symmetric factorial distribution is the Gaussian distribution [13], [25], the marginals cannot be independent.

Hence, the remaining dependencies must be removed by nonlinear mechanisms like an appropriate radial transformation . Fortunately, the joint spherically symmetric distribution of the filter responses already dictates a unique and optimal way to choose : Since a rescaling with will necessarily result in a spherically symmetric distribution again, must be chosen such that is jointly Gaussian distributed. Therefore, we need to choose such that follows the radial distribution of a Gaussian or, in other words, a -distribution. This is a central point for our study: For a spherically symmetric distribution the univariate distribution on determines higher order dependencies in the multi-variate joint distribution of . This means that if we restrict ourselves to radial transformations, it is sufficient to look at radial distributions only. The fact that the Gaussian is the only spherically symmetric factorial distribution implies that the coefficients in can only be statistically independent if follows radial -distribution. Radial factorization finds a transformation which achieves exactly that by using histogram equalization on the distribution of [11], [12, see also Methods]. All these considerations also hold for -spherically symmetric distributions [11], [13].

Note that this does not imply that the neural responses must follow a Gaussian distribution if they are to be independent because the distribution of the single responses can always be altered by applying an elementwise invertible transformation without changing the redundancy. The above considerations only mean that given the two main model components of divisive normalization (and the assumption of spherical symmetry), the best we can do is to choose the to be whitening filters and according to radial factorization.

Radial factorization and divisive normalization are not equivalent.

The goal of this study is to compare the redundancy reduction achieved by divisive normalization and radial factorization. Apart from all similarities between the two models, there is a profound mathematical difference showing that the two mechanisms are not equivalent (as noted by [12]).

Both mechanisms have the formHowever, the radial rescalings of radial factorization and that of divisive normalization, and , have a different range. Since the -distribution is non-zero on all of the range of must be as well. However, in case of divisive normalization, the Naka-Rushton function saturates at . This means that can never transform a radial distribution into a -distribution since values beyond cannot be reached.

While this implies that the two mechanisms are mathematically not equivalent, it could still be that they perform similarly on data if the probability mass of the -distribution in the range beyond is small. Therefore, we choose to be the quantile of the -distribution in all our experiments (see Methods).

Comparison of the redundancy reduction performance.

We compared the amount of redundancy removed by divisive normalization and radial factorization by measuring the multi-information in the plain filter responses and the normalized responses for a large collection of natural image patches (Figure 1b). In both cases the parameters of the radial transformation were chosen to yield the best possible redundancy reduction performance (see Methods). While both divisive normalization and radial factorization remove variance correlations (Figure 1a), the residual amount of dependencies for divisive normalization is still approximately of the total redundancies removed by radial factorization (Figure 1a–b). This demonstrates that divisive normalization is not optimally tailored to the statistics of natural images.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Redundancy reduction and radial distributions for different normalization models.

A: Divisive normalization model used in this study: Natural image patches are linearly filtered. These responses are nonlinearly transformed by divisive normalization or radial factorization (see text). After linear filtering the width of the conditional distribution of two filter responses depends on the value of (conditional log-histograms as contour plots). This demonstrates the presence of variance correlations. These dependencies are decreased by divisive normalization and radial factorization. B: Redundancy measured by multi-information after divisive normalization, extended divisive normalization, and radial factorization: divisive normalization leaves a substantial amount of residual redundancy (error bars show standard deviation over different datasets). C: Distributions on the norm of the filter responses for which divisive normalization (red) and extended divisive normalization (blue) are the optimal redundancy reducing mechanisms. The radial transformation of radial factorization and its corresponding distribution (mixture of five -distributions) is shown in black. While radial factorization (inset, black curve) and extended divisive normalization (inset, blue curve) achieve good redundancy reduction, they lead to physiologically implausibly shaped contrast response curves which are mainly determined by their respective radial transformations shown in the inset. The radial transformation of divisive normalization is shown for comparison (inset, red curve).

https://doi.org/10.1371/journal.pcbi.1002889.g001

To understand this in more detail, we derived the distribution that should have if divisive normalization were the optimal redundancy reducing mechanism and compared it to the empirical radial distribution of represented by a large collection of uniformly sampled patches from natural images. This optimal distribution for divisive normalization can be derived by transforming a -distributed random variable with (see Methods). Since has limited range we actually have to use a -distribution which is truncated at . The parametric form of the resulting distribution is given in the Methods Section. We refer to is as Naka-Rushton distribution in the following. The parameters of the Naka-Ruston distribution are and . Since is already determined by fixing the range of to the quantile of the -distribution, the remaining free parameter is . In the Naka-Rushton function this parameter is called half-saturation constant and controls the horizontal position of the contrast response curve in model neurons.

We fitted via maximum likelihood (see Methods) and found that even for the best fitting there is a pronounced mismatch between the Naka-Rushton distribution and the empirical distribution given by the histogram (Figure 1c). This explains the insufficient redundancy reduction because the Naka-Rushton distribution expects most of the responses to fall into a much narrower range than responses to natural images do in reality. The Naka-Rushton function would map the red radial density in Figure 1c perfectly into a truncated -distribution. However, it maps a profound part of the true radial distribution of (gray histogram) close to , since this part is located to the right of the mode of the Naka-Rushton distribution where it expects almost no probability mass. Additionally, the Naka-Rushton distribution exhibits a small gap of almost zero probability around zero. This gap, however, also contains a portion of empirical distribution. This part gets mapped close to zero. To understand why this leaves significant redundancies, imagine the most extreme case in which all the probability mass of would either be mapped onto or on onto . The corresponding distribution on would consist of a point mass at zero and a spherical shell at . Such a distribution would clearly exhibit strong dependencies.

Augmenting divisive normalization by more parameters.

It is clear that the suboptimal redundancy reduction performance of divisive normalization is due to its restricted parametric form. Therefore, we explored two options how to increase its degrees of freedom and thereby its redundancy reduction performance: the first option endows static divisive normalization with additional parameters , the second option allows for a dynamic temporal adaptation of .

The simplest way to increase the degrees of freedom in divisive normalization is to introduce two additional parameters in the Naka-Rushton functionThese parameters allow for more flexibility in the scale and shape of the corresponding Naka-Rushton distribution. We label all models that use this parametrization as extended in the following. Note that the extended Naka-Rushton function only saturates for . This means that it could in principle transform into such that is -distributed. For and , the original Naka-Rushton function is recovered. As before, we derived the corresponding extended Naka-Rushton distribution by transforming a (truncated) -distributed random variable with . We fitted the resulting distribution to a large collection of , used the maximum likelihood parameters for extended divisive normalization, and measured the redundancy via multi-information in the resulting normalized responses .

We found that an extended divisive normalization transform achieves substantially more redundancy reduction and that the extended Naka-Rushton distribution on fits the image data significantly better (Figure 1b–c). However, we also find that the best extended Naka-Rushton function for redundancy reduction would yield biologically implausible contrast response curves which capture the firing rate of a neuron upon stimulation with gratings of different contrast at the neuron's preferred spatial frequency and orientation.

In the divisive normalization and the radial factorization model, the shape of the contrast response curve is determined by the shape of the radial rescaling function (Figure 1c, inset) [8]. In contrast to the normal Naka-Rushton function (Figure 1c, inset, red curve), the extended version (Figure 1c, inset, blue curve) exhibits a physiologically unreasonable shape: it starts at a non-zero value, increases without saturation, and does not resemble any sigmoidal shape at all. The non-zero level for low contrasts is a direct consequence of the optimization for redundancy reduction: redundancy reduction implies that the target radial distribution is a (truncated) -distribution which has only very little probability mass close to zero. Therefore, the radial rescaling function must map the substantial portion of low contrast values in the empirical distribution upwards in order to match the -distribution. This results in the immediate non-zero onset. This is a pronounced mismatch to the typical contrast response curves measured in cortical neurons (see Figure 2 in [14]). In fact, the addition of more parameters merely leads to a contrast response curve which is more similar to radial factorization (Figure 1, inset, black) which does not have a plausible shape, too. Therefore, we dismiss the option of adding more parameters to the Naka-Rushton function and turn to the option in which is allowed to dynamically adapt to the ambient contrast level.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Simulated eye movements and adapted contrast distributions.

A: Simulated eye movements on a image from the van Hateren database [31]. Local microsaccades are simulated with Brownian motion with a standard deviation of px. In this example, patches are extracted around the fixation location and whitened. B: Values of for the extracted patches plotted along the -axis. Vertical offset was manually introduced for visibility. Colors match the ones in A. The different curves are the maximum likelihood Naka-Rushton distributions estimated from the data points of the same color.

https://doi.org/10.1371/journal.pcbi.1002889.g002

Dynamic divisive normalization.

Previous studies found that single neurons adapt to the ambient contrast level via horizontal shifts of their contrast response curve along the log-contrast axis [8], [14]. In the divisive normalization model, this shift is realized by changes in the half-saturation constant . This means, however, that there is not a single static divisive normalization mechanism, but a whole continuum whose elements differ by the value of (Figure 2). This is equivalent to a continuum of Naka-Rushton distributions which can be adapted to the ambient contrast level by changing the value of . Since this kind of adaptation increases the degrees of freedom, it could also lead to a better redundancy reduction performance.

In order to investigate adaptation to the local contrast in a meaningful way, we used a simple model of saccades and micro-saccades on natural images to sample fixation locations and their corresponding filter responses (see Methods). Previous studies on redundancy reduction with divisive normalization [9], [11], [12] ignored both the structure imposed by fixations between saccades in natural viewing conditions, and the adaptation of neural contrast response curves to the ambient contrast level via the adaptation of [14]. Figure 2 shows an example of simulated eye movements on a natural image from the van Hateren database. For each sample location, we computed the corresponding values of and fitted a Naka-Rushton distribution to it. The right hand side show the resulting Naka-Rushton distributions. One can see that the mode of the distribution shifts with the location of the data, which itself depends on the ambient contrast of the fixation location.

A dynamically adapting predicts that the distribution of across time should be well fit by a mixture of Naka-Rushton distributions. Let (we use to emphasize that the radial distribution is a univariate density and not a multivariate density on ), then averaged over all time points , the distribution of is given by(5)where denotes a single Naka-Rushton distribution at a specific point in time.

We fitted such a mixture distribution to samples from simulated eye movements (see Methods). Figure 3a shows that the mixture of Naka-Rushton distributions fits the empirical data very well, thus confirming the possibility that a dynamic divisive normalization mechanism may be used to achieve optimal redundancy reduction.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Radial distribution and redundancy reduction achieved by the dynamically adapting model.

A: Histogram of for natural image patches sampled with simulated eye movements: The distribution predicted by the dynamically adapting model closely matches the empirical distribution. B: Same as in Fig. 1B but for simulated eye movement data. The dynamically adapting achieves an almost optimal redundancy reduction performance. C: Each colored line shows the distribution of a random variable from 3A transformed with a Naka-Rushton function. Different colors correspond to different values of . The dashed curve corresponds to a truncated -distribution. A mixture of the colored distributions cannot resemble the truncated -distribution since there will either be peaks on the left or the right of the dashed distribution that cannot be canceled by other mixture components.

https://doi.org/10.1371/journal.pcbi.1002889.g003

The next step is to find an explicit dynamic adaptation mechanism that can achieve optimal redundancy reduction. To this end, we sought for a way to adapt such that the redundancies between the output responses were small. Our temporally adapting mechanism chooses the current based on the recent stimulation history by using correlations between the contrast values at consecutive time steps. We estimated for the present set of filter responses from the immediately preceding responses by sampling from a -distribution whose parameters were determined by the mean and the variance of the posterior which was derived from the mixture distribution above (see Methods). We found that this temporal adaptation mechanism significantly decreased the amount of residual redundancies to about (Figure 3B). Note that the proposed mechanism is a simple heuristic that does not commit to a particular biophysical implementation of the adaptation, but it demonstrates that there is at least one mechanism that can perform well under realistic conditions a neural system would face.

Looking at the joint dynamics of and its (Figure 4) we find them to be strongly and positively correlated. Therefore, a higher value of is accompanied by a higher value of . This is analogous to the adaptation of neural contrast response curves observed in vivo where a higher contrast (higher ) shifts the contrast response curve to the right (higher ), and vice versa [14].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Dynamics of the adaptive .

The scatter plot shows the values of plotted against the used to transform in the dynamic divisive normalization model. The two values are clearly correlated. This indicates that the shift of the contrast response curve, which is controlled by , tracks the ambient contrast level, which is proportional to . Single elements in the plot are colored according to the quantile the value of falls in. When the ambient contrast level changes abruptly (e.g. when a saccade is made), this value is large. If the ambient contrast level is relatively stable (e.g. during fixation), this value is small. In those situations (blue dots), and exhibit the strongest proportionality.

https://doi.org/10.1371/journal.pcbi.1002889.g004

In order to demonstrate that improved redundancy reduction is a true adaptation mechanism which relies on correlations between temporally subsequent sample, we need to preclude the possibility that can be sampled independently (i.e. context independent). For strong redundancy reduction, the normalized responses should follow a (possibly truncated) -distribution (see Methods). The history-independent choice of predicts that this truncated -distribution should be expressible as a mixture of distributions that result from transforming random variables, that follow a mixture of Naka-Rushton distributions from Figure 3C, with Naka-Rushton functions for different values of (see Methods for the derivation). We transformed the input distribution with Naka-Rushton functions that differed in the value of (Figure 3C, colored lines). Different colors in Figure 3C refer to different values of . If was history-independent, a positively weighted average of the colored distributions should be able to yield a truncated -distribution (Figure 3C, dashed line). It is obvious that this is not possible. Every component will either add a tail to the left of the -distribution or a peak to the right of it. Since distributions can only be added with non-negative weight in a mixture, there is no way that one distribution can make up for a tail or peak introduced by another. Therefore, cannot be chosen independently of the preceding stimulation, but critically relies on exploiting the temporal correlation structure in the input.

van Hateren data.

For the static experiments, we used randomly sampled patches from the van Hateren database [31]. For all experiments we used the logarithm of the raw light intensities. We sampled pairs of training and test sets of patches which we centered on the pixel mean.

For the simulated eye movements, we also used pairs of training and test sets. For the sampling procedure, we repeated the following steps until samples were drawn: We first drew an image randomly from the van Hateren database. For each image, we simulated ten saccades to random locations in that image. For each saccade location which was uniformly drawn over the entire image, we determined the number of patches to be sampled from around that location by where was the assumed sampling frequency and was a sample from an exponential distribution with average fixation time (i.e. ). The actual locations of the patches were determined by Brownian motion starting at the saccade location and then propagating with a diffusion constant of . This means that each patch location was drawn relative to the previous one based on an isotropic Gaussian centered at the current location with a standard deviation of .

Kienzle data.

The van Hateren database is a standard dataset for static natural image statistics. To make sure that our results also hold for real fixations, we sampled data from the images used by Kienzle et al. [29]. We computed the and percentiles, as well as the width of the interval between them, for both datasets for Figure 5.

We constructed two datasets: One where the patches were uniformly drawn from the images, and one where we again used Brownian motion with a similar standard deviation around human fixation spots to simulate human fixational data. We applied the same preprocessing as for the van Hateren data: centering and whitening.

Filters.

The receptive fields of our model neurons, i.e. the linear filters of our models, are given by the rows of a matrix . In summary, the filters are obtained by (i) projecting the data onto the dimensional subspace that is insensitive to the DC component in the image patches, (ii) performing dimensionality reduction and whitening using principal component analysis, and (iii) training an independent subspace analysis algorithm (ISA) to obtain :

1. The projection of the data onto the dimensional subspace that is insensitive to the DC component is achieved via the matrix . This matrix is a fixed matrix for which the coefficients in each row sum to zero and all rows are mutually orthogonal. The matrix we used has been obtained via a QR-decomposition as described in the Methods Section of [7].
2. The dimensionality reduction and whitening is achieved by . The matrix contains the principal components of such that . As it is common practice, we kept only the first principal components to avoid “noisy” high frequency filters. However, our analysis would also be valid and lead to the same conclusions if we kept the full set of filters.
3. The last matrix is constrained to be an orthogonal matrix because the covariance of whitened data remains white under orthogonal transformations. This additional degree of freedom is used by Independent Subspace Analysis (see below) to optimize the filter shapes for redundancy reduction beyond removing second-order correlations. While the matrix has a large effect on the particular filter shapes, the same results would have been obtained with any type of whitening filter, i.e. for any orthogonal matrix , because they only differ by an orthogonal rotation. Since we use the Euclidean norm in the divisive normalization model, the rotation would not change the norm of the filter responses and therefore all radial distributions would be the same. The only aspect in our analysis for which the filter choice would make a (small) difference is the multi-information of the raw filter responses. When using ICA filter, the multi-information could be a bit lower. However, since even for rather drastic changes of filter shapes (within the class of whitening filters) there is only a small effect on redundancy reduction [6], the particular choice of filter shapes does not affect any of our conclusions. The same is true for any choice of parametric filters as long as the covariance matrix of the filter responses is proportional to the identity matrix. Since the second-order correlations provide the dominant contribution to the multi-information any substantial deviation from the class of whitening filters is likely to yield suboptimal results.

The independent subspace analysis (with two-dimensional subspaces) used to obtain the matrix is based on the model by Hyvärinen [16]:(7)where denotes the list of free parameters for each . More specifically, consists of the value for the -norm and the parameters of the radial distribution for each of the -spherically symmetric distributions. Each single was chosen to be a two-dimensional -spherically symmetric distribution [32]with a radial -distribution with shape and scale . Therefore, the parameters were given by . In the denominator, denotes the surface area of the -norm unit sphere in two dimensions [32]. During training, we first fixed ; after initial convergence, we retrained the model with free and .

The likelihood of the data under equation (7) was optimized by alternating between optimizing for fixed , and optimizing the for fixed . The gradient ascent on the log-likelihood of over the orthogonal group used the backprojection method by Manton [19], [33], [34]. Optimizing over yields filter pairs that resemble quadrature pairs like in the energy model of complex cells [17], [18].

Optimal contrast gain control: radial factorization.

In the following we describe the general mechanism of radial factorization. The spherical symmetric case mostly used in this study is obtained by setting .

Radial factorization is the optimal redundancy reduction mechanism for -spherically symmetric distributed data [11], [32]. Samples from -spherically symmetric distributions with identical -norm are uniformly distributed on the -sphere with that radius. A radial distribution determines how likely it is that a data point is drawn from an -sphere with that specific radius. Since the distribution on the sphere is uniform for any -spherically symmetric distribution, the radial distribution determines the specific type of distribution. For example, and yields an isotropic Gaussian since the Gaussian distribution is spherically symmetric () and has a radial -distribution (). One can show that, for a fixed value of , there is only one type of radial distribution such that the joint distribution is factorial [13]. For this radial distribution is the -distribution corresponding to a joint Gaussian distribution. For , the radial distribution is a generalization of the -distribution and the joint distribution is the so called -generalized Normal [35].

Radial factorization is a mapping on the -norm of the data points that transforms a given source -spherically symmetric distribution into a -generalized Normal. To this end, it first models the distribution of with a flexible distribution and then nonlinearly rescales such that the radial distribution becomes a generalized -distribution. This is achieved via histogram equalization where the denote the respective cumulative distribution functions. On the level of joint responses , radial factorization first normalizes the radius to one and then rescales the data point with the new radius:In our case was chosen to be a mixture of five -distributions.

When determining the optimal redundancy reduction performance on the population response, we set in order to use the same norm as the divisive normalization model. Only when estimating the redundancy of the linear filter responses, we use [11].

Note that the divisive normalization model and the radial factorization model used in this study are invariant with respect to the choice of since the Euclidean norm () is invariant under orthogonal transforms. However, the choice of would affect the redundancies in the plain filter responses in Figure 1B. But even if we had chosen a different , i.e. another set of whitening filters, the redundancy between the coefficients of would not vary much as previous studies have demonstrated [6], [7].

Divisive normalization model and Naka-Rushton distribution.

We use the following divisive normalization transformwhich is the common model for neural contrast gain control [8] and redundancy reduction [9].

Divisive normalization acts on the Euclidean norm of the filter responses . Therefore, divisive normalization can only achieve independence if it outputs a Gaussian random variable. While in radial factorization the target and source distribution were fixed, and the goal was to find a mapping that transforms one into the other, we now fix the mapping to divisive normalization, the target distribution on the normalized response to be Gaussian ( to be -distributed) and search for the corresponding source distribution that would lead to a factorial representation when divisive normalization is applied. Since divisive normalization saturates at , we will actually have to use a truncated -distribution on . becomes the truncation threshold. Note that radial truncation actually introduces some dependencies, but we keep them small by choosing the truncation threshold to be the percentile of the radial -distribution which is approximately . The was chosen to keep the target distribution close to a factorial Gaussian. However, it could still be that another cut-off (value of ) leads to a better redundancy reduction even though the target distribution is less factorial for lower values of (quantiles lower than ). We made sure that this is not the case by choosing different values of , computing the best via a maximum likelihood fit of a Naka-Rushton distribution (see below), and estimating the multi-information in the transformed outputs. We found that the choice of has virtually no effect on the residual multi-infomation (it varies by for and takes its optimum within this interval). Therefore, we kept the choice as it is most similar to the target distribution of radial factorization.

Note also that choosing a Gaussian target distribution does not contradict the finding that cortical firing rates are found to be exponentially distributed [36] since each single response can always be transformed again to be exponentially distributed without changing the redundancy of .

The distribution on such thatis truncated -distributed can be derived by a simple change of variables. In the resulting distributionthe truncation threshold , the half-saturation constant , and the scale of the -distribution become parameters of the model. The parameter of the Naka-Rushton distribution controls the variance of the corresponding Gaussian and was always chosen such that the Gaussian was white with variance one. was determined by the -percentile. The only remaining free parameter of the Naka-Rushton distribution is which simultaneously affects both shape and scale. is the regularized-incomplete-gamma function which accounts for the truncation at . We call the distribution Naka-Rushton distribution and denote it with .

To derive the distribution on for which the extended divisive normalization transformation yields a -distribution, the steps are exactly the same as for the plain divisive normalization transform above. This yieldsfor . The parameters of the distribution are now and .

The parameters for all divisive normalization transforms were estimated via maximum likelihood of the Naka-Rushton distribution on the Euclidean norms of the filter responses to natural image patches. As before, we did not optimize for in the extended Naka-Rushton distribution but fixed it such that the corresponding Gaussian was white.

Dynamically adapting .

For the model with dynamically adapting , we first model the Euclidean norms of the filter responses to the patches from the simulated eye movement data with a mixture of Naka-Rushton distributionsusing EM [37]. denotes the probability that . The values of where chosen in equidistant steps from to .

How much redundancy reduction can be achieved with a dynamically adapting , depends on the dynamics according to which it is selected based on the recent history. While there might be many strategies, we chose a parsimonious one based on the mean and the standard deviation of the posterior over . Our heuristic consists of two steps: First the mean and the standard deviation of the posterior derived from the mixture distribution is approximated with piecewise linear functions and , then we sample used to transform from a -distribution with mean and standard deviation and . This strategy emphasizes that the first two moments of the posterior are the important features for obtaining a good .

In more detail, we evaluated the posteriorof the mixture distribution at equidistant locations between and , computed the posterior mean and standard deviation at those locations, rescaled the standard deviation by , and fitted the piecewise linear functions on the intervals to each set of values. In the first interval, the linear function was constraint to start at zero. From these two functions and , we computed two functions for the scale and the shape of a -distributionvia moment matching. We obtained the value for transforming a value with a Naka-Rushton function by sampling from a -distribution with shape and scale determined by and .

Computation of percentiles for Figure 5.

For the dynamically adapting in Figure 5, we sampled fromand computed the percentiles based on the sampled dataset. For the sampling procedure, we drew from the -distribution with shape and scale computed from and then sampled from the Naka-Rushton distribution with that . We repeated that for all from a test set of simulated eye movement radii. This procedure was carried out for all pairs of training and test sets, and the distributions fitted to them.

For the static case, we sampled data from single Naka-Rushton distributions for different values of and computed the percentiles from the samples.

History-independent choice of .

In the following, let and be the unnormalized and normalized responses at time , respectively, and be the recent history of responses. The underlying generative structure of the model for temporally correlated data is the following: given a fixed history , and are sampled from and . Then, is generated from and through divisive normalization.

For strong redundancy reduction, should follow a truncated -distribution, which means that for given history and , the unnormalized response energy must have a Naka-Rushton distributionbecause normalizing this response via yields a truncated -distribution. Averaged over all histories and half-saturation constants the distribution of is a mixture of Naka-Rushton distributions(8)If depends deterministically on we obtain equation (5).

If could be chosen independently of the preceding history the distribution of would be given bywhere is the marginal distribution of transformed with divisive normalization and a specific value of . Since redundancy reduction requires to be truncated -distributed, can be chosen independently only if the truncated -distribution can be modelled as mixture of the different . Since we assume stationarity, we can drop the index in the equation.