Representation

Consider the case of a population of M units representing the memory of all items seen in a trial. The simultaneous population activity of these units is read during recall to infer the feature of the item of interest. The finiteness of the population, the nature of the representation employed and the influence of noise jointly constitute the limited resource associated with our memory.

In terms of the representation, we assume that units have continuous firing rates, and are tuned to specific combinations of features. Unit i has a preferred angle and colour, with separate tuning widths to each feature, and its mean activity follows a Von Mises curve as shown in Equation (1). We use Bivariate Von Mises [22, 23] tuning curves as they provide a convenient parametrisation of the sensitivity to a pair of angular features.

(1)

Here, ϕ and ψ are respectively the orientation and colour of the item to be represented. θ_m and γ_m are the preferred angle and colour of unit m. τ_1,m and τ_2,m are called concentration parameters, which control the size of the receptive field, as well as the sensitivity of each unit to the different features. Units have continuous valued firing-rate responses, and suffer from independent Gaussian noise about these mean activities. To examine the scaling behaviour of the model, we use a normalization scheme that constrains the mean summed overall network activity induced by any item to be constant as the receptive field concentrations change (although the total activity in the memory grows with the number of items stored). We use independent Gaussian noise for simplicity, although it would be straightforward to examine a more neurally plausible, Poisson, noise model.

Writing x_m as the firing rate of unit m, the population activity x = [x₁,…,x_M]^T is (2)

Depending on the distribution of τ_1,m and τ_2,m, several types of population code can be generated (see Fig. 3 and 10). τ_1,m = τ_2,m = τ ∀i corresponds to a “conjunctive” population code, in which each unit is sensitive to a combination of the two features. Conversely, a “feature” population code employs two subpopulations; one has τ_1,m = τ,τ_2,m = 0, and is sensitive only to the first feature; the other has τ_1,m = 0,τ_2,m = τ, and is only sensitive to the second. We also consider a “mixed” population code including both conjunctive and feature units, and entertain various possibilities for the relative proportions of the two types. This “mixed” population code provides an easy way to parametrise the relative information required to store feature values accurately (these are mostly encoded in the feature sub-population) versus the binding information required to link features together into item-like representations (only encoded by the conjunctive sub-population). Moreover, the different types of population code will require different number of neurons to cover the entire stimulus space appropriately. This will become increasingly important as the number of features increases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Example population codes.

Top: Receptive fields of units (one standard deviation), shown for the three different types of population codes: conjunctive, feature and mixed. Bottom: Activity profile over the entire stimulus space for the two shaded units on the left.

https://doi.org/10.1371/journal.pcbi.1004003.g003

We study the effects of different types of representation on the nature and quality of recall, and show that aspects of human experimental data are better accounted for by population codes that might at first seem sub-optimal.

Storage and recall process

According to our restricted palimpsest memory process (“restricted”, because, as mentioned in the introduction, we do not assume erasure of the palimpsest between storage steps), the noisy population activities associated with all the items are simply summed to produce the final memory. As can be expected, the characteristics of the representation used will determine how readily possible it is to extract items when they are overlaid.

The storage process is depicted in Fig. 1B, in which N items are stored simultaneously. Again, for simplicity, assuming that the final memory suffers from spherical Gaussian noise, we derive: (3) (4)

Here, ϕ_i and ψ_i represent the feature values of item i. x_i is the population representation of item i. Multiple items are then summed to produce the final memory state y_N. Extraction of stored information is based on the memory state y_N, along with any prior information. Examples of memory states for a chosen set of stimuli and population codes are shown in Fig. 4. For completeness, these expressions include two generalizations that we do not consider further here: the terms β_i allow different items to be stored with different strengths in the memory (to accommodate tasks involving explicit attentional instructions); however, here we set β_i = 1 ∀i. The parameter ${\sigma }_y^2$ allows for extra memory noise, but is set to a very small value in our experiments (σ_y = 10⁻⁵).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Recall model and posterior for different population codes.

(A) Example memory states for the different population codes, when three items are stored. Coloured circles indicate the veridical feature values. Left: Conjunctive population code, involving little interaction even between nearby items. Middle: Feature population code. Right: Mixed population code — a few conjunctive units provide just enough binding information to recall the features associated with the appropriate items. (B): Cued posterior probabilities, given the veridical colour to be recalled (the three curves correspond to cueing the three possible colours; vertical bars indicate the true stored orientations). (C) Graphical model representation of the process of recall. The final memory state and colour are observed; the orientation must be inferred.

https://doi.org/10.1371/journal.pcbi.1004003.g004

Having produced this final memory state, the next step is to recall the correct feature based on the recall cue. Bayes optimal recall would require marginalising over the non-target items that were simultaneously presented. Given the final memory state y_N, and a cued feature value (e.g. a colour ψ), this would lead to the posterior distribution over the value of the other feature of this item. However, this marginalisation would be computationally penal, since it would likely require explicitly representing and processing all the non-cued items. Instead, we make the simplifying assumption that only the item to be recalled is explicitly modelled, with the non-targets being collapsed together and treated as background noise. Conceptually, this corresponds to extracting a specific item of interest out of irrelevant noise. This approach was adopted by [21, 24, 25], in the context of retrieval from long-term memory in multistate synapses.

The algorithm is illustrated in Fig. 4. Given a memory state y_N and the cued colour ψ, we compute the posterior distribution over ϕ explicitly (Fig. 4B). No closed-form solution exists for this posterior in general, because of the non-linear transform associated with the population code μ(⋅). Therefore we sample from it using slice sampling [26]. We treat a single sample as the output of recalling a feature from our model for this trial. The use of sampling instead of a maximum likelihood (or MAP) solution has two main consequences: the variance of the posterior has a direct effect on the variance of the recalled orientation, and multi-modal posteriors will reflect situations in which another orientation may be reported in place of the appropriate one.

We formalize this process by writing m_N–1 as the contribution of the noise process to the mean of the final memory state and Σ_N as the contribution of the noise to the full memory covariance, see Fig. 4C. r is the index of the item to be recalled, which we integrate over, as it is unknown during recall.

(5)

(6)

This posterior is usually peaked around the appropriate orientation; however, depending on the population code used and the number of stored memories, additional modes can appear (see Fig. 4B, middle and right). These correspond to the effects of noise and other items on the recall of the item of interest; the latter allows us to study the question of binding.

We now consider various characteristics of our models in the context of visual short-term memory experiments.

Modelling visual working memory experiments

First, the model reproduces the baseline, apparently uniform, component of the distribution of errors (see Fig. 5 upper row, compared to Fig. 2). However, this does not arise from pure random guessing. Rather, a sample is always taken from the posterior distribution given a memory state composed after storing all items. Nevertheless, interactions between items and the overall background noise in the memory imply that the model sometimes samples values away from the target, so producing output resembling guessing. On the lower row of Fig. 5, we see that our model can also reproduce misbinding errors, shown by the over-abundance of small errors towards non-target items values during recall. This central tendency is reduced compared to the experimental data from Fig. 2B, but is still significantly present. In addition, the magenta curve and penumbra represent the distribution of samples from the model when inter-items correlations have been removed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Distribution of errors of the model.

The model is capable of recreating error distributions seen in the literature, such as those shown in Fig. 2. (Top row) Distribution of errors around the target angle. The central bump is at 0^o, showing that recall is normally accurate. The distribution has a non-zero baseline which combines all sources of error. (Bottom row) Distribution of errors relative to all non-target angles. A central tendency in those plots has been interpreted as supporting evidence for the presence of misbinding errors in the responses. Histogram computed on 5000 samples of the model (no standard deviation is shown as all samples are equally probable) The p-values for a resampling analysis of the non-target mixture proportion are shown in each panel. The null hypothesis of no misbinding error can be significantly rejected for all item numbers. The magenta curves represent the resampling-based histograms assuming no misbinding error. Mixed population, M = 200, σ_x = 0.25.

https://doi.org/10.1371/journal.pcbi.1004003.g005

The second experimental observation captured by the model is the decrease in recall precision as a function of the set size, which is the number of stored items. Here we study the precision of recall using the procedure defined by [7]. This involves fitting a mixture of Von Mises components on the recall samples, using a procedure based on the EM algorithm [27]. This mixture model consists of one Von Mises component per item (target or non-target) and a uniform random component. All Von Mises components share a single concentration parameter κ. This mixture model approach turns out to be substantially more robust to outliers than computing the circular standard deviation of the raw errors directly. We refer to κ as the memory fidelity, and show how it depends on set size. In addition to this memory fidelity, two other types of errors are specifically captured by this analysis: misbinding errors, the probability of recalling from a non-target, and random errors, the probability of recalling from the uniform random component. These will be analysed more thoroughly in the Misbinding errors section.

Fig. 6 shows the fit of our model (in green) to human data (dark blue) from [13], where we report the memory fidelity. The shaded region indicates one standard deviation, computed over multiple reruns of the model (or across different subjects for the human data). The smooth decay in performance as set size increases is appropriately captured by our model. This decay arises in our model from the increase in recall noise as the number of stored items increases, but also from interference between items in the memory. We report in both cases the memory fidelity, the concentration κ of the Von Mises component obtained from fitting the mixture model on the responses from human subjects and our model. Here, we used a mixed population code, optimizing the fit to the experimental curve by adjusting the ratio of conjunctive to feature units and the encoding noise σ_x, for a population of M = 200 units (see Methods for the optimisation procedure). The model does not capture the reduced decay rate for 4 and 6 items to its full extent. However, this is a rather specific characteristic of this dataset. For comparison, the inset in Fig. 6 shows the fit of our model to the data from [1]. In this case, the model captures the memory fidelity dependence more accurately.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 6. Memory curve fit.

Mixed population code. This shows a qualitative fit of the model (green; the shaded area represents one standard deviation) to the human experimental data (blue; data from [7]). M = 144, conjunctivity ratio = 0.85, σ_x = 0.1). Inset: similar data fits, for [1] (M = 200, ratio = 0.85, σ_x = 0.4). Observe the different decrease in memory fidelity for an increasing number of items.

https://doi.org/10.1371/journal.pcbi.1004003.g006

Fisher information analysis

A common theoretical technique used to study the representational capacity of a population code is the Fisher information (FI), which, via the Cramer-Rao lower bound, limits the precision of any estimator based on the output of the code [28–30]. If the posterior distribution in our model can be well approximated as being Gaussian, the FI will accurately characterize memory fidelity, allowing us to examine the effects of different parameters and representations.

In our case, the FI should readily be able to characterise the spread of the errors around the correct target value when a single item is stored (when there is sufficient signal [31, 32]). In this section (and the Supplementary information), we study this case.

When there are multiple items, complexity arises from the fact that errors are distributed around both the target feature value and misbound, non-target, features, with the posterior distribution being multi-modal (and therefore not Gaussian). Nevertheless, as we will see in the next section, the Fisher information, calculated assuming storage of just a single item, can still characterise the memory fidelity around each mode.

Assuming a population code with Gaussian noise and signal-independent noise, the Fisher information is defined as follows: (7)

where μ is the mean response of the population, and C the covariance of the population response. In our case, θ = [ϕ ψ]^T, so the Fisher information is a 2-by-2 matrix.

We can easily compute it for the single item case (see Methods), obtaining: (8)

In the large population limit in which preferred values have density ρ, it is possible to obtain an analytical closed-form solution for this equation which is easier to interpret, (see Section, 1 in S1 Text for the complete derivation) (9) where f(τ₁,τ₂) is an increasing, approximately power-law, function of τ₁ and τ₂ that is given explicitly in the Supplementary information. These values depend on the parameters of the code just as one would expect from classical results for non-circular, uni-dimensional, receptive fields [31, 33]: Increasing the concentration τ increases the Fisher information. This is easy to interpret, as narrower receptive fields will be more precise in their encoding of the features. Similarly, increasing the coverage density has the same effect, as more units are available to store information. Finally, the item encoding noise σ decreases the Fisher information, as less signal can be extracted from the final memory.

The Cramer-Rao lower bound transforms the Fisher information into an estimate of performance in the task. Fig. 7 compares the Fisher information for the finite and large population limit with the curvature of the log-posterior at its maximum value (as in the definition of the Fisher information); and to the variance of samples given a memory state. We use again the memory fidelity, by fitting the mixture model onto model samples. We convert this memory fidelity from its units of κ into an inverse variance, by converting κ to the σ² of the approximated Wrapped Gaussian (see Methods for details). Note that the latter procedure, reporting the variance of samples given a memory state, generates a doubly-stochastic process, hence increasing the variance observed. It can be shown that if the posterior is close to being Gaussian, the variance from those samples will be twice that of the curvature of the log-posterior considered above (see Section, 2 in S1 Text). This is shown by the dashed light blue bar.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 7. Fisher information fit for one object.

Comparison between similar metrics: the memory fidelity (fitted κ) of single samples collected for different memory states associated with a single memory state (double the value is shown in dashed blue to take account of the doubly stochastic nature of single sampling); the theoretical Fisher information derived in (8); the large M limit for the Fisher information (35); the average inverse variance of samples from the posterior distribution; and the average curvature of the log-posterior at its maximum. This refers to a Conjunctive population code with M = 200, τ = 4, σ_x = 0.1, σ_y = 10⁻⁵ and 500 samples.

https://doi.org/10.1371/journal.pcbi.1004003.g007

We see that they are all similar on average; the most important being the match between samples from our model and the Fisher information analysis.

When more than one item is stored, errors arise from two sources: variance around a mode, and mistakenly reporting the wrong mode (misbinding error). One can adapt the Fisher information analysis to characterize the former, capturing the variability about each mode, conditioned on the fact that the posterior is close to Gaussian. However, it does not capture the component of variance coming from misbinding errors. Further analysis that quantifies both sources of variability will be required to account in a theoretical manner for the full distribution of errors observed in the data.

Misbinding errors

As noted above, several groups have shown that a significant proportion of the errors made by humans can be explained as arising from “misbinding”, i.e., recalling (at least approximately) the appropriate feature of an inappropriate item, i.e., of a non-target item that also formed part of the array. Such mistakes are shown in Fig. 2B, and could contribute to the appearance of a baseline of errors seen in experiments (Fig. 2A), since these stimuli are drawn randomly from a uniform distribution across all possible angles [7].

The proportion of errors classified as misbinding varies between experiments [1, 7, 13, 34–36]. Although some studies seem to show none at all [37]; in others, they are reported as making up to 30% of all errors when the memory load is high. Misbinding has not been well addressed in the theoretical literature on visual working memory, since current models typically assume distinct subpopulations storing the different items, hence removing any possibility for direct misbinding errors.

Our model uses a single population of units for storage, and so can account for misbinding when the posterior distribution (see Equation 6) becomes multimodal. This usually happens when there is insufficient information in the representation of items to bind the features together (i.e., when the codes are insufficiently conjunctive); the different modes arise from the different items that are stored. The relative heights of the modes of the posterior determine the frequency of misbinding errors. The classical conjunctive population code represents one extreme, offering near perfect binding information, being limited only by the size of each unit’s receptive field. Feature-based population codes, on the other hand, constitute the other extreme: they do not perform binding at all.

For a mixed population code, Fig. 8 shows that the proportion of conjunctive units has a strong effect on misbinding errors and posterior multimodality. We construct a situation with two possible angles, $\pm \frac{3\pi }{5}$, where $\frac{3\pi }{5}$ is to be recalled. In this case, using a mixed population code with around 40% of conjunctive units dramatically reduces the number of misbinding errors produced by the model. This proportion will depend on the number of items to be stored, as more items will require more precise binding information.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 8. Misbinding errors when varying the proportion of conjunctive units.

These plots are based on a mixed population code recalling the orientation of one of two stored items (the correct value is indicated by the red vertical bar). There is a fixed total number of 200 units; the ratio of feature to conjunctive units increases for the graphs going from top to bottom. Left: Average (and standard deviation, shown by the penumbra) of the log-posterior distributions over orientation, given the stored memory states averaged over 1000 instantiations of the noise. If the population code only consists of feature units, the posterior comprises two equal modes the incorrect mode disappears as the fraction of conjunctive units increases. However, feature units improve the localization; as their number decreases, the widths of the posterior modes increases. Right: Distribution of 1000 sampled responses, showing how misbinding errors tend to disappear when sufficient conjunctive information is available. The red (respectively green) vertical lines indicate the target (respectively non-target) item orientation. The red Gaussian curve shows the probabilit distribution of a Gaussian distribution centred at the correct target value and with a standard deviation derived from the Fisher information of the associated population code.

https://doi.org/10.1371/journal.pcbi.1004003.g008

The widths of the posterior modes depend directly on the amount of information provided by feature and conjunctive units. Feature units are more efficient than conjunctive units at representing single features, and so the cost of reducing misbinding by increasing the proportion of conjunctive units is to increase the width of the posterior over the recalled feature. This can be seen in Fig. 9, where we fitted the mixture model presented in the Modelling visual working memory experiments section to the recall samples, we report the concentration (an inverse width) of the Von Mises component in panel A, and the mixture proportions in panel B.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 9. Memory fidelity and mixture proportions as a function of the ratio of conjunctive units.

(A) Standard deviation of the Von Mises component (in blue) from the mixture model fitted to samples of the model shown in Fig. 8 as a function of the fraction of conjunctive units. The (theoretically-calculated) Fisher information is shown in green for the associated population codes. (B) Mixture proportions of the mixture model fitted on the model samples. This metric is less sensitive to random fluctuations of the samples, and shows that if 50% of the units are conjunctive, then 75% of responses will be correctly associated with the appropriate target angle. (C) P-value for a resampling-based estimation of the probability of the non-target mixture proportion to be different than zero. We see that the null hypothesis of the non-target mixture proportion being zero can be rejected from 70% of conjunctive units and less.

https://doi.org/10.1371/journal.pcbi.1004003.g009

Fig. 9A confirms the relationships of the width of the posterior mode with the proportion of conjunctive units. The concentration of the Von Mises component (in blue), closely follows the theoretical Fisher information (in green), although overestimating it. The Fisher information provides a good local estimate of the variability around a mode, as can be seen in Fig. 8 on the right, where we overlap in red a Von Mises PDF with a concentration predicted from the Fisher information (with a height set to be aligned with the histogram of the right mode).

The mixture proportions corresponding to the target, non-target and random responses are shown in Fig. 9B as a function of the fraction of conjunctive units. They show that for around 50% or more conjunctive units, more than 75% of responses are on target. The mixture proportion associated with the random component appears to be overestimated, compared visually to the distribution of the samples of Fig. 8. However, the mixture model well characterizes the proportion of misbinding errors.

Finally, as a last check, we verified that the mixture model estimates of non-target proportions were reliable. To do this, we performed a resampling-based analysis of the mixture of non-target responses, by randomizing the assumed locations of non-target angles and re-fitting the mixture model. Using the empirical cumulative distribution over those samples, we could then compute a p-value for the null hypothesis that the mixture probability for non-target would be zero. The results are shown in Fig. 9C, where the p-values as a function of the proportion of conjunctive units in the mixed population code are reported. For proportions of conjunctive units below 70%, the null hypothesis can be rejected significantly (at a 5% level), consistent with the presence of misbinding errors.

We applied this resampling analysis to the human experimental data shown in Fig. 2, as well as to our model’s fit to these data (Fig. 5). The p-values for the data collapsed across subjects are shown above the histograms of biases towards non-target angles; they all are significant. Redoing the analysis per subject indicates that for 2 items, 8 out of 12 subjects show significant misbinding errors; for 4 items, 7 out of 12 are significant; and finally for, 6 items 10 out of 12 subjects show misbinding errors.

Hierarchical population code

In addition to the “mixed” population code that we have so far described, one might imagine an “hierarchical” population code, shown in Fig. 10. This uses two layers, the lower of which can either be a conjunctive or feature population, parametrised as described above. Units in the higher layer are randomly connected to a subset of the lower layer units, with activities that are a nonlinear (sigmoidal) function of the weighted sum of the sampled units’ activities. More formally, where μ⁽¹⁾ is the mean response of the lower layer, σ_Θ the rectified linear function with threshold Θ: (10) (11) (12) (13)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 10. Hierarchical population code.

The hierarchical code comprises two layers: the lower layer receives the input, and is randomly connected to the upper one, which provides (possibly additional) binding information. Bottom: layer one consisting of either a feature population code or a conjunctive population code. Receptive fields of units of a feature population code are shown (one standard deviation). Top: effective receptive fields of three layer two units are shown. Layer two units randomly sample a subset of the activity of layer one units, and pass a weighted sum of their inputs through a nonlinearity.

https://doi.org/10.1371/journal.pcbi.1004003.g010

Such an hierarchical code can be considered an abstract representation of a layered neural architecture [38].

The “mixed” and “hierarchical” population codes were specifically introduced to parametrise subtly different forms of binding, controlled by the ratio of binding to non-binding units. In the “mixed” population code, conjunctive units introduce binding information independently from the rest of the feature units. In the “hierarchical” population code, the random layer two units bind together the activity of layer one units, generating seemingly arbitrary combinations of feature values, yet providing sufficient conjunctive information. It allows us to check how structured the binding information should be for the results to hold.

Fig. 11 shows the behaviour of recall for a hierarchical population code based on a feature population code at the lower layer. The total number of units was fixed (at M = 200); the ratio of upper to lower units was varied. The optimal arrangement changes markedly when multiple items must be stored. Having few random binding units is very efficient in the single item case, but this breaks down completely when multiple items are stored and interfere with each other. The dependence of the memory fidelity on the ratio of upper to lower units is similar for increasing number of items, with the exception of the overall scale. Unsurprisingly, memory fidelity is lower when increasing the number of items and conjunctivity, see Fig. 11A. As shown in Fig. 11B, the probability of the response being related to the correct target changes completely going from one to many items, with non-target responses becoming prevalent for small ratios of upper to lower units. Moreover, there is an optimal ratio of upper to lower units when storing multiple items, if one tries to optimise the proportion of correct target angle recall.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 11. Memory fidelity and misbinding errors as function of conjunctivity in hierarchical population code.

Left: Memory fidelity based on model samples, while varying the ratio of lower to upper layer units in a hierarchical population code with a constant number of 200 units. The number of (randomly placed) items increases from top to bottom. The memory fidelity decays with increasing item number and conjunctivity. Right: Mixture proportions based on model samples. For a single item, the correct target angle is always retrieved (blue curve). The drop for high ratio of upper to lower layer is expected, as few units are left in the lower layer to represent the item appropriately. For increasing numbers of items, nontarget responses are prevalent (green curve), but including a suitable proportion of upper layer units does allow the appropriate angle to be retrieved. Random responses are marginal with the parameters used here. M = 200, σ_x = 0.2.

https://doi.org/10.1371/journal.pcbi.1004003.g011

Fig. 12 shows the fit of the memory fidelity to the experiments in [1, 7], as was done in Fig. 6 for the mixed population code. Despite being drastically different in its implementation of conjunctivity, it provides a good fit to the experimental data. The hierarchical code is able to capture the trend of decay in both experiments to a greater extend than the mixed population code (main plot shows a fit to [7], inset shows a fit to [1]). However, the fit for 4 and 5 items for [1] does show discrepancies with the experimental data. The optimal parameters obtained for those fits resemble those for the mixed population code, namely a high ratio of higher-level binding units and large input noise. These render promising this class of hierarchical codes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 12. Memory curve fit for hierarchical population code.

Model fit (green; the penumbra represents one standard deviation) to the human experimental data (blue; data from [7]). These qualitative fits are similar to those obtained for a mixed population code (see Fig. 6), despite the significantly different implementation. (M = 200, ratio = 0.9, σ_x = 0.3). Inset: fit for [1]. Notice the difference in performance for large number of items. (M = 200, ratio = 0.9, σ_x = 0.55)

https://doi.org/10.1371/journal.pcbi.1004003.g012

Comparisons of population codes

Effects on experimental data fits.

The patterns of errors arising from specific choices of population codes can be used to help discriminate between different representations. Misbinding, which we quantify via the mixture model approach of [7], is of particular value, since, as observed, it is rare for conjunctive codes; but ubiquitous for feature codes. We therefore compare the misbinding exhibited by human subjects with the output of our model based on different population codes (see Methods for details about the optimisation).

As can be seen in Fig. 13, there are clear differences in the mixture weights associated with misbinding errors, errors arising from local deviations from the correct feature to be recalled, and the uniform component.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 13. Error types for different population codes.

The graphs quantify different sorts of error in terms of the weights in a mixture model capturing local variability around an item, misbinding errors and random choices [7]. Human experimental curves are shown on the bottom right. This shows how misbinding errors are crucial components to fit human performance. Conjunctive population code: M = 225 units, σ_x = 0.3, Feature population code: M = 100 units, σ_x = 0.08, Mixed population code: M = 144 units, conjunctivity ratio = 0.85, σ_x = 0.1

https://doi.org/10.1371/journal.pcbi.1004003.g013

As expected, the feature code makes a large number of misbinding errors when more than one item is stored. On the other hand, the conjunctive code makes only a few errors that appear to arise from random guesses. Misbinding errors are highly unlikely in this configuration. In total, a mixed code provides a better fit to the human data, matching the increase in non-target responses as well as a baseline random response rate.

The canary.

This analysis suggests that stimuli specifically designed to induce patterns of misbinding could be useful for understanding representations in population codes. Consider three stimuli, arranged on a diagonal, separated by a variable distance in feature-space (illustrated in Fig. 14). These create clear interference patterns for feature codes, with multi-modal posteriors and misbinding errors. These errors will be expected to change as a function of the characteristics of the population code. We therefore call this stimuli pattern the “canary” in honour of its capacity to reveal such characteristics.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 14. Stimulus pattern to induce misbinding.

Feature-space representation of three stimuli used to study misbinding errors and characteristics of the population codes. Three items are separated by a distance Δx. This set of items will generate interference patterns as shown by the dotted lines. The circles represent one standard deviation of the receptive field response levels. The green circles represent a population code in which the three stimuli are well separated. The blue circle represents a code for which all the stimuli lie inside a single receptive field and would generate misbinding errors. The target is randomly chosen on each trial as one of the three items.

https://doi.org/10.1371/journal.pcbi.1004003.g014

In particular, by making the stimuli close to each other in feature space, this pattern allows intra-receptive field misbinding to be examined. This happens when the pattern lies entirely in a single receptive field of a conjunctive unit, and can thus provide a somewhat crude and indirect measure of the receptive field size of a mixed conjunctive code. Note, though, that hierarchical conjunctive codes cannot be expected to have such a simple signature; and indeed even mixed codes are ultimately likely to be multi-scale in character.

In Fig. 15 (left panels), we show what happens for a mixed population codes. We report how the parameters of the mixture model we considered before vary with conjunctivity in several conditions, using a population code of 200 units, and allowing the ratio of conjunctive to feature units to vary from 0 to 1 (corresponding to full-feature and full-conjunctive, respectively). We set the item noise σ_x = 0.25, a level compatible with experimental data fits, and show two characteristic distances between stimuli, Δx = {0.22,1} rad. The goal is to recall one of the three items, randomly chosen on different trials. We characterize the errors using the mixture model presented before and report the mixture proportions and the fitted κ from the Von Mises component.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 15. Recall of stimuli shown in Fig. 14.

100 individual samples from the model are generated for specific parameters (M = 200, σ_x = 0.25), mixed (left) or hierarchical (right) population codes and inter-stimulus distances Δx = {0.22, 1} rad. Shaded regions correspond to one standard deviation around the mean over 10 repetitions. Top row: Fitted mixture proportions from a mixture model (with one Von Mises component per target/non-target and a random uniform component, similar to [7]). For small Δx, no amount of conjunctivity can improve the results, indicating intra-receptive field misbinding. For large Δx, there is a change from non-target to target responses as the proportion of conjunctive units increases. The target is randomly chosen for each trial.. Second row: Width of the Von Mises component of the mixture model (represented as the standard deviation corresponding to the fitted concentration κ). The dotted black line corresponds to the distance Δx between items in the stimuli pattern.

https://doi.org/10.1371/journal.pcbi.1004003.g015

For the large separation, Δx = 1.0, the mixed population code behaves in a regular manner as the degree of conjunctivity increases. For a feature-based population, recall suffers from much misbinding; it is only when more than 50% of the units are conjunctive that correct binding typically occurs. The mixed population code increases rapidly at around 70% of conjunctive units and saturates.

The outcome is quite different for the small separation Δx = 0.2. In this case, no amount of conjunctivity can help the discrimination between the three stimuli. This corresponds to a situation in which intra-receptive field misbinding occurs. Even for a fully conjunctive population code, the size of the receptive field is larger than the distance between two items (M = 200, τ = 5.5 ⇒ width of 0.44 rad for one standard deviation of a receptive field).

For the single-scale receptive fields that we employed to create the mixed population code, it is possible to recover the scale from the error patterns as a function of the separation between the stimuli. This is shown in Fig. 16 for two mixed population, with 50% and 98% of conjunctive units. This plots the target (blue) and non-target (green) mixture probabilities (normalized by their joint sum). These start at the same value, but diverge after the point when conjunctive information becomes available and hence when intra-receptive field misbinding become less prevalent. The black vertical dotted line indicates half the size of the receptive field for the conjunctive subpopulation—misbindings stop being prevalent once the stimuli covers multiple receptive fields. The red line for the case of 98% conjunctive units corresponds to two times the size of the receptive field for the conjunctive subpopulation. Once this point is reached, each stimulus lies in its own receptive field, so misbinding should not happen. This is again in agreement with the results, with very few non-target responses in this regime.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 16. Patterns of errors as a function of stimulus separation for different proportions of conjunctive units.

This shows data as in Fig. 15, but as a function of the varying distances in radians between stimuli in the diagonal pattern, for two mixed populations with 50% and 98% conjunctivity. We compute the ratio between the target mixture proportion and the sum of the target and non-target mixture proportions (in blue). We do the same for a non-target mixture proportion (in green). The black vertical bars show half the size of a conjunctive receptive for each population. We see that for separations smaller than the size of a receptive field, misbinding errors are prevalent. This changes as soon as the pattern of stimuli covers more than one receptive field. The vertical red dashed bar shows twice the size of a receptive field. In this situation, each stimulus occupies one receptive field, and misbinding should rarely occur.

https://doi.org/10.1371/journal.pcbi.1004003.g016

We originally expected a hierarchical population code to perform differently, since it encodes binding information in a quite different manner. However, surprisingly, we find consistent results, as can be seen in Fig. 15 (right panels). Again, we show two characteristic distances between stimuli in the canary pattern, Δx = {0.15,0.8} rad (we chose different values than in the mixed code situation, as the population codes behave slightly differently).

When the separation is large, the hierarchical code also behaves in a regular fashion similar to that of the mixed code as the degree of conjunctivity increases. When conjunctivity is low, the memory performs poorly, as no binding information is present. However, as conjunctivity increases, performance does as well. Interestingly, performance with a hierarchical code increases monotonically with conjunctivity (before dropping sharply when the input lower layer population decreases past the required precision needed to discriminate the stimuli). This architecture uses conjunctive information quite effectively, but does not attain the same maximum performance.

The situation is less clear for a small distance between stimuli. Having a large proportion of conjunctive units is actually detrimental in this case, as the input lower layer decreases in size, and thus the encoding precision decreases with it. Hence there is an optimal proportion of conjunctive units for a given required minimum discrimination. The smallest distance for which the target and non-target responses can be discriminated when analysing the results is Δx ≈ 0.30, using a hierarchical code with a conjunctivity of 80% (see Section 3 in S1 Text). Hence the hierarchical code seems to discriminate smaller patterns for a given population size, which is surprising for such a crude representation of a hierarchical representation.

Thus we find that even this simple stimulus pattern can provide something of a formal window into misbinding and the structure of receptive fields.

Representation

We assume continuous firing-rate style units. They have Bivariate Von Mises tuning curves, corrupted by isotropic additive Gaussian noise: (14)

ϕ and ψ are respectively the orientation and colour of the item to be represented. θ_m and γ_m identify the preferred angle and colour of unit i. τ_1,m and τ_2,m control the size of the receptive field, as well as the sensitivity of each unit to the different features.

Let the population firing rate state be x = [x₁,…,x_M]^T, x_m. The firing rate of unit m is: (15)

Differences in the choices of τ_1,m and τ_2,m across the population will generate different types of representation.

The hierarchical population code is defined as follows, with μ⁽¹⁾ being the mean response of the lower layer.

(16)

(17)

(18)

(19)

The receptive field sizes were set automatically to achieve maximum coverage given a population of M units. Given a fixed number of units with preferred stimuli arranged uniformly over the feature space, the receptive field sizes were modified such that one standard deviation of the receptive field would cover the space uniformly without redundancy.

In the case of a conjunctive code, we have: where g_{σ → τ} converts the standard deviation of a Wrapped Gaussian into the τ of a Von Mises. No closed-form solution of g_{σ → τ} exists; it can be computed numerically by finding the argmin_τ ${(\exp (-\frac{{\sigma }^2}{2})-\frac{I_1(\tau )}{I_0(\tau )})}^2$.

For a feature code, we set: (20) (21) Where τ₁ and τ₂ correspond to the two receptive field sizes of one subpopulations (here assumed to be sensitive along the τ₁ direction).

Storage and recall process

The storage process for N items is probabilistic and follows the following model: (22) (23)

x_i is the representation of item i by the population code. ϕ_i and ψ_i represent the feature values of item i. Multiple items are summed to produce the final memory state y_N, which is, in turn, corrupted by additional, independent, Gaussian, noise. β_i models different strengths of storage in the memory (to accommodate tasks involving explicit attentional instructions).

Recall is based on the simplifying assumption that a single item is modelled, while others are collapsed into a single source of noise. m_N-1 is the contribution of the noise process to the mean of the final memory state and Σ_N is the contribution of the noise to the full memory covariance. r is the index of the item to be recalled, which we integrate over as it is unknown during recall. The posterior over the feature ϕ to be recalled is defined as follows: (24) (25)

We use uniform prior distributions over r and ϕ (circularly uniform for ϕ).

The collapsed noise mean m_N-1 and covariance Σ_N can be estimated from random samples of the storage process. m_N-1 is the mean memory built from N−1, marginalising over feature values: (26) (27)

Similarly, Σ_N is the covariance of N items, marginalising over feature values. We obtain estimates by sampling 5000 memory items from the storage process before estimating those two empirical estimates.

We use a slice sampling scheme to obtain samples of ϕ given a memory state. In addition to the classical slice sampling algorithm, we introduce Metropolis-Hastings jumps, which can randomly set the sampler in another part of the state space. This allows to jump between modes in a multi-modal posterior setting. The jump probability is set to 10% and a jump is accepted depending on a Metropolis-Hastings acceptance ratio. We discard the first 500 samples as burn-in steps for the slice sampler. We perform step-out and shrinkage to determine the slice width (initially set to $w=\frac{\pi }{40}$) [26]. We constrain the sampler to the [−π,π] interval. This allows us to sample appropriately from the full posterior.

Mixture model fitting

We use the mixture model of [7], allowing for a mixture of target, non-target and random responses. We fit the following mixture component, using the expectation-maximization algorithm: (28) (29)

where p_t is the mixture proportion associated with the target, p_r the random mixture proportion and p_nt the non-target mixture proportion. μ_t and μ_k are the true locations of the target and non-targets. All Von Mises share the same κ; this is because the concentrations (though not the mixing proportions) of the posterior modes around each target are determined by the Cramer-Rao lower bound associated with the local Fisher information, which are all identical. The values of p_t,p_r,p_nt and κ are fit during the EM procedure; the μ’s are assumed to be known.

To check for the significance of non-zero mixture proportion p_nt, associated with non-target responses, we perform a resampling analysis. Given a set of responses, targets and non-target angles, we randomly resample the non-target angles and refit the mixture model. We perform this procedure K times and obtain K samples of p_nt (K = 1000). We then construct the empirical cumulative distribution function Φ(p_nt) for p_nt given those samples. Finally, we compare the mixture proportion $p_{nt}^{*}$ obtained given the original non-target angles, and reject the null hypothesis “p_nt = 0” when $p=1-\Phi (p_{nt}^{*})<0.01$.

Fisher information derivation

The Fisher information for a population code with Gaussian noise is: (30) where f is the mean response of the population, and C the covariance of the population response. In our case, θ = [ϕ ψ]^T, so the Fisher information is a 2-by-2 matrix.

Consider the case that the memory only contains a single item, with β = 1. Then (31) where we assume ${\tilde{\Sigma }}_N={\sigma }_x^{2}I$. Since the covariance ${\tilde{\Sigma }}_N$ does not depend on θ, the trace term in the Fisher information is 0.

The FI about the angle is given by (32) (33) (34)

The other components of the Fisher information matrix can be derived similarly.

By taking a large population limit in which preferred values have density ρ, we obtain a closed-form approximation to the Fisher information (see Section, 1 in S1 Text for the complete derivation): (35)

Parameter optimization

We perform a grid search over several population code parameters to provide a qualitative fit to human experiments. For the mixed population code, we varied σ_x and the ratio of conjunctivity, as β, σ_y were kept fixed. For the hierarchical code, we set p = 1, λ = 1 and Θ = 1 and varied σ_x and the ratio of conjunctivity (defined as $\frac{M_2}{M_1+M_2}$, where M₁ (respectively M₂) is the size of the layer one subpopulation (respectively layer two)). A full fit, which is the subject of future work, would require at least the consideration of heterogeneous and multi-scale population representations.