The model

We consider a recognition task from positive examples [20–22]. The perceptron should learn to give a response whenever a sample from a given category is presented. In contrast to the standard perceptron algorithm, which ‘unlearns patterns’ for which the neuron should not be active, the synapses are not modified for negative samples. It has been argued that this setup is relevant to biology in particular when the set of negative samples is very large and/or its statistics unknown [22]. For instance, one might want to train a neuron to recognise fruits, but not update the synapses for all other objects. This setup is also relevant when studying reinforcement learning, where learning is gated by reward feedback elicited by positive samples. Finally, it resembles the one-class support vector machine used in statistical learning, which detects whether a sample belongs to a class and which has applications in anomaly detection [23, 24].

The setup is illustrated in Fig 1. A single postsynaptic neuron calculates the weighted sum of its N excitatory inputs and compares it to a positive threshold $\theta \sqrt{N}$. Whenever $h={\sum }_{i=1}^N{w}_i{x}_i-\theta \sqrt{N}$ is non-negative, the perceptron fires. The inputs x_i are randomly chosen to be -1 or +1 with equal probability, and independently of the other inputs (see below for extensions). The $\sqrt{N}$ in the threshold is a mathematical convenience that ensures scaling of the system as the number of inputs is varied [11, 25].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Diagram of our single neuron setup.

A group of N input presynaptic neurons are connected to a single postsynaptic neuron. The input activity can be low, x_i = −1, or high, x_i = 1. The postsynaptic neuron performs a weighted sum of the inputs and fires whenever that sum is larger than a threshold $\theta \sqrt{N}$, otherwise it remains quiet. Each synapse w_i is adjusted as a function of the input activity so that the neuron remembers a set of previously seen patterns. Ideally, only these patterns should trigger the neuron; all other patterns should not.

https://doi.org/10.1371/journal.pcbi.1004265.g001

During learning the neuron is provided with a set of K positive patterns, x¹, …, x^k, …, x^K. As in the standard perceptron, we cycle through the set of patterns until the task is learned. The goal of the perceptron is to ‘fire’ for all these patterns. This should be contrasted to setups in which samples are presented only once (one-shot learning), which generally lead to a lower capacity [25]. We assume that initially all weights w_i are zero (tabula rasa). The learning rule is as follows: whenever a positive pattern is presented and only if it does not lead to postsynaptic activity, the synapse is updated. For high inputs, i.e., x_i = 1, potentiation occurs (1) where Θ(⋅) is the Heaviside step function which is zero if its argument is negative and one otherwise, and a ≪ 1 is the potentiation rate. Similarly, when an input x_i is low, the synapse depresses (2) where b is the amount of depression. Depression is followed by rectification so that all synapses remain excitatory, w_i ≥ 0 [26–30]. If the pattern does already lead to firing of the perceptron, no synapse is altered. This stop-learning condition is also present in a standard perceptron; possible biophysical mechanisms are discussed in [31].

For the simple, random pattern statistics used here, the non-negativity constraint limits the maximal amount of patterns that can be learned to K_max = N [9, 11], which is half of the number of patterns an unconstrained perceptron can learn. Below this limit the learning process finishes with high probability in a number of steps that is polynomial in N. We define the memory load α = K/N, which becomes α_max = 1 at the maximal load in the balanced case.

Imbalancing plasticity promotes sparseness

Unlike the traditional perceptron rule, we allow for distinct amounts of potentiation and depression. By introducing imbalance in favour of depression the learning dynamics is biased towards the hard bound of the weight at zero. We rewrite the plasticity rule using the learning rate ε ≡ (a+b)/2 and an imbalance parameter λ ≡ (b−a)/2ε. Provided the synapse does not hit the zero bound, the weight update is (3) The parameter λ is zero for balanced learning; depression is stronger than potentiation if 0 < λ ≤ 1. We find somewhat improved faster learning when we also depress even when the pattern has already been learned, i.e. (4) For that case it can be shown that the learning dynamics minimises the energy function (5) where [⋅]₊ denotes rectification. The first term of the energy sums over all patterns and promotes low false negative rates; it is zero if the perceptron fires, while it attributes a cost proportional to the distance to the firing threshold whenever a pattern is not yet learned. The second term acts as a linear regulariser; the depression-potentiation imbalance λ penalises synaptic weight configurations that have large linear norms $\mid \mathbf{\text{w}}\mid \equiv {\sum }_{i=1}^N{w}_i$. The regularisation term has a simple interpretation, as it is proportional to the mean synaptic weight, ∣w∣ = N⟨w⟩. The plasticity rule, Eq 4, minimises this energy by performing a stochastic sub-gradient descent [32], projected onto the subspace {w: w_i ≥ 0, i = 1, …, N}.

Rewriting the learning rule as the minimisation of the energy Eq (5) shows explicitly why introducing imbalance towards depression promotes weight sparseness. In linear regression and classification, optimising over regularised energy functions that penalise the L₁-norm $\mid \mid \mathbf{\text{w}}\mid {\mid }_1\equiv {\sum }_{i=1}^N\mid w_i\mid$ of the weights is well-known to induce sparseness [33–35]. Below the critical load α_max the weight configuration with minimal linear norm is known to be sparse [27]. Thus, the learning rule Eq (4) with imbalance λ > 0 will try to find solutions that satisfy the learning conditions but that are sparser than those obtained when λ = 0.

While the linear norm constraint promotes sparseness, it is not guaranteed to produce the sparsest possible solution. The true optimisation problem would be to minimise the L₀-pseudo-norm ∣∣w∣∣₀. The L₀-pseudo-norm simply counts the number of non-zero synapses. However, this leads to a difficult NP-hard combinatorial optimisation task [12, 13]. Instead, optimising under the L₁-norm constraint is a convex relaxation of the original problem for which efficient computer algorithms exist (e.g. [36]). Moreover, imbalancing plasticity has the advantage of being an online procedure that only requires tuning the potentiation and depression event sizes and is thus biologically plausible.

Information and efficiency

Ideally our perceptron learns all examples, and minimises the false positive rate. To characterise the performance we present the perceptron with learned samples and lures (other random patterns), both presented with equal probability. The mutual information between the class of the input pattern and the perceptron’s output on a given trial is (6) where P(x) = 1/2 is the probability that the test pattern is a positive pattern (p) or negative lure pattern (l), P(r) is the probability that the perceptron remains silent or fires, and P(r∣x) is the conditional probability that we observe a given response given the true pattern class.

The information can be expressed in terms of the false positive rate p₀₁ and the false negative rate p₁₀. Below the critical capacity (α ≤ α_max), the positive samples are recognised perfectly after learning, i.e. there are no false negatives (p₁₀ = 0), so that the information is determined by the false positive rate only. As we have 2K trials, the total information normalised per synapse, $C=\frac{2K}{N}I$, equals (7) Although this type of information calculation is common, we note that testing with equiprobable lures and learned patterns is somewhat sub-optimal in terms of information [37]. For the one-class perceptron, testing exhaustively with all 2^N−K possible lures gives about 60.6% more information when p₀₁ = 1/2 with a weak dependence on p₀₁.

As the mutual information does not take energy efficiency into account, we consider a recently suggested capacity measure that includes the sparseness of the final weight configuration [3]. The memory efficiency S measures the information per non-zero synapse by normalising the information to the fraction of non-zero synapses F, (8) Memory efficiency is thus measured in bits per functional synapse. Learning rules that achieve high information C using few resources will have high efficiency. If one assumes that a non-zero synapse has a certain energy cost (independent of synaptic weight) and a zero synapse has none, the memory efficiency S measures the energy cost of the stored memory.

Imbalanced plasticity improves memory efficiency

A variant of the sign-constrained perceptron convergence theorem (see Methods) shows that the learning algorithm Eq 3 converges below a critical imbalance λ_max(α) that depends on the memory load α. In computer simulations we focus on the two extreme cases, i.e., balanced (λ = 0) and maximally-imbalanced λ = λ_max(α) plasticity. In principle it is possible to find the maximum imbalance by trying various values of λ and checking convergence of the learning process. However, it is much quicker to use that the problem is equivalent to learn the patterns while minimising the linear norm ∣w∣, see Eq 5. This was done with a linear programming solver (see Methods) which requires no manual search for the maximal imbalance.

For strongest depression (λ = λ_max), the information C is only slightly below the information of balanced learning, Fig 2A (magenta vs. blue curve). However, imbalanced plasticity provides a large increase in memory efficiency S, Fig 2B. The reason is that the learning dynamics converges to synaptic configurations with a considerably larger number of silent synapses, Fig 2C. As the memory load α increases, the efficiency approaches that of the balanced solution. This is expected; by increasing the task difficulty we are imposing additional constraints on the synaptic weights. As a result the volume of the solution space shrinks and the constraint on the mean weight has to be relieved, therefore leading to smaller gains in memory efficiency. As α approaches its critical value, the space of solutions collapses to a single point, i.e., no additional constraints can be imposed at critical capacity and λ_max = 0 [7].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Information C in bits per synapse (bps), memory efficiency S in bits per functional synapse (bpfs) and the fraction of silent synapses 1−F as a function of the memory load α = K/N.

Results from a simulation with N = 1000 synapses. Shown are: balanced learning where depression equals potentiation (λ = 0); maximal imbalance learning; the maximal-information solution found with offline quadratic programming (QP); minimal-value synapse deletion, where all weights below some threshold are set to zero; and random pruning. The two latter rules were set to delete the same number of synapses as imbalanced learning. The results for online learning were obtained under a large threshold (θ = 1, learning rate ε = 1/N) to maximise information (see Methods). A. Information. Imbalanced plasticity leads to a small information decrease and significantly outperforms thresholded pruning. Random deletion performs very poorly. Truly maximising information (QP) gives only a slight improvement in performance. B. Memory efficiency (information per non-zero synapse). In particular at low α, the imbalanced perceptron finds sparser weight configurations, boosting the memory efficiency. The curves converge as the critical loading α = 1 is approached. The maximal information solution (QP) is more efficient than balanced learning, but still inferior to imbalanced learning. C. The fraction of silent synapses. Balanced online learning (λ = 0) under a large threshold always leads to the appearance of silent synapses, due to the imposed hard bound at zero together with the large firing threshold. Imbalanced plasticity significantly increases sparseness, especially at lower memory loads. QP learning leads to a few more zero-weight synapses compared to balanced learning, the fraction of which remains close to 50% irrespectively of the memory load.

https://doi.org/10.1371/journal.pcbi.1004265.g002

We also considered alternative learning algorithms: first, a minimal-value pruning rule, where all weights below a certain threshold are set to zero after learning has converged. We set the deletion threshold of the offline pruning algorithm to produce the same number of zero-weight synapses as the imbalanced solution. This is optimal in the one-shot learning case [14, 15]. In this case we find a more pronounced loss of information and, interestingly, almost no efficiency increase (dark green curve). The superiority of imbalancing makes intuitive sense: imbalanced plasticity is an online protocol that accommodates for sparseness constraints by redistributing weights dynamically, while the pruning procedure is performed after learning and does not allow for further re-adjustments. Finally, we also tried random pruning after learning, which as expected, performs very poorly (light green curve).

For completeness, we compared these results to the solution that maximises information without requiring sparseness. The optimisation can be formulated as a quadratic programming (QP) problem (see Methods), and the best solution can be found with a high performance barrier method convex optimiser [38]. This algorithm clearly lacks biological plausibility, and does not provide a significant improvement in information over balanced (λ = 0) online learning, Fig 2A. In other words, perceptron learning works well for our problem, provided that the firing threshold θ is large enough (see Methods). Under QP the fraction of silent synapses slightly increases to around 50%, Fig 2C, which leads to a moderate improvement in memory efficiency, Fig 2B. Finally, one can resort to the min-over learning rule, which only applies a weight update for the pattern that evokes the minimal output h [39]. The synaptic weights are guaranteed to asymptotically converge (as θ → ∞) to the QP solution and unsurprisingly the information matches that which is obtained with the quadratic solver. This procedure is difficult to reconcile with biology as well, as each single learning iteration requires access to every pattern.

Synaptic weight distributions

The learning algorithm and the threshold setting also determine the shape of the synaptic weight distribution. This distribution is of importance, as it can be compared to experimental data. For instance, the electro-physiologically determined synaptic weight distribution was used to link Purkinje cell learning to perceptron learning theory [28, 40]. We recorded the obtained synaptic weight histograms (see Methods), averaged over many trials (each with different pattern sets). While collecting results across trials is strictly only approximates the synaptic weight density, it is a good estimate of the actual observed distribution for a single realisation of the system, since the underlying weight density is strongly self-averaging [27, 28].

Balanced learning (λ = 0) leads to an approximately exponential distribution, Fig 3A. Interestingly, although the QP solution did not increase information compared to online balanced learning (Fig 2A), the shape of the distribution of synaptic weights changes considerably (cf. Fig 3A and 3B). At any memory load α ≤ α_max the fraction of zero-weight synapses always remains close to 50% while the remaining weights assume a truncated Gaussian distribution centred around w = 0. The problem that we are dealing with is thus not ‘intrinsically sparse’ in weight space. This should be contrasted with the non-negative perce ptron classifier with 0/1-coded inputs that was recently studied [28–30]. In that case, maximising information in the presence of postsynaptic noise automatically leads to sparse weight configurations (F < 0.5), provided that the memory load is below the critical point. Interestingly, at the critical load, the distribution becomes identical to the truncated Gaussian that we report here as the optimal one.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Synaptic weight histograms, information and memory efficiency at low memory load (α = 0.1).

Data obtained averaging over a thousand simulations (N = 1000). A. For balanced learning the distribution is stretched due to the optimised learning (large threshold choice θ = 1 under a small learning rate ε = 1/N). As with the non-negative perceptron classifier [28], a large number of zero synapses appear. B. Maximal-information solution obtained via quadratic programming, with the objective set at minimising the Euclidean norm ∣∣w∣∣₂. The quadratic objective function leads to a hemi-Gaussian weight distribution, again with a large fraction of silent synapses arising from the non-negativity constraint. C. Minimal linear norm solution (largest imbalance). As the learning task is ‘easy’ (low α), strong depression leads to a highly sparse synaptic configuration.

https://doi.org/10.1371/journal.pcbi.1004265.g003

Imbalanced plasticity boosts the fraction of zero-weight synapses and stretches the weight distribution, Fig 3C. Although the mean weight is lower due to the increased sparseness of the weight configuration, the surviving synapses are stronger. This can be understood through theoretical arguments (see Methods). It can be shown that learning rules that lead to a large minimum postsynaptic sum, ${\mathrm{\text{min}}}_k{\sum }_{i=1}^N{w}_i{x}_i^k$ (together with a normalisation condition that fixes the Euclidean norm ∣∣w∣∣₂) give better recognition performance against lures. As some synapses are zeroed-out, specific strengthening keeps the postsynaptic sum large for learned patterns.

The non-zero weight distribution for maximal imbalance can be reasonably fitted to a compressed exponential P(w) ∼ exp(−cw^β), with an exponent β = 1.4. The two-class perceptron model yields β = 2 (a truncated Gaussian) at critical capacity [28]. The best fit of this type of distribution to the cerebellar data published [40] has an exponent β = 0.7±0.4, however it should be noted that the limited amount of data allows for a broad range of possible β.

Homeostatic excitability regulation and sparse codes

Next we explore if our findings depend on the details of the coding. So far we assumed the inputs were -1 or +1, as in earlier studies of the non-negative perceptron [9, 26, 27]. This is hard to imagine biologically, unless an inhibitory partner neuron is introduced [19, 31, 41, 42]. An arguably more faithful biological model is obtained by representing low inputs as silent, x_i = 0 [16, 19, 20, 28, 43]. Furthermore, we wish to generalise to a case where the probability for a high input is variable rather than fixed to 1/2.

The capacity of the above model can be fully recovered without drastically changing the neural circuit. In fact, two ingredients suffice: one has to rebalance the plasticity rules as a function of the activity level f, and, secondly, introduce a dynamic mechanism that adapts the firing threshold as a function of the linear norm ∣w∣. With these modifications, both the information C and the memory efficiency S are exactly identical to those reported in the previous section.

First, we generalise the model to deal with an arbitrary coding level f. When f = 1/2, the original model is recovered up to scale factors. To preserve the zero mean, we consider activity patterns that are coded as z_i ∈ {−f,1−f}, with P(z_i = 1−f) = f. Stochastic sub-gradient descent dynamics over the energy Eq (5) gives the adjusted potentiation rule for high inputs (9) while depression at low inputs becomes (10) followed by rectification. Here $h={\sum }_{i=1}^N{w}_i{z}_i-\theta \sqrt{fN}$.

Next, a zero-mean input z_i is related to 0/1 coding by the simple relation x_i = z_i+f, x_i ∈ {0,1}. Therefore the net input of the neuron in response to a 0/1 pattern can be written through a change of variables as (11) where we defined a new threshold variable Note that this threshold grows during learning so as to compensate the increasing weights. This can be viewed as a kind of homeostatic adaptation process: as learning progresses, the neuron self-regulates so that it becomes harder to reach the firing threshold. While the incorporation of an auxiliary feed-forward inhibition circuit has been used in related models to increase capacity in the presence of non-negativity constraints [19, 31, 41, 42], the mechanism here does not directly depend on the precise pattern x of the presented input. It thereby obviates the need for coordinated plasticity with a partner interneuron as well as for precise temporal integration of inhibitory signals. Instead it could be implemented sub-cellularly without the aid of additional circuitry. Using the adaptive threshold, the information becomes independent of the input coding level f (Fig 4 solid line), while it decreases when the threshold is fixed (dashed curve). We note that, unlike for two-class learning, for one-class learning a high threshold suffices to implement a large-margin classifier.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Information C in bits per synapse for binary (0 or 1) input patterns as a function of the input coding level f.

Average values for the dynamic-threshold model, where h is given by Eq 11, and average values obtained with a fixed threshold θfN — note the threshold scaling with fN instead of $\sqrt{fN}$ due to the 0/1 input activity. Potentiation and depression were balanced (Eqs 9 and 10) to match the coding level. While the adjusted model is insensitive to f, the information achieved by the uncorrected model approaches that of the original one for sparse input patterns. Simulations performed at moderate memory load α = 0.5 and system size N = 1000.

https://doi.org/10.1371/journal.pcbi.1004265.g004

An alternative route to recover capacity is to employ sparse coding, a finding that has been previously reported for the non-negative perceptron in a more general classification framework [43]. Here the asymptotic situation is rather simple, because as f → 0 and N → ∞ the original model is recovered and performance at low f approaches the ideal performance, Fig 4.

Input correlations

Activity correlations can severely limit the performance of learning rules, depending on the task and the nature of the correlations. For instance, in supervised memory tasks, Hebbian learning deteriorates under almost any type of correlation in the patterns [25, 44]. In contrast, more powerful plasticity rules equipped with a stop-learning condition, like the perceptron rule, are resistant to spatial input correlations [45], and can in some cases take advantage of input-output redundancies to store more patterns [29, 46].

To test the robustness of imbalanced plasticity to correlated activity we draw random patterns from a generative model that induces spatial presynaptic activity correlations (characterised by a parameter g, see Methods, [21, 45]). We first correlated the patterns such that the mean activity remained homogeneous across the inputs. Consistent with the standard two-class perceptron without synaptic sign-constraints [45], neither the imbalanced learning, nor the balanced rule are affected by input correlation, Fig 5A.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Memory efficiency vs input correlations.

A. In case the mean input remains homogeneous, the three learning algorithms considered — balanced (λ = 0), maximally-imbalanced (λ = λ_max) and maximal-information (QP) — are unaffected by spatial presynaptic activity correlations. B. In case of heterogenous inputs, the balanced rule (λ = 0) and the QP algorithm deteriorate. Imbalanced plasticity performs well, however, as it regularises the high-activity synapses while ignoring the remaining ones. As a result the memory efficiency of the maximally-imbalanced solution is approximately constant. Data obtained by averaging a hundred independent simulations at α = 0.1, f = 1/2, and N = 1000 synapses.

https://doi.org/10.1371/journal.pcbi.1004265.g005

Next, we implemented a variation of the generative model that introduces heterogeneities in the input activity rates where some inputs tend to be active more often than others. Interestingly the imbalanced rule is robust to this type of correlation, Fig 5B. Whereas the efficiency of the other rules drops off, the efficiency of the imbalanced rule remains constant. The intuitive explanation is that the high activity synapses effectively experience balanced net potentiation and depression for non-zero imbalance λ. The imbalanced rule finds a high-information solution by silencing and ignoring the low activity inputs and subjecting the remaining synapses to the usual imbalanced learning protocol.

Robustness to noise

So far we have considered the recall of noise-free patterns, however, in the light of the many noise sources in the nervous system, it is important to confirm the noise robustness of the results.

First, we introduce transmission failures and spontaneous presynaptic activity, and test the learning with corrupted patterns, denoted $\widetilde{\mathbf{\text{x}}}$. An active input is switched off with probability ${\delta }_{10}=\mathrm{\text{P}}({\tilde{x}}_i=0\mid x_i=1)$, while an otherwise silent presynaptic input fires with probability ${\delta }_{01}=\mathrm{\text{P}}({\tilde{x}}_i=1\mid x_i=0)$. The lures are generated with a matching mean activity, ⟨x⟩ = (1−f)δ₀₁+f(1−δ₁₀), to ensure that lure statistics match the patterns.

We examined the performance of the balanced and maximally-imbalanced rules, as well as thresholded synaptic pruning, under this stochastic synapse model, Fig 6A and 6B. The information of all three rules decreases smoothly as the input distortion increases. For dense patterns, f = 1/2, the efficiency of the maximally-imbalanced rule is initially the most affected by the introduction of noise, and becomes comparable to the thresholded deletion one for higher noise levels. For sparse patterns, Fig 6B, the efficiency is affected similarly by the noise for all three rules. The maximally-imbalanced and the thresholded solutions remain more efficient than balanced plasticity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Information C and memory efficiency S versus noise level.

The three solutions — balanced (λ = 0) and maximally-imbalanced (λ = λ_max) plasticity, and thresholded synaptic pruning — were obtained once for a single set of K = 0.1N positive patterns (N = 1000 synapses) and then tested against a large number 100K of distorted learned patterns and lures, generated for each noise level. The firing threshold of each solution is numerically optimised to maximise information. The presynaptic noise level varied under the setting δ₀₁ = δ₁₀ = δ (see main text for details). The scale of the postsynaptic noise standard deviation was set by normalising the weights to give a unit size mean response to learned patterns. A. For dense patterns, f = 1/2, the falloff in information is steeper for imbalanced plasticity than thresholded deletion. The two solutions remain more efficient than balanced learning for all noise levels. B. For sparse input patterns, f = 0.01, the balanced solution also suffers and as long as the information is not practically zero, both the maximally-imbalanced and the thresholded pruning rules are more efficient than the balanced one. C. Results for a postsynaptic noise model, where the current h is perturbed with an additive zero-mean Gaussian random variable with standard deviation σ. As the postsynaptic noise does not depend on the actual learned weights, imbalanced and balanced plasticity show similar noise robustness profiles.

https://doi.org/10.1371/journal.pcbi.1004265.g006

Next, we examined the role of postsynaptic current noise by adding a zero-mean Gaussian variable to the postsynaptic current h [28], the variance of which sets the noise intensity, Fig 6C. In contrast to the above, the magnitude of the random contributions is decoupled from the actual learned weights. For this noise model, the relative information reduction is comparable for both balanced and imbalanced plasticity.

Tuning of the imbalance parameter

In the above the imbalance parameter λ was optimised for automatically in an unbiological fashion. To examine suboptimal values we simulated learning while raising λ towards the critical imbalance λ_max, above which the learning algorithm no longer converges. The memory task difficulty, set by the memory load α, limits the allowed imbalance (see Methods). Indeed, we find that λ_max shrinks as α increases, Fig 7. Akin to the margin parameter which sets the noise robustness of the non-negative perceptron [28, 29], the actual λ_max depends on the exact set of patterns the neuron should learn. However, for random patterns drawn from the same distribution, the system is self-averaging as N → ∞ [7]. In simulations we observe a similar behaviour across different runs, although some finite-size effects are still apparent in networks of moderate dimension, Fig 7 (rightmost curves). In other words, λ_max can be reasonably estimated independent of the precise pattern set. Finally note that the figure implies that the parameter can be set conservatively, based on the maximum number of patterns to be expected. Of course, the efficiency gain is not maximised in this case, but still better than the balanced case.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Memory efficiency S increases with imbalance λ.

Efficiency is a function of imbalance for a given set of patterns. The curve stops when the learning dynamics no longer converges. Dashed horizontal lines indicate the corresponding efficiency values achieved by the linear programming solver (see Methods). The results for five independent runs at α = 0.1 (rightmost curves) are very similar, although finite-size effects are visible as the number of inputs is not particularly large. As predicted, the critical imbalance λ_max decreases with the memory load α. The learning rule only updated the synapses for patterns that did not yet lead to firing activity, Eq 3. Simulations of a neuron with N = 1000 synapses and coding level f = 0.01.

https://doi.org/10.1371/journal.pcbi.1004265.g007

Criteria for optimising information

Provided that the memory problem is realisable, perceptron learning ensures that each of the K patterns leads to postsynaptic firing activity (h ≥ 0), i.e., the FN error probability is zero, p₁₀ = 0. In this case the information increases as the FP error probability p₀₁ decreases (see main text, Eq 7). With the additional assumption that the lures are uncorrelated to the learned patterns, it can be shown that our perceptron learning rule leads to a decrease of the FP error.

To see why, we write p₀₁ as a function of the learned synaptic weights. As the lure patterns are uncorrelated to the learned ones, each input x_i will be uncorrelated to its corresponding weight w_i. The total synaptic current is given by a sum of many terms. Assuming that there are sufficient non-zero weights, the probability distribution p(h_l) of the net input h_l in response to a lure is Gaussian, $h_l\sim 𝓝(⟨h_l⟩,⟨\delta h_l^2⟩)$. Under this approximation, (12) where $\mathrm{\text{erfc}}(x)=\frac{2}{\sqrt{\pi }}{\int }_x^{\infty }{e}^{-t^2}\mathrm{\text{d}}t$ is the complementary error function. The approximation improves as N → ∞, as the fraction of non-zero synapses F remains finite irrespective of the imbalance λ (for λ ≤ λ_max) and as long as the memory load α does not vanish [10].

As the inputs are in zero-mean bipolar form, ⟨x⟩ = 0, ⟨x²⟩ = 1. The mean current elicited by lures is just $⟨h_l⟩=N⟨x⟩⟨w⟩-\theta \sqrt{N}=-\theta \sqrt{N}$, independent of the weights. The variance in the current (13) is proportional to the second raw moment $⟨w^2⟩={\int }_0^{\infty }\mathrm{\text{d}}wp(w)w^2$ of the weight distribution. For a particular realisation of the system one has $N⟨w^2⟩=\mid \mid \mathbf{\text{w}}\mid {\mid }_2^2$, the squared Euclidean norm of the synaptic weight vector. This is illustrated in Fig 8. The information of the system is thus given by the Euclidean norm of the weight vector alone. This is true for the learned-vs-lure discrimination task as long as the Gaussianity of the lure current h_l holds, irrespective of the particular learning rule that is employed. For instance, p₀₁ takes the same form for postsynaptic-independent learning [19] or for rate-coded inputs that are learned via the offline pseudo-inverse rule [22].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Schematic illustration of the postsynaptic current distributions.

In the large N limit, the postsynaptic current elicited by lures (dashed line) is well described by a zero-mean Gaussian, whose variance $⟨\delta h_l^2⟩$ is determined by the squared Euclidean norm $\mid \mid \mathbf{\text{w}}\mid {\mid }_2^2$ of the weight vector. Assuming that the learning dynamics converged, the postsynaptic current distribution provided that the input pattern is a learned one (solid line) is characteristic of perceptron learning: a significant number of patterns lie on the decision boundary and thus provoke a current that is exactly at the firing threshold, while the remaining ones generate super-threshold Gaussian tail currents [28]. The integral of the shaded area gives the FP probability p₀₁, which depends on the variance of the lure current distribution.

https://doi.org/10.1371/journal.pcbi.1004265.g008

Thus, the perceptron with the most information satisfies the firing condition h ≥ 0 for every learned pattern, but has a minimal Euclidean length weight vector. This coincides exactly with the traditional perceptron that is optimal with respect to the maximal-stability criterion [39], that prescribes the weight configuration with largest stability $\Delta \equiv \theta \sqrt{N}/\mid \mid \mathbf{\text{w}}\mid {\mid }_2$. This is a widely used principle that enlarges the basins of attraction in recurrent networks and improves the ability to generalise in classifiers [39, 61]. Notice that for a fixed threshold, increasing Δ can only increase information, as it is inversely proportional to the Euclidean weight vector length. Information maximisation thus reveals an interesting close link between recognition memory and the more usual two-class learning problems.

Furthermore, at least for random patterns, we can expect the perceptron learning rule to perform well. Below the critical load α_max the algorithm is known to converge to solutions with large Δ [62]. In other words, although the learning dynamics is not guaranteed to maximise information, it should achieve high C in the recognition task. As shown in the main text, Fig 2, the improvement indeed is minimal when the full quadratic program is actually solved.

The crucial condition that must be observed to achieve good performance is that the firing threshold θ should be large. Here θ plays the role of an indirect (unnormalised) stability parameter. It can be shown [39] that raising θ will indirectly lead to solutions with larger Δ. Lower bounds on how close the learning rule gets to maximal stability with a certain setting of θ and a, b can be extracted from the perceptron convergence proof [39].

Note that the above reasoning requires zero-mean inputs and balanced plasticity. For 0 or 1 inputs, the distribution of the unthresholded output h_l that is obtained in response to lures is still well characterised by a Gaussian, as an uncorrelated input pattern gives a sum over on average fN randomly selected weights. The expressions for the mean ⟨h_l⟩ and the variance $⟨\delta h_l^2⟩$ now include terms that depend on first- and second-order moments of the synaptic weight distribution. For a particular realisation of the random system the mean is $⟨h_l⟩=fN⟨w⟩-\theta \sqrt{N}=f\mid \mathbf{\text{w}}\mid -\theta \sqrt{N}$, and the variance $\langle \delta h_l^2\rangle =N\left(f\langle w^2\rangle \right)-f^2{\langle w\rangle }^2=f\Vert w{\Vert }_2^2-f^2{N}^{-1}\mid w{\mid }^2$. Thus, when the inputs are in 0 or 1 form, the information per synapse C is no longer a simple function of the squared Euclidean norm as before. The output error probability p₀₁, and therefore the information, is affected by the coding level f and the linear norm ∣w∣ as well.

Imbalanced plasticity affects convergence of the learning dynamics

To gain further insight on the effects of allowing a depression-potentiation imbalance, we prove the convergence of perceptron learning rule Eq 3 for non-zero λ, a variation of the detailed proof given by [29]. Besides the inclusion of the parameter λ, differences arise because our inputs are in bipolar form and because all patterns should elicit a high output.

We study a problem that can provably be solved in a finite number of learning steps by balanced postsynaptic-dependent learning (λ = 0). Therefore we can assume the existence of a weight configuration w* that solves the recognition task (14) while simultaneously satisfying the N non-negativity constraints $w_i^{*}\ge 0$, i = 1, …, N. The variable κ ≥ 0 relates the threshold $(\theta +\kappa )\sqrt{N}$ of the solution to the threshold $\theta \sqrt{N}$ that is used in the learning algorithm.

We assume that initially all synapses are silent, i.e., we start from the tabula rasa condition w_i = 0, i = 1, …, N. Learning proceeds by presenting patterns in random order. Since plasticity only occurs when the postsynaptic current $h={\sum }_{i=1}^N{w}_i{x}_i-\theta \sqrt{N}$ is not large enough to activate the perceptron, we index time with m = 1, …, M, m being incremented only when h < 0. Whenever each synapse w_i changes, it does so according to the update, Eq 3 (15) where η_i(m) = x_i(m)−λ is the weight update before rectification and x(m) ∈ {x¹, …, x^K} is the pattern that led to the update at time m.

The analysis is carried out by tracking the quantity (16) over time. If we find that after a finite number of updates a(m) would become larger than one, then the learning process is convergent, as the Cauchy-Schwarz inequality implies that a(m) ≤ 1. To monitor the time evolution of a(m) we bound the scalar product w* ⋅ w(m) from below and the norm ∣∣w(m)∣∣₂ from above.

After one update, the change Δ(w* ⋅ w(m)) ≡ w* ⋅ w(m+1)−w* ⋅ w(m) in the scalar product is (17) where B(m) = {i: w_i(m) < ε+ελ ∧ x_i(m) = −1, i = 1, …, N} is the set of all synapses that are set to zero due to the lower bound. Note that the lower bound can only be triggered by depression, which in turn can only occur for low inputs. The inequality is obtained by plugging in the definition Eq (14) of w*.

A bound on the scalar product w* ⋅ w(m) itself after m such updates can then be obtained by iteratively applying Eq (17): (18)

Meanwhile, the change $\Delta \mid \mid \mathbf{\text{w}}(m)\mid {\mid }_2^2\equiv \mid \mid \mathbf{\text{w}}(m+1)\mid {\mid }_2^2-\mid \mid \mathbf{\text{w}}(m)\mid {\mid }_2^2$ in the squared norm of w(m) after one step can be obtained by expanding the square $\mid \mid \mathbf{\text{w}}(m+1)\mid {\mid }_2^2=\mid \mid \mathbf{\text{w}}(m)+\Delta \mathbf{\text{w}}(m)\mid {\mid }_2^2$, so that (19) We have Δw_i(m) ∈ {εη_i(m), −w_i(m)}, with w_i(m) < ε+ελ, as Δw_i(m) = −w_i(m) only for i ∈ B(m). Thus, the squared norm of the update is dominated by the terms that come from low inputs at synapses that do not cross the lower bound. This gives the inequality (20) where $q\equiv {\max }_{k}1/N{\sum }_{i=1}^N{\delta }_{x_i^k,-1}$ denotes the maximum fraction of low inputs observed across the K patterns.

The scalar product is expanded as before: (21) Note that the update condition h < 0 is always satisfied at time m, so that $\epsilon \mathbf{\text{w}}(m)\cdot \mathbf{\text{x}}(m)<\epsilon \theta \sqrt{N}$. Together with the bound Eq (20), iterating over Eq (19) gives (22) (23) The last inequality is obtained by noticing that w_i(l) < ε+ελ inside the sum over l; the factor q arises from the iteration over the N synapses, conditioning on the low inputs. The bound Eq (23) implies that as learning proceeds ∣∣w(m)∣∣₂ cannot grow faster than $\sqrt{m}$.

From Eq (22) we collect (24) Turning back to Eq (18) and using the previous result Eq (24) yields (25) The last inequality stems from w_i(l) < ε+ελ. The first bracketed factor is always larger than −(ε+ελ), while the second one is bounded from above by ε+ελ. Iterating over the constrained sum introduces the factor Nq as before.

We now have a bound for the cosine a(m). Substituting in Eqs (23) and (25) gives (26) Note that while the neural parameters {ε, θ, λ} can be set at will, for a certain task the solution margin κ and the norms are constrained by the existence of a vector w* that can satisfy the learning conditions. Thus, they cannot be varied arbitrarily. In fact, if one keeps ∣∣w*∣∣₂ fixed, it will only be possible to increase κ up to a certain point, where we will have found the maximally-stable configuration. Similarly, the linear norm ∣w*∣ will have a minimum value. Furthermore, in general it is not possible to achieve simultaneously minimal ∣w*∣ and maximal κ with a single configuration.

From Eq (26) a number of conclusions can be drawn. The straightforward condition for convergence is to check whether that bound becomes larger than one. Another way to show that the learning algorithm stops is to check if a(m) is a monotonically increasing function of m. When λ = 0, the process is convergent, as long as $\epsilon \le 2\kappa /[\sqrt{N}(1+2q)]$. For λ > 0, the crucial observation is that we can only show that learning converges if κ can be raised so as to compensate for the negative terms in the numerator.

Thus, as expected, we find that the imbalance λ is related to the linear norm of the solution vector (one can increase λ as ∣w*∣ can be made smaller), and to the occurrence of depression events (through q). But more importantly, λ_max writes directly as a function of κ as well, which here sets the task difficulty, since the maximal value for κ shrinks as the memory load α increases. What is more, the minimum of ∣w*∣ depends itself on α. This theoretical prediction is confirmed by our numerical work. As α increases, the achievable imbalance λ_max becomes closer to zero, and the fraction of silent synapses approaches that which is obtained with balanced (λ = 0) learning, cf. Fig 2C.

Generating correlated patterns

We generate correlated patterns following previous work in recognition memory [21]. In the first model we generate a template pattern $\widehat{\mathbf{\text{x}}}$ with each input ${\widehat{x}}_i$ being set high (+1) or low (-1) independently and with equal probability 1/2. To maintain balance we also use its negative, $-\mathbf{\text{x}}-\widehat{\mathbf{\text{x}}}$, as a template.

The K patterns the neuron should learn are generated conditioned on either template, such that $\mathrm{\text{P}}(x_i^k={\widehat{x}}_i)=\frac{1+g}{2}$. Lure patterns follow the statistics of the learned patterns and are produced from the same templates. The parameter g controls the level of input correlations; with the choice g = 0 the original statistics are recovered, while at g = 1 the recognition task is impossible, as all patterns are perfect copies or reversals of one another.

In the second model patterns generated according to the process described above, but only using a single template. This procedure introduces inter-pattern correlations at the same presynaptic site x_i, as the arriving patterns become more similar to one another. It also leads to heterogeneous mean activity levels across neurons; although the mean number of active presynaptic neurons per pattern remains 1/2, increasing g leads to a bimodal presynaptic firing distribution. For g > 0, neurons that are active in the template fire more often and, conversely, the remaining neurons fire less frequently.

Computer simulations

All our computer simulations were implemented on Matlab R2013a (MathWorks) and were performed on a standard desktop computer. We simulated a single postsynaptic neuron that was driven by N = 1000 presynaptic random inputs. We varied the memory load parameter within the range α ∈ [0.1,0.8] to avoid both the appearance of unsolvable problem instances and excessive simulation time. We chose a small learning rate ε = 1/N and a sufficiently large firing threshold at $\sqrt{N}$ (i.e., θ = 1) except when otherwise noted. The threshold was set so that typically no increase in information could be obtained by raising it further. In the figures we included second-degree polynomial fits to average values.

The online perceptron learning rule was iterated until all patterns were learned. To obtain the maximally-imbalanced solution (λ = λ_max) we minimised the linear norm ∣w∣ using a linear programming algorithm [38], subject to the set of inequality constraints that ensured that every pattern would lead the neuron to fire. Specifically, using Matlab’s interior-point solver, available via the linprog command (Optimization Toolbox), we minimised ∣w∣ subject to N non-negativity constraints w_i ≥ 0 and K linear pattern imprinting constraints specified in matrix form as ${\mathbf{\text{X}}}^{\top }\mathbf{\text{w}}\ge \theta \sqrt{N}\mathbf{\text{1}}$, where X^⊤ is the K × N design matrix whose rows are the positive examples.

For the balanced case, the maximum-information weight configurations were obtained using the Krauth-Mézard min-over algorithm [39], followed by rectification after each learning step in order to enforce the non-negativity synaptic constraints. This is a batch learning algorithm that employs the balanced rule (Eq 3, λ = 0). At each step the pattern ${\mathbf{\text{x}}}^{k_{\mathrm{\text{min}}}}$ with lowest stability, $k_{\mathrm{\text{min}}}=\arg {\min }_{k=1}^K{\sum }_{i=1}^N{w}_i{x}_i^k$, is determined on the forehand. Then, only ${\mathbf{\text{x}}}^{k_{\mathrm{\text{min}}}}$ is learned; plasticity is silenced for all other patterns. To confirm optimality and validate our mathematical results we also resorted to an interior-point convex optimiser [38] and solved the quadratic programming problem of finding the weight vector with minimal Euclidean norm ∣∣w||₂. We resorted to Matlab’s quadprog command (Optimization Toolbox) to minimise $\mid \mid \mathbf{\text{w}}\mid {\mid }_2^2$ subject to the same N non-negativity and the K pattern imprinting constraints imposed on the linear program. Up to numerical precision the obtained pattern stabilities Δ matched those given by the min-over algorithm.

To calculate the information Eq (7) we tested the neuron with a set of K lures generated with the same statistics as the K learned patterns and recorded the number of FP errors. To determine the fraction of silent synapses, one has to take care of numerical rounding errors as it might be unclear when a synapse can truly be considered zero. We removed the weakest synapses one by one while probing the neuron with a large number of lures, until a drop in information occurred. With this procedure we could distinguish the true zero-weight synapses from small ones while avoiding numerical precision issues and arbitrary threshold setting. The results did not qualitatively change if we simply counted the number of synapses below some small weight $w_{\mathrm{\text{zero}}}\ll {\mathrm{\text{max}}}_{i=1}^N{w}_i$, held constant across trials.

Since we expected self-averaging of the synaptic weights distribution from the validity of the replica trick [7], the averaged synaptic weight histograms were collected from 1000 trials. To set a common weight scale across different learning rules and input statistics, we normalised the synaptic weights so that the threshold became unity, i.e., we re-scaled the weights by a factor $w_i/{\mathrm{\text{min}}}_{k=1}^K{\sum }_{i=1}^N{x}_i^k{w}_i$.