Conceived and designed the experiments: IHS KW KPK. Performed the experiments: IHS HLF IV. Analyzed the data: IHS HLF KW. Wrote the paper: IHS HLF IV KW KPK.
The authors have declared that no competing interests exist.
A large number of experiments have asked to what degree human reaching movements can be understood as being close to optimal in a statistical sense. However, little is known about whether these principles are relevant for other classes of movements. Here we analyzed movement in a task that is similar to surfing or snowboarding. Human subjects stand on a force plate that measures their center of pressure. This center of pressure affects the acceleration of a cursor that is displayed in a noisy fashion (as a cloud of dots) on a projection screen while the subject is incentivized to keep the cursor close to a fixed position. We find that salient aspects of observed behavior are well-described by optimal control models where a Bayesian estimation model (Kalman filter) is combined with an optimal controller (either a Linear-Quadratic-Regulator or Bang-bang controller). We find evidence that subjects integrate information over time taking into account uncertainty. However, behavior in this continuous steering task appears to be a highly non-linear function of the visual feedback. While the nervous system appears to implement Bayes-like mechanisms for a full-body, dynamic task, it may additionally take into account the specific costs and constraints of the task.
There is a growing body of work demonstrating that humans are close to statistically optimal in both their perception of the world and their actions on it. That is, we seem to combine information from our sensors with the constraints and costs of moving to minimize our errors and effort. Most of the evidence for this type of behavior comes from tasks such as reaching in a small workspace or standing on a force plate passively viewing a stimulus. Although humans appear to be near-optimal for these tasks, it is not clear whether the theory holds for other tasks. Here we introduce a full-body, goal-directed task similar to surfing or snowboarding where subjects steer a cursor with their center of pressure. We find that subjects respond to sensory uncertainty near-optimally in this task, but their behavior is highly non-linear. This suggests that the computations performed by the nervous system may take into account a more complicated set of costs and constraints than previously supposed.
Recent studies have shown that, for many motor tasks, human subjects take uncertainty in their sensory feedback into account. They often use knowledge of uncertainty in a way that is close to optimal in a statistical sense both in their perception of the world
In studies of Bayesian behavior, the problem of how the brain uses sensory estimates to control movement has often been formulated as an optimization problem. That is, given the constraints and costs of the movement as well as sensory information, the nervous system computes how to move to minimize the cost. A range of human movement studies have been conducted confirming that humans often move in a way that is close to statistically optimal, in this sense
We introduce a new goal-directed, visuomotor task where whole-body movements are required to interact with the environment. In this task subjects steer a noisy, dynamic visual cursor by forward-backward shifts of body weight similar to surfing or snowboarding. Our purposes are two-fold. First, we aim to test whether Bayesian predictions of the behavioral responses to visual feedback still hold when the task dynamics are more complex. Second, we aim to test whether, as in studies of reaching and quiet standing, subjects appear to use a linear feedback control rule with a quadratic cost function. We find that many aspects of behavior are well captured by optimal control models incorporating Bayesian estimation of feedback uncertainty. However, behavior during this task differs in an important way from previous work on simple movements such as hand reaching and quiet standing. In this steering task human subjects appear to combine two well-known control strategies: bang-bang control and linear-quadratic regulation. Importantly, our results suggest that humans still take uncertainty into account during a full-body, dynamical control task.
All experimental protocols were approved by IRB and in accordance with Northwestern University's policy statement on the use of humans in experiments. Informed consent was obtained from all participants.
Here we use a novel approach to analyze the influence of uncertainty on the dynamical control of subject's movement (see
A) The experimental setup. Subjects steer a cursor by shifts in center of pressure (COP) along the anterior-posterior axis. Noisy feedback of the cursor position (small, medium or large variance) is given while subjects are incentivized to steer the cursor to be close to the midline of the screen (target). B) Subject's movements affect the center of pressure, which is measured by a force plate. The resulting sensor readings then steer the on-screen cursor. Subjects receive noisy visual feedback about the cursor position and react to reduce errors. C) COP, cursor velocity and cursor position are shown as a function of time during one trial for a typical subject (red). The observed feedback (noisy dots) is shown in blue. D) The phase portrait of cursor position and velocity is shown for 10 successive trials. Data from (C) are highlighted in red.
The goal of this experiment is to examine how subjects control a noisy dynamical system during a goal-directed, full-body steering task. 10 healthy volunteers participated in the experiment. (4 female, 6 male; age 30.7 ± 5.0 years; weight 67.6 ± 8.3 kg). Subjects were instructed to stand perpendicular to a rear-projection screen (1.41 m ×0.79 m), ∼0.6m away, on a 4-sensor force-plate (Nintendo Wii Balance Board, recorded at 500 Hz) (see
The experiment was divided into 180 trials with each trial lasting for a random duration evenly distributed between 11.5 and 15 seconds. Every 20 ms a new dot with low contrast was shown on the screen with a position drawn from a radially isotropic Gaussian distribution centered on the true position of the cursor, while the previously shown dot disappeared. Due to persistence of vision, subjects perceive a rapidly fluctuating cloud of ∼5–10 dots. The width of this Gaussian cloud changed randomly from trial to trial with three categories: small, medium, or large variance (
At the end of each trial the true cursor position was revealed. Subjects were subsequently given a score based on the squared distance between the cursor and the mid-line of the display. The random trial duration incentivizes subjects to minimize the error over the entire trial, not simply the final error. The monetary rewards were arranged such that the minimum reward obtainable over the course of the experiment was $$ 10 and the maximal reward obtainable was $$ 20.
To account for the possibility that the cursor dynamics in this task cause subjects to approach biomechanical limits and behave atypically, we ran a similar experiment (N = 5, 1 female, 4 male, separate from the original group) in which the control gain was increased by a factor of four (
The cursor dynamics in this task are based on a stochastic linear dynamical system, where the state of the world evolves linearly with some process noise and subjects receive noisy feedback. Uncertainty arises from both the state evolution, through the process noise
We compare four different models of behavior for this task. Our objective is to predict subject's center of pressure
In model 1, the proportional-integral-derivative controller (PID), we assume that the observer ignores the dynamics of the cursor and simply estimates the best policy based on the noisy observations
In models 2 and 3 we use a standard Kalman filter to compute the estimated state of the cursor from the observations
An important feature of the Kalman filter as it relates to this experiment is how estimation changes as function of feedback uncertainty. The best estimate of the state at time
The following models use the Kalman filter state estimates. However, to be optimal we must define an underlying cost function, which will determine the control policy. In model 2 we consider a linear-quadratic regulator
To fit the free parameters, we optimize over
Model 3 again uses an ideal observer; however, here we assume that subjects use another type of control policy: a bang-bang controller. This model assumes two-state control with a threshold determined by a combination of the estimated position and velocity:
Here
Finally, in model 4, we consider a non-linear extension of the linear-quadratic regulator. This model estimates the optimal control for a standard linear-quadratic regulator. Then, to approximate the constraints of human behavior during this task (not wanting to fall over or biomechanical limits on posture), we pass the control predicted by the linear-quadratic regulator through a static non-linearity (a logistic function). Although this control scheme is sub-optimal for the two classes of cost-functions we consider in models 2 and 3, the static non-linearity serves to interpolate between bang-bang control and LQR. Bang-bang control is limited in the sense that it must explain a continuous signal using only two states, and LQR is limited in that it does not appropriately model the constraints and costs of the task, such as not wanting to fall off the board.
We find that human subjects readily learn our task. While the noise introduced into the cursor dynamics constantly perturbs the movement of the cursor, subjects are able to change their COP and stabilize the cursor position (see
A) Average errors across subjects over the course of the experiment binned in blocks of 10 trials. B) The influence of feedback type on task errors. All comparisons between feedback uncertainty levels were significant (one-sided t-test). In both plots errorbars denote SEM across 10 subjects. * denotes p<0.05. *** denotes p<0.001.
In trials where the feedback is better human subjects have lower mean squared errors (MSEs) on average (
One direct way of analyzing the behavior in this task is to observe subjects' responses to fluctuations in the time domain. Taking the cross-correlation between the fluctuations in cursor dynamics (process noise,
A) Cross-correlation between the fluctuations in cursor acceleration (process noise,
These results are qualitatively predicted by the Kalman filter models, since the Kalman update decreases with increasing feedback uncertainty. Small Kalman updates then lead to longer integration times and smaller excursions. For reference we include results from a simulation showing the cross-correlation between fluctuations and the Kalman update for three levels of feedback uncertainty (
The focus of the high-gain experiment is whether the range of center of pressure required for the task affects subject's control strategies. We do not expect any qualitative differences in how subjects estimate the cursor position. Indeed, we find similar trends for the case where the control gain is much larger. For the 5 subjects in the high-gain condition, the mean-squared target errors are 0.022±0.007 scr2, 0.027±0.007 scr2, and 0.054±0.016 scr2 for
It is important to note that the predictions of the ideal observer model (Kalman filter) describe perception alone. Since we measure postural responses, the above analyses serve as indirect evidence for near-optimal Bayesian integration. However, the ordering of peak time and peak amplitude responses clearly indicates that subjects take feedback uncertainty into account. Moreover, this ordering is consistent with an ideal observer using a monotonic feedback control rule,
Although subjects respond differently to different types of feedback, we can also look in detail at the strategies subjects used during the task – their control policies. To do this we compute the average center of pressure (the response) given the true cursor position and cursor velocity (the state) for each of feedback condition (
A) Distributions of the cursor position (left), cursor velocity (center) and center of pressure normalized by the standard deviation (right), averaged across subjects for the low-gain (top row) and high-gain (bottom row) conditions. In the low-gain condition, note the bimodal distribution of the center of pressure, despite the unimodal distribution of errors. This may indicate a bang-bang-like strategy. B) Policy-maps of the center of pressure averaged across subjects as a function of the true cursor position and velocity for two different levels of feedback uncertainty and across all conditions. Note that in the low-gain condition subject's responses saturate at large cursor velocities and positions. In the high-gain condition responses are much more linear.
This non-linear control strategy may be due to the wide range of center of pressures required for the task. In the high-gain condition, where center of pressure excursions can be much smaller for a given error level, subject's behavior appears much more linear. The COP distribution appears more unimodal (
We also examined how subject's controlled their center of pressure as a function of the cursor position alone (
A) Center of pressure as a function of cursor position for typical subjects in the low and high-gain conditions. Black lines denote median responses for a given range of cursor positions. Red and blue points denote samples along the COP trajectory. B) Average responses across subjects with thin lines denoting the responses of individual subjects. C) The predicted responses from the LQR, Bang-bang, and Non-linear LQR models. Error bars denote SEM across subjects (in B and C) and sample points (in A).
The bang-bang controller appears qualitatively very similar to human behavior (
A) Observed center of pressure for a typical subject and trial along with the center of pressure predicted by each of the three ideal observer models. Note that the linear-quadratic-regulator and the bang-bang controllers produce qualitatively very different estimates. Note also that the non-linear LQR model has some ability to interpolate between the two. B) Cross-validated fraction of variance explained for each model for both the low and high-gain experiments (two-fold cross validation). In the low-gain condition the ideal optimal observer models explain a significantly larger fraction of variance than the PID controller (p<0.05, one-sided paired t-test), and the non-linear LQR explains a significantly larger fraction of variance than all others (p<0.001, one-sided paired t-test). Error bars denote SEM across subjects.
Model 2, the bang-bang controller captures the bimodal strategy observed in human behavior but is limited by the fact that it attempts to model a continuous signal using only two discrete states (
Finally, by combining aspects of the bang-bang and standard LQR controllers, a non-linear LQR model (model 4) out-performs all other models. This model captures the continuous character of the signal, and also allows for saturation-like effects where the nature of the task constrains behavior (
It should be noted that
Here we have shown that ideal observer and optimal control models can describe many aspects of human behavior in a surfing-like task where movements of the body steer the movements of a cursor. We have found that there is a clear influence of uncertainty on motor behavior. As predicted by Bayesian statistics (Kalman filter model), subjects respond more slowly and with lower amplitude to higher uncertainty feedback suggesting that they are integrating information over longer periods of time. Unlike previous (predominantly reaching) experiments examining the effects of uncertainty on behavior, we find that under certain conditions subjects use highly non-linear strategies similar to bang-bang control. These results suggest that human subjects take the uncertainty of sensory information into account and use this information during motor control, even during full-body behavior when the task is continuous and constrained by biomechanical factors.
Several studies have examined behavior during tasks involving control of the center of pressure including skiing on a simulator
The present study provides strong evidence that feedback uncertainty affects online control of continuous movements. When feedback is more uncertain the behavioral responses are significantly slower, indicating the nervous system needs to integrate information over a longer period of time. Similar results have been reported for reaching tasks where reaction time increases with increasing uncertainty about the target
Previous studies of optimal control in reaching have found that human behavior is accurately modeled by linear-quadratic regulation
The models presented here aim to describe the factors that drive motor control in dynamical situations. However, unlike in reaching tasks where two-link systems provide fairly accurate biomechanical models, the experiment here needed to be simplified dramatically to allow for productive modeling. Specifically, we ignore the biomechanical factors that link the motor commands driving body stabilization with actual movements of the center of pressure. This simplifying assumption makes modeling much more tractable but could potentially be extended with more realistic biomechanics. We should note, however, that the dynamics of the body should have a small effect on the results presented here. Although the natural frequency of quiet standing is on the order of one second
Despite this difference in timescales, the cursor dynamics in the low-gain condition apparently do cause subjects to use the full range of their center of pressure, allowing us to observe control strategies near the biomechanical limits of posture. The high-gain experiment was designed to make the task much easier and requires subjects to use a much smaller range of postures. In this case, subjects use much more linear control strategies. Importantly, both these regimes, near equilibrium and near biomechanical limits, exist in normal human behavior, and appear to be well-described by control models that use optimal state estimation. We should also note that, for the results presented here, the problem of how subjects estimate the cursor position is inter-twined with the problem of how subjects control the cursor. The timescales of estimation alone are likely to be faster than those shown.
In addition to computational implications, the results presented above may also have implications for neurophysiological studies. In the past decade several studies have made progress investigating the neural correlates of uncertainty and Bayesian computations
Here we have combined aspects of typical experiments that ask if the nervous system employs Bayesian strategies with aspects of typical experiments that analyze the dynamical control of movements. We have found that salient aspects of optimal control and optimal Bayesian estimation can be observed for a complex task where whole-body movements are controlled continuously. This may indicate that these principles describe general properties of the human movement system and that people can rapidly learn to control a system in a near-optimal way – even if a non-linear control scheme such as bang-bang-like control is necessary.
Thanks to Max Berniker for helpful discussions and the PIAW project for inspiration.