Conceived and designed the experiments: NGC DHO KS EWM. Performed the experiments: NGC DHO DH LP. Analyzed the data: NGC DHO DH LP AH SP KS. Contributed reagents/materials/analysis tools: NGC DHO DH LP. Wrote the paper: NGC KS EWM.
The authors have declared that no competing interests exist.
We have developed software for fully automated tracking of vibrissae (whiskers) in high-speed videos (>500 Hz) of head-fixed, behaving rodents trimmed to a single row of whiskers. Performance was assessed against a manually curated dataset consisting of 1.32 million video frames comprising 4.5 million whisker traces. The current implementation detects whiskers with a recall of 99.998% and identifies individual whiskers with 99.997% accuracy. The average processing rate for these images was 8 Mpx/s/cpu (2.6 GHz Intel Core2, 2 GB RAM). This translates to 35 processed frames per second for a 640 px×352 px video of 4 whiskers. The speed and accuracy achieved enables quantitative behavioral studies where the analysis of millions of video frames is required. We used the software to analyze the evolving whisking strategies as mice learned a whisker-based detection task over the course of 6 days (8148 trials, 25 million frames) and measure the forces at the sensory follicle that most underlie haptic perception.
“This is a
Rats and mice move their large whiskers (vibrissae), typically in a rhythmic pattern, to locate object features
High-speed (>500 Hz) videography is a non-invasive method for measuring whisker movements and forces acting on whiskers and yields nearly complete information about whiskers during behavior
Tracking whiskers is challenging. Whiskers can move at high speeds
To address these challenges, we have developed software for tracking a single row of whiskers in a fully automated fashion. Over a manually curated database of 400 video sequences of head fixed mice (1.32×106 cumulative images, 4.5×106 traced whiskers, 8 mice), whiskers were correctly detected and identified with an accuracy of 99.997% (1 error per 3×104 traced whiskers). In other whisker tracking systems, models constraining possible motions and whisker shapes have been used to aid tracking. In contrast, our approach uses statistics gleaned from the video itself to estimate the most likely identity of each traced object that maintains the expected order of whiskers along the face. As a result, the shapes of highly strained whiskers (curvature>0.25/mm) can be traced with sub-pixel accuracy, and tracking is faithful despite occasional fast motion (deflections >10,000 degrees/s).
Our method consists of two steps performed in succession: tracing and linking. Tracing produces a set of piecewise linear curves that represent whisker-like objects for each individual image in a video. Each image is analyzed independently. The linking algorithm is then applied to determine the identity of each traced curve. The trajectory of a whisker is described by collecting curves with the same identity throughout the video. Importantly, once a faithful tracing is completed, the original voluminous image data set is no longer required for downstream processing.
Once tracing is complete, a small set of features is tabulated for each curve. Based on these features, a heuristic is used to make an initial guess as to the identity of curves in a set of images where it works well. These initial identifications are then used to get a statistical description of the shapes and motions of whiskers. These statistics are then used to compute a final optimal labeling for each traced curve subject to the constraint that whiskers are ordered along the face.
We validated this approach using manually curated video acquired from a head-fixed, whisker-dependant object localization task
(A) A typical field of view. (B) Typical imaging configuration. (C–G) Automated results of tracing and linking. (C) Facial hairs and whiskers are traced in each video frame and then identified by a separate tracking step. (D) A whisker (blue) touches the pole. (E) Two whiskers (blue & green) are bent by the pole. The most posterior whisker is strongly retracted so that only a small segment is visible. (F) Tracking measures whisker orientation, such as the angle at base. (G) Tracking measures whisker shape, such as mean curvature, which can be observed over time. Changes in curvature allow the calculation of forces acting on the whisker follicle
The process of tracing the backbone of a whisker consists of three phases: initiation, extension, and termination. Tracing starts by using the angle and position estimated at the highest scoring candidate initiation site. As a curve is extended, initiation sites that fall under the traced curve are removed from consideration. When none remain, potentially duplicate curves are resolved.
Before any analysis, images were preprocessed to remove imaging artifacts specific to particular cameras (
A variety of methods have been developed to detect interest points in natural images
(A) Lines passing through a 7×7 pixel box are detected by finding the location of intensity minima (black dots) in regions from two partitions. The eccentricity of the best Gaussian fit (indicated by ellipses) to these points is used to score salience. (B) The score computed at each point in an image. (C) For high scoring seeds (eccentricity>0.95), the orientation of the line is indicated by the major axis of the ellipse. Whisker tracing is initiated at these sites using the measured angle.
First, a 7×7 square box is centered on a pixel of interest (
One advantage of this approach is that the entire image need not be queried for line-like objects. For example, in the data analyzed here, whiskers span hundreds of pixels. A grid of lines spaced appropriately (50 px; this is an adjustable parameter) will cross each whisker at least once. Restricting the search for line-like features to this grid greatly reduces the amount of time required for this step. Additionally, since the algorithm relies on local minima, demands on background subtraction and other preprocessing is minimal. The structure of the partitions ensures that lines in any orientation are detected. Greater than 80% of the pixels that could be used to start tracing a whisker (typically 1000 per whisker over the full image) are detected using an eccentricity threshold of 0.95; only 1 is required to fully trace a whisker. This detector systematically avoids potential starting locations near the face, near occlusions or where whiskers cross or nearly touch.
Curves are bidirectionally extended from candidate initiation sites by performing a small step (1 px) along the measured direction, and repeating until one of several criteria is met. The sub-pixel position and tangent angle are estimated by optimizing the correlation between an oriented line detector that is parameterized as a function of sub-pixel position, angle and width (
(A) Two parallel step edge detectors are separated by a distance,
The line detector is designed based on modeling the intensity profile of a whisker as a rectangular valley in the image with variable position, width and angle. The center of the whisker is estimated with sub-pixel precision by finding a position that minimizes the Laplacian of the correlation between the model and the image
Tracing is stopped if any one of several criteria indicates that the optimization procedure cannot be trusted to yield accurate results. This is necessary to handle cases where whiskers were partially occluded, such as when whiskers crossed or contacted an obstructing object, such as the stimulus pole in an object localization experiment (
(A,B) As a curve is extended from an initiation point, a local test region within the image is queried at each step to detect cases where the line detector may be unreliable. This happens near (C) crossing whiskers and (D) whisker-pole contacts. (E,F) When such a cases are encountered, the curve is linearly extended from the last trusted point, up to a threshold distance. (G,H) If all tests are satisfied at one of the points, a line segment is used to jump the gap and normal tracing is resumed. Otherwise, the trace is terminated at the last trusted point.
Occasionally, overlapping curves are generated during the tracing step. These need to be resolved before tracking so that individual whiskers have exactly one associated curve per frame. Searching for curves that cross the same pixel identifies pairs of potentially redundant curves. If all the points within an interval of one of the curves is within a small distance (typically 2 px) of the other curve and that interval consists of over 50% of the total length, then it is considered redundant. The shorter of the two curves is discarded; the shapes of shorter curves were not reliable enough to average. This procedure is repeated until there are no redundant curves left.
Identifying individual whiskers is challenging. First, the tracing algorithm generates curves (false positives) for facial hairs, trimmed whiskers, or other artifacts in the scene (
Our approach is based on the observation that whiskers are relatively long and maintain a consistent ordering along the face. We use a statistical model to describe the shapes and motions that are characteristic of whiskers. The model is then used to assign each traced whisker the most probable identity subject to the constraint that whiskers are ordered along the face.
For the majority of images, a simple length threshold can be used to remove false positives and leave exactly one traced curve per whisker. The linking algorithm is either supplied the number,
The curves in the frames
Rules constrain the labeling to enforce consistent anterior-posterior ordering of whiskers. The most proximal point of curves labeled Wi or Fi must be posterior to that of curves labeled Wj or Fj when
In (1) above, the features are assumed to be conditionally independent in order to simplify estimating feature distributions, even though this not strictly true in practice. Moreover, errors in the heuristic may introduce a systematic sampling bias. For example, at high deflection angles near the edge of the field of view, only a small segment of a whisker is imaged (
Appearance alone is not sufficient to uniquely identify individual whiskers in some cases. To address this, we designed a naive Bayes' classifier to determine the most probable identity of each traced curve subject to ordering constraints. The traced curves are ordered into a sequence,
Applying Bayes' theorem, the most probable sequence of labels,
The likelihood,
The second component of (4),
Given these change histograms, one can estimate the correspondence likelihood according to the formula,
Directly solving for the most probable labeling (2) is complicated by the fact that the likelihood of a labeling in one frame depends on the labeling in neighboring frames. Our approach is to initially score each frame by computing
Average processing time was 8 Mpx/s/cpu (35 fps for 640×352 pixel video) measured on a 2.6 GHz Intel Core2 Duo Macbook Pro with 2 GB of RAM, and was dominated by the CPU time required to trace curves; linking time was 0.5 ms per frame. This is faster than published speeds of similar whisker analysis packages (typically, 1–5 fps.)
Tracing was accurate to within 0.2 px as estimated by analyzing 39 hand-traced whiskers. Individual hand-tracings had an accuracy of 0.23±0.2 pixels when compared against consensus curves. Mouse whiskers (row C) were automatically traced in images with resolutions from 5 µm/px to 320 µm/px using default settings and in noisy, low-contrast images (
Linking accuracy was measured against a hand-annotated set of videos selected by choosing 100 random trials from behavioral sessions of 4 mice (1.32 million frames)
There are two principle sources of error in the linking. First, covariance between different features is ignored. For example, when the field of view is small, whiskers at high deflection angles might not be fully contained within the image. Any curve tracing one of these whiskers would appear shorter than the full whisker length. Under these conditions there is a strong correlation between angle and whisker length. The current model penalizes the shorter length because it is rarely observed; accounting for the covariance should eliminate this problem. Second, the estimated feature distributions are affected by systematic biases introduced from the heuristic used to initially guess whisker identity. A heuristic relying on a length threshold, like the one used here, may systematically reject these short curves at high deflection angle. This will result in a bias against strongly deflected whiskers.
Fundamentally, our linking method is applicable only to single rows of whiskers. The whiskers in videos imaging multiple rows of whiskers will be traced, but, because whisker images no longer maintain a strict anterior-posterior ordering, the linking will likely fail. Although this software was validated with images of one side of the mouse's whiskers, we have successfully tracked whiskers in images containing both whisker fields (
We analyzed data acquired during learning of an object detection task (see
(A–B) Normalized histograms of whisker angle (1° resolution, whisker C1, log scale, 0° lies on the lateral axis) were computed over 100 ms time bins during the first 6 training sessions over correct rejection trials. Each histogram shows data from 21–150 trials. (A) Some mice, such as JF25607, increase the frequency of large deflections during stimulus presentation (trial counts: 99, 150, 135, 109, 102, and 90 respectively). (B) Others, such as JF27332, do not (trial counts: 21, 96, 90, 109, 107 and 86 respectively). (C) During stimulus presentation, an increase in mean deflection angle was correlated with a decrease in the false alarm (FA) rate, a measure of behavioral performance, for two mice (JF25609, JF25607). Two mice did not exhibit this correlation (R2: 0.14, 0.61, 0.97, 0.84 for JF27332, JF26706, JF25609, and JF25607 respectively).
The forces at the base of the whisker constitute the information available to an animal for somatosensory perception. These forces are proportional to changes in whisker curvature
To illustrate this feature we measured curvature in the following manner: For each video frame (
(A–E) The sequence of steps used to extract detailed curvature measurements. (A) Whiskers in raw video frames are automatically traced and linked, yielding (B) an identified set of curves for each frame. (C) The raw curve is fit with a 5th-degree parametric polynomial. (D) A mask is specified to determine where the curve intersects the face. Within a small interval (1–2.5 mm path length) about an interest point chosen for high signal to noise, the raw curve is re-fit to ensure measurements are not biased by whisker shape outside the interval. This new fit is to a 2nd-degree polynomial. The curvature at the interest point is then measured as the curvature of this 2nd fitted curve. (E) Follicle position is estimated by extrapolating a fixed distance into the face from the mask. Similarly, curves are extrapolated, when necessary, to contact points on the pole. Trajectories for curvature (F) and the angle of the whisker at its base (G) are shown for the first contacting whisker in 10 trials grouped by whether the first contact was during a retraction (top 5) or protraction (bottom 5). Trajectories are aligned to first contact. The intervals when the whisker is in whisker-pole contact are highlighted in red. (H) Histograms of peak contact curvature change (from resting) for the first whisker-pole contact in each trial (green) and all whisker-pole contacts prior to an answer-lick (red).
Time series of curvature measurement (
The software and documentation are freely available online (
Sample data and tutorial.
(ZIP)
Line bias correction. (A) Raw data exhibits a fixed-pattern artifact where odd-numbered lines are systematically darker than bright lines. (B) This artifact is corrected by multiplying odd-lines by a factor estimated from the raw data itself. This removes the stripes without blurring or reducing the contrast of whiskers. Scale bar, 0.5 mm.
(TIF)
Methodological details.
(DOC)
Video overlaid with high scoring seeds (eccentricity>0.99) for each frame. Pixels are colored according to the orientation indicated by the seed.
(MP4)
Results of automatic tracing applied to video 500 Hz video (1.7 s duration) of a freely behaving rat exploring a corner with its full whisker field. The video was obtained with permission from the BIOTACT Whisker Tracking Benchmark
(MP4)
Results of automatic tracing applied to a video acquired at 100 Hz video (8 s duration) of a freely behaving mouse with its full whisker field. Background subtraction was performed before tracing to reduce the effects of non-uniform illumination. Because no image of the scene without the animal present was available, the background was estimated for each pixel as the maximum intensity observed at that point throughout the video. The video was obtained with permission from the BIOTACT Whisker Tracking Benchmark
(MP4)
Results from a full trial. The results of automated tracing and linking applied to a video (500 Hz, 9.2 s duration) of a head fixed mouse trimmed to a single row of 4 whiskers interacting with a pole. Curves that were classified as whiskers are colored according to their identity, and otherwise they are not shown. Multiple whiskers simultaneously interact with the pole at 1.2–1.4 sec into the trial. Protracted bouts of whisking can be observed throughout the video.
(MP4)
Tracing and linking is robust to rapid changes in whisker angle. The results of automated tracing and linking applied to a video (500 Hz, 150 ms) of a head fixed mouse trimmed to a single row of 3 whiskers interacting with a pole. Curves that were classified as whiskers are colored according to their identity, and otherwise they are not shown. One whisker (red) has been trimmed so it cannot contact the pole. The green whisker presses against the pole, and quickly flicks past it as it is removed from the field. This is the fastest angular motion (16°/ms) observed in the data set used to measure tracking accuracy.
(MP4)
Linking is robust to whiskers that leave the field of view. The results of automated tracing and linking applied to a video (500 Hz, 190 ms) of a head fixed mouse trimmed to a single row of 3 whiskers interacting with a pole. Curves that were classified as whiskers are colored according to their identity, and otherwise they are not shown. Two whiskers (green and blue) are frequently occluded by the lick-port (black bar, lower right), but they are properly identified before and after such events.
(MP4)
Tracing and linking of whiskers bilaterally. The results of automated tracing and linking applied to a video (500 Hz, 1 s duration) of a bilateral view to a head fixed mouse trimmed to a single row of whiskers (2 on each side).
(MP4)