3  Pose estimation with SLEAP

Before we proceed, make sure you have installed SLEAP v1.6.3 (Pereira et al. 2022) and activated the corresponding conda environment (see prerequisites A.3.2). You will also need to download the CalMS21 dataset (Sun et al. 2021) video file mouse044_task1_annotator1.mp4 from Dropbox (see prerequisites A.4 for details on the folder contents).

3.1 Single-animal vs multi-animal pose estimation

Single-animal pose estimation, source: Pereira et al. (2020).

Single-animal pose estimation focuses on detecting keypoints for one animal per frame, which is considered a landmark-localisation task where each body part has a unique coordinate. This approach is simpler and faster to train and run.

The part-grouping problem in multi-animal pose estimation.

Multi-animal pose estimation aims to detect and track multiple animals simultaneously within the same frame. This is essential for studying social behaviours, group dynamics, or any scenario where animals interact, as it addresses the unique challenges of assigning detections reliably to individuals both within an image (part-grouping problem) and across frames (identity-tracking problem).

3.2 Top-down vs bottom-up approaches

For multi-animal pose estimation, SLEAP offers both top-down and bottom-up approaches.

Source:Pereira et al. (2022)

Top-down approaches use two models in sequence. First, an anchor detection model (e.g. a centroid model) locates each animal in the frame. Then, for each detected animal, a pose estimation model processes an anchor-centred crop to predict confidence maps for the body parts of the centred animal.

This approach typically yields more accurate pose estimates and is well-suited to datasets with few animals. As the second stage of the network runs once per animal, inference speed scales linearly with the number of animals.

Source: Pereira et al. (2022)

Bottom-up approaches use a single model that processes the entire frame in a single pass. This model outputs confidence maps for all body parts in the image, along with Part Affinity Fields (PAFs)—vector fields that represent spatial relationships between parts and are used to group them into individual animal instances.

Due to their single-stage construction, bottom-up models scale efficiently with increasing numbers of animals and are particularly effective in crowded or high-occupancy scenes.

3.3 Identity tracking approaches

SLEAP addresses the challenge of maintaining consistent animal identities across frames using two primary strategies: temporal-based and appearance-based cues.

Temporal-based tracking uses optical flow to estimate pose displacement across frames, associating past with current poses without requiring model training. This makes it well-suited for animals that are visually similar, as it avoids the need to label consecutive frames. However, errors, such as identity switches, can accumulate and propagate, limiting its reliability in long videos or real-time settings where post-hoc correction is not feasible.

Appearance-based tracking (ID models) assigns identities based on visual features while simultaneously detecting and grouping landmarks. This approach mitigates error propagation but relies on animals having distinguishable visual traits that allow manual identification during labelling.

  • Top-down ID models extend the centered-instance network to predict class probabilities for each animal-centred crop.
  • Bottom-up ID models replace PAFs with multi-class segmentation maps, collapsing body part masks into separate channels for each unique class ID. Grouping is implicit in the ID assignment.

3.4 Resident–intruder assay

The resident–intruder assay (Koolhaas Jaap M. 2013) is a behavioural test used to study social interactions, especially aggression and territoriality, in rodents. A resident mouse, habituated to its home cage, is confronted with an unfamiliar intruder, and their interactions—such as chasing, attacking, or investigating—are observed and quantified. This assay is widely used in neuroscience to explore the neural and genetic basis of social behaviour.

3.5 Dataset

In this tutorial, we will use SLEAP to train a multi-animal top-down identity model to simultaneously perform pose estimation and identity tracking of two mice in a short video (mouse044_task1_annotator1.mp4) from the CalMS21 dataset.

This video captures a brief interaction between two mice in a resident–intruder assay, where the black mouse (the resident), implanted with a head-mounted microendoscope, has established territory and the white mouse (the intruder) is newly introduced into the resident’s cage.

3.6 SLEAP workflow

Note

The workflow described here closely follows the official SLEAP tutorial, adapted to our dataset. Refer to the official docs for additional details and alternative workflows.

graph LR
    videos("Videos<br>(1,2,...,n)") --> |extract| frames[/Sampled<br>frames/]
    frames --> |"label<br>body parts<br>and ID"| labels[/Training<br>dataset/]
    labels --> |train| model[/Model/]

    videos --> test[/Unseen<br>frames/]
    test --> model
    model --> |infer| predictions[/Predictions/]
    
    predictions --> |fix labels<br>and merge| labels

A typical SLEAP workflow for multi-animal pose estimation and identity tracking consists of the following key steps:

  1. Create project: Start a new SLEAP project and import your video(s).
  2. Define skeleton: Create a Skeleton that defines the Nodes (each representing a keypoint or body part of interest, e.g. nose, left ear, right ear) and Edges (each representing the connections between keypoints, e.g. nose–left ear, nose–right ear) for the animals to be tracked.
  3. Sample frames: Extract frames from your video(s) to create a set of frames for annotation.
  4. Label frames: Annotate the sampled frames by marking the body parts and assigning identities (Tracks) to each animal. These labelled frames together form the training dataset.
  5. Train model: Use the training dataset to train a pose estimation and identity tracking model.
  6. Predict on new data: Apply the trained model to new, unlabelled video frames to generate pose and identity predictions for each animal.
  7. Proofread predictions: Review and correct the predictions as needed.
  8. Refine model: Corrected predictions can be merged back into the training dataset to retrain the model as needed.

3.6.1 Create a new project

Activate the sleap environment and launch the SLEAP GUI.

conda activate sleap
sleap

Add a video by navigating to the “Videos” panel and clicking on “Add Videos” (or go to “File” → “Add Videos”). Since the video is in greyscale, enable the “Grayscale” option in the video import dialogue. This ensures SLEAP processes the input as a single-channel image, which can improve performance and reduce memory usage for greyscale videos. Further details can be found in SLEAP’s Import videos guide.

Tip

This tutorial uses a single video for simplicity, but actual projects should include all videos you intend to analyse. Add them all to the project upfront, then sample frames from across as many of them as possible when building your training set. A model trained on frames from only one video tends to overfit to that video’s specific conditions (lighting, background, camera angle) and generalises poorly to others.

3.6.2 Define skeleton

In SLEAP, Skeletons are defined as a set of Nodes (body parts or keypoints of interest) and Edges (connections between body parts or keypoints). With the exception of bottom-up models, Edges serve primarily for visualisation.

Switch to the Skeleton panel and add Nodes for each body part of interest, e.g.:

  • nose
  • right_ear
  • left_ear
  • neck
  • right_hip
  • left_hip
  • tail_base

Then, define the Edges to connect the Nodes using the drop-down menus, e.g.:

  • nose–left_ear
  • nose–right_ear
  • left_ear–neck
  • right_ear–neck
  • neck–left_hip
  • neck–right_hip
  • left_hip–tail_base
  • right_hip–tail_base

Once you have defined the skeleton, save the project by clicking on “File” → “Save” (or with Ctrl/Cmd+S).

3.6.3 Sample frames

For assembling a set of frames for annotation, you can either pick your own frames, or let SLEAP suggest a set of frames using the “Labeling Suggestions” panel, which offers several automated sampling strategies to help select informative and diverse frames.

In this example, we will use the “Labeling Suggestions” panel to randomly sample 20 frames from the video.

3.6.4 Label frames

To begin labelling, click on “Labels” → “Add Instance” in the top menu bar. The initial instance will have its nodes placed randomly. Adjust each point to its correct location by dragging it with the mouse.

In this example, the black mouse’s nose is occluded by an implant. To mark the node as hidden, right-click the node or its name to toggle visibility. If you can reasonably infer its location, you can also mark it as “visible” to help the model learn to predict occluded nodes.

For tracking the identities of the animals, we will also assign identities to each instance by adding a Track. This step is optional if you are only tracking a single animal or if the animals are visually indistinguishable.

To assign a track, select an instance and click on “Tracks” → “Set Instance Track” in the top menu bar. A new Track (“1”) will be created. Tracks can be renamed in the “Track” column. In this example, we will name the black mouse resident_b and the white mouse intruder_w.

Once you have labelled the initial frame, you can navigate to the previous or next suggested frame by clicking “Previous” or “Next” in the “Labeling Suggestions” panel.

To speed up labelling in subsequent frames, right-click on the frame and choose from several options for adding a new instance. For example, selecting “Copy prior frame” will duplicate the instance(s) from the previous labelled frame, allowing you to quickly adjust only the necessary points. This is especially useful when animal poses change gradually between frames.

Important

Remember to save your progress frequently by clicking “File” → “Save” (or with Ctrl/Cmd+S).

You may also find the following keyboard shortcuts and mouse actions helpful (go to “Help” → “Keyboard Shortcuts” for the full list and optionally customise them):

Function Keyboard shortcut/Mouse actions
Add instance Ctrl/Cmd+I
Add new track Ctrl/Cmd+0
Assign track Select instance, then Ctrl/Cmd+1-9 (number = track ID)
Toggle node visibility (occlusion) Right-click on node or its label
Move entire instance Hold Alt (Windows) / Option (Mac) and drag any node
Rotate instance Hold Alt (Windows) / Option (Mac), click on any node, and scroll mouse wheel
Zoom in/out Place cursor over area to zoom in or out, then scroll mouse wheel
Delete instance Select instance, then Ctrl/Cmd+Backspace
Navigate between frames Left/Right
Go to the next suggested frame Space
Go to the previous suggested frame Shift + Space
Go to the next labelled frame Alt (Windows) / Option (Mac) + right
Go to the previous labelled frame Alt (Windows) / Option (Mac) + left

See also SLEAP’s GUI guide on “Labels” for a complete reference of all labelling functions.

TipDiscuss
  • How did you find the labelling process? What challenges did you encounter?
  • What strategies could annotators use to ensure consistency throughout the labelling process?
  • How might the quality of annotations affect model performance?

3.6.5 Configure and train models

Once you have labelled a sufficient number of frames, you can configure and train your first model. To do this, go to “Predict” → “Run training” in the top menu bar.

Here, we will employ the appearance-based tracking approach for maintaining consistent identities of individual animals across frames. To do so, select “multi-animal top-down-id” as the “Pipeline Type”. We will also configure the training pipeline to predict on 20 more frames randomly sampled from the video (skipping user-labelled frames) once training is complete.

See also SLEAP’s Configuring models guide for further details on model types and training options.

3.6.5.1 Centroid model

We will now configure the Centroid model, which predicts the location of each animal in each frame.

Since we have labelled only a small number of frames, increase the validation fraction (e.g. 0.2) in “Data” to obtain more stable and representative validation metrics. To reduce the risk of overfitting, in “Optimization”, set the number of training epochs to a low value (e.g. 10–20 epochs).

Next, adjust the receptive field size to match the scale of the features you want the model to detect. This can be achieved by adjusting “Input Scaling” in “Data” and “Max Stride” in “Model”.

To help the model generalise to animals in different orientations, in “Augmentation”, set “Rotation” to match how much orientation varies in your data. The default (±180°) is reasonable when animals can appear at any orientation, but unnecessarily broad when they cannot. For example, in a head-fixed pupillometry setup, eye rotation is minimal, so a narrow range (e.g. ±15°) is more appropriate and avoids forcing the model to learn orientations it will never encounter.

Tip

As a rule of thumb, the receptive field (blue box in the preview) should be large enough to cover the whole animal. Since the centroid model only needs to locate animals coarsely, you can downsample the input more aggressively (e.g. to 0.25×) than for the top-down ID model, which needs to resolve fine details. This saves computation, especially in early labelling rounds with limited training data.

3.6.5.2 Top-down ID model

The Top-down ID model predicts the full pose (locations of all defined nodes) and assigns identities to each animal in each frame. It has two output heads: confmaps (confidence maps), which predict the spatial location of each body part, and class_vectors, which predict the identity of each animal.

As with the Centroid model, configure the validation split, number of training epochs, random rotation augmentation, and receptive field size accordingly.

You may also want to increase the sigma of the confmaps head (e.g. to 5.0) to produce wider peaks that are easier to learn from limited training data, increase the loss weight of the class_vectors head (e.g. to 1.0) so that the model places more emphasis on learning identity prediction, and reduce the batch size (e.g. to 4) to fit within memory constraints, especially when using limited hardware resources (e.g. CPU or lower-end GPUs).

Once you have configured the models, click “Run” to begin training.

3.6.6 Monitor training progress

You should now see a training progress window showing the loss curves and metrics for each model as it trains.

Centroid model training progress

Top-down ID model training progress

You can stop a training run at any time and resume it later, or start a new run using the weights from a previous model. For details, see Section 3.6.9.

Tip

SLEAP supports remote training and inference workflows, allowing users to leverage external computational resources, particularly those with GPU support, for batch training or inference. For detailed instructions and guidance, see the Running SLEAP remotely guide.

3.6.7 Run inference and proofread predictions

After the models are trained, if you had configured the training pipeline to run inference upon completion, SLEAP will automatically apply the trained models to the selected frames.

Alternatively, to manually run inference on random frames with your trained models, go to “Predict” → “Run Inference” in the top menu bar.

Note

Since we have trained an appearance-based ID model, which outputs identity assignments directly for each detected animal in each frame, there is no need to configure a “Tracker” (typically used for temporal-based tracking). Further details on the Tracker module is available in SLEAP’s Tracking and proofreading guide.

Labelled frames (frames with user or predicted labels) will be marked in the seekbar.

To step through labelled frames, click on “Go” → “Next Labeled Frame” (or use Alt (Windows) / Option (Mac) + left/right).

With only a handful of labelled frames, the first models may produce poor predictions, but you can use these initial predictions as a starting point: review and correct them, merge the corrections back into your training dataset, and optionally generate additional labelling suggestions to broaden the range of poses and variations represented in your data. Once you have added and/or corrected more instances, you can begin the iterative workflow: train a new model, predict on more frames, correct those predictions, and repeat this cycle until the model is ready to be applied to the entire video.

Although early rounds use a low number of epochs for faster iteration, with very few labelled frames the model may occasionally fail to learn anything and inference returns no instances. If this happens, increase the number of training epochs for that round. Even though the model will overfit such a small dataset, the extra training can help it latch onto the limited signal and produce usable predictions for the next iteration. “Stop training on plateau” is enabled by default, so training will stop automatically once the model stops improving. You can also increase the number of filters (e.g. to 32) under “Model” to increase model capacity, at the cost of longer training time.

See SLEAP’s Prediction-assisted labeling guide for more details on this iterative process.

Important

Predicted instances will not be used for model training until you convert them into editable instances (by double-clicking on them).

Tip

SLEAP includes a Label Quality Control (QC) module that automatically detects common labelling errors using anomaly detection. Use it to check your labels for errors before training or after proofreading predictions.

3.6.8 Evaluate trained models

The metrics of trained models can be accessed by clicking on “Predict” → “Evaluation Metrics for Trained Models” in the top menu bar.

See SLEAP’s Model evaluation guide for examples on generating accuracy metrics for your trained model.

3.6.9 Continue training across sessions

SLEAP lets you stop a run early, pick it up again later, or start over from scratch.

3.6.9.1 Stop a run early

While a model is training, the training window offers two ways to end it early:

  • Stop Early ends the current training loop and saves the model weights from the epoch with the lowest validation loss.
  • Cancel Training aborts the run and discards it entirely; no model is saved.

3.6.9.2 Choose a starting point for the next run

To resume training, go to “Predict” → “Run Training” as before, then on the “Model Configuration” tab, use the dropdown at the top to select a previously trained config (prefixed with [Trained]). You can then choose one of three options:

Option What it does When to use it
Reuse config (train from scratch) Keeps the previous run’s settings but discards its weights, training a new model from a random initialisation on all currently available labelled frames. You’ve added new labelled frames since the last run—especially early on, while your dataset is still small. Full retrains are fast at this scale, and starting fresh avoids inheriting weaknesses from a model trained on very little data.
Resume training (fine-tune) Initialises the new model with the previous run’s weights, then trains further on all currently available labelled frames. You stopped early and want to resume the same run with no new data; your dataset/model has grown large enough that a full retrain would be slow; you proofread predictions and corrected a systematic mistake (e.g. a consistently misplaced tail) and want the model to learn from those specific corrections; or you want transfer learning—adapting an accurate model from one setting (e.g. black mice) to a related one (e.g. white mice in the same cage).
Reuse model (don’t retrain) Uses the previous run’s weights as-is, with no further training For top-down approaches: e.g. you’re satisfied with the centroid model and only want to retrain the top‑down ID model.

To train a brand-new model instead, leave the dropdown on its default baseline.*.yaml entry rather than selecting a previous run.

See also SLEAP’s Interactive and resumable training guide for customising your own training workflow using sleap-nn’s low-level API.