Tongue tracking

Segmenting the tongue in high-speed behavior video

With Bruno Cruz.

In licking tasks such as dynamic foraging, a mouse reports each choice with its tongue, so the tongue is the effector of every decision. Tongue tracking segments it in every frame of high-speed side-camera video, using a U-Net trained on hand-labeled frames.

Bruno Cruz started the project. My contributions were the labeling loop and inference at scale:

  • Labeling. I moved the labeling tool to SAM2’s video model, so a few clicks on one frame propagate a mask across neighbouring frames, and added a review mode that ranks the model’s own predictions by mask-quality statistics, so corrected frames flow back into the training set.
  • Training. I added optional temporal context frames as input and tuned the loss and augmentation for the tongue’s edges, where errors concentrated.
  • Inference. A four-stage pipeline with batched mixed-precision inference, frame-exact seeking and optional GPU video decoding streams masks for long sessions into compact COCO-RLE Parquet files. A Code Ocean capsule runs it in batch across sessions.