Tongue tracking
Segmenting the tongue in high-speed behavior video
With Bruno Cruz.
In licking tasks such as dynamic foraging, a mouse reports each choice with its tongue, so the tongue is the effector of every decision. Tongue tracking segments it in every frame of high-speed side-camera video, using a U-Net trained on hand-labeled frames.
Bruno Cruz started the project. My contributions were the labeling loop and inference at scale:
- Labeling. I moved the labeling tool to SAM2’s video model, so a few clicks on one frame propagate a mask across neighbouring frames, and added a review mode that ranks the model’s own predictions by mask-quality statistics, so corrected frames flow back into the training set.
- Training. I added optional temporal context frames as input and tuned the loss and augmentation for the tongue’s edges, where errors concentrated.
- Inference. A four-stage pipeline with batched mixed-precision inference, frame-exact seeking and optional GPU video decoding streams masks for long sessions into compact COCO-RLE Parquet files. A Code Ocean capsule runs it in batch across sessions.