Computer vision pipelines for killer whale and humpback analysis
Built two end-to-end CV pipelines that turn raw aerial drone imagery of killer whales and humpbacks into standardized morphometric data — landmarks, segmentation masks, and structured JSON — for Ocean Wise's whale health research.
Challenge
Ocean Wise conducts long-term research on whale health using aerial drone imagery, measuring body condition, tracking pregnancy, and monitoring population health across killer whales and humpbacks. The process was scientifically rigorous but fully manual: each image required expert review to place landmarks and extract anatomical measurements. Landmark placement was time-intensive, measurements were difficult to standardize across large datasets, image quality varied significantly, and scaling analysis across thousands of drone images was not feasible.
Our approach
- Designed a shared pipeline structure for both species: Detection → Crop → Quality Gate → Landmark Extraction → JSON Output
- Killer whales: optimized the existing whale detector for better bounding box consistency and recall, then built a quality gate to keep only frames where both eye patches are visible
- Trained a keypoint model to detect 4 eye patch landmarks for body condition and pregnancy studies, plus a 6-point pregnancy landmark model across the whale's body
- Humpbacks: trained a new object detector, a quality gate classifier, an instance segmentation model for full whale outlines, and a keypoint model returning 8 anatomical landmarks
- Combined model predictions with geometric reasoning — deriving the final 2 landmarks algorithmically from the segmentation mask and the rostrum-to-tail-notch body axis — to produce all 10 required points reliably
- Integrated every model into Roboflow Workflows and deployed the system as a single API call returning detections, quality classification, keypoints, segmentation masks, and standardized JSON