CNN object detection: the heart region in microscope videos
For my bachelor’s thesis I trained an object detector to find the heart region in microscope videos of medaka embryos, a small fish used in biological research. The detector was built for single color photos; a heart only shows itself as movement over time. Most of the work went into changing what the model sees.
- Role
- Bachelor’s thesis, sole author
- Where
- Heidelberg University, with the Centre for Organismal Studies
- Result
- mAP@0.5 0.86, precision 0.93, recall 0.84
- Limits
- About 100 videos; validation values
Figures and frames are taken from the thesis (January 2023, graded 1.0, unpublished). The embryo recordings come from the partner research group.
Detections at stage 2 and stage 10
Problem: a heart is only visible as motion
Heart rate is the main signal for how these embryos respond to stress and other stimuli. The research group located the heart with one frequency analysis and measured the beat with a second. My thesis replaces the first step with a neural network, YOLOv5.
In a single frame, the heart looks like the tissue around it. You see it because it moves: one beat takes 14 to 15 frames. YOLOv5 expects one color image. The dataset had about 100 videos of 240 frames each, about a fifth of them hard cases with opaque embryos or hidden hearts.
Failed approaches: pretrained weights, heavy augmentation, CLAHE
- Pretrained weights: mAP@0.5 of 0.015 with three grayscale frames, worse with 20 channels (about 90 percent came from the first three). Only training from scratch learned.
- Heavy augmentation hurt (all videos were recorded under the same conditions); only scaling and flipping helped.
- Contrast enhancement made a few dark cases visible and lowered every other value.
Approach: difference maps instead of RGB
Movement instead of color. I replaced the RGB image with 20 grayscale frames, then with difference maps between frames, which the thesis identifies as the most important step. Comparing every frame with one base frame also revealed faint movement that frame-to-frame differences missed. Stage 10 added the last frame itself.
Cropping the videos from 900 to 480 pixels cut the training time per epoch from 360 seconds to 6, which made the many small experiments possible.
Difference maps: base frame vs. frame to frame
-
Stage 4: Trained from scratch
Pretrained weights did not transfer: fed three grayscale frames, the pretrained detector reached an mAP@0.5 of 0.015 (stage 2). Training from scratch produced the first learning effect, 0.10 on all three values.
-
Stage 5: Scaling and flipping
Heavy augmentation hurt, because every video was recorded under the same conditions. Only scaling and flipping helped: precision and recall rise to 0.30.
-
Stage 6: Difference maps
Movement instead of color. The network gets differences between frames instead of the frames themselves. The thesis names this the most important step: mAP@0.5 goes from 0.15 to 0.77.
-
Stage 9: Base frame method
Comparing every frame with one base frame, instead of with the frame before, revealed faint movement. All three values reach 0.84 or more.
-
Stage 10: Plus last frame
Stage 10 adds the last frame itself. Precision climbs to 0.93, recall stays at 0.84, mAP@0.5 at 0.86. That is the result of the thesis, on the validation set.
Experiment map: 11 stages, 3 side branches
- Stage 1 pretrained 1× RGB
- Stage 2 pretrained 3× grayscale
- Stage 3 pretrained 20× grayscale
- Stage 4 scratch 20× grayscale
- Stage 5 scratch + augment
- Stage 6 scratch + diff-maps
- Side branch 6.1 + rebalanced dataset
- Stage 7 scratch + cos-lr
- Stage 8 scratch + keep-base
- Side branch 8.1 + AHE
- Side branch 8.2 + CLAHE
- Stage 9 scratch + new diff-maps
- Stage 10 scratch + keep-last-and-base
- Stage 11 scratch 20× + 20× grayscale
Tuned along the way: batch size, model, channels, loss, augmentation, optimizer, NMS, image size.
Redrawn from figure 12 of the thesis. Each stage keeps the changes of the stage before it; colors follow the rating in the original, from red to dark green.
Limitations
- Fully hidden hearts are only partly found; hard cases come with low confidence.
- Small dataset, so the numbers are of limited statistical weight; all of them come from the validation set.
- The thesis finds the heart region; whether that is good enough for the heart rate measurement was left open.
- 3.96 seconds per video on a workstation GPU, 28.6 seconds on 12 CPU cores: slower than the target.
Built with: Python, PyTorch, YOLOv5