Skip to content
Ben Dangelmayr
mAP@0.5 per stageValidation set, bachelor’s thesisDifference maps0.100.150.770.860.86Stage 456910
Research Bachelor’s thesis: CNN object detection

All work

CNN object detection: the heart region in microscope videos

For my bachelor’s thesis I trained an object detector to find the heart region in microscope videos of medaka embryos, a small fish used in biological research. The detector was built for single color photos; a heart only shows itself as movement over time. Most of the work went into changing what the model sees.

Role
Bachelor’s thesis, sole author
Where
Heidelberg University, with the Centre for Organismal Studies
Result
mAP@0.5 0.86, precision 0.93, recall 0.84
Limits
About 100 videos; validation values

Figures and frames are taken from the thesis (January 2023, graded 1.0, unpublished). The embryo recordings come from the partner research group.

Detections at stage 2 and stage 10

Grayscale embryo frame cluttered with many stacked blue boxes, each labelled heart 0.001, far from the heart.
Start (stage 2): a pretrained detector fed three grayscale frames. Boxes pile up at random with a confidence of 0.001.
Grayscale microscope frame of a medaka embryo inside its egg, with two overlapping rectangles over the heart region and the label heart 0.86.
End (stage 10): predicted and annotated box almost coincide on an easy case, confidence 0.86.

Problem: a heart is only visible as motion

Heart rate is the main signal for how these embryos respond to stress and other stimuli. The research group located the heart with one frequency analysis and measured the beat with a second. My thesis replaces the first step with a neural network, YOLOv5.

In a single frame, the heart looks like the tissue around it. You see it because it moves: one beat takes 14 to 15 frames. YOLOv5 expects one color image. The dataset had about 100 videos of 240 frames each, about a fifth of them hard cases with opaque embryos or hidden hearts.

Failed approaches: pretrained weights, heavy augmentation, CLAHE

  • Pretrained weights: mAP@0.5 of 0.015 with three grayscale frames, worse with 20 channels (about 90 percent came from the first three). Only training from scratch learned.
  • Heavy augmentation hurt (all videos were recorded under the same conditions); only scaling and flipping helped.
  • Contrast enhancement made a few dark cases visible and lowered every other value.
Left: an original frame. Right: the same frame after contrast enhancement (CLAHE). It helped in single cases and made the overall result worse.

Approach: difference maps instead of RGB

Movement instead of color. I replaced the RGB image with 20 grayscale frames, then with difference maps between frames, which the thesis identifies as the most important step. Comparing every frame with one base frame also revealed faint movement that frame-to-frame differences missed. Stage 10 added the last frame itself.

Cropping the videos from 900 to 480 pixels cut the training time per epoch from 360 seconds to 6, which made the many small experiments possible.

Difference maps: base frame vs. frame to frame

Base frame: every frame minus frame 1 1 2 3 4 20 … Frame to frame: every frame minus the one before 1 2 3 4 20 …
Redrawn from figure 19 of the thesis.
  1. Stage 4: Trained from scratch

    Pretrained weights did not transfer: fed three grayscale frames, the pretrained detector reached an mAP@0.5 of 0.015 (stage 2). Training from scratch produced the first learning effect, 0.10 on all three values.

  2. Stage 5: Scaling and flipping

    Heavy augmentation hurt, because every video was recorded under the same conditions. Only scaling and flipping helped: precision and recall rise to 0.30.

  3. Stage 6: Difference maps

    Movement instead of color. The network gets differences between frames instead of the frames themselves. The thesis names this the most important step: mAP@0.5 goes from 0.15 to 0.77.

  4. Stage 9: Base frame method

    Comparing every frame with one base frame, instead of with the frame before, revealed faint movement. All three values reach 0.84 or more.

  5. Stage 10: Plus last frame

    Stage 10 adds the last frame itself. Precision climbs to 0.93, recall stays at 0.84, mAP@0.5 at 0.86. That is the result of the thesis, on the validation set.

Stage 4, Trained from scratch: mAP@0.5 0.10, precision 0.10, recall 0.10; Stage 5, Scaling and flipping: mAP@0.5 0.15, precision 0.30, recall 0.30; Stage 6, Difference maps: mAP@0.5 0.77, precision 0.73, recall 0.80; Stage 9, Base frame method: mAP@0.5 0.86, precision 0.85, recall 0.84; Stage 10, Plus last frame: mAP@0.5 0.86, precision 0.93, recall 0.84 0.00 0.25 0.50 0.75 1.00 stage 456910
mAP@0.5 precision recall Validation set. Values from the milestone table of the thesis, which lists these five stages; the 0.015 of stage 2 is from its text.

Experiment map: 11 stages, 3 side branches

  1. Stage 1 pretrained 1× RGB
  2. Stage 2 pretrained 3× grayscale
  3. Stage 3 pretrained 20× grayscale
  4. Stage 4 scratch 20× grayscale
  5. Stage 5 scratch + augment
  6. Stage 6 scratch + diff-maps
    • Side branch 6.1 + rebalanced dataset
  7. Stage 7 scratch + cos-lr
  8. Stage 8 scratch + keep-base
    • Side branch 8.1 + AHE
    • Side branch 8.2 + CLAHE
  9. Stage 9 scratch + new diff-maps
  10. Stage 10 scratch + keep-last-and-base
  11. Stage 11 scratch 20× + 20× grayscale

Tuned along the way: batch size, model, channels, loss, augmentation, optimizer, NMS, image size.

Redrawn from figure 12 of the thesis. Each stage keeps the changes of the stage before it; colors follow the rating in the original, from red to dark green.

Limitations

  • Fully hidden hearts are only partly found; hard cases come with low confidence.
  • Small dataset, so the numbers are of limited statistical weight; all of them come from the validation set.
  • The thesis finds the heart region; whether that is good enough for the heart rate measurement was left open.
  • 3.96 seconds per video on a workstation GPU, 28.6 seconds on 12 CPU cores: slower than the target.
Grayscale frame of a darker, more opaque medaka embryo; a small box labelled heart 0.44 sits in a low-contrast area.
Hard case: an opaque embryo with little contrast at the heart. The model still finds the region, with a confidence of 0.44.

Built with: Python, PyTorch, YOLOv5