Frame Reduction: Auto-Curating a Multi-Photo Shelf Capture Burst

📸

TL;DR — A retail shelf is often too long to capture in one photo, so the app takes a rapid burst instead. This post walks through the pipeline that turns a raw burst of 20-80 photos into a clean, gap-annotated sequence ready to stitch into one panorama: a two-stage frame-reduction algorithm (per-photo quality gate, then a sequential anchor-walk), a batched person-detector that keeps NN inference cheap, and a series-merge algorithm for stitching together photos shot in separate passes. Every constant and decision rule below is taken directly from the shipping C# implementation, cross-checked against two generations of the original Java/Cordova reference.

Why this exists

When a field rep photographs a store shelf for planogram compliance, one photo rarely covers the whole run — shelves are long, the rep is close, and the camera's field of view is limited. So the capture flow lets them fire a burst: hold the shutter (or tap repeatedly) and walk the length of the shelf, producing anywhere from a handful to 80 photos at full native camera resolution.

Most of those photos are redundant (near-identical overlap with the previous frame), some are blurry or tilted from a moving hand, and a few might catch a person in frame. Frame Reduction is the algorithm that automatically sorts this out — before the rep ever has to manually review anything — so the photos that get stitched into a single panorama are the minimum clean set that covers the shelf.

The pipeline, end to end

Stage 1 and Stage 2 run automatically right after capture, with zero user interaction. The Review screen then shows the surviving photos as a filmstrip, with colored "pillars" marking any discontinuity the algorithm couldn't resolve on its own — the user's only job at that point is to either accept a gap as a real shelf break, or fill it by capturing/importing more photos, which triggers the Series Merge algorithm (more on that below).

Stage 1 — the per-photo quality gate

Every captured photo is checked independently, in this order, and rejected at the first failure:

Blur

A 3×3 box blur, then the variance of the Sobel-X response on full-native-resolution grayscale:

Tilt

Hough-line detection on a 2×-downscaled grayscale image, looking for the shelf's horizontal edges:

Parameter	Value
Canny thresholds	min 30 / max 160
Hough rho / threshold / aperture	1.0 / 30 / 3
Min line length / max gap	50 / 3
Angle histogram bins	3
Per-line noise filter	20°
Accept/reject threshold	8°

⚠️

A hard-won lesson: there are two different angle constants here and they are easy to conflate. The 20° value is an internal per-line filter — any individual Hough line steeper than 20° is discarded before it gets averaged into the final tilt estimate. The 8° value is the external accept/reject threshold applied to that averaged result. During a migration attempt, the 20° constant was changed to 8° to match a newer reference's call site — and on-device testing immediately showed tilt detection breaking (heavily tilted photos stopped getting flagged). The two constants aren't interchangeable: the internal filter has to stay well above the external threshold, or genuinely tilted lines get thrown out before they can ever push the average past the decision point. The lesson: a bare constant only means what its consuming function makes it mean — verify the full dependency chain before porting a single number.

Person detection — batched to stay fast

Running a neural-net person detector on every single photo in an 80-photo burst is expensive. Instead, surviving photos (post blur/tilt) are tiled into composite grid images and checked in batches:

This only works correctly with one crucial correction: the area-ratio threshold has to scale with the grid size. A person detector reports "found" when a detected box's area exceeds some fraction of the whole image — but tiling 9 photos into one composite shrinks each photo's contribution to 1/9th of the total canvas before the person's own size is even considered. Using the same threshold for a 3×3 composite as for a single photo would silently miss anyone who isn't already a large fraction of the frame. The fix: divide the base threshold by gridDim² at each tier (÷4 for 2×2, ÷9 for 3×3).

The other piece that makes this efficient rather than just "cheap but sloppy": when a composite comes back positive, the detector's bounding box is mapped back to the specific grid cell it falls in (by pixel position — each cell overlapping the box counts, not just its center, so a box near a tile boundary can't be misattributed). That means a positive 3×3 hit doesn't have to re-check all 9 photos individually — it only has to open the specific sub-group(s) that actually contain a flagged cell.

Stage 2 — the sequential anchor walk

Once a photo survives Stage 1, it's compared against its neighbors to find real discontinuities (shelf breaks) versus redundant near-duplicates. The algorithm walks forward through the surviving photos, maintaining one "anchor" at a time:

Resize both the anchor and the candidate to a working resolution — longest side normalized to 680px, regardless of native camera resolution. (An earlier version used a fixed divide-by-4, which meant distance thresholds behaved inconsistently across different camera resolutions.)

Detect SIFT features (capped at 3000 per image) and match anchor↔candidate with a Lowe's-ratio test (ratio 0.75, RANSAC reprojection threshold 20, minimum 6 good matches).

Compute a homography and warp the candidate's four corners into the anchor's frame.

Run a 4-corner geometric overlap check (height ratio 0.20) to decide stitchingCheck, and compute translationRatio = the homography's x-translation divided by the working width.

stitchingCheck	translationRatio	Last photo?	Outcome
true	≤ 0.4	no	Trimmed as redundant; anchor unchanged, walk continues
true	≤ 0.4	yes (+ valid distances)	Force-kept anyway ("always keep the last frame")
true	> 0.4	—	Clean new anchor
false	—	no	New anchor anyway, flagged as a gap (`RequireLeft`)
false	—	yes	Straggler dropped; previous anchor gets a trailing gap instead

The 0.4 is MinDistanceCoefficient. There's also a MaxDistanceCoefficient of 0.8, but it isn't a standalone comparison — it's baked into the geometric overlap check itself (step 4), bounding how far the warped corners are allowed to land. This is a deliberate simplification from an earlier design: that older version explicitly compared translation against a max-distance threshold and, on overshoot, stepped the anchor back to retry against an earlier candidate. The current design has no such step-back at all — the anchor only ever advances forward, and "too far to stitch" just falls out of the same geometric overlap check as any other stitching failure. Removing that recursive step-back also removed a class of bug where an already-rejected duplicate frame could get re-selected as the anchor on the bounce-back.

Series Merge — stitching together separately-shot passes

Sometimes one continuous burst still isn't enough — a shelf run gets shot in 2-3 separate passes (left section, middle section, right section), and those passes almost always overlap a little at the seams. The Series Merge algorithm finds that overlap and trims it, rather than letting duplicate content survive into the final stitch.

It reuses almost the entire Stage 2 pipeline (SIFT/RANSAC matching, 680px normalization, homography corner transforms) with one genuinely new piece: directional overlap checks. The sequential anchor walk only ever needs to know "does this stitch," but merging independent series needs to know which side the overlap is on — a candidate's warped corners have to land specifically to the right of a left-anchor, or to the left of a right-anchor, not just within some generic distance bound.

How it's actually triggered

This isn't a separate menu action — it runs automatically every time the user inserts new photos at any of the pager's camera icons, and which series play which role falls out entirely from where the insertion happened:

Insertion point	Left series	Middle series	Right series
Leftmost pillar (before everything)	— (none)	new photos	all existing photos
Rightmost pillar (after everything)	all existing photos	new photos	— (none)
A gap in the middle	existing photos before the gap	new photos	existing photos after the gap

The new batch is always the "middle" series being trimmed; whatever's already been reviewed on either side is the anchor. Since the overlap-trim only ever walks inward from an edge, the surviving new photos are always a contiguous range — there's no risk of it carving a hole out of the middle of a freshly captured batch.

Rebuild reference — every constant that matters

Constant	Value	Used for
Working resolution (longest side)	680 px	Stage 2 + Series Merge working size
Min distance coefficient	0.4	Redundant-vs-new-anchor decision
Max distance coefficient	0.8	Geometric overlap bound (not standalone)
Min blur factor	1000.0	Sobel-X variance threshold
Tilt threshold	8°	Accept/reject on averaged tilt angle
Internal tilt line filter	20°	Per-Hough-line noise filter (NOT the same as the 8° above)
Height ratio for stitching	0.20	4-corner overlap check vertical tolerance
Lowe's ratio	0.75	SIFT match filtering
RANSAC reprojection threshold	20.0	Homography estimation
Minimum good matches	6	Below this, treated as a stitching failure
Max SIFT features	3000	Feature cap per image
Person detector thresholds	0.05 / 0.45	Confidence / IoU (NMS)
Person area-ratio threshold	0.02 (÷ gridDim² per tier)	Person-detection decision
Person grid group size	9 (3×3), then 4 (2×2)	Batched NN-call tiers
Canny thresholds (tilt)	30 / 160	Edge detection for Hough lines
Hough params	rho 1.0, threshold 30, aperture 3, min length 50, max gap 3	Line detection
Angle histogram bins	3	Tilt angle averaging
Down-size factor (angle)	2.0×	Tilt-check working resolution

Where things stand

Piece	Status
Blur / tilt / all-blurry failsafe	Stable, unaffected by any recent changes
Stage 2 anchor walk (no step-back, 680px normalization)	Confirmed on real devices, every decision branch exercised
Person-detection batching + threshold scaling	Implemented; validated against people occupying 23%+ of frame — smaller/farther people not yet stress-tested
Series Merge, wired into the pillar-insert flow	Implemented, compiles clean; on-device testing in progress
Foreign-object insertion edge case	Still needs a clean capture that survives Stage 1's blur/tilt gate before Stage 2 can be exercised against it

🔑

The single biggest source of correctness bugs in this whole effort wasn't algorithmic — it was constants that look interchangeable but aren't (two angle thresholds with the same name-shape but different roles), and enum branches that silently no-op when a new status value is added but a downstream switch statement isn't updated to handle it. Both are the kind of bug that produces plausible-looking wrong output rather than a crash, which is exactly why log-driven, on-device verification — reading real decision traces rather than trusting the code to be self-evidently correct — did more to catch real issues here than reasoning about the algorithm in the abstract.