Cropping Prerequisites — Manual Shelf Mask, Polygon Generation, and ROI Adjustment¶

Overview¶

Before automated bottle-cell cropping can run reliably, the system must first know the exact polygon geometry of each shelf compartment.

This geometry is not derived directly from the raw camera image.
Instead, the workflow uses a hybrid manual + automated preparation process:

create a rectified shelf image with take_clean_snapshot.py
manually draw blue separator lines over the wood structure of the shelf
convert the resulting mask image into cell polygons with generate_mask_for_polc.py
optionally fine-adjust the generated polygons with ROI_ADjustment_from_PNG.py

This preparation stage is a prerequisite for later crop-based processing.

The workflow is intentionally semi-manual because:

the shelf geometry is fixed but visually complex
the wooden dividers are diagonal and perspective-sensitive
fully automatic initial ROI extraction would be less reliable
once created, the polygon JSON can be reused and only occasionally adjusted

Position in the Overall Vision Workflow¶

flowchart TD

SNAP["take_clean_snapshot.py"]
RECT["shelf_rectified.jpg"]

MASK["Manual blue-line mask drawing<br>Canvas / image editor"]
GEN["generate_mask_for_polc.py"]
JSON["cells_polygons.json"]
ADJ["ROI_ADjustment_from_PNG.py"]
CROP["crop_with_json.py"]

SNAP --> RECT
RECT --> MASK
MASK --> GEN
GEN --> JSON
JSON --> ADJ
ADJ --> JSON
JSON --> CROP

Why a Manual Mask Is Needed¶

The shelf consists of many diagonal wooden separators forming diamond-shaped bottle cells.

Even after perspective rectification:

wood edges may vary in tone
bottle presence can obscure separators
lighting can vary
the exact visible boundary of each cell is not always easy to derive robustly from vision alone

Because of this, the system uses a deliberate manual markup step:

the user opens shelf_rectified.jpg
draws blue lines over the visible wood separator structure
saves the annotated image
the script extracts the non-blue connected components as the shelf cells

This gives a very practical result:

human defines the structural separators visually
script converts them into machine-usable polygons

Input Prerequisite: shelf_rectified.jpg¶

The manual mask must always be based on a previously created:

shelf_rectified.jpg

This image is produced by the snapshot/rectification stage.
It is the normalized front-facing shelf image and is the only appropriate base image for mask creation.

Why this matters:

the mask must align with the corrected shelf geometry
the mask should not be drawn on the raw RTSP frame
all downstream polygon coordinates are expected in the rectified image coordinate system

Important Upstream Dependency¶

The rectified shelf image itself depends on the manually configured main_points stored in pince_shelf.ini, which are set from the kiosk ROI page. :contentReference[oaicite:1]{index=1}

This means the chain is:

user manually defines the four main shelf points in kiosk
take_clean_snapshot.py uses those points to rectify the shelf
user draws the mask on shelf_rectified.jpg
mask is converted into cell polygons

So the cropping prerequisite workflow depends on the calibration workflow being already completed.

Manual Mask Creation¶

Goal¶

Create an image in which blue separator lines cover the wooden shelf divider structure.

These blue lines are interpreted as cell borders.

The script then treats the regions between those lines as candidate shelf cells.

Recommended Base Image¶

Use:

<snapshot_dir>/shelf_rectified.jpg

Open it in a drawing-capable tool such as:

Canva
Preview / image annotation app
any editor capable of drawing clear blue lines
any Mac-friendly graphical application

The exact app is not important.
What matters is that the saved output preserves visibly blue separators.

Drawing Rule¶

The blue lines should cover the wood structure boundaries that separate the bottle cells.

That means:

trace the diagonal wooden dividers
close the visible shelf cell boundaries
ensure the line network separates each compartment from its neighbors

The blue line structure acts like an artificial segmentation grid.

Practical Interpretation of the Example¶

In the provided example image, the blue lines overlay the shelf wood lattice and create separated polygonal/diamond compartments.

The intention is:

blue = separator/border
non-blue = cell interior candidate

This image is not the final crop input itself.
It is a preprocessing mask source used to build ROI polygons.

Manual Mask Requirements¶

To work well with generate_mask_for_polc.py, the mask image should follow these principles.

1. Use clearly blue lines¶

The script detects blue in HSV space.
Therefore the separator lines must be distinctly blue and not close to gray, black, or purple.

The code uses:

lower_blue = np.array([100, 80, 80])
upper_blue = np.array([140, 255, 255])

So the line color must fall inside this broad blue range. :contentReference[oaicite:2]{index=2}

2. Lines should be continuous¶

Gaps in the blue divider network may cause two neighboring cells to merge into one connected component.

This would create incorrect polygon extraction.

3. Lines should approximately follow the wood separators¶

The lines do not need pixel-perfect artistic precision, but they should match the shelf divider structure closely enough that each cell interior remains isolated.

4. Outer frame should also be closed where needed¶

If the shelf border is not effectively separated from the outside image background, the outer region may merge with the background and be rejected.

5. Do not draw blue inside the true cell interior¶

Only the dividers should be blue.
The interior should remain non-blue so it can be extracted as a connected component.

Conceptual Segmentation Logic¶

flowchart LR

RECT["shelf_rectified.jpg"]
DRAW["Draw blue divider lines"]
MASK["Blue-line mask image"]
HSV["HSV blue detection"]
NB["Invert to non-blue"]
CC["Connected components"]
POLY["Extract polygons"]

RECT --> DRAW
DRAW --> MASK
MASK --> HSV
HSV --> NB
NB --> CC
CC --> POLY

generate_mask_for_polc.py¶

Purpose¶

This script converts the manually prepared blue-line shelf image into:

cells_polygons.json

It is the initial polygon generator.

The script does not adjust existing ROIs interactively.
Its role is to detect the cell areas from the blue-line mask image and assign stable IDs. :contentReference[oaicite:3]{index=3}

Input Expectations¶

The script expects an image where:

blue pixels represent cell borders
non-blue regions represent potential cell interiors

The input can be provided by:

--mask-image /path/to/file
or a GUI picker if available and no path is provided

Runtime and Cross-Platform Behavior¶

The script prints runtime diagnostics including:

operating system
hostname
Raspberry Pi detection
GUI availability
working directory
CONF_DIR

It then loads config using OS-specific logic:

on macOS: load_pince_config_darwin()
otherwise: load_pince_config()

This is important because your project supports different path roots on Raspberry Pi and Mac.

Why This Script Should Preferably Run on the Mac¶

This stage is often interactive or visually supervised:

selecting the correct mask image
checking output cell count
validating the generated polygons

Because of that, it is more comfortable to run on a Mac with a normal screen and local GUI.

Also, settings.py explicitly supports Darwin path resolution using:

load_pince_config_darwin()

with:

PINCE_DATA_DIR_DARWIN

This allows your Mac environment to resolve the same logical data tree using Mac-specific filesystem roots. :contentReference[oaicite:5]{index=5}

Internal Processing of generate_mask_for_polc.py¶

Step 1 — Load Mask Image¶

The script reads the selected mask image using OpenCV:

img = cv2.imread(str(mask_image_path))

Then obtains:

image width
image height

These are later stored in the output JSON.

Step 2 — Detect Blue Separator Lines¶

The image is converted to HSV:

hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

Blue is detected using:

lower_blue = np.array([100, 80, 80])
upper_blue = np.array([140, 255, 255])
blue_mask = cv2.inRange(hsv, lower_blue, upper_blue)

Variable roles¶

`lower_blue`¶

Lower HSV threshold for blue detection.

`upper_blue`¶

Upper HSV threshold for blue detection.

`blue_mask`¶

Binary image where detected blue separator pixels are foreground.

Role: - identify the manually drawn separator network

Step 3 — Invert to Get Cell Interior Candidates¶

The script computes:

non_blue = cv2.bitwise_not(blue_mask)

Interpretation: - blue areas are borders - everything else becomes cell/background candidate space

Step 4 — Morphological Cleanup¶

The script applies:

kernel = np.ones((3, 3), np.uint8)
non_blue = cv2.morphologyEx(non_blue, cv2.MORPH_OPEN, kernel, 1)

Variable roles¶

`kernel`¶

3×3 structuring element for morphology.

`MORPH_OPEN`¶

Removes small noise and isolated artifacts.

Purpose: - clean tiny speckles - reduce accidental non-blue noise before connected-components extraction

Step 5 — Connected Components¶

The script runs:

num_labels, labels = cv2.connectedComponents(non_blue)

Each connected non-blue region becomes a candidate component.

These components are then filtered.

Step 6 — Area Filtering¶

Each connected component is turned into a mask and its area is measured.

Only components with area greater than:

--min-area

are kept.

Default:

Variable role: `min_area`¶

This is one of the most important parameters.

Purpose: - reject tiny noise regions - keep only substantial cell-like components

Effect: - too low → noise may become fake cells - too high → valid small edge cells may be lost

Step 7 — Edge-Touch Filtering¶

The script computes a bounding box for each component:

x, y, w_box, h_box = cv2.boundingRect(component_mask)

Then rejects components touching image edges:

if x == 0 or y == 0 or x + w_box >= w or y + h_box >= h:
    continue

Why: - components touching outer edges are often background, not real shelf cells

This is especially useful when the outside-of-shelf region remains connected.

Step 8 — Polygon Approximation¶

For each valid component, the largest contour is extracted and simplified:

eps = 0.01 * cv2.arcLength(cnt, True)
approx = cv2.approxPolyDP(cnt, eps, True)

Variable roles¶

`cnt`¶

Largest external contour of the component.

`eps`¶

Approximation tolerance, proportional to contour perimeter.

Role: - simplify the contour into a manageable polygon - preserve overall shape while reducing point count

`approx`¶

Simplified polygon vertices.

This becomes the stored polygon for that shelf cell.

Step 9 — Centroid Computation¶

Moments are computed:

M = cv2.moments(component_mask)
cx = int(M["m10"] / M["m00"])
cy = int(M["m01"] / M["m00"])

The centroid is used for:

row grouping
column sorting
stable ID assignment

Stable Cell Ordering¶

The extracted cells are not kept in arbitrary connected-component order.

Instead, the script assigns stable IDs using:

rows from bottom to top
columns left to right inside each row

This is critical because later stages need reproducible cell IDs.

`_group_by_cy()`¶

This helper clusters cells into row groups using centroid Y coordinates.

It first tries a greedy grouping with tolerance, then falls back to 1D k-means if needed. :contentReference[oaicite:6]{index=6}

Important Variables in Row Grouping¶

`row_groups`¶

Default:

Meaning: - expected number of shelf rows

This is a structural assumption about the shelf layout.

`row_tol`¶

Default:

40.0

Meaning: - tolerance window in pixels for grouping centroids into the same row

Effect: - larger value tolerates more perspective distortion - smaller value makes row grouping stricter

ID Format¶

Each cell gets an ID in the form:

rr_cc

Example:

01_01

Where: - rr = row index - cc = column index within that row

Rows are ordered bottom first.

This is important to remember when comparing visual shelf position to JSON ID order.

Output JSON¶

The script writes:

{
  "width": W,
  "height": H,
  "cells": [
    {
      "id": "01_01",
      "centroid": [x, y],
      "polygon": [[x1, y1], [x2, y2], ...]
    }
  ]
}

This file is written by default to:

<snapshot_dir>/cells_polygons.json

If the filename already exists, the script auto-increments using:

_1
_2
etc.

This protects previous outputs from accidental overwrite. :contentReference[oaicite:7]{index=7}

Role of settings.py in Path Resolution¶

Your workflow is intentionally dual-environment:

Raspberry Pi runtime
Mac local interactive tooling

settings.py supports this via two loaders:

load_pince_config()
load_pince_config_darwin()

These loaders differ mainly in how relative data/... paths are resolved. :contentReference[oaicite:8]{index=8}

Path Resolution Logic¶

For Linux / Pi:

_resolve_path(raw)

For macOS:

_resolve_path_darwin(raw)

Both call:

_resolve_path_base(raw, data_root_override=...)

Rules:

absolute paths stay unchanged
relative data/... paths are anchored to environment-specific data roots
other relative paths are anchored to PROJECT_ROOT

This allows the same logical INI file to work in both environments.

Why This Matters for Cropping Preparation¶

The following files must resolve correctly on both Pi and Mac:

shelf_rectified.jpg
mask image location
cells_polygons.json
crop output directories

Because the mask-generation and ROI-adjustment steps prefer a GUI, it is practical to run them on the Mac while keeping the same repository logic.

ROI_ADjustment_from_PNG.py¶

Purpose¶

This script is the interactive ROI correction editor.

It does not generate new polygons from scratch.
It only adjusts existing ones. :contentReference[oaicite:9]{index=9}

That means the proper sequence is:

first run generate_mask_for_polc.py
produce cells_polygons.json
then run ROI_ADjustment_from_PNG.py
shift polygons until they fit the actual shelf cells correctly
save the updated JSON

Input Files for ROI Adjustment¶

The script loads:

image: <snapshot_dir>/shelf_rectified.jpg
ROI JSON: <snapshot_dir>/cells_polygons.json

So it always works in the rectified image coordinate space.

Why This Script Also Preferably Runs on the Mac¶

This tool uses a real OpenCV GUI window and keyboard control:

ROI selection
arrow-key movement
save on keypress

That is much more convenient on a Mac than on the Pi 7-inch touch screen.

The script itself detects platform and GUI availability, and uses Darwin config loading when appropriate.

ROI Adjustment Controls¶

The script supports:

Arrow keys: move selected ROI
TAB: next ROI
b: previous ROI
f: toggle fast movement
s: save JSON
q or ESC: quit

This is a lightweight geometric correction tool.

Internal Processing of ROI_ADjustment_from_PNG.py¶

Step 1 — Load Image and JSON¶

The script reads:

snapshot_dir / "shelf_rectified.jpg"
snapshot_dir / "cells_polygons.json"

The JSON includes:

original image width
original image height
list of cell polygons

Step 2 — Scale JSON Coordinates into Current Image Coordinates¶

Because the displayed image may not match stored JSON dimensions exactly, scaling factors are computed:

sx = w / json_w
sy = h / json_h

Variable roles¶

`sx`¶

Scale factor in X direction.

`sy`¶

Scale factor in Y direction.

These are used to convert stored polygon coordinates into screen/image coordinates.

Step 3 — Editable Polygon Representation¶

Each polygon from JSON is scaled into frame coordinates and stored as:

polygon_frame

This is the editable in-memory version used during GUI interaction.

ROIs are sorted by centroid position for easier navigation:

rois.sort(key=lambda r: (centroid_y, centroid_x))

This makes TAB traversal more predictable visually.

Step 5 — Drawing Overlay¶

Function:

draw_rois()

renders all polygons on top of the image.

Visual logic: - selected ROI: green, thicker line - others: red, thinner line - each ROI labeled by ID at centroid

This makes it easy to identify which polygon is being adjusted.

Step 6 — Keyboard-Based Translation¶

Selected polygon is translated by:

translate_polygon(poly_xy, dx, dy)

Movement step is controlled by:

--step default 2
--step-fast default 10

And fast mode is toggled with f.

Step 7 — Boundary Clamping¶

After movement, polygon coordinates are clamped with:

clamp_polygon(poly_xy, w, h)

This prevents polygons from drifting outside the visible image.

Step 8 — Save Back to JSON¶

When user presses s:

frame coordinates are converted back to JSON coordinate system using unscale_polygon()
coordinates are rounded to integers
centroids are recomputed
updated cells_polygons.json is written

This means the output remains in the original normalized coordinate space.

Division of Responsibility Between the Two Scripts¶

`generate_mask_for_polc.py`¶

Use when: - creating ROI polygons for the first time - major shelf structure changed - mask image has been redrawn - cell set itself must be regenerated

`ROI_ADjustment_from_PNG.py`¶

Use when: - polygons exist already - only position correction is needed - fine alignment is required - no new cells must be created

This division is important and is explicitly noted inside the ROI adjustment script. :contentReference[oaicite:11]{index=11}

Recommended Operational Sequence¶

sequenceDiagram

participant User
participant Rectified as shelf_rectified.jpg
participant MaskApp as Canvas/Image App
participant Gen as generate_mask_for_polc.py
participant Json as cells_polygons.json
participant Adj as ROI_ADjustment_from_PNG.py

User->>Rectified: open rectified shelf image
User->>MaskApp: draw blue lines on wood separators
MaskApp-->>User: save mask image

User->>Gen: run polygon generator on mask image
Gen->>Json: write initial cells_polygons.json

User->>Adj: open interactive ROI editor
Adj->>Json: read polygons
User->>Adj: nudge ROIs with keyboard
Adj->>Json: save corrected polygons

Practical Notes for Wine Platfor use¶

Because the solution supports both Raspberry Pi and Mac, the most practical approach is:

Raspberry Pi¶

Use for: - kiosk solution - Stock & Master Data management - automated runtime execution - headless scheduled stock detection - final production pipeline

Mac¶

Use for: - mask drawing - polygon generation - ROI adjustment - visual validation

This matches architecture and the dual path-resolution strategy already implemented in settings.py. :contentReference[oaicite:12]{index=12}

Failure Modes¶

During manual mask creation¶

Possible problems: - separator lines not blue enough - gaps in lines - merged cells - missing outer borders - overpainting inside cell interiors

During `generate_mask_for_polc.py`¶

Possible problems: - wrong image selected - min-area too strict or too loose - row grouping not matching shelf layout - edge-touching valid cells rejected - background merged with shelf exterior

During `ROI_ADjustment_from_PNG.py`¶

Possible problems: - missing cells_polygons.json - missing shelf_rectified.jpg - no GUI available - keyboard handling differences across OS - saving not performed before quitting

Key Data Artifacts¶

File	Role
`shelf_rectified.jpg`	base image for mask drawing and ROI editing
manually annotated mask image	blue-line separator image
`cells_polygons.json`	machine-readable shelf cell polygons
later crop outputs	per-cell images generated downstream

Summary¶

The cropping prerequisite workflow is intentionally hybrid:

automated rectification produces a stable shelf image
manual mask drawing captures the wooden divider structure precisely
automatic polygon extraction converts the mask into ROI geometry
interactive ROI adjustment fine-tunes the final polygons

This design is well suited to your cellar shelf because it combines:

visual human reliability for complex structure definition
automated repeatable polygon generation
convenient Mac-based GUI tooling
consistent Pi/Mac path handling through settings.py

As a result, the downstream crop stage can rely on a stable and explicit:

cells_polygons.json

which is the real prerequisite for consistent bottle-cell cropping.

Cropping Prerequisites — Manual Shelf Mask, Polygon Generation, and ROI Adjustment¶

Overview¶

Position in the Overall Vision Workflow¶

Why a Manual Mask Is Needed¶

Input Prerequisite: shelf_rectified.jpg¶

Important Upstream Dependency¶

Manual Mask Creation¶

Goal¶

Recommended Base Image¶

Drawing Rule¶

Practical Interpretation of the Example¶

Manual Mask Requirements¶

1. Use clearly blue lines¶

2. Lines should be continuous¶

3. Lines should approximately follow the wood separators¶

4. Outer frame should also be closed where needed¶

5. Do not draw blue inside the true cell interior¶

Conceptual Segmentation Logic¶

generate_mask_for_polc.py¶

Purpose¶

Input Expectations¶

Runtime and Cross-Platform Behavior¶

Why This Script Should Preferably Run on the Mac¶

Internal Processing of generate_mask_for_polc.py¶

Step 1 — Load Mask Image¶

Step 2 — Detect Blue Separator Lines¶

Variable roles¶

lower_blue¶

upper_blue¶

blue_mask¶

Step 3 — Invert to Get Cell Interior Candidates¶

Step 4 — Morphological Cleanup¶

Variable roles¶

kernel¶

MORPH_OPEN¶

Step 5 — Connected Components¶

Step 6 — Area Filtering¶

Variable role: min_area¶

Step 7 — Edge-Touch Filtering¶

Step 8 — Polygon Approximation¶

Variable roles¶

cnt¶

eps¶

approx¶

Step 9 — Centroid Computation¶

Stable Cell Ordering¶

_group_by_cy()¶

Important Variables in Row Grouping¶

row_groups¶

row_tol¶

ID Format¶

Output JSON¶

Role of settings.py in Path Resolution¶

Path Resolution Logic¶

Why This Matters for Cropping Preparation¶

ROI_ADjustment_from_PNG.py¶

Purpose¶

Input Files for ROI Adjustment¶

Why This Script Also Preferably Runs on the Mac¶

ROI Adjustment Controls¶

Internal Processing of ROI_ADjustment_from_PNG.py¶

Step 1 — Load Image and JSON¶

Step 2 — Scale JSON Coordinates into Current Image Coordinates¶

Variable roles¶

sx¶

sy¶

Step 3 — Editable Polygon Representation¶

Step 4 — ROI Sorting for Navigation Convenience¶

Step 5 — Drawing Overlay¶

Step 6 — Keyboard-Based Translation¶

Step 7 — Boundary Clamping¶

Step 8 — Save Back to JSON¶

Division of Responsibility Between the Two Scripts¶

generate_mask_for_polc.py¶

ROI_ADjustment_from_PNG.py¶

Recommended Operational Sequence¶

Practical Notes for Wine Platfor use¶

Raspberry Pi¶

Mac¶

Failure Modes¶

`lower_blue`¶

`upper_blue`¶

`blue_mask`¶

`kernel`¶

`MORPH_OPEN`¶

Variable role: `min_area`¶

`cnt`¶

`eps`¶

`approx`¶

`_group_by_cy()`¶

`row_groups`¶

`row_tol`¶

`sx`¶

`sy`¶

`generate_mask_for_polc.py`¶

`ROI_ADjustment_from_PNG.py`¶

During `generate_mask_for_polc.py`¶

During `ROI_ADjustment_from_PNG.py`¶