# Model blocks — Roboflow-trained & OCR

Category dir: `inference/core/workflows/core_steps/models/roboflow/**` and `.../models/foundation/ocr/**`. See the universal invariants section in `SKILL.md` first — this file only adds category-specific nuance.

**What / when.** Reach here when the block's job is to *run a model and emit predictions*: a Roboflow-trained model (object detection, instance/semantic segmentation, keypoint detection, single/multi-label classification) selected by `model_id`, or a text-extraction model (DocTR OCR core model, Google Vision OCR). These are the canonical "producer" blocks — everything downstream (visualizations, filters, UQL) consumes what they output.

**Mental model.** Every block in this category is a thin, *stateless* adapter over the model backend, all cut from one template:
- Manifest subclasses `WorkflowBlockManifest`; `block_type: "model"`, `ui_manifest.section: "model"`. Roboflow-trained blocks take `images: Selector(kind=[IMAGE_KIND])` (via `ImageInputField`) + `model_id: Union[Selector(kind=[ROBOFLOW_MODEL_ID_KIND]), str]` (via `RoboflowModelField`) plus per-task knobs (`confidence`, `iou_threshold`, `class_filter`, NMS params, `active_learning_*`).
- `images` is **batch-oriented**: `get_parameters_accepting_batches()` returns `["images"]`, so `run` receives `Batch[WorkflowImageData]` and returns a `BlockResult` list — one dict per input image (no dimensionality change; batch in, batch out).
- Init params are always `["model_manager", "api_key", "step_execution_mode"]`; `run` dispatches on `StepExecutionMode` to `run_locally` (via `self._model_manager` / `load_core_model` for core models) vs `run_remotely` (via `InferenceHTTPClient` + `WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_*`). Both funnel into a shared `_post_process_result`.
- Detection-family blocks produce a typed detections **kind** (`OBJECT_DETECTION_PREDICTION_KIND`, `INSTANCE_SEGMENTATION_PREDICTION_KIND`, `KEYPOINT_DETECTION_PREDICTION_KIND`, `SEMANTIC_SEGMENTATION_PREDICTION_KIND`) carried as `sv.Detections`; classification produces `CLASSIFICATION_PREDICTION_KIND` (a dict, not `sv.Detections`); OCR produces a `STRING_KIND` `result` (+ boxes as `OBJECT_DETECTION_PREDICTION_KIND`).

**Nuances an author MUST know.**
- **Shared post-processing pipeline, not ad-hoc.** Detection blocks convert raw inference dicts to `sv.Detections` and enrich them through a fixed util chain in `core_steps/common/utils.py`: `convert_inference_detections_batch_to_sv_detections` → `attach_prediction_type_info_to_sv_detections_batch` → `filter_out_unwanted_classes_from_sv_detections_batch` → `attach_parents_coordinates_to_batch_of_sv_detections` (see `_post_process_result` in `object_detection/v2.py`). Reuse these; don't hand-roll `sv.Detections` construction or coordinate attachment.
- **Classification carries parent metadata differently.** Instead of the coordinate-attach util, classification pulls `parent_id`/`root_parent_id` from `image.parent_metadata` / `image.workflow_root_ancestor_metadata` (`_post_process_result` in `multi_class_classification/v1.py`).
- **Output name vs constant.** The *output name* differs from the constant in some blocks (`OutputDefinition(name=INFERENCE_ID_KEY, ...)` in keypoint v2 vs literal `"inference_id"` in OD v2) — be deliberate about the string; `describe_outputs()` names must match `run` keys exactly.
- **sv.Detections data density differs by task.** Keypoint blocks call an *extra* `add_inference_keypoints_to_sv_detections` step to fold keypoints into `detections.data` after the standard conversion (`keypoint_detection/v2.py`); instance/semantic segmentation carry masks. A new per-detection field must round-trip through the kind's serializer or it is dropped on REMOTE / API boundaries.
- **Model selection & variants.** Roboflow blocks gate on `get_compatible_task_types` (e.g. `["object-detection"]`); OCR/core-model blocks advertise `get_supported_model_variants` (e.g. `["doctr/default"]`) and load via `load_core_model(core_model="doctr")` rather than a user `model_id`. Segmentation exposes task-specific knobs (`mask_decode_mode`, `tradeoff_factor`, `enforce_dense_masks_in_inference_models`) that must be threaded into both the local request and the `InferenceConfiguration`.
- **LOCAL vs REMOTE parity is a correctness contract.** The two paths hit different backends and the REMOTE path must mirror every knob into `InferenceConfiguration` (compare `confidence` → `confidence_threshold` renaming in OD v2 `run_remotely`). A knob added to only one path is a correctness bug, not a style nit. `WORKFLOWS_REMOTE_API_TARGET == "hosted"` also forces `client.select_api_v0()`.
- **These blocks are stateless** — no per-video state or eviction here (that lives in tracker/video blocks). Keep new model blocks stateless too.

**Example blocks to pattern-match.**
- `models/roboflow/object_detection/v2.py` — the reference template for the whole category: manifest fields, LOCAL/REMOTE split, the full `sv.Detections` post-process chain. Read this first. (`object_detection/v3.py` shows the newer `confidence: Literal["best","default","custom"]` variant selection.)
- `models/roboflow/instance_segmentation/v2.py` — **mask-carrying detections** and task-specific decode knobs (`mask_decode_mode`, `tradeoff_factor`, `enforce_dense_masks_in_inference_models`). (Latest is `instance_segmentation/v4.py`.)
- `models/roboflow/keypoint_detection/v2.py` — **augmenting `sv.Detections` with extra structured data** via `add_inference_keypoints_to_sv_detections`.
- `models/roboflow/multi_class_classification/v1.py` — the **non-`sv.Detections` output shape** (dict prediction + manual `parent_id`/`root_parent_id`); pair with `multi_label_classification/v1.py`.
- `models/foundation/ocr/v1.py` — a **core-model** block: `load_core_model`, `EXPECTED_OUTPUT_KEYS` contract fed to `post_process_ocr_result`, `STRING_KIND` result + detection boxes. Contrast with `foundation/google_vision_ocr/v1.py` (external API, `SECRET_KIND` key, `rf_key:` proxy, `AirGappedAvailability`). Both commonly feed `transformations/stitch_ocr_detections/v1.py`.

Author-facing user docs for these blocks are **autogenerated** by `development/docs/build_block_docs.py` from the manifest field descriptions and long-descriptions — do not hand-write `docs/workflows/blocks/*.md`.
