# Foundation & VLM/LMM model blocks

Category dir: `inference/core/workflows/core_steps/models/foundation/**`. See the universal invariants section in `SKILL.md` first.

**What / when.** Wrappers around generic ("foundation") vision or vision-language models — CLIP/SAM2/YOLO-World/depth/OCR core models, and prompt-driven VLM/LLM blocks (Gemini, Claude, GPT, Florence-2, LLaMA-vision, Qwen, gaze). Reach here when your block *invokes a model* rather than transforming data, and the model is either a Roboflow core model or a third-party LLM API.

**Mental model.** Every block subclasses `WorkflowBlock` with a `BlockManifest(WorkflowBlockManifest)`, `block_type="model"`, and `ui_manifest.section` = `"model"` (or `"video"`). Two archetypes dominate:
- **Core-model blocks** (CLIP, SAM2, YOLO-World, depth, gaze) init with `["model_manager", "api_key", "step_execution_mode"]` and split `run()` into `run_locally()` (via `load_core_model` + `self._model_manager.infer_from_request_sync`) vs `run_remotely()` (via `InferenceHTTPClient`) keyed on `StepExecutionMode`. See `foundation/clip/v1.py` (simplest) and `foundation/segment_anything2/v1.py`.
- **VLM/LLM API blocks** (Gemini, Claude, OpenAI) init with just `["model_manager", "api_key"]`, have **no** local/remote split (always call an external API), and expose a `task_type` enum that drives which fields are required. See `foundation/google_gemini/v3.py`, `foundation/anthropic_claude/v3.py`.

Kinds consumed: `IMAGE_KIND` (+ `STRING_KIND` for prompts/text). Kinds produced vary — `EMBEDDING_KIND` (clip), `INSTANCE_SEGMENTATION_PREDICTION_KIND` (SAM2), `OBJECT_DETECTION_PREDICTION_KIND` (yolo_world), `LANGUAGE_MODEL_OUTPUT_KIND` (VLMs), `NUMPY_ARRAY_KIND` (depth), `KEYPOINT_DETECTION_PREDICTION_KIND` (gaze). Image-consuming blocks declare `get_parameters_accepting_batches()` and receive `Batch[WorkflowImageData]`, returning one dict per element (`clip/v1.py` is the scalar exception). Most are **stateless**; the video trackers are the notable stateful ones.

**Nuances an author MUST know.**
- **VLM blocks do NOT parse into detections themselves.** They emit a single `"output"`/`"raw_output"` string of `LANGUAGE_MODEL_OUTPUT_KIND` plus `"classes"`. The `task_type` field carries a `recommended_parsers`/`recommended_parser` map in `json_schema_extra` pointing downstream to `roboflow_core/vlm_as_detector@v2`, `vlm_as_classifier@v2`, `json_parser@v1` (see the `task_type` field in `anthropic_claude/v3.py`). Mirror this — don't bolt parsing onto the model block.
- **Conditional-required fields via `relevant_for` + `model_validator`.** `task_type` gates `prompt`/`classes`/`output_structure`; each field carries `json_schema_extra={"relevant_for": {...}}` for the UI *and* a `@model_validator(mode="after")` enforces it server-side (`google_gemini/v3.py`). Both are needed — UI hint alone doesn't validate.
- **Prompt forwarding + coordinate systems.** SAM2 accepts upstream `boxes` (OD/seg/keypoint kinds) as prompts, converts `xyxy → center+wh` per box, and **forwards the prompt box's `class_name`/`detection_id` onto the output mask**; unprompted, it invents `"foreground"`/integer ids (`segment_anything2/v1.py`). Note the 6-element `sv.Detections` iteration `for xyxy, _, confidence, class_id, _, bbox_data in boxes_for_image` — that unpack shape (`xyxy, mask, confidence, class_id, tracker_id, data`) is the `sv.Detections` contract (the 7-tuple form belongs to the tensor-native `NativeDetections` path).
- **Post-process sv.Detections consistently.** Detection-producing blocks end with the trio `convert_inference_detections_batch_to_sv_detections` → `attach_prediction_type_info_to_sv_detections_batch` → `attach_parents_coordinates_to_batch_of_sv_detections` (`segment_anything2/v1.py`). Skipping the parent-coords attach breaks coordinate re-anchoring for cropped/tiled inputs.
- **Statefulness ⇒ LOCAL-only + per-video eviction.** `segment_anything2_video/v1.py` holds `self._sessions: Dict[str, VideoSessionBookkeeping]` keyed on `video_metadata.video_identifier`, and **raises `NotImplementedError`** (message constant `_REMOTE_EXECUTION_NOT_SUPPORTED_MESSAGE`) for any non-LOCAL mode because remote execution would fan frames across processes and destroy temporal memory. Switching `model_id` clears all sessions (`self._sessions.clear()`). Stateful blocks also register `STATEFUL_VIDEO_HTTP_SOFT_RESTRICTION` / `STILL_IMAGE_INPUT_SOFT_RESTRICTION` in `get_restrictions()`.
- **Env-gated core models need `RuntimeRestriction`s.** Blocks behind a feature flag (`CORE_MODEL_SAM2_ENABLED`, `FLORENCE2_ENABLED`, `CORE_MODEL_GAZE_ENABLED`) add a `Severity.HARD` restriction for `HOSTED_SERVERLESS`+`REMOTE` (endpoint returns 404) and for `SELF_HOSTED_CPU`+`LOCAL` when CUDA is required (`get_restrictions` in `segment_anything2/v1.py`).
- **API-key kinds & managed keys.** VLM blocks accept `Selector(kind=[STRING_KIND, SECRET_KIND, ROBOFLOW_MANAGED_KEY])` with `private=True` and default `"rf_key:account"` (`anthropic_claude/v3.py`); they also set `get_air_gapped_availability(available=False, reason="requires_internet")`.
- **Deprecation stays as a live block.** Removed features (gaze) keep the manifest with `"deprecated": True` in `json_schema_extra` and raise `FeatureDeprecatedError` (HTTP 410) at `run()` rather than being deleted (`gaze/v1.py`) — matches the MediaPipe-deprecation scope for gaze/cog_vlm.

**Example blocks to pattern-match (simplest → representative).**
- `foundation/clip/v1.py` — the **minimal core-model shape**: `str`-or-`WorkflowImageData` input, LOCAL/REMOTE split, an `LRUCache` for text embeddings, single scalar `EMBEDDING_KIND` output.
- `foundation/depth_estimation/v1.py` — a **numpy-tensor output** (`NUMPY_ARRAY_KIND`) alongside a passthrough `IMAGE_KIND`; canonical "non-detection dense output".
- `foundation/yolo_world/v1.py` — **class-prompted zero-shot OD**: dynamic `classes` list → `OBJECT_DETECTION_PREDICTION_KIND`; contrast with SAM2's box-prompted flow.
- `foundation/segment_anything2/v1.py` — the **full detection block**: batch handling, upstream-boxes-as-prompts with class/id forwarding, the sv.Detections post-process trio, env-gated `RuntimeRestriction`s.
- `foundation/google_gemini/v3.py` (peer: `anthropic_claude/v3.py`) — the **VLM/LLM archetype**: `task_type` enum, `relevant_for`+validator conditional fields, `recommended_parsers` handoff, `LANGUAGE_MODEL_OUTPUT_KIND` output, concurrency via `run_in_parallel`.
- `foundation/segment_anything2_video/v1.py` — the **stateful video tracker**: per-`video_identifier` session dict, prompt-vs-track gating, LOCAL-only enforcement, `_streaming_video_common.py` helpers, model-switch eviction.
