# Plan: Conform first/last frame reference images to the canvas aspect ratio

## Motivation

The "whiteboard marker drawing animation" workflow generates a finished slide image and asks an
image-to-video model (e.g. `wan-video/wan-2.2-i2v-fast`) to animate a hand drawing it, by passing:

- first frame = a blank whiteboard (canvas-sized)
- last frame = the finished slide (`movieParams.lastFrameImageName`)

This already works end-to-end in MulmoScript: beat-local `images` entries with
`type: "imagePrompt"` are generated by `resolveBeatLocalRefs` (with the global
`imageParams.style` appended), and `movieParams.firstFrameImageName` / `lastFrameImageName`
resolve against the merged refs.

The one gap: generated images come back at the provider's fixed sizes (gpt-image-2 emits
1536×1024 for a landscape canvas), while the movie is generated at the canvas aspect ratio
(e.g. 1280×720). Today `movieReplicateAgent` / `movieGenAIAgent` send frame images **as-is**,
so the model receives mismatched first/last frames and reconciles them unpredictably.
The `nurtured-at-home` deck worked around this manually:

```sh
ffmpeg -i <src> -vf "scale=1080:720,pad=1280:720:100:0:color=0xF7F6F4" <dst>
```

This plan moves that padding step into the pipeline.

## Schema change (`src/types/schema.ts`)

Add one optional field to `mulmoMovieParamsSchema`:

```ts
frameFillColor: z
  .string()
  .optional()
  .describe("Fill color used when padding firstFrame/lastFrame reference images to the canvas aspect ratio (e.g. '#F7F6F4'). Default: black"),
```

Notes:
- No new object/style option for now: conforming is always **aspectFit + centered pad**
  (cropping would destroy slide content). `aspectFill` can be added later if needed.
- Since `src/types/` changes, the `@mulmocast/types` package must be published together with
  the next `mulmocast` release.

## New utilities (`src/utils/ffmpeg_utils.ts`)

```ts
ffmpegGetImageDimensions(filePath): Promise<{ width, height }>
// ffprobe, first video stream

padImageToCanvas(srcPath, destPath, width, height, fillColor): Promise<void>
// -vf scale=w:h:force_original_aspect_ratio=decrease,pad=w:h:(ow-iw)/2:(oh-ih)/2:color=<fillColor>
// '#RRGGBB' is normalized to '0xRRGGBB' for ffmpeg
```

## Pipeline change (`src/actions/image_agents.ts`)

In `imagePreprocessAgent`, where `movieParams.firstFrameImageName` / `lastFrameImageName`
are resolved into `firstFrameImagePath` / `lastFrameImagePath`, conform the resolved path:

- If the source file does not exist (mock agents / unit tests), return it unchanged.
- Probe dimensions; if the aspect ratio already matches the canvas (tolerance 0.01), return
  the original path — **no behavior change for users who pass pre-padded images**.
- Otherwise write `<imageProjectDir>/<imageName>_fit_<w>x<h>.png` (aspect-fit pad, centered,
  `frameFillColor` ?? black) and return that path.
- Cache: skip regeneration when the conformed file exists and is newer than the source
  (`-f` regenerates the ref image, which makes the source newer and cascades here).
- Concurrent beats can reference the same image name; an in-flight promise map keyed by the
  destination path prevents two ffmpeg runs writing the same file.

Downstream needs no changes: `movieGenerator` (both replicate and genai agents) and the
`referenceImageForMovie` fallback already consume these paths.

Scope notes:
- The beat's own generated image used as an implicit first frame
  (`imagePrompt` + `moviePrompt` without frame names) is intentionally **not** conformed —
  that is long-standing behavior that many scripts rely on.
- `movieParams.referenceImages` (Veo style/asset references) are intentionally **not**
  conformed either: they describe aesthetics or objects, not frames, so they need not
  match the canvas aspect ratio.
- Frame-name/fill-color resolution shallow-merges style-level and beat-level
  `movieParams` (like `getMovieAgentInfo`), so a beat setting only `lastFrameImageName`
  keeps the global `frameFillColor`.

## Sample script (`scripts/samples/whiteboard_marker_animation.json`)

Demonstrates the fully self-contained whiteboard workflow:

- global `imageParams.style` = whiteboard marker style, model `gpt-image-2`
- global `imageParams.images.board` = a checked-in solid-color blank board PNG
  (`scripts/samples/images/whiteboard/blank_board.png`, 1280×720, `0xF7F6F4`).
  Generating the blank board via imagePrompt does not work: `generateReferenceImage`
  appends the global `imageParams.style` to ref prompts, and a style full of
  "boxes, arrows, stick figures, labels" makes the model fill the "blank" board with content.
- each beat: beat-local `images` entry (imagePrompt for the finished slide, drawn onto the
  shared board via `referenceImageName: "board"`), `moviePrompt` = hand-drawing time-lapse,
  `movieParams: { firstFrameImageName: "board", lastFrameImageName, frameFillColor: "#F7F6F4" }`
- Each beat animates blank board → finished slide (the nurtured-at-home deck pattern).
  Chaining slide N as the first frame of beat N+1 is not used: beat-local refs are only
  visible to their own beat, and global imagePrompt refs support one level of
  `referenceImageName` (stage-2 entries resolve in parallel).
- `frameFillColor` sits in the global `movieParams` and applies to every beat via the
  shallow merge.
- global `movieParams.model` = `wan-video/wan-2.2-i2v-fast`

## Tests

- `test/utils/test_ffmpeg_image.ts`: generate a small solid PNG via ffmpeg `lavfi`,
  assert `ffmpegGetImageDimensions`, pad it with `padImageToCanvas`, assert output size.
- `test/actions/test_frame_conform.ts`: `imagePreprocessAgent` with a real temp
  image project dir — mismatched ref is conformed to `<name>_fit_1280x720.png` (file exists,
  correct size); matching-aspect ref passes through unchanged; missing file passes through.

## Affected files

- `src/types/schema.ts` — `frameFillColor`
- `src/utils/ffmpeg_utils.ts` — `ffmpegGetImageDimensions`, `padImageToCanvas`
- `src/actions/image_agents.ts` — conform on frame-ref resolution
- `scripts/samples/whiteboard_marker_animation.json` — new sample
- `test/utils/test_ffmpeg_image.ts`, `test/actions/test_frame_conform.ts` — new tests
