# Image Messages

MindRoom can process images sent to Matrix rooms, passing them to vision-capable agents and teams for analysis.

## Overview

When a user sends an image in a Matrix room:

1. The responder determines whether it should answer (via mention, thread participation, or DM)
2. The image is downloaded and decrypted (if E2E encrypted)
3. The image is wrapped as an `agno.media.Image` and passed to the AI model
4. The responder replies with its analysis

Image support works automatically for agents and teams -- no configuration is needed.
The selected model must support vision (e.g., Claude, GPT-5.6).

## Supported Formats

MindRoom detects image format from file byte signatures:

- PNG
- JPEG
- GIF
- WebP
- BMP
- TIFF

If the declared MIME type in the Matrix event does not match the detected byte signature, MindRoom logs a warning and uses the detected type.

## How It Works

```
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│ Image Msg   │────>│ Download &  │────>│ Pass to AI  │
│ (Matrix)    │     │ Decrypt     │     │ Model       │
└─────────────┘     └─────────────┘     └─────────────┘
                                              │
                                              v
                                        ┌─────────────┐
                                        │ Responder   │
                                        │ Replies     │
                                        └─────────────┘
```

## Usage

Send an image in a Matrix room and mention the agent or team in the caption:

- **With caption**: `@assistant What does this diagram show?` -- the caption is used as the prompt
- **Without caption**: The agent receives `[Attached image]` as the prompt and describes what it sees
- **Bare filename**: If the body is just a filename (e.g., `IMG_1234.jpg`), it is treated the same as no caption

Images work in both direct messages and threads, and with both individual agents and teams.

## Captions (MSC2530)

If the Matrix event's `filename` field differs from `body`, the `body` is used as a user caption.
This follows [MSC2530](https://github.com/matrix-org/matrix-spec-proposals/pull/2530) semantics and works with clients that set the caption in the body.

## Image Persistence

Images are saved under `mindroom_data/attachments/` and `mindroom_data/incoming_media/` and registered as attachment records with 30-day retention.
In addition to being passed to the AI model as vision input, each image is also registered as an `att_*` attachment ID so agents can reference it via tool calls.
See [Attachments](https://docs.mindroom.chat/attachments/) for details on retention and context scoping.

## Encryption

Both unencrypted and E2E encrypted images are supported. Encrypted images are decrypted transparently using the key material from the Matrix event.

## Caching

AI response caching is automatically skipped when images are present, since image payloads are large and unlikely to repeat.

## Media Fallback

If a model rejects inline media (images, audio, video, or documents), MindRoom automatically retries the request without the inline media.
The retried prompt includes `[Inline media unavailable for this model]` to inform the agent that attachments were dropped.
Agents can still reference the files via attachment IDs and tools.

This fallback is transparent — no user action is required.
Any failure of a media-bearing request triggers one retry without media — no error wording decides whether to retry, so unknown provider prose degrades gracefully instead of surfacing a raw provider error.
When the retry succeeds, the model route learns that the dropped media kinds are unsupported, and later requests omit them up front instead of paying a failed API call.
This learned capability state is process-local and resets on restart.
Payload-size and context-overflow rejections never teach the capability state, since dropping media can shrink an oversized request for reasons unrelated to media support.
Transient failures (HTTP 5xx and 429 status codes on the provider exception) also never teach, since their retry can succeed simply because the outage or rate limit passed.

## Limitations

- **Routing with multiple eligible responders** -- without an `@mention`, the router uses the image caption to select among candidates only when room configuration and reply permissions leave multiple eligible agents or teams.
- **Bridge mention detection** uses `m.mentions` in the event, falling back to parsing HTML pills from `formatted_body` when `m.mentions` is absent (e.g., mautrix-telegram). Bridges that set neither may not trigger agent responses.
- **Model support** -- vision input requires a model that supports it. Text-only models reject inline images, and the [media fallback](#media-fallback) retries without them so the agent still answers with a note that it cannot view the attachment.
