# Conversion, Transformation and Projection Pipeline

This section describes all the processing stages that are applied to data given to a plot on its way to being displayed.

## Overview

The pipeline can be broadly be summarized in 3 parts each with a few steps:

```@raw html
<ol>
<li>
```
Conversions which mainly normalize types
1. `expand_dimensions()` adds defaulted/generated data (e.g. x, y in `image()`)
2. `dim_convert` processes special types like Units
3. `convert_arguments()` normalizes numeric types & data formats
```@raw html
</li>

<li>
```
Transformations which transform data on a per-plot basis
1. `transform_func` is a function applied to data
2. `model` matrix applies linear transformations
```@raw html
</li>

<li>
```
Projections which project data from one coordinate system to another
1. `view` matrix moves data from "world" space to a camera "view/eye" space
2. `projection` matrix moves from the camera space to "clip" space
3. `viewport` moves "clip" space to "pixel/screen" space
```@raw html
</li>
</ol>
```

As a **user** you have direct control over the `model` matrix (1.2) with the `scale!()`, `translate!()` and `rotate!()` functions.
You have indirect control over projections (3) with the `space` attribute.
It sets what coordinate system is used as the initial space and adjusts the projections as a result.
You also have indirect control over `transform_func`, which can be set by passing a `Transformation()` directly to a plot.
However it usually inherited and controlled by the `Axis`.

As a **developer**, i.e. someone who wants to extend Makie, you can interact with most these steps.
Most likely you will extend with `convert_arguments()` to allow special types to be plotted.
But you can also implement more `dim_converts`, add methods for `expand_dimensions()`, implement more transform functions or add a camera which produces its own `view` and `projection` matrix.
Only `model` and `viewport` handling as well as the interpretation of `space` are set.

## Argument Conversions

When calling a plot function, e.g. `scatter!(axis_or_scene, args...)` a new plot object is constructed.
It keeps track of the original input arguments as an Observables in `plot.args`.
Those input arguments are then converted by the conversion pipeline and stored in `plot.converted`.
The pipeline consists of 3 steps as mentioned above:

### Data Generation

The first step is to generate "missing" data.
For example, you can create an image plot with just `image(rand(10, 10))`.
The most general form however also includes a `ClosedInterval` for the x and y dimensions, declaring the size of the image.
This data is generated by `expand_dimensions(::Trait, args...)` where the `Trait = conversion_trait(::PlotType, args...)`.

### Special Type Processing

The second step handles special types like `Unitful` types, `Dates` types or categorical values which need to be synchronized within the scene.
For example, if one plot uses "hours" as unit for its x values other plots need to also use time units for x.
If the scale of the unit differs between plots, i.e. one uses hours, the other minutes, then a common unit must be found and the values need to be scaled appropriately.
This is what **dim_converts** handles.
You can find more documentation on them in the [Dimension conversions](@ref) docs.

### Convert Arguments

The last step and main work-horse in the conversion pipeline is the `convert_arguments()` function.
It's purpose is to convert different data types and layouts into one or a select few formats.
For example, any data passed to `scatter()` is converted to a `Vector{Point{D, T}}` where `D = 2` or `3` and `T = Float32` or `Float64`.
These conversions can happen based on the plot type or its conversion trait.
For `scatter()` the conversion trait `PointBased` is used.

`convert_arguments()` can also accept keyword arguments sourced from plot attributes.
For this the attribute needs to be marked with `used_attribute(::PlotType) = (names...)`.
Any name in that list will be removed from the attributes of the final plot and be passed to `convert_arguments()` instead.

If you want to plot your own custom types you may want to extend `convert_arguments()`.
Let's say you have some custom type `MySimulation` and some function `positions(::MySimulation)` which returns positions you want to plot when calling `scatter(::MySimulation)`.
In this case you can define

```julia
function Makie.convert_arguments(PT::PointBased, sim::MySimulation)
    return Makie.convert_arguments(PT, positions(sim))
end
```

to make that possible.
You can use the plot type (i.e. `Scatter`) as the first argument as well.


## [Transformations](@id pipeline_transformations)

After conversions have normalized the type and layout of plot data, transformations can now adjust it.
They are handled by the `Transformation` object, which exists both on the plot and scene level in the `transformation` field.
It contains a `transform_func` which is a function-like object applied to the data, and a `model` matrix which handles linear transformations of the data.

The plot `Transformation` object may inherit from it's parent scene or plot if it acts in the same `space`.
For `transform_func` this means using the same function as the parent.
For `model` it means merging the parents `model` matrix with the local one as `plot.model = parent.model * local_model`.

### Transformation Function

The transformation function or `transform_func` is part of the `Transformation` object.
It handles non-linear transformations like log transform of a logarithmic axis.

Each `transform_func` implements at least
```julia
Makie.apply_transform(transform_func, arg::VecTypes{N, T}) where {N, T}
```
where the transform function can be represented by any type, not just a `Function`.
That way it can carry auxiliary information that may be important to the transformation.
Additionally methods with other `arg` types such as numbers of `Vector`s thereof may also be implemented to more efficiently apply the `transform_func`.

Typically a `transform_func` also implements
```julia
Makie.inverse_transform(transform_func)
Makie.apply_transform(transform_func, arg::Rect3)
```
The inverse allows transforming data back, which is used for example in Axis limits.
If a reasonable inverse exists (even if it is incomplete or ambiguous) it should be given by `inverse_transform`.
The other `apply_transform` method is used in `boundingbox()` and defaults to transforming the corners of the bounding box.
It should be implemented if the default returns wrong results, e.g. with a Polar transform.

### Model Transformations

The model matrix takes care of linear transformations of plot data.
This includes scaling with `scale!()`, translations with `translate!()` and rotations with `rotate!()`.
See the [Transformation](@ref transformations_reference_docs) reference docs for more details.

## Projections

Projections are matrix transformations that move data from one coordinate system or "space" to another.
The matrices are part of the `camera(scene)` and managed either by the `cameracontrols(scene)` or by the parent `Block`.
The plot cannot change these matrices but can control which will be used via the `space` attribute.

### Camera Controller

The camera controller, be that the object in the Scene or the parent Block, generates the `view` and `projection` matrices.
These matrices are stored in `camera(scene)` which also combines them into `projectionview = projection * view`.
The `view` matrix moves plot data from a "world" coordinate system to centered and oriented based on the viewer.
This coordinate system is typically called camera, eye or view space.
From there the `projection` matrix may apply perspective projection and scales data to move to "clip" space.
This space is a normalized space where everything outside a -1 .. 1 box is clipped.
The final projection to pixel or screen space is handled implicitly by the Graphics API or explicitly in CairoMakie based on `viewport(scene)`.
The `camera(scene)` also holds onto a `pixel_space` matrix, which transforms pixel space to clip space, the scenes resolution as well as some auxiliary information about the cameras orientation.
Note that the camera controller is often referred to as the "camera" because the `camera(scene)` is just inactive storage.

### Space Attribute

The `space` attribute controls which projection matrices the plot uses.
The options refer to the input space, which generally transforms to clip space.
The options include:
- `space = :data`: Apply the cameras `view` and `projection` matrices. (This is usually called world space in Graphics APIs.)
- `space = :pixel`: Apply the camera `pixel_space` matrix.
- `space = :clip`: Apply an identity matrix.
- `space = :relative`: Apply a constant translation-scale matrix.

Note that all of these act after transformations, i.e. after the `model` matrix is applied.

### Marker Space Attribute

A few plots include a `markerspace` attribute.
For these, the projections above are split into two steps, going from `space` to `markerspace` to clip space.
The same options as above apply.
If needed, some of these matrices may also be inverted (e.g. going from :pixel -> :data -> :clip space).


## Float32Convert

The Float32Convert is an optional step that doesn't fit as cleanly into the pipeline.
It's job is to make sure that the data, projection matrices and model matrix are save to convert to Float32 types and thus can be passed to the graphics API without precision issues.

Currently only `Axis` actually defines this transformation.
When calling `plot!(axis, ...)` the `data_limits()` of the plot are recorded by the Axis and combined with existing limits.
The limits eventually trigger a camera update, where the `transform_func` gets applied.
The result gets passed to the `Float32Convert`, which updates its linear transformation to make the transformed limits Float32-safe.
The projection matrices are then derived from the safe limits.
At this point the linear transformation of the Float32Convert exists just before `view`.
If possible, it is permuted with `model` so that the model matrix can processed by the graphics API, i.e. on the GPU.


## [Projecting in Recipes](@id pipeline_recipe_projections)

As of Makie 0.24.3 the function `register_projected_positions!()` can be used to project point-like data that exists in a plot.

```@docs
register_projected_positions!
```

The most common cases are that you either want to project positions to pixel space, e.g. to mix in attributes that apply in pixel space, or apply the `transform_func` as it is a potentially non-linear transform.

To project to pixel space you can simply call `register_projected_positions!(plot, input_name = ...)`.
For this positions need to exist as a compute node and their name needs to be passed as `input_name`.
If your positions are an observable you can add them to the compute graph with `Makie.add_input!(plot.attributes, name, observable)`.
The function will then step through the whole transformation-projection pipeline, applying `transform_func`, `float32convert`, `model` and a `plot.space -> :pixel` projection matrix.
The final output is written to a node with the name `Symbol(:pixel_, input_name)`, which can be addressed in `map!()` afterwards.
The node is also returned.

If you just want to apply `transform_func` you can turn off all the other steps by setting the appropriate keyword arguments.

```julia
register_projected_positions!(
    plot, input_name = ...,
    apply_transform = false, # turn off model, transform_func, float32convert
    apply_transform_func = true, # turn transform_func back on
    output_space = :space # turn off projections
)
```

Alternatively you can also call `Makie.register_positions_transformed!(plot, input_name = ..., output_name = ...)`.

