// Module included in the following assemblies:
//
// * microshift_ai/microshift-rhoai.adoc

:_mod-docs-content-type: CONCEPT
[id="microshift-rhoai-serving-ai-models-con_{context}"]
= Serving AI models on {microshift-short}

[role="_abstract"]
You can review the following information to learn how to serve models on the {rhoai} single-model serving platform in {microshift-short} by configuring a model-serving runtime using the `ServingRuntime` and `InferenceService` custom resource (CRs).

Model-serving runtimes for AI models in {microshift-short}::

A model-serving runtime is an environment for deploying and managing an AI model, providing integration with a specified model server and the model frameworks that it supports. Creating the model-serving runtime means configuring the objects that select the right model format for the AI model and serve the queries, among other detailed functions specific your deployment.

The `ServingRuntime` custom resource::

The `ServingRuntime` CR is a YAML file that defines the templates for pods that can dynamically load and unload AI model formats and exposes a service endpoint for querying the model through the API. Each `ServingRuntime` CR contains the information needed to run AI models, including the container image of the runtime and a list of the model formats that the model-serving runtime supports. Other configuration settings for the model-serving runtime can be set with environment variables defined in the container specification.

The `InferenceService` custom resource::

The `InferenceService` CR is a YAML file that creates a server or inference service to process inference queries, pass them to the model, then return the inference output. In {microshift-short}, the output is returned in the CLI. This inference service configuration file can also include many other options, such as specifying a hardware accelerator.

[IMPORTANT]
====
As a single-node Kubernetes distribution, {microshift-short} does not support multi-model deployments. You must use the single-model serving platform. In each {microshift-short} deployment, you can use one AI model, but potentially more than one model runtime.
====

Workflow for configuring a model-serving runtime::

* Select the model-serving runtime that supports the format of your AI model.

* Create the `ServingRuntime` CR in your workload namespace.
//CRD is shipped with product; the CR is what users are creating.

* If the {microshift-short} node is already running, you can export the required `ServingRuntime` CR to a file and edit it.

* If the {microshift-short} node is not running, or if you want to manually prepare a manifest, you can use the original definition on the disk, which is part of the `microshift-ai-model-serving` RPM.

* Create the `InferenceService` CR in your workload namespace.
//CRD is shipped with product; the CR is what users are creating.
