Managed Inference Job

A Managed Inference Job runs an open-source model inside a KubeVirt virtual machine instance (VMI) on your cluster and serves it behind an OpenAI-compatible API. The job uses one of two runtimes, vLLM for language models or Parakeet for speech-to-text. You send requests the same way you would to any OpenAI-compatible endpoint.

When to use one

A Managed Inference Job fits when you want to call a model over an API. You pick an open-source model, and CosmicAC serves it.

If you instead want direct control of a GPU to run your code, a GPU Container Job is the better fit. It gives you a machine and a shell, and you set up the environment yourself.

What you get

OpenAI-compatible API — your existing clients and SDKs work without changes.
Open-source models — served on your cluster with vLLM or Parakeet.
Managed serving — CosmicAC provisions and runs the model server, so you don't set up the runtime or GPU environment yourself.

Runtimes

A Managed Inference Job uses one of two runtimes.

vLLM — serves open-source language models behind an OpenAI-compatible chat endpoint.
Parakeet — serves the NVIDIA Parakeet speech-to-text model nvidia/parakeet-tdt-0.6b-v3 behind an OpenAI-compatible transcription endpoint.

Supported models

A vLLM job serves one of these supported models, each with a model master of recommended serving parameters and hardware.

Qwen3-VL-235B-A22B-Thinking-FP8 — Qwen/Qwen3-VL-235B-A22B-Thinking-FP8
MiniMax M2.5 — MiniMaxAI/MiniMax-M2.5
Qwen2-VL-2B-Instruct — Qwen/Qwen2-VL-2B-Instruct

For each model's serving parameters and hardware, see Recommended model parameters.

A Parakeet job serves one speech-to-text model, nvidia/parakeet-tdt-0.6b-v3.

Model masters

A model master stores what CosmicAC knows about a model, such as its runtime image and default serving parameters. It saves you from configuring every field each time you serve the model.

When you select a model while creating a Managed Inference Job, its model master prefills the Serving configuration. You can adjust the prefilled values before you submit the job.

You manage model masters through the app-node API. After you deploy CosmicAC on your host machine, add a model master for each supported model. See Set up model masters. To update, add, or archive them, see Manage model masters. For the values to use, see Recommended model parameters.

Replicas

A replica is one running copy of the job's model. All replicas serve requests through a single endpoint. When you create a Managed Inference Job, you choose how many replicas to run.

Running more than one replica gives the endpoint two benefits.

Capacity — CosmicAC balances requests across every healthy replica, so more replicas serve more traffic.
Resilience — if a replica fails, the endpoint keeps serving on the remaining replicas at reduced capacity.

The endpoint's model health reflects the health of its replicas.

Model health

Alongside its status, a Managed Inference Job reports its model health, which measures how well its replicas respond to inference requests. CosmicAC determines each replica's health from health probes and recent request success, then combines the replica values into a single health value for the endpoint.

Healthy — every replica is passing its health checks and serving requests successfully. A replica that has not been probed yet also counts as healthy.
Degraded — some replicas are not healthy, but not every replica is down. A replica becomes degraded when it starts failing its health checks, or when its recent success rate falls below the configured threshold even though its health check passes. The endpoint stays up and serves on its healthy replicas.
Down — every replica is down, or no replicas are registered for the endpoint. A replica becomes down when it keeps failing its health checks for longer than the configured threshold, or when its recent success rate reaches zero.

Model health is separate from the job's status. A job can be Running while its endpoint is Degraded, because the job is up but one replica is not serving well. To check the model health of your endpoints, see Check endpoint health.

How it works

When you create a Managed Inference Job, CosmicAC allocates the GPUs each replica needs and provisions a VMI for each replica to serve the model with the chosen runtime. After the model is serving, you call it through the OpenAI-compatible endpoint. The endpoint authenticates your requests and routes them to the running model.

Restarting the job replaces its VMIs but keeps its storage and resources. Deleting the job removes its VMIs, resources, and storage.

For how a request flows through the CosmicAC components, see Architecture.

How you connect

You call the model in two ways.

OpenAI-compatible client — send requests over HTTP.
cosmicac-cli — run inference on a vLLM endpoint directly from the terminal.

Both authenticate with an API key. See Create an API key.

To connect a client to a vLLM endpoint, see Connect to a Managed Inference endpoint (vLLM). For a Parakeet endpoint, see Transcribe audio with a Parakeet endpoint.