OpenAI Frontend: set_openai_frontend()

This notebook demonstrates ServingRuntime.set_openai_frontend(), a one-call API that wires a serving function with OpenAI-compatible endpoints so the official OpenAI Python SDK can invoke an MLRun function as if it were OpenAI.

set_openai_frontend() registers two endpoint groups (OpenAIEndpoint.CHAT_COMPLETIONS and OpenAIEndpoint.RESPONSES) along with their input and output body mappings. Extra fields are filtered, mandatory fields are validated, and path parameters are extracted. Endpoint groups are selectable via the OpenAIEndpoint enum; passing no arguments registers every supported group.

A flow graph has a single terminal handler, so to serve both endpoint groups from one deployment you add a small dispatcher (openai_router) that routes each request to the right handler based on the URL path and HTTP method. Enabling include_url_info=True on the API handler config injects both mlrun_request_path and mlrun_request_method into the handler, which the dispatcher uses to choose between chat_completion_handler and response_handler (and to disambiguate routes that share a path template but differ by method).

In this section

• SDK

• Supported endpoint groups

• Setup

• Define the serving handlers

• Deploy one function for both endpoint groups

• Invoke responses

• Mandatory output validation

• Path prefix

• Custom endpoints alongside

• See also

SDK

• set_openai_frontend

Supported endpoint groups

Selected via the OpenAIEndpoint enum:

Value	OpenAI operation group	Paths registered
OpenAIEndpoint.CHAT_COMPLETIONS	/chat/completions	POST /chat/completions, GET /chat/completions, GET / POST / DELETE /chat/completions/{completion_id}, GET /chat/completions/{completion_id}/messages
OpenAIEndpoint.RESPONSES	/responses	POST /responses, GET / DELETE /responses/{response_id}, GET /responses/{response_id}/input_items, POST /responses/input_tokens, POST /responses/{response_id}/cancel, POST /responses/compact

Each group ships pre-built input and output body mappings that:

• Extract the documented OpenAI request fields from the request body as keyword arguments (e.g. model, messages, input, instructions, …).

• Filter the handler's response down to the OpenAI response contract so the SDK's typed deserializers (ChatCompletion, Response, …) accept it.

Mandatory fields (per the OpenAI spec) are enforced via mandatory=True on the relevant mappings — missing fields fail the request with HTTP 422 (Unprocessable Entity).

Setup

import httpx
import openai

import mlrun
from mlrun.serving.endpoint_mapping import APIHandlerConfig
from mlrun.serving.openai_mappings import OpenAIEndpoint

project_name = "openai-frontend-demo"
image = "mlrun/mlrun"
project = mlrun.get_or_create_project(project_name, context="./")

> 2026-06-25 13:51:39,957 [warning] Failed resolving version info. Ignoring and using defaults
> 2026-06-25 13:51:42,317 [warning] Server or client version is unstable. Assuming compatible: {"client_version":"0.0.0+unstable","server_version":"0.0.0+unstable"}
> 2026-06-25 13:51:57,702 [info] Created and saved project: {"context":"./","from_template":null,"name":"openai-frontend-demo","overwrite":false,"save":true}
> 2026-06-25 13:51:57,706 [info] Project created successfully: {"project_name":"openai-frontend-demo","stored_in_db":true}

Define the serving handlers

Two endpoint handlers and a small dispatcher:

• chat_completion_handler: returns a hard-coded ChatCompletion-shaped dict

• response_handler: returns a hard-coded Response-shaped dict

• openai_router: dispatches to one or the other, based on the URL path

Each handler includes an extra_field so the output body mapping has something to filter out.

%%writefile ./openai_handler.py
"""Serving graph handlers for the OpenAI frontend demo."""

CHAT_COMPLETION_ID = "chatcmpl_system_test_123"
RESPONSE_ID = "resp_system_test_123"


def chat_completion_handler(body, **kwargs) -> dict:
    """Return a hard-coded ChatCompletion-shaped response."""
    return {
        "id": CHAT_COMPLETION_ID,
        "choices": [
            {
                "finish_reason": "stop",
                "index": 0,
                "logprobs": None,
                "message": {"role": "assistant", "content": "Hello from MLRun!"},
            }
        ],
        "created": 1234567890,
        "model": kwargs.get("model", "gpt-4"),
        "object": "chat.completion",
        "service_tier": "default",
        "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15},
        "extra_field": "should_be_filtered",
    }


def response_handler(body, **kwargs) -> dict:
    """Return a hard-coded Response-shaped response."""
    return {
        "id": RESPONSE_ID,
        "object": "response",
        "created_at": 1741476542,
        "status": "completed",
        "completed_at": 1741476543,
        "error": None,
        "incomplete_details": None,
        "instructions": None,
        "max_output_tokens": None,
        "model": kwargs.get("model", "gpt-4"),
        "output": [
            {
                "type": "message",
                "id": "msg_system_test_001",
                "status": "completed",
                "role": "assistant",
                "content": [
                    {
                        "type": "output_text",
                        "text": "Hello from MLRun!",
                        "annotations": [],
                    }
                ],
            }
        ],
        "parallel_tool_calls": True,
        "previous_response_id": None,
        "reasoning": {"effort": None, "summary": None},
        "store": True,
        "temperature": 1.0,
        "text": {"format": {"type": "text"}},
        "tool_choice": "auto",
        "tools": [],
        "top_p": 1.0,
        "truncation": "disabled",
        "usage": {
            "input_tokens": 36,
            "input_tokens_details": {"cached_tokens": 0},
            "output_tokens": 87,
            "output_tokens_details": {"reasoning_tokens": 0},
            "total_tokens": 123,
        },
        "metadata": {},
        "extra_field": "should_be_filtered",
    }


def openai_router(
    body,
    mlrun_request_path: str,
    mlrun_request_method: str,
    **kwargs,
) -> dict:
    """Dispatch to the right OpenAI handler based on the request URL path and HTTP method."""
    if mlrun_request_method == "POST" and mlrun_request_path == "/chat/completions":
        return chat_completion_handler(body, **kwargs)
    if mlrun_request_method == "POST" and mlrun_request_path == "/responses":
        return response_handler(body, **kwargs)
    raise RuntimeError(
        f"no handler for {mlrun_request_method} {mlrun_request_path}"
    )

Writing ./openai_handler.py

---

Deploy one function for both endpoint groups

set_openai_frontend() (with no arguments) registers every supported endpoint group, both /chat/completions and /responses, on a single function. Enable include_url_info=True on the API handler config so the request path and HTTP method are injected into the handler as mlrun_request_path and mlrun_request_method. The openai_router dispatcher then forwards each request to the matching handler. The OpenAI SDK can talk to the same function for both APIs.

openai_fn = project.set_function(
    func="openai_handler.py",
    name="openai-frontend",
    kind="serving",
    image=image,
)
# include_url_info=True - injects the request path and HTTP method into the handler
# as `mlrun_request_path` and `mlrun_request_method`.
# set_openai_frontend() merges its endpoints into this existing config.
openai_fn.set_api_handler_config(APIHandlerConfig(include_url_info=True))
openai_fn.set_openai_frontend()  # registers both /chat/completions and /responses

graph = openai_fn.set_topology("flow", engine="sync")
graph.to(name="router", handler="openai_router").respond()
openai_fn.deploy()

Invoke /chat/completions via the OpenAI SDK:

client = openai.OpenAI(
    base_url=openai_fn.get_url(),
    api_key="dummy",
    http_client=httpx.Client(verify=mlrun.mlconf.httpdb.http.verify),
)

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response)
# Typed ChatCompletion object - 'extra_field' from the handler was filtered out by the output mapping

ChatCompletion(id='chatcmpl_system_test_123', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Hello from MLRun!', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None))], created=1234567890, model='gpt-4', object='chat.completion', moderation=None, service_tier='default', system_fingerprint=None, usage=CompletionUsage(completion_tokens=5, prompt_tokens=10, total_tokens=15, completion_tokens_details=None, prompt_tokens_details=None))

assert isinstance(response, openai.types.chat.ChatCompletion)
assert response.id == "chatcmpl_system_test_123"
assert response.choices[0].message.content == "Hello from MLRun!"

---

Invoke /responses

No redeploy needed - the same function already has /responses registered alongside /chat/completions. The OpenAI SDK call is the only thing that changes.

# Same client, same URL - just a different SDK call
response = client.responses.create(model="gpt-4", input="Hello")
print(response)

Response(id='resp_system_test_123', created_at=1741476542.0, error=None, incomplete_details=None, instructions=None, metadata={}, model='gpt-4', object='response', output=[ResponseOutputMessage(id='msg_system_test_001', content=[ResponseOutputText(annotations=[], text='Hello from MLRun!', type='output_text', logprobs=None)], role='assistant', status='completed', type='message', phase=None)], parallel_tool_calls=True, temperature=1.0, tool_choice='auto', tools=[], top_p=1.0, background=None, completed_at=1741476543.0, conversation=None, max_output_tokens=None, max_tool_calls=None, moderation=None, previous_response_id=None, prompt=None, prompt_cache_key=None, prompt_cache_retention=None, reasoning=Reasoning(context=None, effort=None, generate_summary=None, summary=None), safety_identifier=None, service_tier=None, status='completed', text=ResponseTextConfig(format=ResponseFormatText(type='text'), verbosity=None), top_logprobs=None, truncation='disabled', usage=ResponseUsage(input_tokens=36, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=87, output_tokens_details=OutputTokensDetails(reasoning_tokens=0), total_tokens=123), user=None, output_text=None, store=True)

assert isinstance(response, openai.types.responses.Response)
assert response.id == "resp_system_test_123"
assert response.output[0].content[0].text == "Hello from MLRun!"

---

Mandatory output validation

If the handler omits a mandatory output field, the framework returns HTTP 422 before the OpenAI SDK can attempt to deserialize a malformed response. Below is a second handler file, broken_handler.py, whose handler intentionally omits the mandatory id field of /responses.

%%writefile ./broken_handler.py
"""Broken Responses handler: omits the mandatory 'id' output field."""


def response_handler_missing_mandatory(body, **kwargs) -> dict:
    return {
        "object": "response",
        "created_at": 1741476542,
        "model": kwargs.get("model", "gpt-4"),
        # 'id' intentionally omitted - mandatory output field
    }

Writing ./broken_handler.py

broken_fn = project.set_function(
    func="broken_handler.py",
    name="openai-responses-missing-mandatory",
    kind="serving",
    image=image,
)
broken_fn.set_openai_frontend([OpenAIEndpoint.RESPONSES])
graph = broken_fn.set_topology("flow", engine="sync")
graph.to(name="handler", handler="response_handler_missing_mandatory").respond()
broken_fn.deploy()

client = openai.OpenAI(
    base_url=broken_fn.get_url(),
    api_key="dummy",
    http_client=httpx.Client(verify=mlrun.mlconf.httpdb.http.verify),
)

try:
    client.responses.create(model="gpt-4", input="Hello")
except openai.APIStatusError as exc:
    print(f"status: {exc.status_code}")
    print(exc)
    # 422 - MLRunUnprocessableEntityError: Mandatory field 'id' not found in body

status: 422
MLRunUnprocessableEntityError: Failed to process output body mapping: Mandatory field 'id' not found in body, caused by: Mandatory field 'id' not found in body

---

Path prefix

The optional prefix= argument prepends a path segment to every registered endpoint (e.g. prefix="/v1" registers /v1/chat/completions, /v1/responses, …). The prefix must start with / — a missing leading slash raises MLRunInvalidArgumentError.

---

Custom endpoints alongside

set_openai_frontend() is additive — it merges its endpoints into the existing APIHandlerConfig. You can add custom endpoints (e.g. health checks, admin) on top:

from http import HTTPMethod
from mlrun.common.schemas.serving import APIHandlerAction

fn.set_openai_frontend(prefix="/v1")
config = APIHandlerConfig.from_dict(fn.spec.api_handler_config)
config.add_endpoint_handler("/health", HTTPMethod.GET, APIHandlerAction.ALLOW)
fn.set_api_handler_config(config)

---

See also

• API handler — the underlying mechanism set_openai_frontend() builds on.

• OpenAIEndpoint enum — pass a subset (e.g. [OpenAIEndpoint.CHAT_COMPLETIONS]) to register only that group, or call set_openai_frontend() with no arguments to register every supported group.