Application runtime#

You can use the ApplicationRuntime() to provide an image that runs on top of your deployed model.

The application runtime deploys a container image (for example, a web application) that is exposed on a specific port, and a command to run the HTTP server. The runtime is based on top of Nuclio, and adds the application as a side-car to a Nuclio function pod while the actual function is a reverse proxy to that application.

You can set an existing image to run in the application, or let the application runtime build the side-car image for you, by specifying the source code, and pulling at build-time.

Note: The default base image is python:3.9 when not specifying an image for the application.

An API Gateway, by default, is in front of the application and can provide different authentication methods, or none.

Typical use cases are:

  • Deploy a Vizro dashboard that communicates with an external source (for example, a serving model) to display graphs, data, and inference.

  • Deploy a model and a UI — the model serving is the backend and the UI is the side car.

  • Deploy a fastapi web-server with an MLRun model. In this case, the Nuclio function is a reverse proxy and the user web-app is the side car.

Usage examples#

Deploy a Vizro dashboard from a pre-built image:

# Create an application runtime (with pre-built image)
application = project.set_function(
    name="my-vizro-dashboard", kind="application", image="repo/my-vizro-image:latest"
)
# Set the port that the side-car listens on
application.set_internal_application_port(port=8050)

# Deploy
application.deploy()

Deploy a Vizro dashboard from a source archive or git:

# Specify the source to be loaded at build-time or run-time
application = project.set_function(
    name="my-vizro-dashboard", kind="application", requirements=["vizro"]
)
application.set_internal_application_port(8050)
application.spec.command = "gunicorn"
application.spec.args = [
    "<my-app>:<my-server>",
    "--bind",
    "0.0.0.0:8050",
]

# Provide code artifacts
application.with_source_archive(
    "git://github.com/org/repo#my-branch", pull_at_runtime=False
)

# Build the application image via MLRun and deploy the Nuclio function
# Optionally add mlrun
application.deploy(with_mlrun=False)

Reusing an already built reverse proxy image is done when:

  • Redeploying a function that built the reverse proxy once and has application.status.container_image enriched.

  • It was already built manually with mlrun.runtimes.ApplicationRuntime.deploy_reverse_proxy_image().

Using one of the above options can help minimize redundant builds and streamline the development cycle.

Authentication modes#

An application runtime can be accessed through an API Gateway that supports various authentication methods.

The default authentication mode is none for open source and access-key for the Iguazio platform.

The different authentication modes can be configured as follows (see APIGateway() for further information):

from mlrun.common.schemas.api_gateway import APIGatewayAuthenticationMode

# Unless disabled, the default API gateway is created when the application is deployed
application.deploy(create_default_api_gateway=False)

# Create API gateway without authentication
application.create_api_gateway(
    authentication_mode=APIGatewayAuthenticationMode.none,
)

# Basic authentication mode.
# This means that the application can be invoked only using the provided credentials
application.create_api_gateway(
    authentication_mode=APIGatewayAuthenticationMode.basic,
    authentication_creds=("my-username", "my-password"),
)

# Access-key authentication mode. the application can be invoked only with a valid session
application.create_api_gateway(
    authentication_mode=APIGatewayAuthenticationMode.access_key,
)

Other API gateway configurations#

There are additional parameters that can be configured when creating an API gateway:

application.create_api_gateway(
    # The name of the API gateway
    name="my-api-gateway",
    # Optional path of the API gateway, default value is "/". The given path should be supported by the deployed application
    path="/",
    # Set to True to allow direct port access to the application sidecar
    direct_port_access=False,
    # Set to True to force SSL redirect, False to disable. Defaults to mlrun.mlconf.force_api_gateway_ssl_redirect()
    ssl_redirect=True,
    # Set the API gateway as the default for the application (`status.api_gateway`)
    set_as_default=False,
)

An API gateway can also be created manually. See the instructions here.

Application runtime in a dark environment#

To use application runtime in a dark (air-gapped) environment, you need to build the reverse proxy image and push it to a private registry, following the steps below:

# 1. Create the reverse proxy image in a non air-gapped system
import mlrun.runtimes

mlrun.runtimes.ApplicationRuntime.deploy_reverse_proxy_image()

# 2. The created image name is saved on the ApplicationRuntime class:
mlrun.runtimes.ApplicationRuntime.reverse_proxy_image

# 3. Push the created image to the system’s docker registry

# 4. On the air-gapped environment, set the image on the class property:
mlrun.runtimes.ApplicationRuntime.reverse_proxy_image = (
    "registry/reverse-proxy-image:<tag>"
)

# 5. When creating application functions, this image will be used as the reverse proxy image, and it won’t be built again.

Application and serving function integration#

This example demonstrates deploying a serving function and using it with the application.

Serving creation:
First, create the model file and save it as a .py file.

%%writefile /your/path/add_ten_model.py

from mlrun.serving import V2ModelServer

class AddTenModel(V2ModelServer):
    def load(self):
        # No actual model to load, just a demo
        pass

    def predict(self, request):
        input = request['inputs'][0]
        result = input + 10
        return {"outputs": [result]}

Now, deploy this model as a serving function:

import mlrun

project_name = "app-demo-flask"
model_name = "add-ten-model"
model_path = "/your/path/add_ten_model.py"

project = mlrun.get_or_create_project(project_name)

# Create the serving function
function = project.set_function(
    name="add-ten-serving",
    kind="serving",
    filename=model_path,
)

# Add the model to the function (even though there's no real model in this case)
function.add_model(model_name, model_path=model_path, class_name="AddTenModel")
function.deploy()

# An example of invoke:
function.invoke(f"/v2/models/{model_name}/infer", {"inputs": [20]})["outputs"][
    "outputs"
]

Application creation:
First, create the flask server application.
The application includes several different endpoints - to check its functionality:

%%writefile flask_app_example.py

from flask import Flask
import requests
import mlrun

app = Flask(__name__)


@app.route("/internal")
def internal():
    # Test access to the serving function with MLRun.
    project_name = "app-demo-flask"
    project = mlrun.get_or_create_project(project_name)
    function = project.get_function("add-ten-serving")
    response = function.invoke("/v2/models/add-ten-model/infer", {"inputs": [20]})
    output = response["outputs"]["outputs"]
    return {"result": output}


@app.route("/external")
def external():
    # Test access to the serving function without MLRun (externally).
    project_name = "app-demo-flask"
    project = mlrun.get_or_create_project(project_name)
    function = project.get_function("add-ten-serving")
    url = f"https://{function.status.external_invocation_urls[0]}"
    response = requests.post(url, json={"inputs": [50]}).json()
    output = response["outputs"]["outputs"]
    return {"result": output}

Archive the application code into a .tar.gz file:

!tar -czvf archive.tar.gz flask_app_example.py

Now, deploy this flask application:

import mlrun

project = mlrun.get_or_create_project("app-demo-flask")
# Create a demo secret for testing
project.set_secrets(secrets={"secret-example": "project_secret_example"})

# Specify source to be loaded on build time
# The image or the requirements should include mlrun, flask and gunicorn.
application = project.set_function(
    name="flask_app",
    kind="application",
    requirements_file="/your/path/requirements.txt",
)

# Provide code artifacts
application.with_source_archive(
    "v3io:///your/path/archive.tar.gz", pull_at_runtime=False
)

application.set_internal_application_port(5000)
application.spec.command = "gunicorn"
application.spec.args = [
    "flask_app_example:app",
    "--bind",
    "127.0.0.1:5000",
    "--log-level",
    "debug",
]
application.deploy(with_mlrun=True)

# Test the deployment:
application.invoke("/internal").json()

application.invoke("/external").json()