Realtime monitoring and drift detection#

This tutorial illustrates the basic model monitoring capabilities of MLRun: deploying a model to a live endpoint and calculating data drift.

See the overview to model monitoring in Model monitoring, and make sure you have reviewed the basics in MLRun Quick Start Tutorial.

MLRun installation and configuration#

Before running this notebook make sure mlrun is installed and that you have configured the access to the MLRun service.

Set up the project#

First, import the dependencies and create an MLRun project. This contains all of the models, functions, datasets, etc.

%config Completer.use_jedi = False
import os
import pandas as pd
from sklearn.datasets import load_iris
import mlrun
from mlrun import import_function, get_dataitem, get_or_create_project
import uuid

project_name = "mm-app-project"
project = get_or_create_project(project_name, context="./")
> 2024-05-21 19:58:56,161 [info] Project loaded successfully: {'project_name': 'mm-app-project'}

Note

This tutorial does not focus on training a model. Instead, it starts with a trained model and its corresponding training dataset.

Enable model monitoring#

Model monitoring is enabled per project. enable_model_monitoring() brings up the controller and schedules it according to the base_period, and deploys the writer.

The controller runs, by default, every 10 minutes, which is also the minimum interval. You can modify the frequency with the parameter base_period. To change the base_period, first run disable_model_monitoring(), then run enable_model_monitoring with the new base_period value.

project.enable_model_monitoring(base_period=1)
{'kind': 'job',
 'metadata': {'name': 'model-monitoring-controller',
  'tag': 'latest',
  'hash': 'b2a49d5dcc32e715d708d0859ea13efc1dd1369d',
  'project': 'mm-app-project',
  'categories': [],
  'updated': '2024-05-21T19:58:00.032834+00:00',
  'credentials': {'access_key': '$ref:mlrun-auth-secrets.f74b03408698823c96b9a2c456fe27f0192177bdb8a2416d16bfbf21'}},
 'spec': {'command': '',
  'args': [],
  'image': 'mlrun/mlrun',
  'build': {'functionSourceCode': 'IyBDb3B5cmlnaHQgMjAyMyBJZ3VhemlvCiMKIyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKIyB5b3UgbWF5IG5vdCB1c2UgdGhpcyBmaWxlIGV4Y2VwdCBpbiBjb21wbGlhbmNlIHdpdGggdGhlIExpY2Vuc2UuCiMgWW91IG1heSBvYnRhaW4gYSBjb3B5IG9mIHRoZSBMaWNlbnNlIGF0CiMKIyAgIGh0dHA6Ly93d3cuYXBhY2hlLm9yZy9saWNlbnNlcy9MSUNFTlNFLTIuMAojCiMgVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQojIGRpc3RyaWJ1dGVkIHVuZGVyIHRoZSBMaWNlbnNlIGlzIGRpc3RyaWJ1dGVkIG9uIGFuICJBUyBJUyIgQkFTSVMsCiMgV0lUSE9VVCBXQVJSQU5USUVTIE9SIENPTkRJVElPTlMgT0YgQU5ZIEtJTkQsIGVpdGhlciBleHByZXNzIG9yIGltcGxpZWQuCiMgU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAojIGxpbWl0YXRpb25zIHVuZGVyIHRoZSBMaWNlbnNlLgoKaW1wb3J0IG1scnVuCmZyb20gbWxydW4ubW9kZWxfbW9uaXRvcmluZy5jb250cm9sbGVyIGltcG9ydCBNb25pdG9yaW5nQXBwbGljYXRpb25Db250cm9sbGVyCgoKZGVmIGhhbmRsZXIoY29udGV4dDogbWxydW4ucnVuLk1MQ2xpZW50Q3R4KSAtPiBOb25lOgogICAgIiIiCiAgICBSdW4gbW9kZWwgbW9uaXRvcmluZyBhcHBsaWNhdGlvbiBwcm9jZXNzb3IKCiAgICA6cGFyYW0gY29udGV4dDogdGhlIE1MUnVuIGNvbnRleHQKICAgICIiIgogICAgbW9uaXRvcl9hcHBfY29udHJvbGxlciA9IE1vbml0b3JpbmdBcHBsaWNhdGlvbkNvbnRyb2xsZXIoCiAgICAgICAgY29udGV4dD1jb250ZXh0LAogICAgICAgIHByb2plY3Q9Y29udGV4dC5wcm9qZWN0LAogICAgKQogICAgbW9uaXRvcl9hcHBfY29udHJvbGxlci5ydW4oKQo=',
   'commands': [],
   'code_origin': '/mlrun/mlrun/model_monitoring/controller_handler.py',
   'origin_filename': '/mlrun/mlrun/model_monitoring/controller_handler.py',
   'requirements': []},
  'entry_points': {'handler': {'name': 'handler',
    'doc': 'Run model monitoring application processor',
    'parameters': [{'name': 'context',
      'type': 'mlrun.run.MLClientCtx',
      'doc': 'the MLRun context'}],
    'outputs': [{'type': 'None'}],
    'lineno': 19,
    'has_varargs': False,
    'has_kwargs': False}},
  'description': '',
  'default_handler': 'handler',
  'disable_auto_mount': True,
  'volumes': [],
  'volume_mounts': [],
  'env': [{'name': 'MODEL_MONITORING_ACCESS_KEY',
    'valueFrom': {'secretKeyRef': {'key': 'mlrun.model-monitoring.MODEL_MONITORING_ACCESS_KEY',
      'name': 'mlrun-project-secrets-mm-app-project'}}},
   {'name': 'V3IO_API', 'value': 'http://v3io-webapi:8081'},
   {'name': 'V3IO_USERNAME', 'value': 'pipelines'},
   {'name': 'V3IO_ACCESS_KEY',
    'valueFrom': {'secretKeyRef': {'key': 'accessKey',
      'name': 'mlrun-auth-secrets.8ccbe9615ecd005bf03eccc855e7d970c514b5ba685d21dc302a0790'}}},
   {'name': 'V3IO_FRAMESD', 'value': 'http://framesd:8080'},
   {'name': 'MLRUN_AUTH_SESSION',
    'valueFrom': {'secretKeyRef': {'key': 'accessKey',
      'name': 'mlrun-auth-secrets.f74b03408698823c96b9a2c456fe27f0192177bdb8a2416d16bfbf21'}}}],
  'resources': {'requests': {'memory': '1Mi', 'cpu': '25m'},
   'limits': {'memory': '20Gi', 'cpu': '2'}},
  'priority_class_name': 'igz-workload-medium',
  'preemption_mode': 'prevent',
  'state_thresholds': {'pending_scheduled': '1h',
   'pending_not_scheduled': '-1',
   'image_pull_backoff': '1h',
   'executing': '24h'},
  'affinity': None,
  'tolerations': None,
  'security_context': {}},
 'verbose': False,
 'status': {}}

Log the model artifacts#

See full parameter details in log_model().

First download the pickle file.

iris = load_iris()
train_set = pd.DataFrame(
    iris["data"],
    columns=["sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm"],
)

model_name = "RandomForestClassifier"
project.log_model(
    model_name,
    model_file="_static/model.pkl",
    training_set=train_set,
    framework="sklearn",
)
<mlrun.artifacts.model.ModelArtifact at 0x7faa31ee1850>

Import, enable monitoring, and deploy the serving function#

Import the model server function from the MLRun Function Hub, add the model that was logged via experiment tracking, and enable drift detection.

The model monitoring infrastructure was already enabled in Enable model monitoring. Now, you enable monitoring on this specific function and its related models with set_tracking. This activates all inferences and predictions, which is used for drift detection.

Then you deploy the serving function with drift detection enabled with a single line of code.

The result of this step is that the model-monitoring stream pod writes data to Parquet, by model endpoint. Every base period, the controller checks for new data. Each time it finds new data, it sends it to the relevant app.

# Import the serving function
serving_fn = import_function(
    "hub://v2_model_server", project=project_name, new_name="serving"
)

# Add the model to the serving function's routing spec
serving_fn.add_model(
    model_name, model_path=f"store://models/{project_name}/{model_name}:latest"
)

# Enable monitoring on this serving function
serving_fn.set_tracking()

serving_fn.spec.build.requirements = ["scikit-learn"]

# Deploy the serving function
project.deploy_function(serving_fn)
> 2024-05-21 19:58:56,822 [info] Starting remote function deploy
2024-05-21 19:58:57  (info) Deploying function
2024-05-21 19:58:57  (info) Building
2024-05-21 19:58:57  (info) Staging files and preparing base images
2024-05-21 19:58:58  (warn) Using user provided base image, runtime interpreter version is provided by the base image
2024-05-21 19:58:58  (info) Building processor image
2024-05-21 20:00:03  (info) Build complete
2024-05-21 20:00:11  (info) Function deploy complete
> 2024-05-21 20:00:18,995 [info] Successfully deployed function: {'internal_invocation_urls': ['nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['mm-app-project-serving.default-tenant.app.dev19.lab.iguazeng.com/']}
DeployStatus(state=ready, outputs={'endpoint': 'http://mm-app-project-serving.default-tenant.app.dev19.lab.iguazeng.com/', 'name': 'mm-app-project-serving'})

View deployed resources#

At this point, you should see the controller and the model-monitoring-batch jobs in the UI under Projects | Jobs and Workflows.

Invoke the model#

See full parameter details in invoke().

import json
from time import sleep
from random import choice, uniform

iris = load_iris()
iris_data = iris["data"].tolist()

model_name = "RandomForestClassifier"
serving_1 = project.get_function("serving")
0
for i in range(150):
    # data_point = choice(iris_data)
    data_point = [0.5, 0.5, 0.5, 0.5]
    serving_1.invoke(
        f"v2/models/{model_name}/infer", json.dumps({"inputs": [data_point]})
    )
    sleep(choice([0.01, 0.04]))
> 2024-05-21 20:00:19,087 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,406 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,433 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,460 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,518 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,576 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,632 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,687 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,718 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,745 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,801 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,824 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,852 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,879 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,905 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,964 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:19,986 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,043 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,099 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,155 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,211 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,233 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,255 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,281 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,307 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,334 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,386 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,410 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,432 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,457 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,514 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,568 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,625 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,656 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,713 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,768 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,827 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,880 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,933 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:20,967 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,030 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,054 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,078 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,106 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,171 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,228 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,284 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,311 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,362 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,384 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,408 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,437 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,495 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,520 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,542 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,568 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,620 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,646 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,698 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,750 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,775 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,833 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,887 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,912 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,970 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:21,996 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,023 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,046 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,068 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,153 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,176 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,229 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,281 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,337 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,374 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,397 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,449 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,504 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,556 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,581 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,637 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,662 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,690 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,747 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,801 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,825 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,882 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,905 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,932 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:22,984 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,006 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,057 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,109 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,132 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,159 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,216 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,241 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,267 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,319 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,340 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,392 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,415 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,436 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,488 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,545 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,598 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,668 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,721 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,774 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,796 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,848 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,870 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,891 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:23,943 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,002 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,024 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,075 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,131 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,154 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,175 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,197 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,219 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,241 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,293 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,345 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,367 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,419 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,442 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,464 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,538 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,566 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,619 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,641 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,693 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,745 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,767 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,789 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,841 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,863 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,885 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,910 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,932 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:24,984 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:25,006 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:25,028 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:25,080 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:25,103 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:25,124 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:25,146 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-21 20:00:25,168 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}

At this stage you can see the model endpoints and minimal meta data (for example, last prediction and average latency) in the Models | Model Endpoints page.

../_images/model_endpoint_1.png

You can also see the basic statistics in Grafana.

Register and deploy the model-monitoring apps#

The next step is to deploy the model-monitoring app to generate the full meta data. Add the monitoring app to the project using set_model_monitoring_function(). Then, deploy the app using deploy_function().

This example illustrates two monitoring aps:

After deploying the jobs they show in the UI under Real-time functions (Nuclio).

Batch monitoring app#

First download the demo_app.

# register the first app named "demo_app"
my_app = project.set_model_monitoring_function(
    func="_static/demo_app.py",
    application_class="DemoMonitoringApp",
    name="myApp",
)

project.deploy_function(my_app)
> 2024-05-21 20:01:43,222 [info] Starting remote function deploy
2024-05-21 20:01:43  (info) Deploying function
2024-05-21 20:01:43  (info) Building
2024-05-21 20:01:43  (info) Staging files and preparing base images
2024-05-21 20:01:43  (warn) Using user provided base image, runtime interpreter version is provided by the base image
2024-05-21 20:01:43  (info) Building processor image
2024-05-21 20:03:28  (info) Build complete
2024-05-21 20:04:25  (info) Function deploy complete
> 2024-05-21 20:04:25,080 [info] Successfully deployed function: {'internal_invocation_urls': ['nuclio-mm-app-project-myapp.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['mm-app-project-myapp.default-tenant.app.dev19.lab.iguazeng.com/']}
DeployStatus(state=ready, outputs={'endpoint': 'http://mm-app-project-myapp.default-tenant.app.dev19.lab.iguazeng.com/', 'name': 'mm-app-project-myapp'})

Evidently app#

First download evidently_app.

# register the second app named "evidently_app"
my_evidently_app = project.set_model_monitoring_function(
    func="_static/evidently_app.py",
    image="mlrun/mlrun",
    requirements=[
        "evidently~=0.4.3",
    ],
    name="MyEvidentlyApp",
    application_class="DemoEvidentlyMonitoringApp",
    evidently_workspace_path=os.path.abspath(
        f"/v3io/projects/{project_name}/artifacts/evidently_workspace"
    ),
    evidently_project_id=str(uuid.uuid4()),
)

project.deploy_function(my_evidently_app)
> 2024-05-22 09:59:53,566 [info] Starting remote function deploy
2024-05-22 09:59:53  (info) Deploying function
2024-05-22 09:59:53  (info) Building
2024-05-22 09:59:53  (info) Staging files and preparing base images
2024-05-22 09:59:53  (warn) Using user provided base image, runtime interpreter version is provided by the base image
2024-05-22 09:59:53  (info) Building processor image
2024-05-22 10:02:09  (info) Build complete
2024-05-22 10:02:33  (info) Function deploy complete
> 2024-05-22 10:02:35,710 [info] Successfully deployed function: {'internal_invocation_urls': ['nuclio-mm-app-project-myevidentlyapp.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['mm-app-project-myevidentlyapp.default-tenant.app.dev19.lab.iguazeng.com/']}
DeployStatus(state=ready, outputs={'endpoint': 'http://mm-app-project-myevidentlyapp.default-tenant.app.dev19.lab.iguazeng.com/', 'name': 'mm-app-project-myevidentlyapp'})

Invoke the model again#

The controller checks for new datasets every base_period to send to the app. Invoking the model a second time ensures that the previous window closed and therefore the data contains the full monitoring window. From this point on, the applications are triggered by the controller. The controller checks the Parquet DB every 10 minutes (or non-default base_period) and streams any new data to the app.

import json
from time import sleep
from random import choice, uniform

iris = load_iris()
iris_data = iris["data"].tolist()

model_name = "RandomForestClassifier"
serving_1 = project.get_function("serving")

for i in range(150):
    data_point = choice(iris_data)
    # data_point = [0.5,0.5,0.5,0.5]
    serving_1.invoke(
        f"v2/models/{model_name}/infer", json.dumps({"inputs": [data_point]})
    )
    sleep(choice([0.01, 0.04]))
> 2024-05-22 10:02:57,822 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,287 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,343 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,399 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,423 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,447 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,502 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,527 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,582 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,638 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,692 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,748 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,771 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,825 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,849 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,901 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,924 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,947 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:58,999 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,023 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,075 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,129 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,183 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,206 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,258 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,312 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,334 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,386 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,409 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,461 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,514 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,541 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,566 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,620 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,644 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,698 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,751 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,804 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,828 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,880 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,936 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:02:59,991 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,044 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,092 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,116 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,169 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,222 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,276 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,329 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,353 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,414 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,438 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,491 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,546 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,599 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,625 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,649 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,672 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,695 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,748 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,802 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,856 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,910 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,933 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,956 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:00,978 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,002 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,057 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,087 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,139 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,163 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,185 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,207 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,260 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,320 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,347 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,374 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,396 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,449 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,505 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,529 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,551 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,572 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,597 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,620 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,673 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,727 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,780 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,803 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,856 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,910 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,934 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:01,957 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,011 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,065 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,088 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,141 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,165 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,189 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,242 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,266 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,319 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,372 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,426 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,480 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,504 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,557 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,581 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,605 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,629 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,683 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,737 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,761 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,814 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,836 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,859 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,912 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,935 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,957 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:02,980 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,002 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,059 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,111 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,135 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,159 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,182 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,204 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,257 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,281 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,303 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,326 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,348 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,371 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,394 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,447 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,499 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,522 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,544 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,596 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,619 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,671 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,693 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,714 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,736 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,788 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,810 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,862 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,885 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,936 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}
> 2024-05-22 10:03:03,989 [info] Invoking function: {'method': 'POST', 'path': 'http://nuclio-mm-app-project-serving.default-tenant.svc.cluster.local:8080/v2/models/RandomForestClassifier/infer'}

View the application results#

../_images/mm-myapp.png

And if you've used Evidently:

../_images/mm-logger-dashb-evidently.png ../_images/mm-evidently.png

View the status of the model monitoring jobs#

View the model monitoring jobs in Jobs and Workflows. Model monitoring jobs run continuously, therefore they should have a blue dot indicating that the function is running. (A green dot indicates that the job completed.)

For more information on the UI, see Model monitoring using the platform UI.

../_images/mm-monitor-jobs.png

View detailed drift dashboards#

Grafana has detailed dashboards that show additional information on each model in the project:

For more information on the dashboards, see Model monitoring using Grafana dashboards.

The Overview dashboard displays the model endpoint IDs of a specific project. Only deployed models with Model Monitoring enabled are displayed. Endpoint IDs are URIs used to provide access to performance data and drift detection statistics of a deployed model.

grafana_dashboard_1

The model monitoring details dashboard displays the real-time performance data of the selected model, including graphs of individual features over time.

grafana_dashboard_2

The model monitoring performance dashboard displaysdrift and operational metrics over time.

grafana_dashboard_3

Batch infer model-monitoring#

Batch inference is not part of the model monitoring flow in this tutorial, it is an alternate method of evaluating data against your logged model without disturbing the model, for example a one-time evaluation of new data. See related details in the Batch inference and drift detection tutorial.

import random

batch_inference_function = import_function(
    "hub://batch_inference_v2", project=project_name
)

endpoint_id = "batch_infer_endpoint_7"
data = random.sample(iris_data, 20)
df = pd.DataFrame(
    data,
    columns=["sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm"],
)
dataset = project.log_dataset("sampel_data_iris", df)
batch_inference_run = batch_inference_function.run(
    handler="infer",
    inputs={"dataset": dataset.uri},
    params={
        "model_path": f"store://models/{project_name}/{model_name}:latest",
        "perform_drift_analysis": True,
        "endpoint_id": endpoint_id,
    },
    local=True,
)
> 2024-05-22 10:04:12,315 [info] Storing function: {'name': 'batch-inference-v2-infer', 'uid': '03bbefd4d5304a1280e38a30d18d86b3', 'db': 'http://mlrun-api:8080'}
> 2024-05-22 10:04:12,630 [info] Loading model...
> 2024-05-22 10:04:13,178 [info] Loading data...
Trying to unpickle estimator DecisionTreeClassifier from version 1.4.0 when using version 1.4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
Trying to unpickle estimator RandomForestClassifier from version 1.4.0 when using version 1.4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
> 2024-05-22 10:04:13,615 [info] Calculating prediction...
> 2024-05-22 10:04:13,626 [info] Logging result set (x | prediction)...
> 2024-05-22 10:04:13,741 [info] Performing drift analysis...
X has feature names, but RandomForestClassifier was fitted without feature names
> 2024-05-22 10:04:14,919 [info] Bumping model endpoint last request time (EP without serving): {'project': 'mm-app-project', 'endpoint_id': 'batch_infer_endpoint_7', 'last_request': '2024-05-22T10:04:13.767835+00:00', 'current_request': '2024-05-22T10:04:13.865900+00:00', 'bumped_last_request': '2024-05-22T10:06:13.865900+00:00'}
project uid iter start state name labels inputs parameters results artifacts
mm-app-project 0 May 22 10:04:12 completed batch-inference-v2-infer
v3io_user=admin
kind=local
owner=admin
host=jupy-jill-58fffdb54d-qjsrl
dataset
model_path=store://models/mm-app-project/RandomForestClassifier:latest
perform_drift_analysis=True
endpoint_id=batch_infer_endpoint_7
batch_id=42b09701606482d3a39636aeb5024eaebb9086a400b09b0fcb3681a2
prediction

> to track results use the .show() or .logs() methods or click here to open in UI
> 2024-05-22 10:04:15,085 [info] Run execution finished: {'status': 'completed', 'name': 'batch-inference-v2-infer'}
batch_inference_run.artifact("prediction").as_df()
sepal_length_cm sepal_width_cm petal_length_cm petal_width_cm predicted_label
0 5.2 3.4 1.4 0.2 0
1 6.8 2.8 4.8 1.4 1
2 6.3 3.3 4.7 1.6 1
3 5.0 3.6 1.4 0.2 0
4 6.0 2.2 4.0 1.0 1
5 6.3 2.5 4.9 1.5 1
6 5.6 2.7 4.2 1.3 1
7 5.0 2.3 3.3 1.0 1
8 6.3 2.5 5.0 1.9 2
9 6.1 2.8 4.7 1.2 1
10 5.1 3.5 1.4 0.2 0
11 6.4 2.9 4.3 1.3 1
12 5.1 3.8 1.6 0.2 0
13 4.9 3.1 1.5 0.2 0
14 6.2 2.2 4.5 1.5 1
15 6.3 2.3 4.4 1.3 1
16 5.1 3.7 1.5 0.4 0
17 4.4 3.0 1.3 0.2 0
18 6.0 2.7 5.1 1.6 1
19 6.1 3.0 4.6 1.4 1

Done!#

Congratulations! You’ve completed Part 5 of the MLRun getting-started tutorial. To continue, proceed to Part 6 Batch inference and drift detection.