Build and run workflows/pipelines#

This section shows how to write a batch pipeline so that it can be executed via an MLRun Project. With a batch pipelineyou can use the MLRun Project to execute several Functions in a DAG using the Python SDK or CLI.

This example creates a project with three MLRun functions and a single pipeline that orchestrates them. The pipeline steps are:

  • get-data — Get iris data from sklearn

  • train-model — Train model via sklearn

  • deploy-model — Deploy model to HTTP endpoint

import mlrun
project = mlrun.get_or_create_project(\"iguazio-academy\", context=\"./\")

Add functions to a project#

Add the functions to a project:

project.set_function(name='get-data', func='functions/', kind='job', image='mlrun/mlrun')
project.set_function(name='train-model', func='functions/', kind='job', image='mlrun/mlrun'),
project.set_function(name='deploy-model', func='hub://v2_model_server')

Write a pipeline#

Next, define the pipeline that orchestrates the three comoponents. This pipeline is simple, however you can create very complex pipelines with branches, conditions, and more.

%%writefile pipelines/
from kfp import dsl
import mlrun

    description=\"Example of batch pipeline for Iguazio Academy\"
def pipeline(label_column: str, test_size=0.2):
    # Ingest the data set
    ingest = mlrun.run_function(
        params={'label_column': label_column},
    # Train a model   
    train = mlrun.run_function(
        inputs={\"dataset\": ingest.outputs[\"iris_dataset\"]},
            \"label_column\": label_column,
            \"test_size\" : test_size
    # Deploy the model as a serverless function
    deploy = mlrun.deploy_function(
        models=[{\"key\": \"model\", \"model_path\": train.outputs[\"model\"]}]

Add a pipeline to a project#

Add the pipeline to your project:

project.set_workflow(name='train', workflow_path=\"pipelines/")