Local vs. remote workflows#

To run multiple functions, one after the other, or in parallel, such as: jobs, serving and nuclio, you can use a kfp pipeline. There are three types of pipeline engines:

  • Remote on KFP

  • KFP

  • Local — Used to simulate a pipeline run without using KFP - it triggers the jobs directly, mainly for testing. Use (set local=True in function.run() to run the functions locally or project.run(local=True) to apply for all functions).

../_images/pipelines-flow.png

All three types are configured by the engine flag, when running the workflow, see mlrun.projects.MlrunProject.run.

Remote-KFP#

Remote workflows are used for scheduled workflows.
Remote workflows are run on the workflow runner pod that runs and load your workflow on a pod named workflow-runner-<workflow-name>. This pod is responsible for loading the files from the remote source (git, tar.gz or zip) and running the KFP by using the files from the remote source.
In some cases you might not want to load the files from the remote source, but instead use the files within the running image. See details in build image. In this case, you need to build an image that contains the workflow file and then change the workflow runner source to point to the project local files in the running image. See the example below.

The remote workflow supports sending notifications when runs are complete.

You can modify the pod image, source, and the pod node selector with:

  • project.set_workflow(name="main",workflow_path="workflow.py,image="<runner-image>") — changing the runner image

  • project.run("main",engine="remote",workflow_runner_node_selector={"key":"value"}) — changing the runner node selector

  • project.run(source=<source-URL>) — changing the runner source

Note

This pod must be based on the image mlrun/mlrun-kfp. See Images and their usage in MLRun.

See an example of a remote GitHub project in mlrun/project-demo.

Note

From MLRun v1.7.1: when running a remote/scheduled workflow, the remote workflow pulls/extracts the remote source content to the running pod but loads the project configuration from the MLRun DB and not from the project.yaml file in the remote source.

The remote files are primarily retrieved for:

  • The project_setup that may affect the project configuration (if it exists).

  • Syncing function files. This behavior may be unexpected for users who rely on project.yaml in the remote source (for the project configuration). Be sure to update MLRun DB with the latest project configuration to ensure consistent configuration management (use project.save()).
    Project configuration in this context could be, for example, project.node_selector or project.artifact_path, and not function configurations like: function resources or function node selector.

import mlrun
project_name = "remote-workflow-example"
source_url = "git://github.com/mlrun/project-demo.git"
source_code_target_dir = "./project" # Optional, relative to "/home/mlrun_code". A different absolute path can be specified.

# Create a new project
project = mlrun.load_project(context=f"./{project_name}", url=source_url, name=project_name)

# Set the project source and workflow
project.set_source(source_url)
project.set_workflow(name="main", workflow_path="kflow.py")

# Build the image, load the source to the target dir and save the project
project.build_image(target_dir=source_code_target_dir)
project.save()

# Run the workflow, load the project from the target dir on the image
project.run("main", source="./", engine="remote", dirty=True)

See also

KFP#

The KFP workflow spec file is created in MLRun, and is compiled and run in the client side, using the files from your local file system. For example:

project.run("main", engine='kfp')