Batch runs and workflows

A workflow is a definition of execution of functions. It defines the order of execution of multiple dependent steps in a directed acyclic graph (DAG). A workflow can reference the project’s params, secrets, artifacts, etc. It can also use a function execution output as a function execution input (which, of course, defines the order of execution).

MLRun supports running workflows on a local or kubeflow pipeline engine. The local engine runs the workflow as a local process, which is simpler for debugging and running simple/sequential tasks. The kubeflow ("kfp") engine runs either locally or as a remote runner over the cluster and supports more advanced operations (conditions, branches, etc.). You can select the engine at runtime. Kubeflow-specific directives like conditions and branches are not supported by the local engine. See Local vs. remote workflows.

Workflows are saved/registered in the project using the set_workflow().
Workflows are executed using the run() method or using the CLI command mlrun project.

See the examples listed below and the Machine learning tutorials for more details.

In this section

• Local vs. remote workflows
• MLRun execution context
• Decorators and auto-logging
• Running a task (job)
• Running a multi-stage workflow
• Running a conditional workflow
• Running multiple functions with ExitHandler
• Running multiple functions in parallel
• Configuring runs and functions
• Scheduled jobs and workflows