Scheduled jobs and workflows#
Oftentimes you may want to run a job
on a regular schedule. For example, fetching from a datasource every morning, compiling an analytics report every month, or detecting model drift every hour.
Schedules have a minimum interval that will be allowed between two scheduled jobs. By default, a job is not allowed to be scheduled twice in a 10-minute period Currently, schedules like */13 * * * * (every 13th minute), in which the job would trigger at the 52nd minute and then again at the start of the next hour (minute 0) (with only 8 minutes between runs) are not allowed. See mlrun.mlconf.httpdb.scheduling for service schedules configuration.
Creating a job and scheduling it#
MLRun makes it very simple to add a schedule to a given job
. To showcase this, the following job runs the code below, which resides in a file titled schedule.py
:
def hello(context):
print("You just ran a scheduled job!")
To create the job, use the set_function
syntax and specify the kind
like below:
import mlrun
project = mlrun.get_or_create_project("schedule")
job = project.set_function(
name="my-scheduled-job", # Name of the job (displayed in console and UI)
filename="schedule.py", # Python file or Jupyter notebook to run
kind="job", # Run as a job
image="mlrun/mlrun", # Use this Docker image
handler="hello", # Execute the function hello() within code.py
)
Running the job using a schedule
To add a schedule, run the job and specify the schedule
parameter using Cron syntax like so:
job.run(schedule="0 * * * *")
This runs the job every hour. An excellent resource for generating Cron schedules is Crontab.guru.
Scheduling a workflow#
Note
Tech Preview
Scheduled jobs are essentially remote-KFP workflows with a schedule.
After loading the project (load_project
), run the project with the scheduled workflow:
project.run("main", schedule='0 * * * *')
You can delete a scheduled workflow in the MLRun UI. To update a scheduled workflow, re-define the schedule in the workflow, for example:
project.run("main", schedule='0 * * * *')