Running Jobs with Code from Archives or File Shares

Archive URL options:

  • Git e.g. git://github.com/mlrun/something.git#master

  • Zip/Tar archives e.g. https://github.com/mlrun/mlrun/raw/run-cli/examples/archive.zip

  • File share e.g. /User/mycode (this require adding a file share to the function)

The archive will be set as the working dir for the function and the file/params to execute should be set using the command parameter (with the relative path inside the archive)

Run from zip using the CLI

!python -m mlrun run --name tst1 --watch --source https://github.com/mlrun/mlrun/raw/development/examples/archive.zip --handler handler --image mlrun/mlrun myfunc.py
> 2021-06-15 11:52:45,847 [warning] Failed resolving version info. Ignoring and using defaults
> 2021-06-15 11:52:48,460 [warning] Unable to parse server or client version. Assuming compatible: {'server_version': '0.6.4', 'client_version': 'unstable'}
> 2021-06-15 11:52:48,469 [info] starting run tst1 uid=ce4a3eab42ff43e0885d82ec27762949 DB=http://mlrun-api:8080
> 2021-06-15 11:52:48,612 [info] Job is running in the background, pod: tst1-6zpsv
> 2021-06-15 11:52:51,885 [info] extracting source from https://github.com/mlrun/mlrun/raw/development/examples/archive.zip to /mlrun/code
Run: tst1 (uid=ce4a3eab42ff43e0885d82ec27762949)
my line
got text: some text
> 2021-06-15 11:52:51,957 [info] run executed, status=completed
final state: completed
> 2021-06-15 11:52:54,715 [info] run executed, status=completed

Using code from Git

import mlrun
from mlrun.platforms import auto_mount
fn = mlrun.new_function('archive', kind='job', image='mlrun/mlrun', command='./myfunc.py',
                        source='git://github.com/mlrun/ci-demo.git#master')
run = fn.run(handler='my_func')
> 2021-06-15 11:58:59,002 [info] starting run archive-my_func uid=0b6195fbf1844880a829d61505bd9a38 DB=http://mlrun-api:8080
> 2021-06-15 11:58:59,468 [info] Job is running in the background, pod: archive-my-func-8frkp
> 2021-06-15 11:59:02,726 [info] extracting source from git://github.com/mlrun/ci-demo.git#master to /mlrun/code
Run: archive-my_func (uid=0b6195fbf1844880a829d61505bd9a38)
Params: p1=1, p2=a-string
> 2021-06-15 11:59:02,764 [info] running function
> 2021-06-15 11:59:02,797 [info] run executed, status=completed
final state: completed
project uid iter start state name labels inputs parameters results artifacts
default 0 Jun 15 11:59:02 completed archive-my_func
v3io_user=admin
kind=job
owner=admin
host=archive-my-func-8frkp
framework=sklearn
accuracy=2
loss=3
to track results use .show() or .logs() or in CLI: 
!mlrun get run 0b6195fbf1844880a829d61505bd9a38 --project default , !mlrun logs 0b6195fbf1844880a829d61505bd9a38 --project default
> 2021-06-15 11:59:05,633 [info] run executed, status=completed

Using code from file share

import mlrun
fn = mlrun.new_function('archive', kind='job', image='mlrun/mlrun', command='./code.py',
                        source='/User/sample2')
# add shared volume mount so the function will have access to the mounted code
fn.apply(auto_mount())
run = fn.run(handler='handler')
> 2021-04-27 13:08:58,586 [info] starting run archive-handler uid=a09a42808aff4551b2ba29c701f78395 DB=http://mlrun-api:8080
> 2021-04-27 13:08:58,801 [info] Job is running in the background, pod: archive-handler-74crq
extracting source from /User/sample2 to ./
cwd=/mlrun, workdir=None
Run: archive-handler (uid=a09a42808aff4551b2ba29c701f78395)
my line, bla bla
> 2021-04-27 13:09:05,095 [info] run executed, status=completed
final state: completed
project uid iter start state name labels inputs parameters results artifacts
default 0 Apr 27 13:09:04 completed archive-handler
v3io_user=admin
kind=job
owner=admin
host=archive-handler-74crq
accuracy=6
loss=12
file_result
to track results use .show() or .logs() or in CLI: 
!mlrun get run a09a42808aff4551b2ba29c701f78395 --project default , !mlrun logs a09a42808aff4551b2ba29c701f78395 --project default
> 2021-04-27 13:09:08,022 [info] run executed, status=completed
<mlrun.model.RunObject at 0x7eff90ab77d0>

Inject parameters into command line

the function command parameter is the command we execute inside the container (the archive path will be set as the working directory), users can pass arguments in the command and also inject the job/task parameters into the command at runtime (by using {} around the parameter).

for example we can define a function with command template and pass a parameter during the run:

fn = mlrun.new_function('withargs', kind='job', image='mlrun/mlrun', command="main.py --myarg {myarg}",
                        source='git://github.com/org/repo')
run = fn.run(params={'myarg': 'xx'})

Execute non Python code

By default MLRun will try and execute Python code, users can run any other code by specifying the Pass (passthrough) mode (mode="pass"), in the pass mode the command is used as is, for example:

fn = mlrun.new_function('withargs', kind='job', image='mlrun/mlrun', command="bash main.sh --myarg {myarg}",
                        source='git://github.com/org/repo', mode='pass')
run = fn.run(params={'myarg': 'xx'})