Using code from archives or file shares¶
In this section
Archive URL options¶
Git e.g. git://github.com/mlrun/something.git#master
Zip/Tar archives e.g. https://github.com/mlrun/mlrun/raw/run-cli/examples/archive.zip
File share e.g.
/User/mycode
(requires adding a file share to the function)
The archive is set as the working dir for the function and the file/params to execute should be set using the command
parameter (with the relative path inside the archive).
Run from zip using the CLI¶
!python -m mlrun run --name tst1 --watch --source https://github.com/mlrun/mlrun/raw/development/examples/archive.zip --handler handler --image mlrun/mlrun myfunc.py
> 2021-06-15 11:52:45,847 [warning] Failed resolving version info. Ignoring and using defaults
> 2021-06-15 11:52:48,460 [warning] Unable to parse server or client version. Assuming compatible: {'server_version': '0.6.4', 'client_version': 'unstable'}
> 2021-06-15 11:52:48,469 [info] starting run tst1 uid=ce4a3eab42ff43e0885d82ec27762949 DB=http://mlrun-api:8080
> 2021-06-15 11:52:48,612 [info] Job is running in the background, pod: tst1-6zpsv
> 2021-06-15 11:52:51,885 [info] extracting source from https://github.com/mlrun/mlrun/raw/development/examples/archive.zip to /mlrun/code
Run: tst1 (uid=ce4a3eab42ff43e0885d82ec27762949)
my line
got text: some text
> 2021-06-15 11:52:51,957 [info] run executed, status=completed
final state: completed
> 2021-06-15 11:52:54,715 [info] run executed, status=completed
Using code from Git¶
import mlrun
from mlrun.platforms import auto_mount
fn = mlrun.new_function('archive', kind='job', image='mlrun/mlrun', command='./myfunc.py',
source='git://github.com/mlrun/ci-demo.git#master')
run = fn.run(handler='my_func')
> 2021-06-15 11:58:59,002 [info] starting run archive-my_func uid=0b6195fbf1844880a829d61505bd9a38 DB=http://mlrun-api:8080
> 2021-06-15 11:58:59,468 [info] Job is running in the background, pod: archive-my-func-8frkp
> 2021-06-15 11:59:02,726 [info] extracting source from git://github.com/mlrun/ci-demo.git#master to /mlrun/code
Run: archive-my_func (uid=0b6195fbf1844880a829d61505bd9a38)
Params: p1=1, p2=a-string
> 2021-06-15 11:59:02,764 [info] running function
> 2021-06-15 11:59:02,797 [info] run executed, status=completed
final state: completed
project | uid | iter | start | state | name | labels | inputs | parameters | results | artifacts |
---|---|---|---|---|---|---|---|---|---|---|
default | 0 | Jun 15 11:59:02 | completed | archive-my_func | v3io_user=admin kind=job owner=admin host=archive-my-func-8frkp framework=sklearn |
accuracy=2 loss=3 |
to track results use .show() or .logs() or in CLI:
!mlrun get run 0b6195fbf1844880a829d61505bd9a38 --project default , !mlrun logs 0b6195fbf1844880a829d61505bd9a38 --project default
> 2021-06-15 11:59:05,633 [info] run executed, status=completed
Using code from file share¶
import mlrun
fn = mlrun.new_function('archive', kind='job', image='mlrun/mlrun', command='./code.py',
source='/User/sample2')
# add shared volume mount so the function will have access to the mounted code
fn.apply(auto_mount())
run = fn.run(handler='handler')
> 2021-04-27 13:08:58,586 [info] starting run archive-handler uid=a09a42808aff4551b2ba29c701f78395 DB=http://mlrun-api:8080
> 2021-04-27 13:08:58,801 [info] Job is running in the background, pod: archive-handler-74crq
extracting source from /User/sample2 to ./
cwd=/mlrun, workdir=None
Run: archive-handler (uid=a09a42808aff4551b2ba29c701f78395)
my line, bla bla
> 2021-04-27 13:09:05,095 [info] run executed, status=completed
final state: completed
project | uid | iter | start | state | name | labels | inputs | parameters | results | artifacts |
---|---|---|---|---|---|---|---|---|---|---|
default | 0 | Apr 27 13:09:04 | completed | archive-handler | v3io_user=admin kind=job owner=admin host=archive-handler-74crq |
accuracy=6 loss=12 |
file_result |
to track results use .show() or .logs() or in CLI:
!mlrun get run a09a42808aff4551b2ba29c701f78395 --project default , !mlrun logs a09a42808aff4551b2ba29c701f78395 --project default
> 2021-04-27 13:09:08,022 [info] run executed, status=completed
<mlrun.model.RunObject at 0x7eff90ab77d0>
Inject parameters into command line¶
The function command
parameter is the command that executes inside the container (the archive path is set as the working directory). You can pass arguments in the command and also inject the job/task parameters into the command at runtime (by using {}
around the parameter).
For example, define a function with the command template and pass a parameter during the run:
fn = mlrun.new_function('withargs', kind='job', image='mlrun/mlrun', command="main.py --myarg {myarg}",
source='git://github.com/org/repo')
run = fn.run(params={'myarg': 'xx'})
Execute non-Python code¶
By default MLRun tries to execute Python code. You can run any other code by specifying the Pass (passthrough) mode (mode="pass"
). In the pass
mode the command
is used as is, for example:
fn = mlrun.new_function('withargs', kind='job', image='mlrun/mlrun', command="bash main.sh --myarg {myarg}",
source='git://github.com/org/repo', mode='pass')
run = fn.run(params={'myarg': 'xx'})