{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "(code-archive)=\n", "# Using code from archives or file shares" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**In this section**\n", "- [Archive URL options](#archive-url-options)\n", "- [Run from zip using the CLI](#run-from-zip-using-the-cli)\n", "- [Using code from Git](#using-code-from-Git)\n", "- [Using code from file share](#using-code-from-file-share)\n", "- [Inject parameters into command line](#inject-parameters-into-command-line)\n", "- [Execute non-Python code](#execute-non-Python-code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Archive URL options\n", "* Git, for example: git://github.com/mlrun/something.git#master\n", "* Zip/Tar archives, for example: https://github.com/mlrun/mlrun/raw/run-cli/examples/archive.zip\n", "* File share, for example: `/User/mycode` (requires adding a file share to the function) \n", "\n", "The archive is set as the working dir for the function and the file/params to execute should be set using the `command` parameter (with the relative path inside the archive)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Run from zip using the CLI" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "> 2021-06-15 11:52:45,847 [warning] Failed resolving version info. Ignoring and using defaults\n", "> 2021-06-15 11:52:48,460 [warning] Unable to parse server or client version. Assuming compatible: {'server_version': '0.6.4', 'client_version': 'unstable'}\n", "> 2021-06-15 11:52:48,469 [info] starting run tst1 uid=ce4a3eab42ff43e0885d82ec27762949 DB=http://mlrun-api:8080\n", "> 2021-06-15 11:52:48,612 [info] Job is running in the background, pod: tst1-6zpsv\n", "> 2021-06-15 11:52:51,885 [info] extracting source from https://github.com/mlrun/mlrun/raw/development/examples/archive.zip to /mlrun/code\n", "Run: tst1 (uid=ce4a3eab42ff43e0885d82ec27762949)\n", "my line\n", "got text: some text\n", "> 2021-06-15 11:52:51,957 [info] run executed, status=completed\n", "final state: completed\n", "> 2021-06-15 11:52:54,715 [info] run executed, status=completed\n" ] } ], "source": [ "!python -m mlrun run --name tst1 --watch --source https://github.com/mlrun/mlrun/raw/development/examples/archive.zip --handler handler --image mlrun/mlrun myfunc.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using code from Git" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "> 2021-06-15 11:58:59,002 [info] starting run archive-my_func uid=0b6195fbf1844880a829d61505bd9a38 DB=http://mlrun-api:8080\n", "> 2021-06-15 11:58:59,468 [info] Job is running in the background, pod: archive-my-func-8frkp\n", "> 2021-06-15 11:59:02,726 [info] extracting source from git://github.com/mlrun/ci-demo.git#master to /mlrun/code\n", "Run: archive-my_func (uid=0b6195fbf1844880a829d61505bd9a38)\n", "Params: p1=1, p2=a-string\n", "> 2021-06-15 11:59:02,764 [info] running function\n", "> 2021-06-15 11:59:02,797 [info] run executed, status=completed\n", "final state: completed\n" ] }, { "data": { "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
projectuiditerstartstatenamelabelsinputsparametersresultsartifacts
default0Jun 15 11:59:02completedarchive-my_func
v3io_user=admin
kind=job
owner=admin
host=archive-my-func-8frkp
framework=sklearn
accuracy=2
loss=3
\n", "
\n", "
\n", "
\n", " Title\n", " ×\n", "
\n", " \n", "
\n", "
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "to track results use .show() or .logs() or in CLI: \n", "!mlrun get run 0b6195fbf1844880a829d61505bd9a38 --project default , !mlrun logs 0b6195fbf1844880a829d61505bd9a38 --project default\n", "> 2021-06-15 11:59:05,633 [info] run executed, status=completed\n" ] } ], "source": [ "import mlrun\n", "from mlrun.platforms import auto_mount\n", "fn = mlrun.new_function('archive', kind='job', image='mlrun/mlrun', command='./myfunc.py',\n", " source='git://github.com/mlrun/ci-demo.git#master')\n", "run = fn.run(handler='my_func')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using code from file share" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "> 2021-04-27 13:08:58,586 [info] starting run archive-handler uid=a09a42808aff4551b2ba29c701f78395 DB=http://mlrun-api:8080\n", "> 2021-04-27 13:08:58,801 [info] Job is running in the background, pod: archive-handler-74crq\n", "extracting source from /User/sample2 to ./\n", "cwd=/mlrun, workdir=None\n", "Run: archive-handler (uid=a09a42808aff4551b2ba29c701f78395)\n", "my line, bla bla\n", "> 2021-04-27 13:09:05,095 [info] run executed, status=completed\n", "final state: completed\n" ] }, { "data": { "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
projectuiditerstartstatenamelabelsinputsparametersresultsartifacts
default0Apr 27 13:09:04completedarchive-handler
v3io_user=admin
kind=job
owner=admin
host=archive-handler-74crq
accuracy=6
loss=12
file_result
\n", "
\n", "
\n", "
\n", " Title\n", " ×\n", "
\n", " \n", "
\n", "
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "to track results use .show() or .logs() or in CLI: \n", "!mlrun get run a09a42808aff4551b2ba29c701f78395 --project default , !mlrun logs a09a42808aff4551b2ba29c701f78395 --project default\n", "> 2021-04-27 13:09:08,022 [info] run executed, status=completed\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import mlrun\n", "fn = mlrun.new_function('archive', kind='job', image='mlrun/mlrun', command='./code.py',\n", " source='/User/sample2')\n", "# add shared volume mount so the function will have access to the mounted code\n", "fn.apply(auto_mount())\n", "run = fn.run(handler='handler')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Inject parameters into command line\n", "\n", "The function `command` parameter is the command that executes inside the container (the archive path is set as the working directory). You can pass arguments in the command and also inject the job/task parameters into the command at runtime (by using `{}` around the parameter).\n", "\n", "For example, define a function with the command template and pass a parameter during the run:\n", "\n", "```python\n", "fn = mlrun.new_function('withargs', kind='job', image='mlrun/mlrun', command=\"main.py --myarg {myarg}\",\n", " source='git://github.com/org/repo')\n", "run = fn.run(params={'myarg': 'xx'})\n", "```\n", "\n", "\n", "## Execute non-Python code\n", "\n", "By default MLRun tries to execute Python code. You can run any other code by specifying the **Pass** (passthrough) mode (`mode=\"pass\"`). In the `pass` mode the `command` is used as is, for example:\n", "\n", "```python\n", "fn = mlrun.new_function('withargs', kind='job', image='mlrun/mlrun', command=\"bash main.sh --myarg {myarg}\",\n", " source='git://github.com/org/repo', mode='pass')\n", "run = fn.run(params={'myarg': 'xx'})\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" } }, "nbformat": 4, "nbformat_minor": 4 }