Train, compare, and register models#

This notebook provides a quick overview of training ML models using MLRun MLOps orchestration framework.

Make sure you reviewed the basics in MLRun Quick Start Tutorial.

Tutorial steps:

MLRun installation and configuration#

Before running this notebook make sure mlrun and sklearn packages are installed (pip install mlrun scikit-learn~=1.0) and that you have configured the access to the MLRun service.

# install MLRun if not installed, run this only once (restart the notebook after the install !!!)
%pip install mlrun scikit-learn~=1.0.0

Requirement already satisfied: mlrun in /opt/conda/lib/python3.8/site-packages (1.1.0rc24)
Requirement already satisfied: scikit-learn~=1.0.0 in /opt/conda/lib/python3.8/site-packages (1.0.2)
Requirement already satisfied: pyarrow<7,>=1 in /opt/conda/lib/python3.8/site-packages (from mlrun) (6.0.1)
Requirement already satisfied: kfp~=1.8.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.8.13)
Requirement already satisfied: python-dotenv~=0.17.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.17.1)
Requirement already satisfied: nest-asyncio~=1.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.5.1)
Requirement already satisfied: v3io-frames~=0.10.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.10.2)
Requirement already satisfied: inflection~=0.5.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.5.1)
Requirement already satisfied: v3io~=0.5.13 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.5.15)
Requirement already satisfied: GitPython~=3.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.1.27)
Requirement already satisfied: pydantic~=1.5 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.9.2)
Requirement already satisfied: sqlalchemy~=1.3 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.4.13)
Requirement already satisfied: dask~=2021.11.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2021.11.2)
Requirement already satisfied: requests~=2.22 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2.25.1)
Requirement already satisfied: click~=8.0.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (8.0.4)
Requirement already satisfied: humanfriendly~=8.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (8.2)
Requirement already satisfied: storey~=1.1.7 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.1.7)
Requirement already satisfied: fastapi~=0.78.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.78.0)
Requirement already satisfied: tabulate~=0.8.6 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.8.10)
Requirement already satisfied: alembic<1.6.0,~=1.4 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.5.8)
Requirement already satisfied: cryptography<3.4,~=3.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.3.2)
Requirement already satisfied: ipython~=7.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (7.23.1)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.26.4)
Requirement already satisfied: chardet<4.0,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.0.4)
Requirement already satisfied: aiohttp~=3.8 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.8.1)
Requirement already satisfied: pyyaml~=5.1 in /opt/conda/lib/python3.8/site-packages (from mlrun) (5.4.1)
Requirement already satisfied: typing-extensions<5,>=3.10.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (4.3.0)
Requirement already satisfied: mergedeep~=1.3 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.3.4)
Requirement already satisfied: fsspec~=2021.8.1 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2021.8.1)
Requirement already satisfied: deepdiff~=5.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (5.8.1)
Requirement already satisfied: numpy<1.23.0,>=1.16.5 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.20.2)
Requirement already satisfied: pymysql~=1.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.0.2)
Requirement already satisfied: nuclio-jupyter~=0.9.1 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.9.2)
Requirement already satisfied: orjson~=3.3 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.7.12)
Requirement already satisfied: distributed~=2021.11.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2021.11.2)
Requirement already satisfied: pandas~=1.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.2.4)
Requirement already satisfied: semver~=2.13 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2.13.0)
Requirement already satisfied: v3iofs~=0.1.7 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.1.11)
Requirement already satisfied: kubernetes~=12.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (12.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from scikit-learn~=1.0.0) (2.1.0)
Requirement already satisfied: scipy>=1.1.0 in /opt/conda/lib/python3.8/site-packages (from scikit-learn~=1.0.0) (1.6.3)
Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.8/site-packages (from scikit-learn~=1.0.0) (1.0.1)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (1.2.0)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (2.1.0)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (4.0.2)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (20.3.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (6.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (1.8.1)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (1.3.1)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.8/site-packages (from alembic<1.6.0,~=1.4->mlrun) (1.0.4)
Requirement already satisfied: Mako in /opt/conda/lib/python3.8/site-packages (from alembic<1.6.0,~=1.4->mlrun) (1.1.4)
Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.8/site-packages (from alembic<1.6.0,~=1.4->mlrun) (2.8.1)
Requirement already satisfied: cffi>=1.12 in /opt/conda/lib/python3.8/site-packages (from cryptography<3.4,~=3.0->mlrun) (1.14.5)
Requirement already satisfied: six>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from cryptography<3.4,~=3.0->mlrun) (1.16.0)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.8/site-packages (from dask~=2021.11.2->mlrun) (20.9)
Requirement already satisfied: toolz>=0.8.2 in /opt/conda/lib/python3.8/site-packages (from dask~=2021.11.2->mlrun) (0.11.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /opt/conda/lib/python3.8/site-packages (from dask~=2021.11.2->mlrun) (2.1.0)
Requirement already satisfied: partd>=0.3.10 in /opt/conda/lib/python3.8/site-packages (from dask~=2021.11.2->mlrun) (1.2.0)
Requirement already satisfied: ordered-set<4.2.0,>=4.1.0 in /opt/conda/lib/python3.8/site-packages (from deepdiff~=5.0->mlrun) (4.1.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (49.6.0.post20210108)
Requirement already satisfied: msgpack>=0.6.0 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (1.0.2)
Requirement already satisfied: tornado>=6.0.3 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (6.1)
Requirement already satisfied: zict>=0.1.3 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (2.0.0)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (3.0.3)
Requirement already satisfied: tblib>=1.6.0 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (2.3.0)
Requirement already satisfied: starlette==0.19.1 in /opt/conda/lib/python3.8/site-packages (from fastapi~=0.78.0->mlrun) (0.19.1)
Requirement already satisfied: anyio<5,>=3.4.0 in /opt/conda/lib/python3.8/site-packages (from starlette==0.19.1->fastapi~=0.78.0->mlrun) (3.6.1)
Requirement already satisfied: gitdb<5,>=4.0.1 in /opt/conda/lib/python3.8/site-packages (from GitPython~=3.0->mlrun) (4.0.9)
Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (0.18.0)
Requirement already satisfied: pickleshare in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (3.0.18)
Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (0.1.2)
Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (4.8.0)
Requirement already satisfied: decorator in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (5.0.7)
Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (5.3.0)
Requirement already satisfied: backcall in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (0.2.0)
Requirement already satisfied: pygments in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (2.9.0)
Requirement already satisfied: typer<1.0,>=0.3.2 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.6.1)
Requirement already satisfied: google-api-python-client<2,>=1.7.8 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.12.11)
Requirement already satisfied: fire<1,>=0.3.1 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.4.0)
Requirement already satisfied: jsonschema<4,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (3.2.0)
Requirement already satisfied: kfp-pipeline-spec<0.2.0,>=0.1.14 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.1.16)
Requirement already satisfied: docstring-parser<1,>=0.7.3 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.14.1)
Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.9.1)
Requirement already satisfied: strip-hints<1,>=0.1.8 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.1.10)
Requirement already satisfied: uritemplate<4,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (3.0.1)
Requirement already satisfied: absl-py<2,>=0.9 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.2.0)
Requirement already satisfied: google-cloud-storage<2,>=1.20.0 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.44.0)
Requirement already satisfied: google-auth<2,>=1.6.1 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.35.0)
Requirement already satisfied: google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (2.8.2)
Requirement already satisfied: Deprecated<2,>=1.2.7 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.2.13)
Requirement already satisfied: kfp-server-api<2.0.0,>=1.1.2 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.8.4)
Requirement already satisfied: protobuf<4,>=3.13.0 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (3.19.4)
Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.8/site-packages (from kubernetes~=12.0->mlrun) (1.3.1)
Requirement already satisfied: certifi>=14.05.14 in /opt/conda/lib/python3.8/site-packages (from kubernetes~=12.0->mlrun) (2020.12.5)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.8/site-packages (from kubernetes~=12.0->mlrun) (1.3.3)
Requirement already satisfied: notebook>=5.2.0 in /opt/conda/lib/python3.8/site-packages (from nuclio-jupyter~=0.9.1->mlrun) (6.4.12)
Requirement already satisfied: boto3>=1.9 in /opt/conda/lib/python3.8/site-packages (from nuclio-jupyter~=0.9.1->mlrun) (1.17.106)
Requirement already satisfied: nbconvert>=5.4 in /opt/conda/lib/python3.8/site-packages (from nuclio-jupyter~=0.9.1->mlrun) (6.5.3)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas~=1.2->mlrun) (2021.1)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests~=2.22->mlrun) (2.10)
Requirement already satisfied: greenlet!=0.4.17 in /opt/conda/lib/python3.8/site-packages (from sqlalchemy~=1.3->mlrun) (1.0.0)
Requirement already satisfied: grpcio-tools<1.42,>1.34.0 in /opt/conda/lib/python3.8/site-packages (from storey~=1.1.7->mlrun) (1.41.1)
Requirement already satisfied: grpcio<1.42,>1.34.0 in /opt/conda/lib/python3.8/site-packages (from storey~=1.1.7->mlrun) (1.41.1)
Requirement already satisfied: future>=0.18.2 in /opt/conda/lib/python3.8/site-packages (from v3io~=0.5.13->mlrun) (0.18.2)
Requirement already satisfied: ujson>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from v3io~=0.5.13->mlrun) (5.4.0)
Requirement already satisfied: googleapis-common-protos>=1.5.3 in /opt/conda/lib/python3.8/site-packages (from v3io-frames~=0.10.2->mlrun) (1.56.4)
Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /opt/conda/lib/python3.8/site-packages (from boto3>=1.9->nuclio-jupyter~=0.9.1->mlrun) (0.4.2)
Requirement already satisfied: botocore<1.21.0,>=1.20.106 in /opt/conda/lib/python3.8/site-packages (from boto3>=1.9->nuclio-jupyter~=0.9.1->mlrun) (1.20.106)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.8/site-packages (from boto3>=1.9->nuclio-jupyter~=0.9.1->mlrun) (0.10.0)
Requirement already satisfied: pycparser in /opt/conda/lib/python3.8/site-packages (from cffi>=1.12->cryptography<3.4,~=3.0->mlrun) (2.20)
Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.8/site-packages (from Deprecated<2,>=1.2.7->kfp~=1.8.0->mlrun) (1.14.1)
Requirement already satisfied: termcolor in /opt/conda/lib/python3.8/site-packages (from fire<1,>=0.3.1->kfp~=1.8.0->mlrun) (1.1.0)
Requirement already satisfied: smmap<6,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from gitdb<5,>=4.0.1->GitPython~=3.0->mlrun) (5.0.0)
Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp~=1.8.0->mlrun) (0.1.0)
Requirement already satisfied: httplib2<1dev,>=0.15.0 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp~=1.8.0->mlrun) (0.20.4)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp~=1.8.0->mlrun) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp~=1.8.0->mlrun) (4.2.4)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp~=1.8.0->mlrun) (4.9)
Requirement already satisfied: google-cloud-core<3.0dev,>=1.6.0 in /opt/conda/lib/python3.8/site-packages (from google-cloud-storage<2,>=1.20.0->kfp~=1.8.0->mlrun) (2.3.2)
Requirement already satisfied: google-resumable-media<3.0dev,>=1.3.0 in /opt/conda/lib/python3.8/site-packages (from google-cloud-storage<2,>=1.20.0->kfp~=1.8.0->mlrun) (2.3.3)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from jedi>=0.16->ipython~=7.0->mlrun) (0.8.2)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.8/site-packages (from jinja2->distributed~=2021.11.2->mlrun) (2.1.1)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema<4,>=3.0.1->kfp~=1.8.0->mlrun) (0.17.3)
Requirement already satisfied: defusedxml in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.7.1)
Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (1.4.2)
Requirement already satisfied: tinycss2 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (1.1.1)
Requirement already satisfied: jupyter-core>=4.7 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (4.7.1)
Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (4.9.3)
Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.8.4)
Requirement already satisfied: entrypoints>=0.2.2 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.3)
Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.1.2)
Requirement already satisfied: nbclient>=0.5.0 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.5.3)
Requirement already satisfied: bleach in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (3.3.0)
Requirement already satisfied: nbformat>=5.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (5.4.0)
Requirement already satisfied: lxml in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (4.9.1)
Requirement already satisfied: pyzmq>=17 in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (22.0.3)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (0.10.1)
Requirement already satisfied: ipykernel in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (5.5.4)
Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (0.9.4)
Requirement already satisfied: Send2Trash>=1.8.0 in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (1.8.0)
Requirement already satisfied: jupyter-client>=5.3.4 in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (6.1.12)
Requirement already satisfied: ipython-genutils in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (0.2.0)
Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (20.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.8/site-packages (from packaging>=20.0->dask~=2021.11.2->mlrun) (2.4.7)
Requirement already satisfied: locket in /opt/conda/lib/python3.8/site-packages (from partd>=0.3.10->dask~=2021.11.2->mlrun) (0.2.0)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.8/site-packages (from pexpect>4.3->ipython~=7.0->mlrun) (0.7.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython~=7.0->mlrun) (0.2.5)
Requirement already satisfied: wheel in /opt/conda/lib/python3.8/site-packages (from strip-hints<1,>=0.1.8->kfp~=1.8.0->mlrun) (0.36.2)
Requirement already satisfied: heapdict in /opt/conda/lib/python3.8/site-packages (from zict>=0.1.3->distributed~=2021.11.2->mlrun) (1.0.1)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from requests-oauthlib->kubernetes~=12.0->mlrun) (3.0.1)
Requirement already satisfied: sniffio>=1.1 in /opt/conda/lib/python3.8/site-packages (from anyio<5,>=3.4.0->starlette==0.19.1->fastapi~=0.78.0->mlrun) (1.2.0)
Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.8/site-packages (from google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp~=1.8.0->mlrun) (1.3.0)
Requirement already satisfied: async-generator in /opt/conda/lib/python3.8/site-packages (from nbclient>=0.5.0->nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (1.10)
Requirement already satisfied: fastjsonschema in /opt/conda/lib/python3.8/site-packages (from nbformat>=5.1->nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (2.16.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.1->kfp~=1.8.0->mlrun) (0.4.8)
Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.8/site-packages (from beautifulsoup4->nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (2.0.1)
Requirement already satisfied: webencodings in /opt/conda/lib/python3.8/site-packages (from bleach->nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.5.1)
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/opt/conda/bin/python -m pip install --upgrade pip' command.
Note: you may need to restart the kernel to use updated packages.

Define MLRun project and a training functions#

You should create, load, or use (get) an MLRun Project that holds all your functions and assets.

Get or create a new project:

The get_or_create_project() method tries to load the project from MLRun DB. If the project does not exist it creates a new one.

import mlrun
project = mlrun.get_or_create_project("tutorial", context="src/", user_project=True)

> 2022-08-24 08:50:23,251 [info] loaded project tutorial from None or context and saved in MLRun DB

Add (auto) MLOps to your training function:

Training functions generate models and various model statistics. You’ll want to store the models along with all the relevant data, metadata, and measurements. MLRun can apply all the MLOps functionality automatically (“Auto-MLOps”) by simply using the framework specific apply_mlrun() method.

In the training function below note the single custom line you need to add to your code:

apply_mlrun(model=model, model_name="my_model", x_test=x_test, y_test=y_test)

apply_mlrun() manages the training process and automatically logs all the framework-specific model object, details, data, metadata, and metrics. It accepts the model object and various optional parameters. When specifying the x_test and y_test data it generates various plots and calculations to evaluate the model. Metadata and parameters are automatically recorded (from MLRun context object) and don’t need to be specified.

Function code:

Run the following cell to generate the trainer.py file (or copy it manually):

Create a serverless function object from the code above, and register it in the project:

trainer = project.set_function("trainer.py", name="trainer", kind="job", image="mlrun/mlrun", handler="train")

Run the training function and log the artifacts and model#

Create a dataset for training:

import pandas as pd
from sklearn.datasets import load_breast_cancer
breast_cancer = load_breast_cancer()
breast_cancer_dataset = pd.DataFrame(data=breast_cancer.data, columns=breast_cancer.feature_names)
breast_cancer_labels = pd.DataFrame(data=breast_cancer.target, columns=["label"])
breast_cancer_dataset = pd.concat([breast_cancer_dataset, breast_cancer_labels], axis=1)

breast_cancer_dataset.to_csv("cancer-dataset.csv", index=False)

Run the function (locally) using the generated dataset:

trainer_run = project.run_function(
    "trainer", 
    inputs={"dataset": "cancer-dataset.csv"}, 
    params = {"n_estimators": 100, "learning_rate": 1e-1, "max_depth": 3},
    local=True
)

> 2022-08-24 08:50:24,614 [warning] it is recommended to use k8s secret (specify secret_name), specifying the aws_access_key/aws_secret_key directly is unsafe
> 2022-08-24 08:50:24,636 [info] starting run trainer-train uid=05c6e41b668f460fa67d7abf9dff9542 DB=http://mlrun-api:8080
project uid iter start state name labels inputs parameters results artifacts
tutorial-jovyan 0 Aug 24 08:50:24 completed trainer-train
kind=
owner=jovyan
host=mlrun-jupyter-6b78bf965-knkrt
dataset
n_estimators=100
learning_rate=0.1
max_depth=3
accuracy=0.956140350877193
f1_score=0.965034965034965
precision_score=0.9583333333333334
recall_score=0.971830985915493
feature-importance
test_set
confusion-matrix
roc-curves
calibration-curve
model
> to track results use the .show() or .logs() methods or click here to open in UI
> 2022-08-24 08:50:32,906 [info] run executed, status=completed

View the auto generated results and artifacts:

trainer_run.outputs

{'accuracy': 0.956140350877193,
 'f1_score': 0.965034965034965,
 'precision_score': 0.9583333333333334,
 'recall_score': 0.971830985915493,
 'feature-importance': 's3://mlrun/trainer-train/0/feature-importance.html',
 'test_set': 'store://artifacts/tutorial-jovyan/trainer-train_test_set:05c6e41b668f460fa67d7abf9dff9542',
 'confusion-matrix': 's3://mlrun/trainer-train/0/confusion-matrix.html',
 'roc-curves': 's3://mlrun/trainer-train/0/roc-curves.html',
 'calibration-curve': 's3://mlrun/trainer-train/0/calibration-curve.html',
 'model': 'store://artifacts/tutorial-jovyan/cancer_classifier:05c6e41b668f460fa67d7abf9dff9542'}

trainer_run.artifact('feature-importance').show()

Export model files + metadata into a zip: (require MLRun 1.1.0 and above)

You can export() the model package (files + metadata) into a zip, and load it on a remote system/cluster (by simply running model = project.import_artifact(key, path)).

trainer_run.artifact('model').meta.export("model.zip")

Hyper-parameter tuning and model/experiment comparison#

Run a GridSearch with a couple of parameters, and select the best run with respect to the max accuracy.
(Read more about MLRun Hyper-Param and Iterative jobs.)

For basic usage you can run the hyperparameters tuning job by using the arguments:

  • hyperparams for the hyperparameters options and values of choice.

  • selector for specifying how to select the best model.

Running a remote function:

In order to run the hyper-param task over the cluster you need the input data to be available for the job, using object storage or the mlrun versioned artifact store.

The following line logs (and uploads) the dataframe as a project artifact:

dataset_artifact = project.log_dataset("cancer-dataset", df=breast_cancer_dataset, index=False)

Run the function over the remote Kubernetes cluster (local is not set):

hp_tuning_run = project.run_function(
    "trainer", 
    inputs={"dataset": dataset_artifact.uri}, 
    hyperparams={
        "n_estimators": [10, 100, 1000], 
        "learning_rate": [1e-1, 1e-3], 
        "max_depth": [2, 8]
    }, 
    selector="max.accuracy", 
)

> 2022-08-24 08:50:34,657 [info] starting run trainer-train uid=22118ca5268e45babf7668ce11064837 DB=http://mlrun-api:8080
> 2022-08-24 08:50:35,094 [info] Job is running in the background, pod: trainer-train-8c6f4
> 2022-08-24 08:51:27,920 [info] best iteration=3, used criteria max.accuracy
> 2022-08-24 08:51:28,380 [info] run executed, status=completed
final state: completed
project uid iter start state name labels inputs parameters results artifacts
tutorial-jovyan 0 Aug 24 08:50:46 completed trainer-train
kind=job
owner=jovyan
mlrun/client_version=1.1.0-rc24
dataset
best_iteration=3
accuracy=0.9649122807017544
f1_score=0.9722222222222222
precision_score=0.958904109589041
recall_score=0.9859154929577465
feature-importance
test_set
confusion-matrix
roc-curves
calibration-curve
model
iteration_results
parallel_coordinates
> to track results use the .show() or .logs() methods or click here to open in UI
> 2022-08-24 08:51:35,352 [info] run executed, status=completed

View Hyper-param results and the selected run in the MLRun UI:

hprun

Interactive Parallel Coordinates Plot:

pcp


List the generated models and compare the different runs:

hp_tuning_run.outputs

{'best_iteration': 3,
 'accuracy': 0.9649122807017544,
 'f1_score': 0.9722222222222222,
 'precision_score': 0.958904109589041,
 'recall_score': 0.9859154929577465,
 'feature-importance': 's3://mlrun/trainer-train/3/feature-importance.html',
 'test_set': 'store://artifacts/tutorial-jovyan/trainer-train_test_set:22118ca5268e45babf7668ce11064837',
 'confusion-matrix': 's3://mlrun/trainer-train/3/confusion-matrix.html',
 'roc-curves': 's3://mlrun/trainer-train/3/roc-curves.html',
 'calibration-curve': 's3://mlrun/trainer-train/3/calibration-curve.html',
 'model': 'store://artifacts/tutorial-jovyan/cancer_classifier:22118ca5268e45babf7668ce11064837',
 'iteration_results': 's3://mlrun/trainer-train/0/iteration_results.csv',
 'parallel_coordinates': 's3://mlrun/trainer-train/0/parallel_coordinates.html'}

# list the models in the project (can apply filters)
models = project.list_models()
for model in models:
    print(f"uri: {model.uri}, metrics: {model.metrics}")

uri: store://models/tutorial-jovyan/cancer_classifier#0:05c6e41b668f460fa67d7abf9dff9542, metrics: {'accuracy': 0.956140350877193, 'f1_score': 0.965034965034965, 'precision_score': 0.9583333333333334, 'recall_score': 0.971830985915493}
uri: store://models/tutorial-jovyan/cancer_classifier#1:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.956140350877193, 'f1_score': 0.965034965034965, 'precision_score': 0.9583333333333334, 'recall_score': 0.971830985915493}
uri: store://models/tutorial-jovyan/cancer_classifier#2:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.956140350877193, 'f1_score': 0.965034965034965, 'precision_score': 0.9583333333333334, 'recall_score': 0.971830985915493}
uri: store://models/tutorial-jovyan/cancer_classifier#3:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9649122807017544, 'f1_score': 0.9722222222222222, 'precision_score': 0.958904109589041, 'recall_score': 0.9859154929577465}
uri: store://models/tutorial-jovyan/cancer_classifier#4:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.6228070175438597, 'f1_score': 0.7675675675675676, 'precision_score': 0.6228070175438597, 'recall_score': 1.0}
uri: store://models/tutorial-jovyan/cancer_classifier#5:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.6228070175438597, 'f1_score': 0.7675675675675676, 'precision_score': 0.6228070175438597, 'recall_score': 1.0}
uri: store://models/tutorial-jovyan/cancer_classifier#6:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.956140350877193, 'f1_score': 0.965034965034965, 'precision_score': 0.9583333333333334, 'recall_score': 0.971830985915493}
uri: store://models/tutorial-jovyan/cancer_classifier#7:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9385964912280702, 'f1_score': 0.951048951048951, 'precision_score': 0.9444444444444444, 'recall_score': 0.9577464788732394}
uri: store://models/tutorial-jovyan/cancer_classifier#8:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9385964912280702, 'f1_score': 0.951048951048951, 'precision_score': 0.9444444444444444, 'recall_score': 0.9577464788732394}
uri: store://models/tutorial-jovyan/cancer_classifier#9:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9385964912280702, 'f1_score': 0.951048951048951, 'precision_score': 0.9444444444444444, 'recall_score': 0.9577464788732394}
uri: store://models/tutorial-jovyan/cancer_classifier#10:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.6228070175438597, 'f1_score': 0.7675675675675676, 'precision_score': 0.6228070175438597, 'recall_score': 1.0}
uri: store://models/tutorial-jovyan/cancer_classifier#11:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.6228070175438597, 'f1_score': 0.7675675675675676, 'precision_score': 0.6228070175438597, 'recall_score': 1.0}
uri: store://models/tutorial-jovyan/cancer_classifier#12:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9385964912280702, 'f1_score': 0.951048951048951, 'precision_score': 0.9444444444444444, 'recall_score': 0.9577464788732394}

# to view the full model object use:
# print(models[0].to_yaml())

# compare the runs (generate interactive parallel coordinates plot and a table)
project.list_runs(name="trainer-train", iter=True).compare()
uid iter start state name parameters results
12 Aug 24 08:51:18 completed trainer-train
n_estimators=1000
learning_rate=0.001
max_depth=8
accuracy=0.9385964912280702
f1_score=0.951048951048951
precision_score=0.9444444444444444
recall_score=0.9577464788732394
11 Aug 24 08:51:16 completed trainer-train
n_estimators=100
learning_rate=0.001
max_depth=8
accuracy=0.6228070175438597
f1_score=0.7675675675675676
precision_score=0.6228070175438597
recall_score=1.0
10 Aug 24 08:51:14 completed trainer-train
n_estimators=10
learning_rate=0.001
max_depth=8
accuracy=0.6228070175438597
f1_score=0.7675675675675676
precision_score=0.6228070175438597
recall_score=1.0
9 Aug 24 08:51:10 completed trainer-train
n_estimators=1000
learning_rate=0.1
max_depth=8
accuracy=0.9385964912280702
f1_score=0.951048951048951
precision_score=0.9444444444444444
recall_score=0.9577464788732394
8 Aug 24 08:51:08 completed trainer-train
n_estimators=100
learning_rate=0.1
max_depth=8
accuracy=0.9385964912280702
f1_score=0.951048951048951
precision_score=0.9444444444444444
recall_score=0.9577464788732394
7 Aug 24 08:51:06 completed trainer-train
n_estimators=10
learning_rate=0.1
max_depth=8
accuracy=0.9385964912280702
f1_score=0.951048951048951
precision_score=0.9444444444444444
recall_score=0.9577464788732394
6 Aug 24 08:51:01 completed trainer-train
n_estimators=1000
learning_rate=0.001
max_depth=2
accuracy=0.956140350877193
f1_score=0.965034965034965
precision_score=0.9583333333333334
recall_score=0.971830985915493
5 Aug 24 08:50:59 completed trainer-train
n_estimators=100
learning_rate=0.001
max_depth=2
accuracy=0.6228070175438597
f1_score=0.7675675675675676
precision_score=0.6228070175438597
recall_score=1.0
4 Aug 24 08:50:57 completed trainer-train
n_estimators=10
learning_rate=0.001
max_depth=2
accuracy=0.6228070175438597
f1_score=0.7675675675675676
precision_score=0.6228070175438597
recall_score=1.0
3 Aug 24 08:50:52 completed trainer-train
n_estimators=1000
learning_rate=0.1
max_depth=2
accuracy=0.9649122807017544
f1_score=0.9722222222222222
precision_score=0.958904109589041
recall_score=0.9859154929577465
2 Aug 24 08:50:50 completed trainer-train
n_estimators=100
learning_rate=0.1
max_depth=2
accuracy=0.956140350877193
f1_score=0.965034965034965
precision_score=0.9583333333333334
recall_score=0.971830985915493
1 Aug 24 08:50:47 completed trainer-train
n_estimators=10
learning_rate=0.1
max_depth=2
accuracy=0.956140350877193
f1_score=0.965034965034965
precision_score=0.9583333333333334
recall_score=0.971830985915493
1 Aug 24 08:50:46 completed trainer-train
n_estimators=10
learning_rate=0.1
max_depth=2
accuracy=0.956140350877193
f1_score=0.965034965034965
precision_score=0.9583333333333334
recall_score=0.971830985915493
2 Aug 24 08:50:46 completed trainer-train
n_estimators=100
learning_rate=0.1
max_depth=2
accuracy=0.956140350877193
f1_score=0.965034965034965
precision_score=0.9583333333333334
recall_score=0.971830985915493
3 Aug 24 08:50:46 completed trainer-train
n_estimators=1000
learning_rate=0.1
max_depth=2
accuracy=0.9649122807017544
f1_score=0.9722222222222222
precision_score=0.958904109589041
recall_score=0.9859154929577465
4 Aug 24 08:50:46 completed trainer-train
n_estimators=10
learning_rate=0.001
max_depth=2
accuracy=0.6228070175438597
f1_score=0.7675675675675676
precision_score=0.6228070175438597
recall_score=1.0
5 Aug 24 08:50:46 completed trainer-train
n_estimators=100
learning_rate=0.001
max_depth=2
accuracy=0.6228070175438597
f1_score=0.7675675675675676
precision_score=0.6228070175438597
recall_score=1.0
6 Aug 24 08:50:46 completed trainer-train
n_estimators=1000
learning_rate=0.001
max_depth=2
accuracy=0.956140350877193
f1_score=0.965034965034965
precision_score=0.9583333333333334
recall_score=0.971830985915493
7 Aug 24 08:50:46 completed trainer-train
n_estimators=10
learning_rate=0.1
max_depth=8
accuracy=0.9385964912280702
f1_score=0.951048951048951
precision_score=0.9444444444444444
recall_score=0.9577464788732394
8 Aug 24 08:50:46 completed trainer-train
n_estimators=100
learning_rate=0.1
max_depth=8
accuracy=0.9385964912280702
f1_score=0.951048951048951
precision_score=0.9444444444444444
recall_score=0.9577464788732394
9 Aug 24 08:50:46 completed trainer-train
n_estimators=1000
learning_rate=0.1
max_depth=8
accuracy=0.9385964912280702
f1_score=0.951048951048951
precision_score=0.9444444444444444
recall_score=0.9577464788732394
10 Aug 24 08:50:46 completed trainer-train
n_estimators=10
learning_rate=0.001
max_depth=8
accuracy=0.6228070175438597
f1_score=0.7675675675675676
precision_score=0.6228070175438597
recall_score=1.0
11 Aug 24 08:50:46 completed trainer-train
n_estimators=100
learning_rate=0.001
max_depth=8
accuracy=0.6228070175438597
f1_score=0.7675675675675676
precision_score=0.6228070175438597
recall_score=1.0
12 Aug 24 08:50:46 completed trainer-train
n_estimators=1000
learning_rate=0.001
max_depth=8
accuracy=0.9385964912280702
f1_score=0.951048951048951
precision_score=0.9444444444444444
recall_score=0.9577464788732394
0 Aug 24 08:50:24 completed trainer-train
n_estimators=100
learning_rate=0.1
max_depth=3
accuracy=0.956140350877193
f1_score=0.965034965034965
precision_score=0.9583333333333334
recall_score=0.971830985915493

Build and test the model serving functions#

MLRun serving can produce managed, real-time, serverless, pipelines composed of various data processing and ML tasks. The pipelines use the Nuclio real-time serverless engine, which can be deployed anywhere. For more details and examples, see the MLRun Serving Graphs.

Create a model serving function from our code

serving_fn = mlrun.new_function("serving", image="mlrun/mlrun", kind="serving")
serving_fn.add_model('cancer-classifier',model_path=hp_tuning_run.outputs["model"], class_name='mlrun.frameworks.sklearn.SklearnModelServer')

<mlrun.serving.states.TaskStep at 0x7ffb4eba6bb0>

# create a mock (simulator of the real-time function)
server = serving_fn.to_mock_server()

my_data = {"inputs"
           :[[
               1.371e+01, 2.083e+01, 9.020e+01, 5.779e+02, 1.189e-01, 1.645e-01,
               9.366e-02, 5.985e-02, 2.196e-01, 7.451e-02, 5.835e-01, 1.377e+00,
               3.856e+00, 5.096e+01, 8.805e-03, 3.029e-02, 2.488e-02, 1.448e-02,
               1.486e-02, 5.412e-03, 1.706e+01, 2.814e+01, 1.106e+02, 8.970e+02,
               1.654e-01, 3.682e-01, 2.678e-01, 1.556e-01, 3.196e-01, 1.151e-01]
            ]
}
server.test("/v2/models/cancer-classifier/infer", body=my_data)

> 2022-08-24 08:51:36,487 [warning] run command, file or code were not specified
> 2022-08-24 08:51:37,660 [info] model cancer-classifier was loaded
> 2022-08-24 08:51:37,661 [info] Loaded ['cancer-classifier']

/opt/conda/lib/python3.8/site-packages/sklearn/base.py:450: UserWarning:

X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names

{'id': 'ec127766ed0d4e229496a61e8790047e',
 'model_name': 'cancer-classifier',
 'outputs': [0]}

Done!#

Congratulation! You’ve completed Part 2 of the MLRun getting-started tutorial. Proceed to Part 3: Model serving to learn how to deploy and serve your model using a serverless function.