Train, compare, and register models
Contents
Train, compare, and register models#
This notebook provides a quick overview of training ML models using MLRun MLOps orchestration framework.
Make sure you reviewed the basics in MLRun Quick Start Tutorial.
Tutorial steps:
MLRun installation and configuration#
Before running this notebook make sure mlrun
and sklearn
packages are installed (pip install mlrun scikit-learn~=1.0
) and that you have configured the access to the MLRun service.
# install MLRun if not installed, run this only once (restart the notebook after the install !!!)
%pip install mlrun scikit-learn~=1.0.0
Requirement already satisfied: mlrun in /opt/conda/lib/python3.8/site-packages (1.1.0rc24)
Requirement already satisfied: scikit-learn~=1.0.0 in /opt/conda/lib/python3.8/site-packages (1.0.2)
Requirement already satisfied: pyarrow<7,>=1 in /opt/conda/lib/python3.8/site-packages (from mlrun) (6.0.1)
Requirement already satisfied: kfp~=1.8.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.8.13)
Requirement already satisfied: python-dotenv~=0.17.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.17.1)
Requirement already satisfied: nest-asyncio~=1.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.5.1)
Requirement already satisfied: v3io-frames~=0.10.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.10.2)
Requirement already satisfied: inflection~=0.5.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.5.1)
Requirement already satisfied: v3io~=0.5.13 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.5.15)
Requirement already satisfied: GitPython~=3.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.1.27)
Requirement already satisfied: pydantic~=1.5 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.9.2)
Requirement already satisfied: sqlalchemy~=1.3 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.4.13)
Requirement already satisfied: dask~=2021.11.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2021.11.2)
Requirement already satisfied: requests~=2.22 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2.25.1)
Requirement already satisfied: click~=8.0.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (8.0.4)
Requirement already satisfied: humanfriendly~=8.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (8.2)
Requirement already satisfied: storey~=1.1.7 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.1.7)
Requirement already satisfied: fastapi~=0.78.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.78.0)
Requirement already satisfied: tabulate~=0.8.6 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.8.10)
Requirement already satisfied: alembic<1.6.0,~=1.4 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.5.8)
Requirement already satisfied: cryptography<3.4,~=3.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.3.2)
Requirement already satisfied: ipython~=7.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (7.23.1)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.26.4)
Requirement already satisfied: chardet<4.0,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.0.4)
Requirement already satisfied: aiohttp~=3.8 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.8.1)
Requirement already satisfied: pyyaml~=5.1 in /opt/conda/lib/python3.8/site-packages (from mlrun) (5.4.1)
Requirement already satisfied: typing-extensions<5,>=3.10.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (4.3.0)
Requirement already satisfied: mergedeep~=1.3 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.3.4)
Requirement already satisfied: fsspec~=2021.8.1 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2021.8.1)
Requirement already satisfied: deepdiff~=5.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (5.8.1)
Requirement already satisfied: numpy<1.23.0,>=1.16.5 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.20.2)
Requirement already satisfied: pymysql~=1.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.0.2)
Requirement already satisfied: nuclio-jupyter~=0.9.1 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.9.2)
Requirement already satisfied: orjson~=3.3 in /opt/conda/lib/python3.8/site-packages (from mlrun) (3.7.12)
Requirement already satisfied: distributed~=2021.11.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2021.11.2)
Requirement already satisfied: pandas~=1.2 in /opt/conda/lib/python3.8/site-packages (from mlrun) (1.2.4)
Requirement already satisfied: semver~=2.13 in /opt/conda/lib/python3.8/site-packages (from mlrun) (2.13.0)
Requirement already satisfied: v3iofs~=0.1.7 in /opt/conda/lib/python3.8/site-packages (from mlrun) (0.1.11)
Requirement already satisfied: kubernetes~=12.0 in /opt/conda/lib/python3.8/site-packages (from mlrun) (12.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from scikit-learn~=1.0.0) (2.1.0)
Requirement already satisfied: scipy>=1.1.0 in /opt/conda/lib/python3.8/site-packages (from scikit-learn~=1.0.0) (1.6.3)
Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.8/site-packages (from scikit-learn~=1.0.0) (1.0.1)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (1.2.0)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (2.1.0)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (4.0.2)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (20.3.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (6.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (1.8.1)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.8/site-packages (from aiohttp~=3.8->mlrun) (1.3.1)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.8/site-packages (from alembic<1.6.0,~=1.4->mlrun) (1.0.4)
Requirement already satisfied: Mako in /opt/conda/lib/python3.8/site-packages (from alembic<1.6.0,~=1.4->mlrun) (1.1.4)
Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.8/site-packages (from alembic<1.6.0,~=1.4->mlrun) (2.8.1)
Requirement already satisfied: cffi>=1.12 in /opt/conda/lib/python3.8/site-packages (from cryptography<3.4,~=3.0->mlrun) (1.14.5)
Requirement already satisfied: six>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from cryptography<3.4,~=3.0->mlrun) (1.16.0)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.8/site-packages (from dask~=2021.11.2->mlrun) (20.9)
Requirement already satisfied: toolz>=0.8.2 in /opt/conda/lib/python3.8/site-packages (from dask~=2021.11.2->mlrun) (0.11.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /opt/conda/lib/python3.8/site-packages (from dask~=2021.11.2->mlrun) (2.1.0)
Requirement already satisfied: partd>=0.3.10 in /opt/conda/lib/python3.8/site-packages (from dask~=2021.11.2->mlrun) (1.2.0)
Requirement already satisfied: ordered-set<4.2.0,>=4.1.0 in /opt/conda/lib/python3.8/site-packages (from deepdiff~=5.0->mlrun) (4.1.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (49.6.0.post20210108)
Requirement already satisfied: msgpack>=0.6.0 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (1.0.2)
Requirement already satisfied: tornado>=6.0.3 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (6.1)
Requirement already satisfied: zict>=0.1.3 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (2.0.0)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (3.0.3)
Requirement already satisfied: tblib>=1.6.0 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /opt/conda/lib/python3.8/site-packages (from distributed~=2021.11.2->mlrun) (2.3.0)
Requirement already satisfied: starlette==0.19.1 in /opt/conda/lib/python3.8/site-packages (from fastapi~=0.78.0->mlrun) (0.19.1)
Requirement already satisfied: anyio<5,>=3.4.0 in /opt/conda/lib/python3.8/site-packages (from starlette==0.19.1->fastapi~=0.78.0->mlrun) (3.6.1)
Requirement already satisfied: gitdb<5,>=4.0.1 in /opt/conda/lib/python3.8/site-packages (from GitPython~=3.0->mlrun) (4.0.9)
Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (0.18.0)
Requirement already satisfied: pickleshare in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (3.0.18)
Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (0.1.2)
Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (4.8.0)
Requirement already satisfied: decorator in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (5.0.7)
Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (5.3.0)
Requirement already satisfied: backcall in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (0.2.0)
Requirement already satisfied: pygments in /opt/conda/lib/python3.8/site-packages (from ipython~=7.0->mlrun) (2.9.0)
Requirement already satisfied: typer<1.0,>=0.3.2 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.6.1)
Requirement already satisfied: google-api-python-client<2,>=1.7.8 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.12.11)
Requirement already satisfied: fire<1,>=0.3.1 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.4.0)
Requirement already satisfied: jsonschema<4,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (3.2.0)
Requirement already satisfied: kfp-pipeline-spec<0.2.0,>=0.1.14 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.1.16)
Requirement already satisfied: docstring-parser<1,>=0.7.3 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.14.1)
Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.9.1)
Requirement already satisfied: strip-hints<1,>=0.1.8 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (0.1.10)
Requirement already satisfied: uritemplate<4,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (3.0.1)
Requirement already satisfied: absl-py<2,>=0.9 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.2.0)
Requirement already satisfied: google-cloud-storage<2,>=1.20.0 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.44.0)
Requirement already satisfied: google-auth<2,>=1.6.1 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.35.0)
Requirement already satisfied: google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (2.8.2)
Requirement already satisfied: Deprecated<2,>=1.2.7 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.2.13)
Requirement already satisfied: kfp-server-api<2.0.0,>=1.1.2 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (1.8.4)
Requirement already satisfied: protobuf<4,>=3.13.0 in /opt/conda/lib/python3.8/site-packages (from kfp~=1.8.0->mlrun) (3.19.4)
Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.8/site-packages (from kubernetes~=12.0->mlrun) (1.3.1)
Requirement already satisfied: certifi>=14.05.14 in /opt/conda/lib/python3.8/site-packages (from kubernetes~=12.0->mlrun) (2020.12.5)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.8/site-packages (from kubernetes~=12.0->mlrun) (1.3.3)
Requirement already satisfied: notebook>=5.2.0 in /opt/conda/lib/python3.8/site-packages (from nuclio-jupyter~=0.9.1->mlrun) (6.4.12)
Requirement already satisfied: boto3>=1.9 in /opt/conda/lib/python3.8/site-packages (from nuclio-jupyter~=0.9.1->mlrun) (1.17.106)
Requirement already satisfied: nbconvert>=5.4 in /opt/conda/lib/python3.8/site-packages (from nuclio-jupyter~=0.9.1->mlrun) (6.5.3)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas~=1.2->mlrun) (2021.1)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests~=2.22->mlrun) (2.10)
Requirement already satisfied: greenlet!=0.4.17 in /opt/conda/lib/python3.8/site-packages (from sqlalchemy~=1.3->mlrun) (1.0.0)
Requirement already satisfied: grpcio-tools<1.42,>1.34.0 in /opt/conda/lib/python3.8/site-packages (from storey~=1.1.7->mlrun) (1.41.1)
Requirement already satisfied: grpcio<1.42,>1.34.0 in /opt/conda/lib/python3.8/site-packages (from storey~=1.1.7->mlrun) (1.41.1)
Requirement already satisfied: future>=0.18.2 in /opt/conda/lib/python3.8/site-packages (from v3io~=0.5.13->mlrun) (0.18.2)
Requirement already satisfied: ujson>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from v3io~=0.5.13->mlrun) (5.4.0)
Requirement already satisfied: googleapis-common-protos>=1.5.3 in /opt/conda/lib/python3.8/site-packages (from v3io-frames~=0.10.2->mlrun) (1.56.4)
Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /opt/conda/lib/python3.8/site-packages (from boto3>=1.9->nuclio-jupyter~=0.9.1->mlrun) (0.4.2)
Requirement already satisfied: botocore<1.21.0,>=1.20.106 in /opt/conda/lib/python3.8/site-packages (from boto3>=1.9->nuclio-jupyter~=0.9.1->mlrun) (1.20.106)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.8/site-packages (from boto3>=1.9->nuclio-jupyter~=0.9.1->mlrun) (0.10.0)
Requirement already satisfied: pycparser in /opt/conda/lib/python3.8/site-packages (from cffi>=1.12->cryptography<3.4,~=3.0->mlrun) (2.20)
Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.8/site-packages (from Deprecated<2,>=1.2.7->kfp~=1.8.0->mlrun) (1.14.1)
Requirement already satisfied: termcolor in /opt/conda/lib/python3.8/site-packages (from fire<1,>=0.3.1->kfp~=1.8.0->mlrun) (1.1.0)
Requirement already satisfied: smmap<6,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from gitdb<5,>=4.0.1->GitPython~=3.0->mlrun) (5.0.0)
Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp~=1.8.0->mlrun) (0.1.0)
Requirement already satisfied: httplib2<1dev,>=0.15.0 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp~=1.8.0->mlrun) (0.20.4)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp~=1.8.0->mlrun) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp~=1.8.0->mlrun) (4.2.4)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp~=1.8.0->mlrun) (4.9)
Requirement already satisfied: google-cloud-core<3.0dev,>=1.6.0 in /opt/conda/lib/python3.8/site-packages (from google-cloud-storage<2,>=1.20.0->kfp~=1.8.0->mlrun) (2.3.2)
Requirement already satisfied: google-resumable-media<3.0dev,>=1.3.0 in /opt/conda/lib/python3.8/site-packages (from google-cloud-storage<2,>=1.20.0->kfp~=1.8.0->mlrun) (2.3.3)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from jedi>=0.16->ipython~=7.0->mlrun) (0.8.2)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.8/site-packages (from jinja2->distributed~=2021.11.2->mlrun) (2.1.1)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema<4,>=3.0.1->kfp~=1.8.0->mlrun) (0.17.3)
Requirement already satisfied: defusedxml in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.7.1)
Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (1.4.2)
Requirement already satisfied: tinycss2 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (1.1.1)
Requirement already satisfied: jupyter-core>=4.7 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (4.7.1)
Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (4.9.3)
Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.8.4)
Requirement already satisfied: entrypoints>=0.2.2 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.3)
Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.1.2)
Requirement already satisfied: nbclient>=0.5.0 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.5.3)
Requirement already satisfied: bleach in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (3.3.0)
Requirement already satisfied: nbformat>=5.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (5.4.0)
Requirement already satisfied: lxml in /opt/conda/lib/python3.8/site-packages (from nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (4.9.1)
Requirement already satisfied: pyzmq>=17 in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (22.0.3)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (0.10.1)
Requirement already satisfied: ipykernel in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (5.5.4)
Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (0.9.4)
Requirement already satisfied: Send2Trash>=1.8.0 in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (1.8.0)
Requirement already satisfied: jupyter-client>=5.3.4 in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (6.1.12)
Requirement already satisfied: ipython-genutils in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (0.2.0)
Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.8/site-packages (from notebook>=5.2.0->nuclio-jupyter~=0.9.1->mlrun) (20.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.8/site-packages (from packaging>=20.0->dask~=2021.11.2->mlrun) (2.4.7)
Requirement already satisfied: locket in /opt/conda/lib/python3.8/site-packages (from partd>=0.3.10->dask~=2021.11.2->mlrun) (0.2.0)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.8/site-packages (from pexpect>4.3->ipython~=7.0->mlrun) (0.7.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython~=7.0->mlrun) (0.2.5)
Requirement already satisfied: wheel in /opt/conda/lib/python3.8/site-packages (from strip-hints<1,>=0.1.8->kfp~=1.8.0->mlrun) (0.36.2)
Requirement already satisfied: heapdict in /opt/conda/lib/python3.8/site-packages (from zict>=0.1.3->distributed~=2021.11.2->mlrun) (1.0.1)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from requests-oauthlib->kubernetes~=12.0->mlrun) (3.0.1)
Requirement already satisfied: sniffio>=1.1 in /opt/conda/lib/python3.8/site-packages (from anyio<5,>=3.4.0->starlette==0.19.1->fastapi~=0.78.0->mlrun) (1.2.0)
Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.8/site-packages (from google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp~=1.8.0->mlrun) (1.3.0)
Requirement already satisfied: async-generator in /opt/conda/lib/python3.8/site-packages (from nbclient>=0.5.0->nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (1.10)
Requirement already satisfied: fastjsonschema in /opt/conda/lib/python3.8/site-packages (from nbformat>=5.1->nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (2.16.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.1->kfp~=1.8.0->mlrun) (0.4.8)
Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.8/site-packages (from beautifulsoup4->nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (2.0.1)
Requirement already satisfied: webencodings in /opt/conda/lib/python3.8/site-packages (from bleach->nbconvert>=5.4->nuclio-jupyter~=0.9.1->mlrun) (0.5.1)
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/opt/conda/bin/python -m pip install --upgrade pip' command.
Note: you may need to restart the kernel to use updated packages.
Define MLRun project and a training functions#
You should create, load, or use (get) an MLRun Project that holds all your functions and assets.
Get or create a new project:
The get_or_create_project()
method tries to load the project from MLRun DB. If the project does not exist it creates a new one.
import mlrun
project = mlrun.get_or_create_project("tutorial", context="src/", user_project=True)
> 2022-08-24 08:50:23,251 [info] loaded project tutorial from None or context and saved in MLRun DB
Add (auto) MLOps to your training function:
Training functions generate models and various model statistics. You’ll want to store the models along with all the relevant data,
metadata, and measurements. MLRun can apply all the MLOps functionality automatically (“Auto-MLOps”) by simply using the framework specific apply_mlrun()
method.
In the training function below note the single custom line you need to add to your code:
apply_mlrun(model=model, model_name="my_model", x_test=x_test, y_test=y_test)
apply_mlrun()
manages the training process and automatically logs all the framework-specific model object, details, data, metadata, and metrics.
It accepts the model object and various optional parameters. When specifying the x_test
and y_test
data it generates various plots and calculations to evaluate the model.
Metadata and parameters are automatically recorded (from MLRun context
object) and don’t need to be specified.
Function code:
Run the following cell to generate the trainer.py
file (or copy it manually):
Create a serverless function object from the code above, and register it in the project:
trainer = project.set_function("trainer.py", name="trainer", kind="job", image="mlrun/mlrun", handler="train")
Run the training function and log the artifacts and model#
Create a dataset for training:
import pandas as pd
from sklearn.datasets import load_breast_cancer
breast_cancer = load_breast_cancer()
breast_cancer_dataset = pd.DataFrame(data=breast_cancer.data, columns=breast_cancer.feature_names)
breast_cancer_labels = pd.DataFrame(data=breast_cancer.target, columns=["label"])
breast_cancer_dataset = pd.concat([breast_cancer_dataset, breast_cancer_labels], axis=1)
breast_cancer_dataset.to_csv("cancer-dataset.csv", index=False)
Run the function (locally) using the generated dataset:
trainer_run = project.run_function(
"trainer",
inputs={"dataset": "cancer-dataset.csv"},
params = {"n_estimators": 100, "learning_rate": 1e-1, "max_depth": 3},
local=True
)
> 2022-08-24 08:50:24,614 [warning] it is recommended to use k8s secret (specify secret_name), specifying the aws_access_key/aws_secret_key directly is unsafe
> 2022-08-24 08:50:24,636 [info] starting run trainer-train uid=05c6e41b668f460fa67d7abf9dff9542 DB=http://mlrun-api:8080
project | uid | iter | start | state | name | labels | inputs | parameters | results | artifacts |
---|---|---|---|---|---|---|---|---|---|---|
tutorial-jovyan | 0 | Aug 24 08:50:24 | completed | trainer-train | kind= owner=jovyan host=mlrun-jupyter-6b78bf965-knkrt |
dataset |
n_estimators=100 learning_rate=0.1 max_depth=3 |
accuracy=0.956140350877193 f1_score=0.965034965034965 precision_score=0.9583333333333334 recall_score=0.971830985915493 |
feature-importance test_set confusion-matrix roc-curves calibration-curve model |
> 2022-08-24 08:50:32,906 [info] run executed, status=completed
View the auto generated results and artifacts:
trainer_run.outputs
{'accuracy': 0.956140350877193,
'f1_score': 0.965034965034965,
'precision_score': 0.9583333333333334,
'recall_score': 0.971830985915493,
'feature-importance': 's3://mlrun/trainer-train/0/feature-importance.html',
'test_set': 'store://artifacts/tutorial-jovyan/trainer-train_test_set:05c6e41b668f460fa67d7abf9dff9542',
'confusion-matrix': 's3://mlrun/trainer-train/0/confusion-matrix.html',
'roc-curves': 's3://mlrun/trainer-train/0/roc-curves.html',
'calibration-curve': 's3://mlrun/trainer-train/0/calibration-curve.html',
'model': 'store://artifacts/tutorial-jovyan/cancer_classifier:05c6e41b668f460fa67d7abf9dff9542'}
trainer_run.artifact('feature-importance').show()
Export model files + metadata into a zip: (require MLRun 1.1.0 and above)
You can export()
the model package (files + metadata) into a zip, and load it on a remote system/cluster (by simply running model = project.import_artifact(key, path)
).
trainer_run.artifact('model').meta.export("model.zip")
Hyper-parameter tuning and model/experiment comparison#
Run a GridSearch
with a couple of parameters, and select the best run with respect to the max accuracy
.
(Read more about MLRun Hyper-Param and Iterative jobs.)
For basic usage you can run the hyperparameters tuning job by using the arguments:
hyperparams
for the hyperparameters options and values of choice.selector
for specifying how to select the best model.
Running a remote function:
In order to run the hyper-param task over the cluster you need the input data to be available for the job, using object storage or the mlrun versioned artifact store.
The following line logs (and uploads) the dataframe as a project artifact:
dataset_artifact = project.log_dataset("cancer-dataset", df=breast_cancer_dataset, index=False)
Run the function over the remote Kubernetes cluster (local
is not set):
hp_tuning_run = project.run_function(
"trainer",
inputs={"dataset": dataset_artifact.uri},
hyperparams={
"n_estimators": [10, 100, 1000],
"learning_rate": [1e-1, 1e-3],
"max_depth": [2, 8]
},
selector="max.accuracy",
)
> 2022-08-24 08:50:34,657 [info] starting run trainer-train uid=22118ca5268e45babf7668ce11064837 DB=http://mlrun-api:8080
> 2022-08-24 08:50:35,094 [info] Job is running in the background, pod: trainer-train-8c6f4
> 2022-08-24 08:51:27,920 [info] best iteration=3, used criteria max.accuracy
> 2022-08-24 08:51:28,380 [info] run executed, status=completed
final state: completed
project | uid | iter | start | state | name | labels | inputs | parameters | results | artifacts |
---|---|---|---|---|---|---|---|---|---|---|
tutorial-jovyan | 0 | Aug 24 08:50:46 | completed | trainer-train | kind=job owner=jovyan mlrun/client_version=1.1.0-rc24 |
dataset |
best_iteration=3 accuracy=0.9649122807017544 f1_score=0.9722222222222222 precision_score=0.958904109589041 recall_score=0.9859154929577465 |
feature-importance test_set confusion-matrix roc-curves calibration-curve model iteration_results parallel_coordinates |
> 2022-08-24 08:51:35,352 [info] run executed, status=completed
View Hyper-param results and the selected run in the MLRun UI:
Interactive Parallel Coordinates Plot:
List the generated models and compare the different runs:
hp_tuning_run.outputs
{'best_iteration': 3,
'accuracy': 0.9649122807017544,
'f1_score': 0.9722222222222222,
'precision_score': 0.958904109589041,
'recall_score': 0.9859154929577465,
'feature-importance': 's3://mlrun/trainer-train/3/feature-importance.html',
'test_set': 'store://artifacts/tutorial-jovyan/trainer-train_test_set:22118ca5268e45babf7668ce11064837',
'confusion-matrix': 's3://mlrun/trainer-train/3/confusion-matrix.html',
'roc-curves': 's3://mlrun/trainer-train/3/roc-curves.html',
'calibration-curve': 's3://mlrun/trainer-train/3/calibration-curve.html',
'model': 'store://artifacts/tutorial-jovyan/cancer_classifier:22118ca5268e45babf7668ce11064837',
'iteration_results': 's3://mlrun/trainer-train/0/iteration_results.csv',
'parallel_coordinates': 's3://mlrun/trainer-train/0/parallel_coordinates.html'}
# list the models in the project (can apply filters)
models = project.list_models()
for model in models:
print(f"uri: {model.uri}, metrics: {model.metrics}")
uri: store://models/tutorial-jovyan/cancer_classifier#0:05c6e41b668f460fa67d7abf9dff9542, metrics: {'accuracy': 0.956140350877193, 'f1_score': 0.965034965034965, 'precision_score': 0.9583333333333334, 'recall_score': 0.971830985915493}
uri: store://models/tutorial-jovyan/cancer_classifier#1:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.956140350877193, 'f1_score': 0.965034965034965, 'precision_score': 0.9583333333333334, 'recall_score': 0.971830985915493}
uri: store://models/tutorial-jovyan/cancer_classifier#2:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.956140350877193, 'f1_score': 0.965034965034965, 'precision_score': 0.9583333333333334, 'recall_score': 0.971830985915493}
uri: store://models/tutorial-jovyan/cancer_classifier#3:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9649122807017544, 'f1_score': 0.9722222222222222, 'precision_score': 0.958904109589041, 'recall_score': 0.9859154929577465}
uri: store://models/tutorial-jovyan/cancer_classifier#4:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.6228070175438597, 'f1_score': 0.7675675675675676, 'precision_score': 0.6228070175438597, 'recall_score': 1.0}
uri: store://models/tutorial-jovyan/cancer_classifier#5:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.6228070175438597, 'f1_score': 0.7675675675675676, 'precision_score': 0.6228070175438597, 'recall_score': 1.0}
uri: store://models/tutorial-jovyan/cancer_classifier#6:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.956140350877193, 'f1_score': 0.965034965034965, 'precision_score': 0.9583333333333334, 'recall_score': 0.971830985915493}
uri: store://models/tutorial-jovyan/cancer_classifier#7:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9385964912280702, 'f1_score': 0.951048951048951, 'precision_score': 0.9444444444444444, 'recall_score': 0.9577464788732394}
uri: store://models/tutorial-jovyan/cancer_classifier#8:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9385964912280702, 'f1_score': 0.951048951048951, 'precision_score': 0.9444444444444444, 'recall_score': 0.9577464788732394}
uri: store://models/tutorial-jovyan/cancer_classifier#9:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9385964912280702, 'f1_score': 0.951048951048951, 'precision_score': 0.9444444444444444, 'recall_score': 0.9577464788732394}
uri: store://models/tutorial-jovyan/cancer_classifier#10:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.6228070175438597, 'f1_score': 0.7675675675675676, 'precision_score': 0.6228070175438597, 'recall_score': 1.0}
uri: store://models/tutorial-jovyan/cancer_classifier#11:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.6228070175438597, 'f1_score': 0.7675675675675676, 'precision_score': 0.6228070175438597, 'recall_score': 1.0}
uri: store://models/tutorial-jovyan/cancer_classifier#12:22118ca5268e45babf7668ce11064837, metrics: {'accuracy': 0.9385964912280702, 'f1_score': 0.951048951048951, 'precision_score': 0.9444444444444444, 'recall_score': 0.9577464788732394}
# to view the full model object use:
# print(models[0].to_yaml())
# compare the runs (generate interactive parallel coordinates plot and a table)
project.list_runs(name="trainer-train", iter=True).compare()
uid | iter | start | state | name | parameters | results |
---|---|---|---|---|---|---|
12 | Aug 24 08:51:18 | completed | trainer-train | n_estimators=1000 learning_rate=0.001 max_depth=8 |
accuracy=0.9385964912280702 f1_score=0.951048951048951 precision_score=0.9444444444444444 recall_score=0.9577464788732394 |
|
11 | Aug 24 08:51:16 | completed | trainer-train | n_estimators=100 learning_rate=0.001 max_depth=8 |
accuracy=0.6228070175438597 f1_score=0.7675675675675676 precision_score=0.6228070175438597 recall_score=1.0 |
|
10 | Aug 24 08:51:14 | completed | trainer-train | n_estimators=10 learning_rate=0.001 max_depth=8 |
accuracy=0.6228070175438597 f1_score=0.7675675675675676 precision_score=0.6228070175438597 recall_score=1.0 |
|
9 | Aug 24 08:51:10 | completed | trainer-train | n_estimators=1000 learning_rate=0.1 max_depth=8 |
accuracy=0.9385964912280702 f1_score=0.951048951048951 precision_score=0.9444444444444444 recall_score=0.9577464788732394 |
|
8 | Aug 24 08:51:08 | completed | trainer-train | n_estimators=100 learning_rate=0.1 max_depth=8 |
accuracy=0.9385964912280702 f1_score=0.951048951048951 precision_score=0.9444444444444444 recall_score=0.9577464788732394 |
|
7 | Aug 24 08:51:06 | completed | trainer-train | n_estimators=10 learning_rate=0.1 max_depth=8 |
accuracy=0.9385964912280702 f1_score=0.951048951048951 precision_score=0.9444444444444444 recall_score=0.9577464788732394 |
|
6 | Aug 24 08:51:01 | completed | trainer-train | n_estimators=1000 learning_rate=0.001 max_depth=2 |
accuracy=0.956140350877193 f1_score=0.965034965034965 precision_score=0.9583333333333334 recall_score=0.971830985915493 |
|
5 | Aug 24 08:50:59 | completed | trainer-train | n_estimators=100 learning_rate=0.001 max_depth=2 |
accuracy=0.6228070175438597 f1_score=0.7675675675675676 precision_score=0.6228070175438597 recall_score=1.0 |
|
4 | Aug 24 08:50:57 | completed | trainer-train | n_estimators=10 learning_rate=0.001 max_depth=2 |
accuracy=0.6228070175438597 f1_score=0.7675675675675676 precision_score=0.6228070175438597 recall_score=1.0 |
|
3 | Aug 24 08:50:52 | completed | trainer-train | n_estimators=1000 learning_rate=0.1 max_depth=2 |
accuracy=0.9649122807017544 f1_score=0.9722222222222222 precision_score=0.958904109589041 recall_score=0.9859154929577465 |
|
2 | Aug 24 08:50:50 | completed | trainer-train | n_estimators=100 learning_rate=0.1 max_depth=2 |
accuracy=0.956140350877193 f1_score=0.965034965034965 precision_score=0.9583333333333334 recall_score=0.971830985915493 |
|
1 | Aug 24 08:50:47 | completed | trainer-train | n_estimators=10 learning_rate=0.1 max_depth=2 |
accuracy=0.956140350877193 f1_score=0.965034965034965 precision_score=0.9583333333333334 recall_score=0.971830985915493 |
|
1 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=10 learning_rate=0.1 max_depth=2 |
accuracy=0.956140350877193 f1_score=0.965034965034965 precision_score=0.9583333333333334 recall_score=0.971830985915493 |
|
2 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=100 learning_rate=0.1 max_depth=2 |
accuracy=0.956140350877193 f1_score=0.965034965034965 precision_score=0.9583333333333334 recall_score=0.971830985915493 |
|
3 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=1000 learning_rate=0.1 max_depth=2 |
accuracy=0.9649122807017544 f1_score=0.9722222222222222 precision_score=0.958904109589041 recall_score=0.9859154929577465 |
|
4 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=10 learning_rate=0.001 max_depth=2 |
accuracy=0.6228070175438597 f1_score=0.7675675675675676 precision_score=0.6228070175438597 recall_score=1.0 |
|
5 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=100 learning_rate=0.001 max_depth=2 |
accuracy=0.6228070175438597 f1_score=0.7675675675675676 precision_score=0.6228070175438597 recall_score=1.0 |
|
6 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=1000 learning_rate=0.001 max_depth=2 |
accuracy=0.956140350877193 f1_score=0.965034965034965 precision_score=0.9583333333333334 recall_score=0.971830985915493 |
|
7 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=10 learning_rate=0.1 max_depth=8 |
accuracy=0.9385964912280702 f1_score=0.951048951048951 precision_score=0.9444444444444444 recall_score=0.9577464788732394 |
|
8 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=100 learning_rate=0.1 max_depth=8 |
accuracy=0.9385964912280702 f1_score=0.951048951048951 precision_score=0.9444444444444444 recall_score=0.9577464788732394 |
|
9 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=1000 learning_rate=0.1 max_depth=8 |
accuracy=0.9385964912280702 f1_score=0.951048951048951 precision_score=0.9444444444444444 recall_score=0.9577464788732394 |
|
10 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=10 learning_rate=0.001 max_depth=8 |
accuracy=0.6228070175438597 f1_score=0.7675675675675676 precision_score=0.6228070175438597 recall_score=1.0 |
|
11 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=100 learning_rate=0.001 max_depth=8 |
accuracy=0.6228070175438597 f1_score=0.7675675675675676 precision_score=0.6228070175438597 recall_score=1.0 |
|
12 | Aug 24 08:50:46 | completed | trainer-train | n_estimators=1000 learning_rate=0.001 max_depth=8 |
accuracy=0.9385964912280702 f1_score=0.951048951048951 precision_score=0.9444444444444444 recall_score=0.9577464788732394 |
|
0 | Aug 24 08:50:24 | completed | trainer-train | n_estimators=100 learning_rate=0.1 max_depth=3 |
accuracy=0.956140350877193 f1_score=0.965034965034965 precision_score=0.9583333333333334 recall_score=0.971830985915493 |
Build and test the model serving functions#
MLRun serving can produce managed, real-time, serverless, pipelines composed of various data processing and ML tasks. The pipelines use the Nuclio real-time serverless engine, which can be deployed anywhere. For more details and examples, see the MLRun Serving Graphs.
Create a model serving function from our code
serving_fn = mlrun.new_function("serving", image="mlrun/mlrun", kind="serving")
serving_fn.add_model('cancer-classifier',model_path=hp_tuning_run.outputs["model"], class_name='mlrun.frameworks.sklearn.SklearnModelServer')
<mlrun.serving.states.TaskStep at 0x7ffb4eba6bb0>
# create a mock (simulator of the real-time function)
server = serving_fn.to_mock_server()
my_data = {"inputs"
:[[
1.371e+01, 2.083e+01, 9.020e+01, 5.779e+02, 1.189e-01, 1.645e-01,
9.366e-02, 5.985e-02, 2.196e-01, 7.451e-02, 5.835e-01, 1.377e+00,
3.856e+00, 5.096e+01, 8.805e-03, 3.029e-02, 2.488e-02, 1.448e-02,
1.486e-02, 5.412e-03, 1.706e+01, 2.814e+01, 1.106e+02, 8.970e+02,
1.654e-01, 3.682e-01, 2.678e-01, 1.556e-01, 3.196e-01, 1.151e-01]
]
}
server.test("/v2/models/cancer-classifier/infer", body=my_data)
> 2022-08-24 08:51:36,487 [warning] run command, file or code were not specified
> 2022-08-24 08:51:37,660 [info] model cancer-classifier was loaded
> 2022-08-24 08:51:37,661 [info] Loaded ['cancer-classifier']
/opt/conda/lib/python3.8/site-packages/sklearn/base.py:450: UserWarning:
X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names
{'id': 'ec127766ed0d4e229496a61e8790047e',
'model_name': 'cancer-classifier',
'outputs': [0]}
Done!#
Congratulation! You’ve completed Part 2 of the MLRun getting-started tutorial. Proceed to Part 3: Model serving to learn how to deploy and serve your model using a serverless function.