Install MLRun on Kubernetes
Install MLRun on Kubernetes#
In this section
Access to a Kubernetes cluster. You must have administrator permissions in order to install MLRun on your cluster. For local installation on Windows or Mac, Docker Desktop is recommended. MLRun fully supports k8s releases 1.22 and 1.23.
The Kubernetes command-line tool (kubectl) compatible with your Kubernetes cluster is installed. Refer to the kubectl installation instructions for more information.
Helm 3.6 CLI is installed. Refer to the Helm installation instructions for more information.
An accessible docker-registry (such as Docker Hub). The registry’s URL and credentials are consumed by the applications via a pre-created secret.
RAM: A minimum of 8Gi is required for running all the initial MLRun components. The amount of RAM required for running MLRun jobs depends on the job’s requirements.
The MLRun Community Edition resources are configured initially with the default cluster/namespace resources limits. You can modify the resources from outside if needed.
Community Edition flavors#
The MLRun CE (Community Edition) chart arrives in 2 flavors - lite and full. The lite version is the default installation and includes the following components:
MLRun - https://github.com/mlrun/mlrun
MLRun DB (MySQL)
Nuclio - https://github.com/nuclio/nuclio
Jupyter - https://github.com/jupyter/notebook (+MLRun integrated)
MPI Operator - https://github.com/kubeflow/mpi-operator
The Full Version also includes:
Spark Operator - https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
Pipelines - https://github.com/kubeflow/pipelines
Prometheus stack - https://github.com/prometheus-community/helm-charts
Installing on Docker Desktop#
Docker Desktop is available for Mac and Windows. For download information, system requirements, and installation instructions, see:
Install Docker Desktop on Windows. Note that WSL 2 backend was tested, Hyper-V was not tested.
Configuring Docker Desktop#
Docker Desktop includes a standalone Kubernetes server and client, as well as Docker CLI integration that runs on your machine. The
Kubernetes server runs locally within your Docker instance. To enable Kubernetes support and install a standalone instance of Kubernetes
running as a Docker container, go to Preferences > Kubernetes and then click Enable Kubernetes. Click Apply & Restart to
save the settings and then click Install to confirm. This instantiates the images that are required to run the Kubernetes server as
containers, and installs the
/usr/local/bin/kubectl command on your machine. For more information, see the Kubernetes documentation.
It’s recommended to limit the amount of memory allocated to Kubernetes. If you’re using Windows and WSL 2, you can configure global WSL options by placing a
.wslconfig file into the root directory of your users folder:
C:\Users\<yourUserName>\.wslconfig. Keep in mind that you might need to run
wsl --shutdown to shut down the WSL 2 VM and then restart your WSL instance for these changes to take effect.
[wsl2] memory=8GB # Limits VM memory in WSL 2 to 8 GB
To learn about the various UI options and their usage, see:
Installing the Lite Version#
These instructions use
mlrun as the namespace (
-n parameter). You can choose a different namespace in your kubernetes cluster.
Create a namespace for the deployed components:
kubectl create namespace mlrun
Add the Community Edition helm chart repo:
helm repo add mlrun-ce https://mlrun.github.io/ce
Update the repo to make sure you’re getting the latest chart:
helm repo update
Create a secret with your docker-registry named
kubectl --namespace mlrun create secret docker-registry registry-credentials \ --docker-server <your-registry-server> \ --docker-username <your-username> \ --docker-password <your-password> \ --docker-email <your-email>
<your-registry-server>is your Private Docker Registry FQDN. (https://index.docker.io/v1/ for Docker Hub).
<your-username>is your Docker username.
<your-password>is your Docker password.
<your-email>is your Docker email.
First-time MLRun users will experience a relatively longer installation time because all required images are being pulled locally for the first time (it will take an average of 10-15 minutes mostly depends on your internet speed).
To install the chart with the release name
mlrun-ce use the following command.
Note the reference to the pre-created
registry-credentials secret in
helm --namespace mlrun \ install mlrun-ce \ --wait \ --timeout 960s \ --set global.registry.url=<registry-url> \ --set global.registry.secretName=registry-credentials \ --set global.externalHostAddress=<host-machine-address> \ mlrun-ce/mlrun-ce
<registry-url>is the registry URL that can be authenticated by the
index.docker.io/<your-username>for Docker Hub).
<host-machine-address>is the IP address of the host machine (or
$(minikube ip)if using minikube).
When the installation is complete, the helm command prints the URLs and Ports of all the MLRun CE services.
Installing the Full Version#
To install the full version of the chart, use the following command:
helm --namespace mlrun \ install mlrun-ce \ --wait \ --timeout 960s \ -f override-full.yaml \ --set global.registry.url=<registry-url> \ --set global.registry.secretName=registry-credentials \ --set global.externalHostAddress=<host-machine-address> \ mlrun-ce/mlrun-ce
Configuring Online Feature Store#
The MLRun Community Edition now supports the online feature store. To enable it, you need to first deploy a REDIS service that is accessible to your MLRun CE cluster. To deploy a REDIS service, refer to the following link.
When you have a REDIS service deployed, you can configure MLRun CE to use it by adding the following helm value configuration to your helm install command:
Your applications are now available in your local browser:
jupyter-notebook - http://
nuclio - http://
mlrun UI - http://
mlrun API (external) - http://
minio API - http://
minio UI - http://
pipeline UI - http://
grafana UI - http://
You can check current state of installation via command
kubectl -n mlrun get pods, where the main information
is in columns
State. If all images have already been pulled locally, typically it will take
a minute for all services to start.
You can change the ports by providing values to the helm install command.
You can add and configure a k8s ingress-controller for better security and control over external access.
Make sure to save your changes in the
data folder within the Jupyter Lab. The root folder and any other folder do not retain the changes when you restart the Jupyter Lab.
Configuring the remote environment#
You can use your code on a local machine while running your functions on a remote cluster. Refer to Set up your client environment for more information.
Advanced chart configuration#
Configurable values are documented in the
values.yaml, and the
values.yaml of all sub charts. Override those in the normal methods.
Uninstalling the Chart#
helm --namespace mlrun uninstall mlrun-ce
Note on terminating pods and hanging resources#
This chart generates several persistent volume claims that provide persistency (via PVC) out of the box. Upon uninstallation, any hanging / terminating pods will hold the PVCs and PVs respectively, as those prevent their safe removal. Since pods that are stuck in terminating state seem to be a never-ending plague in k8s, note this, and remember to clean the remaining PVs and PVCs.
Handing stuck-at-terminating pods:#
kubectl --namespace mlrun delete pod --force --grace-period=0 <pod-name>
Reclaim dangling persistency resources:#
This will result in data loss!
# To list PVCs $ kubectl --namespace mlrun get pvc ... # To remove a PVC $ kubectl --namespace mlrun delete pvc <pvc-name> ... # To list PVs $ kubectl --namespace mlrun get pv ... # To remove a PVC $ kubectl --namespace mlrun delete pvc <pv-name> ...
Upgrading the chart#
In order to upgrade to the latest version of the chart, first make sure you have the latest helm repo
helm repo update
Then upgrade the chart:
helm upgrade --install --reuse-values mlrun-ce mlrun-ce/mlrun-ce