.md .pdf

repository open issue suggest edit

Data stores and feature store

One of the biggest challenge in distributed systems is handling data given the different access methods, APIs, and authentication mechanisms across types and providers.

MLRun provides three main abstractions to access structured and unstructured data:

• Data Store — defines a storage provider (e.g. file system, S3, Azure blob, Iguazio v3io, etc.)

• Data items — represent a data item or collection of such (file, dir, table, etc.)

• Artifacts — Metadata describing one or more data items. see Artifacts.

Working with the abstractions enable you to securely access different data sources through a single API, many continuance methods (e.g. to/from DataFrame, get, download, list, ..), automated data movement, and versioning.

In this section

• Data stores
  • Shared data store
  • Storage credentials and parameters
• Data items
• Feature store
  • Overview
  • How the feature store works
  • Training and serving using the feature store
  • Further reading

Node affinity for MLRun jobs Data stores