mlrun.datastore

class mlrun.datastore.DataItem(key: str, store: mlrun.datastore.base.DataStore, subpath: str, url: str = '', meta=None, artifact_url=None)[source]

Bases: object

Data input/output class abstracting access to various local/remote data sources

property artifact_url

DataItem artifact url (when its an artifact) or url for simple dataitems

as_df(columns=None, df_module=None, format='', **kwargs)[source]

return a dataframe object (generated from the dataitem).

Parameters
  • columns – optional, list of columns to select

  • df_module – optional, dataframe class (e.g. pd, dd, cudf, ..)

  • format – file format, if not specified it will be deducted from the suffix

download(target_path)[source]

download to the target dir/path

get(size=None, offset=0)[source]

read all or a range and return the content

property key

DataItem key

property kind

DataItem store kind (file, s3, v3io, ..)

listdir()[source]

return a list of child file names

local()[source]

get the local path of the file, download to tmp first if its a remote object

ls()[source]

return a list of child file names

property meta

Artifact Metadata, when the DataItem is read from the artifacts store

open(mode)[source]

return fsspec file handler, if supported

put(data, append=False)[source]

write/upload the data, append is only supported by some datastores

stat()[source]

return FileStats class (size, modified, content_type)

property store

DataItem store object

property suffix

DataItem suffix (file extension) e.g. ‘.png’

upload(src_path)[source]

upload the source file (src_path)

property url

//bucket/path

Type

DataItem url e.g. /dir/path, s3

mlrun.datastore.get_store_resource(uri, db=None, secrets=None, project=None)[source]

get store resource object by uri