mlrun.package.packagers.default_packager.DefaultPackager

mlrun.package.packagers.default_packager.DefaultPackager#

class mlrun.package.packagers.default_packager.DefaultPackager[source]#

Bases: Packager

A default packager that handles all types and packs them as pickle files.

The default packager implements all the required methods (except for a bundle-supported packager that should still implement both bundle and unbundle methods) has a default logic that should satisfy most use cases. To work with this class, don't override the abstract class methods, but instead follow the guidelines below:

The class variable PACKABLE_OBJECT_TYPE: The type of object this packager can pack and unpack (used in the is_packable method).
The class variable PACK_SUBCLASSES: A flag that indicates whether to pack all subclasses of the PACKABLE_OBJECT_TYPE (used in the is_packable method). Default is False.
The class variable DEFAULT_PACKING_ARTIFACT_TYPE: The default artifact type to pack as. It is returned from the method get_default_packing_artifact_type.
The class variable DEFAULT_UNPACKING_ARTIFACT_TYPE: The default artifact type to unpack from. It is returned from the method get_default_unpacking_artifact_type.
The class variable BUNDLE_FROM_LIST: A flag that indicates whether the PACKABLE_OBJECT_TYPE can be initialized from a list to be used as a collection bundle. It is used in the can_bundle method. Default is False.
The class variable BUNDLE_FROM_DICT: A flag that indicates whether the PACKABLE_OBJECT_TYPE can be initialized from a dict to be used as a collection bundle. It is used in the can_bundle method. Default is False.
The abstract class method pack(): This method is implemented to get the object and send it to the relevant packing method by the given artifact type using the following naming: "pack_<artifact_type>". (If the artifact type was not provided, it uses the default). For example: if the artifact type is x then the class method pack_x must be implemented. The signature of each pack class method must be:
```
def pack_x(self, obj: Any, key: str, ...) -> Union[Tuple[Artifact, dict], dict]:
    pass
```
Where 'x' is the artifact type, 'obj' is the object to pack, key is the key to name the artifact and ... are additional, custom, log hint configurations. The returned values are the packed artifact and the instructions for unpacking it, or in the case of result, the dictionary of the result with its key and value. configurations are sent by the user and shouldn't be mandatory, meaning they should have a default value (otherwise, the user has to add them to every log hint).
The abstract class method unpack(): The method is implemented to get a DataItem and send it to the relevant unpacking method by the artifact type using the following naming: "unpack_<artifact_type>". (If the artifact type was not provided, it uses the default). For example: if the artifact type stored within the DataItem is x then the class method unpack_x must be implemented. The signature of each unpack class method must be:
```
def unpack_x(self, data_item: mlrun.DataItem, ...) -> Any:
    pass
```
Where 'x' is the artifact type, 'data_item' is the artifact's data item to unpack, ... are the instructions that were originally returned from pack_x. (Each instruction must be optional (have a default value) to support objects from this type that were not packaged but custom-logged.) The returned value is the unpacked object.
The abstract class method is_packable(): The method is implemented to automatically validate
the object type and artifact type by the following rules:
- Object type validation: Checks if the given object type matches the class variable PACKABLE_OBJECT_TYPE with respect to the PACK_SUBCLASSES class variable.
- Artifact type validation: Checks if the given artifact type is in the list returned from get_supported_packing_artifact_types.
The abstract class method is_unpackable(): The method is left as implemented in Packager.
The abstract class method can_bundle(): The method is implemented to automatically check
the bundle type and collection type by the following rules:
- Bundle type validation: Checks if the bundle type to initialize matches the class variable PACKABLE_OBJECT_TYPE with respect to the PACK_SUBCLASSES class variable.
- Collection type validation: Checks if the given collection type appears as True in the matching flags: BUNDLE_FROM_LIST or BUNDLE_FROM_DICT.
Remember, to have a packager that supports bundles, you must also implement the methods bundle() and unbundle().
The abstract class method can_unbundle(): The method is implemented to automatically checks if the packager can be used as a bundle (either class variables BUNDLE_FROM_LIST or BUNDLE_FROM_DICT are true) and then checks that the bundle type matches the class variable PACKABLE_OBJECT_TYPE with respect to the PACK_SUBCLASSES class variable.

Remember, to have a packager that supports bundles, you must also implement the methods bundle() and unbundle().
The abstract class method get_supported_artifact_types(): The method is implemented to return the union of get_supported_packing_artifact_types and get_supported_unpacking_artifact_types.
- The class method get_supported_packing_artifact_types(): Scans for pack_* methods to discover artifact types available for packing.
- The class method get_supported_unpacking_artifact_types(): Scans for unpack_* methods to discover artifact types available for unpacking. A pack_* method without a matching unpack_* method means that artifact type is pack-only.
The abstract class method get_default_packing_artifact_type(): The method is implemented to return the new class variable DEFAULT_PACKING_ARTIFACT_TYPE. You can still override the method if the default artifact type you need could change according to the object that's about to be packed.
The abstract class method get_default_unpacking_artifact_type(): The method is implemented to return the new class variable DEFAULT_UNPACKING_ARTIFACT_TYPE. You can still override the method if the default artifact type you need could change according to the data item that's about to be unpacked.

Important to remember

From the Packager docstring:

Bundles: A bundle means the type of object handled by this packager can be used to hold a collection of other objects - like a list or a dict of packages. A bundle can be sent as a list or dict in a function's run input so the packager manager will receive a list or dictionary of data items. A packager that support bundles means it can initialize an object that will hold the unpacked data items later on - based on the type hint the user required.

A packager can be a bundle if it implements the mandatory methods can_bundle(), can_unbundle(), and the methods: bundle() and unbundle().
Linking artifacts ("extra data" and "metrics" (for models)): In order to link between packages (using the extra data or metrics spec attributes of an artifact), use the key as if it exists and as value ellipses (...). The manager links all packages once it is done packing.
Clearing outputs: Some packagers may produce files and temporary directories that should be deleted after the artifact is logged. The packager can mark paths of files and directories to delete after logging using the class method add_future_clearing_path.

Packager Summary

Packing Type: Any type

Packing Sub-Classes: False

Priority: Default priority (5)

Default Artifact Types:

Packing: object
Unpacking: object

Packing Artifact Types:

object - Pack a python object, pickling it into a pkl file and store it in an artifact.
- pickle_module_name - The pickle module name to use for serializing the object.
result - Pack an object as a result.

Unpacking Artifact Types:

object - Unpack the data item's object, unpickle it using the instructions, and return.Warnings of mismatching python and module versions between the original pickling interpreter and this one may beraised.
- pickle_module_name - Module to use for unpickling the object.
- object_module_name - The original object's module. Used to verify that the current interpreter objectmodule version matches the pickled object version before unpickling the object.
- python_version - The python version in which the original object was pickled. Used to verify thatthe current interpreter python version matches the pickled object version beforeunpickling the object.
- pickle_module_version - The pickle module version. Used to verify that the current interpreter moduleversion matches the one that pickled the object before unpickling it.
- object_module_version - The original object's module version to match to the interpreter's module version.

Attributes

`BUNDLE_FROM_DICT`	Whether the PACKABLE_OBJECT_TYPE can be used as a bundle and be initialized from a dictionary.
`BUNDLE_FROM_LIST`	Whether the PACKABLE_OBJECT_TYPE can be used as a bundle and be initialized from a list.
`DEFAULT_PACKING_ARTIFACT_TYPE`	The default artifact type to pack as.
`DEFAULT_UNPACKING_ARTIFACT_TYPE`	The default artifact type to unpack from.
`PACKABLE_OBJECT_TYPE`	The type of object this packager can pack and unpack.
`PACK_SUBCLASSES`	Whether to also pack all subclasses of the PACKABLE_OBJECT_TYPE.
`PRIORITY`	The priority of this packager in the packagers collection of the manager (lower is better).
`future_clearing_path_list`	Get the packager's future clearing path list.
`priority`	Get the packager's priority.

BUNDLE_FROM_DICT = False#: Whether the PACKABLE_OBJECT_TYPE can be used as a bundle and be initialized from a dictionary.

BUNDLE_FROM_LIST = False#: Whether the PACKABLE_OBJECT_TYPE can be used as a bundle and be initialized from a list.

DEFAULT_PACKING_ARTIFACT_TYPE = 'object'#: The default artifact type to pack as.

DEFAULT_UNPACKING_ARTIFACT_TYPE = 'object'#: The default artifact type to unpack from.

PACKABLE_OBJECT_TYPE: type = Ellipsis#: The type of object this packager can pack and unpack.

PACK_SUBCLASSES = False#: Whether to also pack all subclasses of the PACKABLE_OBJECT_TYPE.

PRIORITY: int = Ellipsis#: The priority of this packager in the packagers collection of the manager (lower is better).

future_clearing_path_list#

Get the packager's future clearing path list.

Returns:: The clearing path list.

priority#

Get the packager's priority.

Returns:: The packager's priority.

Methods

`__init__`()
`add_future_clearing_path`(path)	Mark a path to be cleared by this packager's manager after logging the packaged artifacts.
`bundle`(collection)	Initialize a bundle object with the collection given using this packager.
`can_bundle`(bundle_hint, collection_type)	Check if the packager can be used to initialize a bundle (a collection of packages) of the required type with the provided collection type.
`can_unbundle`(bundled_object)	Check if the packager can unbundle a bundled object of the provided type.
`get_data_item_local_path`(data_item[, ...])	Get the local path to the item handled by the data item provided.
`get_default_packing_artifact_type`(obj)	Get the default artifact type for packing an object of this packager.
`get_default_unpacking_artifact_type`(data_item)	Get the default artifact type used for unpacking a data item holding an object of this packager.
`get_supported_artifact_types`()	Get all the supported artifact types on this packager (union of packing and unpacking).
`get_supported_packing_artifact_types`()	Get the supported artifact types for packing by scanning for `pack_*` methods.
`get_supported_unpacking_artifact_types`()	Get the supported artifact types for unpacking by scanning for `unpack_*` methods.
`is_packable`(obj[, artifact_type, configurations])	Check if this packager can pack an object of the provided type as the provided artifact type.
`is_unpackable`(data_item, type_hint[, ...])	Check if this packager can unpack an input according to the user-given type hint and the provided artifact type.
`pack`(obj[, key, artifact_type, configurations])	Pack an object as the given artifact type using the provided configurations.
`pack_object`(obj, key[, pickle_module_name])	Pack a python object, pickling it into a pkl file and store it in an artifact.
`pack_result`(obj, key)	Pack an object as a result.
`unbundle`(bundled_object)	Unbundle the given object into a collection of objects (for later pack them each separately).
`unpack`(data_item[, artifact_type, instructions])	Unpack the data item's artifact by the provided type using the given instructions.
`unpack_object`(data_item[, ...])	Unpack the data item's object, unpickle it using the instructions, and return.

__init__()#

add_future_clearing_path(path: str | Path)#

Mark a path to be cleared by this packager's manager after logging the packaged artifacts.

Parameters:: path -- The path to clear post logging the artifacts.

bundle(collection: dict | list) → Any#

Initialize a bundle object with the collection given using this packager.

Parameters:: collection -- The collection of objects to bundle.
Returns:: The bundled object.
Raises:: NotImplementedError -- In case the packager does not support bundling.

can_bundle(bundle_hint: type, collection_type: type[dict] | type[list]) → bool[source]#

Check if the packager can be used to initialize a bundle (a collection of packages) of the required type with the provided collection type.

The method is implemented to validate the bundle type by checking if the given type matches the variable PACKABLE_OBJECT_TYPE with respect to the PACK_SUBCLASSES class variable. If it does, it checks if the given collection type's flag is set (either BUNDLE_FROM_LIST or BUNDLE_FROM_DICT).

Parameters:

bundle_hint -- The bundle type hint to check if the PACKABLE_OBJECT_TYPE matches to.
collection_type -- The available collection type that will be used in the bundle type's constructor.

Returns:

True if it can be used as a bundle and False otherwise.

can_unbundle(bundled_object: Any)[source]#

Check if the packager can unbundle a bundled object of the provided type.

The method is implemented to automatically checks if the packager can be used as a bundle (either class variables BUNDLE_FROM_LIST or BUNDLE_FROM_DICT are true) and then checks that the bundle type matches the class variable PACKABLE_OBJECT_TYPE with respect to the PACK_SUBCLASSES class variable.

Parameters:: bundled_object -- The bundled object to check.
Returns:: True if it can unbundle and False otherwise.

get_data_item_local_path(data_item: DataItem, add_to_future_clearing_path: bool | None = None) → str#

Get the local path to the item handled by the data item provided. The local path can be the same as the data item in case the data item points to a local path, or will be downloaded to a temporary directory and return this newly created temporary local path.

Parameters:

data_item -- The data item to get its item local path.
add_to_future_clearing_path -- Whether to add the local path to the future clearing paths list. If None, it will add the path to the list only if the data item is not of kind 'file', meaning it represents a local file and hence we don't want to delete it post running automatically. We wish to delete it only if the local path is temporary (and that will be in case kind is not 'file', so it is being downloaded to a temporary directory).

Returns:

The data item local path.

get_default_packing_artifact_type(obj: Any) → str[source]#

Get the default artifact type for packing an object of this packager.

Parameters:: obj -- The about-to-be packed object.
Returns:: The default artifact type.

get_default_unpacking_artifact_type(data_item: DataItem) → str[source]#

Get the default artifact type used for unpacking a data item holding an object of this packager. The method is used when a data item is sent for unpacking without it being a package, but is a simple url or an old / manually logged artifact.

Parameters:: data_item -- The about-to-be unpacked data item.
Returns:: The default artifact type.

get_supported_artifact_types() → list[str][source]#

Get all the supported artifact types on this packager (union of packing and unpacking).

Returns:: A list of all the supported artifact types.

get_supported_packing_artifact_types() → list[str][source]#

Get the supported artifact types for packing by scanning for pack_* methods.

Returns:: A list of artifact types discovered from pack_* method names.

get_supported_unpacking_artifact_types() → list[str][source]#

Get the supported artifact types for unpacking by scanning for unpack_* methods.

Returns:: A list of artifact types discovered from unpack_* method names.

is_packable(obj: Any, artifact_type: str | None = None, configurations: dict | None = None) → bool[source]#

Check if this packager can pack an object of the provided type as the provided artifact type.

The method is implemented to validate the object's type and artifact type by checking if the given object type matches the variable PACKABLE_OBJECT_TYPE with respect to the PACK_SUBCLASSES class variable. If it does, it checks if the given artifact type is in the list returned from get_supported_packing_artifact_types.

Parameters:

obj -- The object to pack.
artifact_type -- The artifact type to log the object as.
configurations -- The log hint configurations passed by the user.

Returns:

True if packable and False otherwise.

is_unpackable(data_item: DataItem, type_hint: type, artifact_type: str | None = None) → bool#

Check if this packager can unpack an input according to the user-given type hint and the provided artifact type.

The default implementation tries to match the packable object type of this packager to the given type hint. If it matches, it looks for the artifact type in the list returned from get_supported_unpacking_artifact_types.

Parameters:

data_item -- The input data item to check if unpackable.
type_hint -- The type hint of the input to unpack (the object type to be unpacked).
artifact_type -- The artifact type to unpack the object as.

Returns:

True if unpackable and False otherwise.

pack(obj: Any, key: str | None = None, artifact_type: str | None = None, configurations: dict | None = None) → tuple[Artifact, dict] | dict[source]#

Pack an object as the given artifact type using the provided configurations.

Parameters:

obj -- The object to pack.
key -- The key of the artifact.
artifact_type -- Artifact type to log to MLRun. If passing None, the default artifact type is used.
configurations -- Log hints configurations to pass to the packing method.

Returns:

If the packed object is an artifact, a tuple of the packed artifact and unpacking instructions dictionary. If the packed object is a result, a dictionary containing the result key and value.

pack_object(obj: Any, key: str, pickle_module_name: str = 'cloudpickle') → tuple[Artifact, dict][source]#

Pack a python object, pickling it into a pkl file and store it in an artifact.

Parameters:

obj -- The object to pack and log.
key -- The artifact's key.
pickle_module_name -- The pickle module name to use for serializing the object.

Returns:

The artifacts and its pickling instructions.

pack_result(obj: Any, key: str) → dict[source]#

Pack an object as a result.

Parameters:

obj -- The object to pack and log.
key -- The result's key.

Returns:

The result dictionary.

unbundle(bundled_object: Any) → dict | list#

Unbundle the given object into a collection of objects (for later pack them each separately).

Returns:: The unbundled collection of objects - a list or dict.
Raises:: NotImplementedError -- In case the packager does not support bundling.

unpack(data_item: DataItem, artifact_type: str | None = None, instructions: dict | None = None) → Any[source]#

Unpack the data item's artifact by the provided type using the given instructions.

Parameters:

data_item -- The data input to unpack.
artifact_type -- The artifact type to unpack the data item as. If passing None, the default artifact type is used.
instructions -- Additional instructions noted in the package to pass to the unpacking method.

Returns:

The unpacked data item's object.

Raises:

MLRunPackageUnpackingError -- In case the packager could not unpack the data item.

unpack_object(data_item: DataItem, pickle_module_name: str = 'cloudpickle', object_module_name: str | None = None, python_version: str | None = None, pickle_module_version: str | None = None, object_module_version: str | None = None) → Any[source]#

Unpack the data item's object, unpickle it using the instructions, and return.

Warnings of mismatching python and module versions between the original pickling interpreter and this one may be raised.

Parameters:

data_item -- The data item holding the pkl file.
pickle_module_name -- Module to use for unpickling the object.
object_module_name -- The original object's module. Used to verify that the current interpreter object module version matches the pickled object version before unpickling the object.
python_version -- The python version in which the original object was pickled. Used to verify that the current interpreter python version matches the pickled object version before unpickling the object.
pickle_module_version -- The pickle module version. Used to verify that the current interpreter module version matches the one that pickled the object before unpickling it.
object_module_version -- The original object's module version to match to the interpreter's module version.

Returns:

The un-pickled python object.

mlrun.package.packagers.default_packager.DefaultPackager

Contents

mlrun.package.packagers.default_packager.DefaultPackager#