Built-in steps

Contents

Built-in steps#

A step runs a function or class handler or a REST API call: MLRun comes with pre-built steps that include data manipulation, readers, writers and model serving. All steps are supported by the storey engine. Support by any other engines is included in the step description, as relevant.

You can also write your own steps using standard Python functions or custom functions/classes, or a step can be an external REST API (the special $remote class).

See also Data transformations.

In this section

Base Operators
External IO and data enrichment
Models
Model serving
Other

Base Operators#

Class name	Description
`Batch`	Batches events. This step emits a batch every `max_events` events, or when `timeout` seconds have passed since the first event in the batch was received.
`Choice`	Redirects each input element into one of the multiple downstreams.
`Extend`	Adds fields to each incoming event.
`Filter`	Filters events based on a user-provided function.
`FlatMap`	Maps, or transforms, each incoming event into any number of events.
`Flatten`	Flatten is equivalent to FlatMap(lambda x: x).
`ForEach`	Applies the given function on each event in the stream, and passes the original event downstream.
`MapClass`	Similar to Map, but instead of a function argument, this class should be extended and its do() method overridden.
`MapWithState`	Maps, or transforms, incoming events using a stateful user-provided function, and an initial state, which can be a database table.
`Partition`	Partitions events by calling a predicate function on each event. Each processed event results in a Partitioned named tuple of (left=Optional[Event], right=Optional[Event]).
storey.Reduce	Reduces incoming events into a single value that is returned upon the successful termination of the flow.
`SampleWindow`	Emits a single event in a window of `window_size` events, in accordance with `emit_period` and `emit_before_termination`.

External IO and data enrichment#

Class name	Description
`BatchHttpRequests`	A class for calling remote endpoints in parallel.
`DataItem`	Data input/output class abstracting access to various local/remote data sources.
`JoinWithTable`	Joins each event with data from the given table.
JoinWithV3IOTable	Joins each event with a V3IO table. Used for event augmentation.
`QueryByKey`	Similar to AggregateByKey, but this step is for serving only and does not aggregate the event.
`RemoteStep`	Class for calling remote endpoints.
`SendToHttp`	Joins each event with data from any HTTP source. Used for event augmentation.

Models#

Class name	Description
`ONNXModelServer`	A model serving class for serving ONYX Models. A sub-class of the V2ModelServer class.
`PyTorchModelServer`	A model serving class for serving PyTorch Models. A sub-class of the V2ModelServer class.
`SKLearnModelServer`	A model serving class for serving Sklearn Models. A sub-class of the V2ModelServer class.
`TFKerasModelServer`	A model serving class for serving TFKeras Models. A sub-class of the V2ModelServer class.
`XGBModelServer`	A model serving class for serving XGB Models. A sub-class of the V2ModelServer class.

Model serving#

Class name	Description
`EnrichmentModelRouter`	Auto enrich the request with data from the feature store. The router input accepts a list of inference requests (each request can be a dict or a list of incoming features/keys). It enriches the request with data from the specified feature vector (`feature_vector_uri`).
`EnrichmentVotingEnsemble`	Auto enrich the request with data from the feature store. The router input accepts a list of inference requests (each request can be a dict or a list of incoming features/keys). It enriches the request with data from the specified feature vector (`feature_vector_uri`).
`ModelRouter`	Basic model router, for calling different models per each model path.
`VotingEnsemble`	An ensemble machine learning model that combines the prediction of several models.
`ModelRunnerStep`	Runs multiple models on each event. When used in a graph, MLRun automatically imports the default language model class (LLModel) during function deployment.

Other#

Class name	Description
`FeaturesetValidator`	Validate feature values according to the feature set validation policy. Supported also by the Pandas engines.
`ChoiceByField`	Routes events to downstream steps based on an event field that contains the step name or names.
ReduceToDataFrame	Builds a pandas DataFrame from events and returns that DataFrame on flow termination.