.md .pdf

repository open issue suggest edit

Contents

Python API

Module Interface

class meillionen.interface.module_interface.ModuleInterface(classes: List[meillionen.interface.class_interface.ClassInterface])

A collection of class interfaces

__call__(req: meillionen.interface.method_request.MethodRequest)

Call a class method

Parameters

req – a request to a class method

__init__(classes: List[meillionen.interface.class_interface.ClassInterface])

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer)

Builds a module interface from a buffer

serialize(builder: flatbuffers.builder.Builder)

Serialize a module interface into a flatbuffer builder

Parameters

builder – the flatbuffer builder

Class Interface

class meillionen.interface.class_interface.ClassInterface(name, methods: Union[List[meillionen.interface.method_interface.MethodInterface], Dict[str, meillionen.interface.method_interface.MethodInterface]])

A class interface describes the methods available in a model along with the model’s name

__init__(name, methods: Union[List[meillionen.interface.method_interface.MethodInterface], Dict[str, meillionen.interface.method_interface.MethodInterface]])

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer)

Builds a ClassInterface from a buffer

classmethod from_interface(interface: meillionen.interface.class_interface._ClassInterface)

Builds a ClassInterface from a flatbuffer _ClassInterface

get_method(name: str) → meillionen.interface.method_interface.MethodInterface

Retrieve a method from a class given its name

property name

Returns

the name of the model

serialize(builder: flatbuffers.builder.Builder)

Serialized a class interface into a flatbuffer builder

Method Interface

class meillionen.interface.method_interface.MethodInterface(name, args: List[Any], handler=<function default_handler>)

A description of a method. It has a name, arguments (with schema information) and a handler (which runs the method)

__call__(**kwargs)

Call self as a function.

__init__(name, args: List[Any], handler=<function default_handler>)

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer)

Builds a method interface from a buffer

process_kwargs(kwargs: Dict[str, Any])

Preprocesses resource payloads to reduce boilerplate code in the handler

serialize(builder: flatbuffers.builder.Builder)

Serializes a method interface into a flatbuffer builder

Method Request

class meillionen.interface.method_request.MethodRequest(class_name, method_name, kwargs)

A request to call a model method

__init__(class_name, method_name, kwargs)

Initialize self. See help(type(self)) for accurate signature.

classmethod from_partial(settings: meillionen.settings.Settings, class_name: str, method_name: str, resource_payloads, partition)

settings:

get_arg(name)

Get the path metadata for a argument

Parameters

name – the name of the argument

serialize(builder: flatbuffers.builder.Builder)

Serialize a method request into a flatbuffer builder

Parameters

builder – the flatbuffer builder

Schemas

class meillionen.interface.schema.DataFrameSchemaPayload(arrow_schema: pyarrow.lib.Schema)

__init__(arrow_schema: pyarrow.lib.Schema)

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer: _io.BytesIO)

Builds the schema payload from a buffer

Parameters

buffer – the buffer to load from

serialize(builder: flatbuffers.builder.Builder)

Serializes the schema payload into a flatbuffer builder

Parameters

builder – the flatbuffer builder

class meillionen.interface.schema.DirHandler(name: str, mutability: meillionen.interface.mutability.Mutability)

__init__(name: str, mutability: meillionen.interface.mutability.Mutability)

Initialize self. See help(type(self)) for accurate signature.

classmethod from_schema(schema: meillionen.interface.schema.Schema)

Builds the handler from the schema

Parameters

schema – the schema to build the handler with

load(resource)

Loads a resource into memory

Parameters

resource – the resource to load

serialize(builder: flatbuffers.builder.Builder)

Serialize a handler into a flatbuffer builder using only schema information

class meillionen.interface.schema.FileExtensionHandlerMapper

A handler mapper that inspects a resources file extension to determine what handler should wrap a particular schema

__init__()

Initialize self. See help(type(self)) for accurate signature.

class meillionen.interface.schema.Handlable(*args, **kwargs)

abstract classmethod from_schema(schema: meillionen.interface.schema.Schema) → _H

Builds the handler from the schema

Parameters

schema – the schema to build the handler with

abstract load(resource)

Loads a resource into memory

Parameters

resource – the resource to load

abstract save(resource, data)

Save data using the metadata provided by the resource

Parameters

• resource – the metadata desribing where and how to save the data

• data – the data to save

abstract serialize(builder: flatbuffers.builder.Builder)

Serialize a handler into a flatbuffer builder using only schema information

class meillionen.interface.schema.LandLabGridHandler(name: str, data_type: str, mutability: meillionen.interface.mutability.Mutability)

__init__(name: str, data_type: str, mutability: meillionen.interface.mutability.Mutability)

Initialize self. See help(type(self)) for accurate signature.

classmethod from_schema(schema: meillionen.interface.schema.Schema)

Builds the handler from the schema

Parameters

schema – the schema to build the handler with

serialize(builder: flatbuffers.builder.Builder)

Serialize a handler into a flatbuffer builder using only schema information

class meillionen.interface.schema.NetCDFHandler(name: str, data_type: str, dimensions: List[str], mutability: meillionen.interface.mutability.Mutability)

__init__(name: str, data_type: str, dimensions: List[str], mutability: meillionen.interface.mutability.Mutability)

Initialize self. See help(type(self)) for accurate signature.

classmethod from_schema(schema: meillionen.interface.schema.Schema)

Builds the handler from the schema

Parameters

schema – the schema to build the handler with

serialize(builder: flatbuffers.builder.Builder)

Serialize a handler into a flatbuffer builder using only schema information

class meillionen.interface.schema.NetCDFSliceHandler(name: str, data_type: str, dimensions: List[str], mutability: meillionen.interface.mutability.Mutability)

__init__(name: str, data_type: str, dimensions: List[str], mutability: meillionen.interface.mutability.Mutability)

Initialize self. See help(type(self)) for accurate signature.

classmethod from_schema(schema: meillionen.interface.schema.Schema)

Builds the handler from the schema

Parameters

schema – the schema to build the handler with

serialize(builder: flatbuffers.builder.Builder)

Serialize a handler into a flatbuffer builder using only schema information

class meillionen.interface.schema.PandasHandler(name: str, s: pyarrow.lib.Schema, mutability: meillionen.interface.mutability.Mutability)

__init__(name: str, s: pyarrow.lib.Schema, mutability: meillionen.interface.mutability.Mutability)

Initialize self. See help(type(self)) for accurate signature.

classmethod from_schema(schema: meillionen.interface.schema.Schema)

Builds the handler from the schema

Parameters

schema – the schema to build the handler with

serialize(builder: flatbuffers.builder.Builder)

Serialize a handler into a flatbuffer builder using only schema information

class meillionen.interface.schema.SchemaPayloadable(*args, **kwargs)

A schema payload describes the shape of data a method argument expects

abstract classmethod deserialize(buffer) → _SP

Builds the schema payload from a buffer

Parameters

buffer – the buffer to load from

abstract serialize(builder: flatbuffers.builder.Builder)

Serializes the schema payload into a flatbuffer builder

Parameters

builder – the flatbuffer builder

class meillionen.interface.schema.SchemalessPayload

__init__()

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer: _io.BytesIO)

Builds the schema payload from a buffer

Parameters

buffer – the buffer to load from

serialize(builder: flatbuffers.builder.Builder)

Serializes the schema payload into a flatbuffer builder

Parameters

builder – the flatbuffer builder

class meillionen.interface.schema.TensorSchemaPayload(data_type: str, dimensions: List[str])

__init__(data_type: str, dimensions: List[str])

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer: _io.BytesIO)

Builds the schema payload from a buffer

Parameters

buffer – the buffer to load from

serialize(builder: flatbuffers.builder.Builder)

Serializes the schema payload into a flatbuffer builder

Parameters

builder – the flatbuffer builder

meillionen.interface.schema.get_handler(resource_payload, schema: meillionen.interface.schema.Schema)

Retrieve a handler given a resource payload and a schema

Resources

class meillionen.interface.resource.Feather(path)

A Feather file resource payload

__init__(path)

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer)

Builds the resource payload from a buffer

classmethod from_kwargs(kwargs, settings, mra: meillionen.interface.base.MethodRequestArg, partition=None)

Builds a resource payload

Parameters

• kwargs – the arguments to pass to the classes initializer

• settings – the common settings used to complete the partial resource payload

• mra – the metadata describing which argument of a method is being built

• partition – additional context used to build the resource for grid searches and sensitivity analysis

serialize(builder: flatbuffers.builder.Builder)

Serializes the resource payload into the flatbuffer builder

Parameters

builder – the flatbuffer builder

class meillionen.interface.resource.NetCDF(path, variable)

A NetCDF file resource payload

__init__(path, variable)

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer)

Builds the resource payload from a buffer

classmethod from_kwargs(kwargs, settings, mra: meillionen.interface.base.MethodRequestArg, partition=None)

Builds a resource payload

Parameters

• kwargs – the arguments to pass to the classes initializer

• settings – the common settings used to complete the partial resource payload

• mra – the metadata describing which argument of a method is being built

• partition – additional context used to build the resource for grid searches and sensitivity analysis

serialize(builder: flatbuffers.builder.Builder)

Serializes the resource payload into the flatbuffer builder

Parameters

builder – the flatbuffer builder

class meillionen.interface.resource.OtherFile(path)

A file payload resource payload

Together with a name a full resource can be created

__init__(path)

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer)

Builds the resource payload from a buffer

classmethod from_kwargs(kwargs, settings, mra: meillionen.interface.base.MethodRequestArg, partition)

Builds a resource payload

Parameters

• kwargs – the arguments to pass to the classes initializer

• settings – the common settings used to complete the partial resource payload

• mra – the metadata describing which argument of a method is being built

• partition – additional context used to build the resource for grid searches and sensitivity analysis

serialize(builder: flatbuffers.builder.Builder)

Serializes the resource payload into the flatbuffer builder

Parameters

builder – the flatbuffer builder

class meillionen.interface.resource.Parquet(path)

A Parquet file resource payload

__init__(path)

Initialize self. See help(type(self)) for accurate signature.

classmethod deserialize(buffer)

Builds the resource payload from a buffer

classmethod from_kwargs(kwargs, settings, mra: meillionen.interface.base.MethodRequestArg, partition=None)

Builds a resource payload

Parameters

• kwargs – the arguments to pass to the classes initializer

• settings – the common settings used to complete the partial resource payload

• mra – the metadata describing which argument of a method is being built

• partition – additional context used to build the resource for grid searches and sensitivity analysis

serialize(builder: flatbuffers.builder.Builder)

Serializes the resource payload into the flatbuffer builder

Parameters

builder – the flatbuffer builder

class meillionen.interface.resource.Resource(name: str, resource_payload)

A serializable resource. Should only be used internally.

__init__(name: str, resource_payload)

Initialize self. See help(type(self)) for accurate signature.

class meillionen.interface.resource.ResourcePayloadable(*args, **kwargs)

The interface of a resource

classmethod deserialize(buffer) → _R

Builds the resource payload from a buffer

classmethod from_kwargs(kwargs, settings: meillionen.settings.Settings, mra: meillionen.interface.base.MethodRequestArg, partition: Optional[meillionen.settings.Partition]) → _R

Builds a resource payload

Parameters

• kwargs – the arguments to pass to the classes initializer

• settings – the common settings used to complete the partial resource payload

• mra – the metadata describing which argument of a method is being built

• partition – additional context used to build the resource for grid searches and sensitivity analysis

serialize(builder: flatbuffers.builder.Builder)

Serializes the resource payload into the flatbuffer builder

Parameters

builder – the flatbuffer builder

Response

class meillionen.response.Response(handlers, resources)

Response to a request from the client to call a method

__init__(handlers, resources)

Initialize self. See help(type(self)) for accurate signature.

handler(name: str)

Retrieve a handler

Parameters

name – the name of the handler

load(name: str)

Loads a resource using its handler

Parameters

name – the name of the resource and handler

resource(name: str)

Retrieve a resource

Parameters

name – the name of the resource

save(name: str, data)

Saves a resource using its handler

Parameters

• name – the name of the resource and handler

• data – the data to save

Settings

class meillionen.settings.Partitioning(schema: pyarrow.lib.Schema)

Describes additional data to be used to build up partial resource payload objects.

Commonly used when broadcasting a model over a grid with different resources available at each grid point. Also useful for sensitivity analysis.

__init__(schema: pyarrow.lib.Schema)

Initialize self. See help(type(self)) for accurate signature.

complete(**kwargs)

Build a partition object with the data filled in for each missing partition key

to_arrow()

Convert the partition into a pyarrow partition so that final results can be loaded as partitioned parquet files (which are supported by pyarrow, spark, datafusion and other libraries)

class meillionen.settings.Settings(*, base_path: str)

Settings for a module.

Currently used only to turn partial resource payloads into resource payloads. In the future it may have information about logging and how to access particular common resources (such as databases).

trial(name: str)

Used to build ‘trial’ settings objects which extend the base path of the settings object.

Client

class meillionen.client.CLIRef(path)

A CLI reference to a model

The cli program must conform to the Meillionen cli interface

__init__(path)

Initialize self. See help(type(self)) for accurate signature.

class meillionen.client.Client(module_ref, settings=None)

A client to execute and inspect Meillionen models

__init__(module_ref, settings=None)

Initialize self. See help(type(self)) for accurate signature.

run(class_name, method_name, resource_payloads, partition: Optional[meillionen.settings.Partition] = None)

Runs a particular class method on the module

Parameters

• class_name – the name of the class

• method_name – the name of the method to call

• resource_payloads – dict of argument names to resource payloads

• partition – an optional partition which describes how to build nested paths, to add key columns to tabular data or to add dimensions to tensor data

save(mra: meillionen.interface.base.MethodRequestArg, resource, data, partition=None)

Saves a resource

Parameters

• mra – metadata needed to locate the schema of a method argument

• resource – the resource metadata describing how and where to save the data to

• data – the data to save

• partition – the optional partition information. Useful for evaluating a model at many different parameter combinations

class meillionen.client.DockerImageRef(image_name: str)

A Docker Image reference to a model

The docker image must have an entrypoint of a Meillionen cli program and have a working directory of /code in order to work

__init__(image_name: str)

Initialize self. See help(type(self)) for accurate signature.

class meillionen.client.ServerRef(server: meillionen.server.Server)

A server reference to a model

This allows running a model in the same Python process.

Mostly suited for interactive experimentation and model testing.

__init__(server: meillionen.server.Server)

Initialize self. See help(type(self)) for accurate signature.

Server

class meillionen.server.Server(module: meillionen.interface.module_interface.ModuleInterface)

A class that handles method requests from clients

Currently this is only for handling requests from stdin but in the future it may handle requests over http2.

__init__(module: meillionen.interface.module_interface.ModuleInterface)

Initialize self. See help(type(self)) for accurate signature.

cli()

Returns an argparse cli to access the model with

previous

Getting Started

next

Examples