View Source arrow_ipc_message (arrow v0.1.0)
Provides a record and functions to deal with the Encapsulated Message Format.
A Message is a serialized form of a Schema[1] or a RecordBatch[2] (which may be required to read a serialized array) along with some metadata. This module provides a record and a function to manage all the metadata required to represent a message. Metadata such as:
version: The Apache Arrow Format Version. One of v1..v5. Defaults to v5.header: The metadata of the Schema or RecordBatchbody_length: The length of the body in bytescustom_metadata: A list of custom metadata in key-value formatbody: The actual body. Can be undefined (in the case of Schema) or a binary (in the case of Record Batch).
Currently, changing the version and custom metadata are not supported, but they have been added for forwards comapatibility.
This module also provides the to_ipc/1 function which serializes the message into the Encapsulated Message Format[3]. However, this function gives incomplete output with invalid metadata, because of an unsatisfied dependency on flatbuffers, which is required for serializing the metadata.
[1]: https://arrow.apache.org/docs/format/Columnar.html#schema-message
[2]: https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message
[3]: https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-formatLink to this section Summary
Types
Functions
Returns the body of message from a list of arrays.
Link to this section Types
-type key_value() :: #{key => string(), value => string()}.
-type metadata_version() :: v1 | v2 | v3 | v4 | v5.
Link to this section Functions
-spec body_from_erlang(Columns :: [#array{}]) -> Body :: binary().
Returns the body of message from a list of arrays.
Shorthand for: <<<<(arrow_array:to_arrow(Array))/binary>> || Array <- Columns>>-spec from_erlang(Header :: #schema{} | #record_batch{}) -> Message :: #message{}.
-spec from_erlang(Header :: #schema{} | #record_batch{}, Body :: binary()) -> Message :: #message{}.
-spec metadata_len(EMF :: binary()) -> MetadataLen :: non_neg_integer().
-spec to_ipc(Message :: #message{}) -> EMF :: binary().
-spec to_stream(Messages :: [#message{}] | [binary()]) -> Stream :: binary().