arrow
v0.1.0

View Source arrow_offsets (arrow v0.1.0)

Provides support for Apache Arrow's Offsets

Arrow has a concept of offsets[1] in order to tell the length of a slot[2] or a single element in an array of variable-size elements. This module provides support for generating offsets.

There are couple of things to remember about offsets:

  1. Each element in the offsets coresponds to the distance in bytes of the coresponding element in the values from the beginning of the buffer.

    I.E., distance of values[j] from beginning of the buffer = offsets[j].
  2. The very last element in the offsets is the length of the buffer, or distance of the end of the last slot from the beginning of the buffer.

  3. Thus, the offsets is one element longer than the values, or:

    length(offsets) == length(values) + 1

  4. Therefore, in order to find the length of a slot, we subtract the offset the current element from the offset of the next element, or:

    slot[j] = offsets[j + 1] - offsets[j]
  5. Null values have an offset of 0 as they take no memory in the buffer. Thus, the previous offset and the current offset are equivalent if the current element is a null.

[1]: https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout

[2]: https://arrow.apache.org/docs/format/Columnar.html#terminology

Link to this section Summary

Functions

new(Value, Type)
Returns the offsets array given some values and their type as a buffer.
new(Value, Type, Length)
Returns the offsets array given some values, their type and length as a buffer.

Link to this section Functions

Link to this function

new(Value, Type)

View Source 
-spec new(Value :: [arrow_type:native_type()], Type :: arrow_type:arrow_longhand_type()) ->
       Buffer :: #buffer{}.
Returns the offsets array given some values and their type as a buffer.
Link to this function

new(Value, Type, Length)

View Source 
-spec new(Value :: [arrow_type:native_type()],
    Type :: arrow_type:arrow_longhand_type(),
    Length :: pos_integer()) ->
       Buffer :: #buffer{}.
Returns the offsets array given some values, their type and length as a buffer.