The Ion 1.1 Draft Specification

System Macros

Many of the system macros MAY be defined as template macros, and when possible, the specification includes a template. Templates are given here as normative example, but system macros are not required to be implemented as template macros.

The macros that can be defined as templates are included as system macros because of their broad applicability, and so that Ion implementations can provide optimizations for these macros that run directly in the implementations runtime environment rather than in the macro evaluator. For example, a macro such as add_symbols does not produce user values, so an Ion Reader could bypass evaluating the template and directly update the encoding context with the new symbols.

Stream Constructors

none

(macro none () (.values))

none accepts no values and produces nothing (an empty stream).

values

(macro values (v*) v)

This is, essentially, the identity function. It produces a stream from any number of arguments, concatenating the streams produced by the nested expressions. Used to aggregate multiple values or sub-streams to pass to a single argument, or to produce multiple results.

default

(macro default (expr* default_expr*)
    // If `expr` is empty...
    (.if_none (%expr)
        // then expand `default_expr` instead.
        (%default_expr)
        // If it wasn't empty, then expand `expr`.
        (%expr)
    )
)

default tests expr to determine whether it expands to the empty stream. If it does not, default will produce the expansion of expr. If it does, default will produce the expansion of default_expr instead.

flatten

(macro flatten (sequence*) /* Not representable in TDL */)

The flatten system macro constructs a stream from the content of one or more sequences.

Produces a stream with the contents of all the sequence values. Any annotations on the sequence values are discarded. Any non-sequence arguments will raise an error. Any null arguments will be ignored.

Examples:

(:flatten [a, b, c] (d e f))       => a b c d e f
(:flatten [[], null.list] foo::()) => [] null.list

The flatten macro can also be used to splice the content of one list or s-expression into another list or s-expression.

[1, 2, (:flatten [a, b]), 3, 4] => [1, 2, a, b, 3, 4]

parse_ion

Ion documents may be embedded in other Ion documents using the parse_ion macro.

(macro parse_ion (uint8::data*) /* Not representable in TDL */)

The parse_ion macro constructs a stream of values by parsing a blob literal or string literal as a single, self-contained Ion document. All values produced by the expansion of parse_ion are application values. (I.e. it is as if they are all annotated with $ion_literal.)

The IVM at the beginning of an Ion data stream is sufficient to identify whether it is text or binary, so text Ion can be embedded as a blob containing the UTF-8 encoded text.

Embedded text example:

(:parse_ion
    '''
    $ion_1_1
    $ion_encoding::((symbol_table ["foo" "bar"]]))
    $1 $2
    '''
)
=> foo bar

Embedded binary example:

(:parse_ion {{ 4AEB6qNmb2+jYmFy }} )
=> foo bar

Value Constructors

annotate

(macro annotate (ann* value) /* Not representable in TDL */)

Produces the value prefixed with the annotations anns¹. Each ann must be a non-null, unannotated string or symbol.

(:annotate (: "a2") a1::true) => a2::a1::true

make_string

(macro make_string (content*) /* Not representable in TDL */)

Produces a non-null, unannotated string containing the concatenated content produced by the arguments. Nulls (of any type) are forbidden. Any annotations on the arguments are discarded.

make_symbol

(macro make_symbol (content*) /* Not representable in TDL */)

Like make_string but produces a symbol.

make_blob

(macro make_blob (lobs*) /* Not representable in TDL */)

Like make_string but accepts lobs and produces a blob.

make_list

(macro make_list (sequences*) [ (.flatten sequences) ])

Produces a non-null, unannotated list by concatenating the content of any number of non-null list or sexp inputs.

(:make_list)                  => []
(:make_list (1 2))            => [1, 2]
(:make_list (1 2) [3, 4])     => [1, 2, 3, 4]
(:make_list ((1 2)) [[3, 4]]) => [(1 2), [3, 4]]

make_sexp

(macro make_sexp (sequences*) ( (.flatten sequences) ))

Like make_list but produces a sexp.

(:make_sexp)                  => ()
(:make_sexp (1 2))            => (1 2)
(:make_sexp (1 2) [3, 4])     => (1 2 3 4)
(:make_sexp ((1 2)) [[3, 4]]) => ((1 2) [3, 4])

make_struct

(macro make_struct (structs*) /* Not representable in TDL */)

Produces a non-null, unannotated struct by combining the fields of any number of non-null structs.

(:make_struct)    => {}
(:make_struct
  {k1: 1, k2: 2}
  {k3: 3}
  {k4: 4})        => {k1:1, k2:2, k3:3, k4:4}

make_field

(macro make_field (flex_sym::field_name value) /* Not representable in TDL */)

Produces a non-null, unannotated, single-field struct using the given field name and value.

This can be used to dynamically construct field names based on macro parameters.

Example:

(macro foo_struct (extra_name extra_value)
       (make_struct 
         {
           foo_a: 1,
           foo_b: 2,
         }
         (make_field (make_string "foo_" (%extra_name)) (%extra_value))
       ))

Then:

(:foo_struct c 3) => { foo_a: 1, foo_b: 2, foo_c: 3 }

make_decimal

(macro make_decimal (flex_int::coefficient flex_int::exponent) /* Not representable in TDL */)

This is no more compact than the regular binary encoding for decimals. However, it can be used in conjunction with other macros, for example, to represent fixed-point numbers.

(macro usd (cents) (.annotate USD (.make_decimal cents -2))

(:usd 199) =>  USD::1.99

make_timestamp

(macro make_timestamp (uint16::year
                       uint8::month?
                       uint8::day?
                       uint8::hour?
                       uint8::minute?
                       /*decimal*/ second?
                       int16::offset_minutes?) /* Not representable in TDL */)

Produces a non-null, unannotated timestamp at various levels of precision. When offset is absent, the result has unknown local offset; offset 0 denotes UTC. The arguments to this macro may not be any null value.

Example:

(macro ts_today 
       (uint8::hour uint8::minute uint32::seconds_millis)
       (.make_timestamp
         2022
         4
         28
         hour
         minute
         (.make_decimal (%seconds_millis) -3) 0))

Encoding Utility Macros

repeat

The repeat system macro can be used for efficient run-length encoding.

(macro repeat (n! value+) /* Not representable in TDL */)

Produces a stream that repeats the specified value expression(s) n times.

(:repeat 5 0)          => 0 0 0 0 0
(:repeat 2 true false) => true false true false

delta

The delta system macro can be used for directed delta encoding.

(macro delta (flex_int::initial! flex_int::deltas+) /* Not representable in TDL */)

Example:

(:delta 10 1 2 3 -4) => 11 13 16 12

sum

(macro sum (i*) /* Not representable in TDL */)

Produces the sum of all the integer arguments.

Examples:

(:sum 1 2 3) => 6
(:sum (:))   => 0

meta

(macro meta (anything*) (.none))

The meta macro accepts any values and emits nothing. It allows writers to encode data that will be not be surfaced to most readers. Readers can be configured to intercept calls to meta, allowing them to read the otherwise invisible data.

When transcribing from one format to another, writers should preserve invocations of meta when possible.

Example:

(:values
    (:meta {author: "Mike Smith", email: "mikesmith@example.com"})
    {foo:2,foo:1}
)
=>
{foo:2,foo:1}

Updating the Encoding Context

set_symbols

Sets the local symbol table, preserving any macros in the macro table.

(macro set_symbols (symbols*)
       $ion_encoding::(
         (symbol_table [(%symbols)])
         (macro_table $ion_encoding)
       ))

Example:

(:set_symbols foo bar)
=>
$ion_encoding::(
  (symbol_table [foo, bar])
  (macro_table $ion_encoding)
)

add_symbols

Appends symbols to the local symbol table, preserving any macros in the macro table.

(macro add_symbols (symbols*)
       $ion_encoding::(
         (symbol_table $ion_encoding [(%symbols)])
         (macro_table $ion_encoding)
       ))

Example:

(:add_symbols foo bar)
=>
$ion_encoding::(
  (symbol_table $ion_encoding [foo, bar])
  (macro_table $ion_encoding)
)

set_macros

Sets the local macro table, preserving any symbols in the symbol table.

(macro set_macros (macros*)
       $ion_encoding::(
         (symbol_table $ion_encoding)
         (macro_table (%macros))
       ))

Example:

(:set_macros (macro pi () 3.14159))
=>
$ion_encoding::(
  (symbol_table $ion_encoding)
  (macro_table (macro pi () 3.14159))
)

add_macros

Appends macros to the local macro table, preserving any symbols in the symbol table.

(macro add_macros (macros*)
       $ion_encoding::(
         (symbol_table $ion_encoding)
         (macro_table $ion_encoding (%macros))
       ))

Example:

(:add_macros (macro pi () 3.14159))
=>
$ion_encoding::(
  (symbol_table $ion_encoding)
  (macro_table $ion_encoding (macro pi () 3.14159))
)

use

Appends the content of the given module to the encoding context.

(macro use (catalog_key version?)
       $ion_encoding::(
         (import the_module catalog_key (.default (%version) 1))
         (symbol_table $ion_encoding the_module)
         (macro_table $ion_encoding the_module)
       ))

Example:

(:use "org.example.FooModule" 2)
=>
$ion_encoding::(
  (import the_module "org.example.FooModule" 2)
  (symbol_table $ion_encoding the_module)
  (macro_table $ion_encoding the_module)
)