Amazon Ion Schema

RFC: Ion Schema Language Versions

Introduction

The purpose of this document is to specify how the Ion Schema Language will be versioned.

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

Definitions/Glossary

Ion Schema Compatibility/Portability Goal

A schema document MUST have the same behavior on all equivalently configured Ion Schema implementations. If an implementation does not support all features used in a particular schema document, the implementation MUST error when it attempts to load that schema.

Motivation

The Ion Schema 1.0 Specification requires version markers at the start of a schema document, but the specification says nothing else about how Ion Schema is to be versioned. Without clear versioning rules, almost any functional change to the Ion Schema specification is a potentially breaking change for our customers.

Incompatibility across implementation versions

We cannot change the meaning or behavior of any existing language syntax in a backwards compatible way. If the meaning of any syntax is modified, then schemas would no longer portable because they could behave differently in different applications that take a dependency on different releases of a library that implements Ion Schema.

Lack of portability

Any behavioral change to the Ion Schema specification is not guaranteed to be portable across different libraries that implement Ion Schema. These libraries will not be updated at exactly the same time, and users of Ion Schema will not update their dependencies at exactly the same time (or possibly never update their dependencies). Ion Schema needs a clearly defined versioning strategy that allows users to specify which behavior to use for a given schema document.

Underspecification of importing across schema versions

For Ion Schema use cases that have a large number of schemas, perhaps owned by multiple teams, we currently provide no guidance regarding whether schemas with different versions can be used together (i.e. can a schema import another schema with a different version?). Users will not be able to safely use multiple ISL versions or gradually upgrade their ISL version if this remains undefined.

Backwards Incompatibility caused by Open Content

There can be no Ion Schema 1.1—as long as open content is allowed, almost any change is backwards incompatible. ISL 1.0 broadly allows user-defined (“open”) content and so any new syntax has the possibility of name-shadowing user-defined content, causing unexpected behavior. However, restricting user defined content would itself be a backwards incompatible change. This backwards incompatible change will require a new major version, but before that can happen, Ion Schema needs a versioning strategy.

Solution – Ion Schema Language Versioning

This solution assumes that Ion Schema 2.0 also introduces some rules to prevent open content from colliding with Ion Schema features. See Ion Schema 2.0 Open Content

What is versioned?

The Ion Schema Language SHALL be versioned. The documents that describe the Ion Schema Language (the specification) SHALL NOT have versioned releases.

How is it versioned?

The Ion Schema Language version SHALL consist of a major and minor version components. The major and minor version components SHALL be non-negative integers.

Within a schema document, the format SHALL be $ion_schema_<major>_<minor>. In the specification text and other documentation, it SHALL be <major>.<minor>.

Readers may notice that this is similar to Semantic Versioning. However, it differs in two key ways. Semantic Versioning includes patch versions, but Ion Schema Language versioning does not use patch versions. Semantic Versioning is specific that releases must be immutable, but Ion Schema Language versioning does not provide that guarantee. The Ion Schema Language may be modified without changing the version number when the modification is an unsubstantive clarification or bugfix.

Major Versions

A new major version MAY contain any sort of changes, including backwards incompatible changes. Upon creating a new major version, the minor version component SHALL be reset to 0.

Minor Versions

Each new minor version of the Ion Schema Language SHALL allow any Ion Schema document that is valid against any previous minor version of the Ion Schema Language, within the same major version, to be updated to the new Specification version with equivalent semantics. Such an update MUST only require changing the ISL version marker to the new minor version. For example, a valid Ion Schema 2.2 document, upon changing its ISL version marker to $ion_schema_2_3, SHALL be a valid Ion Schema 2.3 document, semantically equivalent to the original Ion Schema 2.2 document. New minor versions of the Ion Schema Specification MUST be written to ensure this form of backward compatibility.

The guarantee of backwards compatibility is the objective standard by which we differentiate a major and minor version. Without some sort of objective standard, all we have are good intentions in order to decide whether a change requires a new major or minor version.

A change that qualifies as a minor version update MAY be released as a major version update as a way to signal to users that it is a particularly large or significant change.

Examples of different types of changes

Any change that is not backwards compatible must be released as a new major version. In other words, given the latest released version N, if we create version N+1, and there are any possible schemas for version N that are not valid and semantically equivalent by simply changing the ISL version marker to N+1, then a new major version is required. Examples of this include placing new restrictions on user-defined content, removing functionality, or changing the ISL versioning rules.

Any backwards compatible behavior or syntax change can be released as a new minor version. Examples of this include adding new behavior with new syntax to an existing constraint (such as a new modifier), adding a new constraint (assuming open content is appropriately limited), or adding syntactical sugar.

Some behavior clarifications may be released as an update to an existing minor version. Examples include reclassifying occurs to not be a constraint (ion-schema#48) and clarifying the interactions between number ranges and special float values (ion-schema#56).

Changes to the Ion Schema Specification that do not affect the Ion Schema Language are out of scope for Ion Schema Language versioning—for example, spelling and grammar changes, adding examples, or renumbering sections of the specification.

ISL Version Marker Syntax and Implementation

The ISL version marker in a schema document will only specify the major and minor version. The patch version is not included in the ISL version marker because patch versions do not change the behavior of the schema, so they have no relevance in a schema document.

The Ion Schema specification requires ISL version markers, but the ion-schema-kotlin implementation does not enforce that requirement. Starting with Ion Schema 2.0, a version marker must be required, but to avoid breaking existing schemas, we must continue to interpret the absence of an ISL version marker as an implied $ion_schema_1_0. (Refer to Backwards Incompatibility caused by Open Content for why we must go directly to Ion Schema 2.0.)

Implications for Schema Imports

From Minor Versions:

Each new minor version of the Ion Schema Specification SHALL allow any Ion Schema document that is valid against any previous minor version of the Specification, within the same major version, to be updated to the new Specification version with equivalent semantics. Such an update MUST only require changing the ISL version marker to the new minor version.

Under this condition, any schema that uses imports must remain semantically equivalent, without requiring anything of the imported schema, so by implication, Ion Schema must allow importing schemas that are still on prior minor versions.

Because ISL supports cycles in schema dependency graphs, we can infer that a schema SHALL be allowed to import another schema that uses any other minor version in the same major version. For example, given a dependency of A → B → A, where both A and B are using Ion Schema 2.0, when A is updated to 2.1, A must be allowed to import B and B must be allowed to import A.

(In other words, any schema using Ion Schema X.a MUST be able to import any other schema using Ion Schema X.b, both minor versions are supported by the Ion Schema implementation. Since an Ion Schema implementation must always provide support for all prior minor versions for a given major version, it is always possible to import a schema with any different minor version, up to the highest minor version supported by that implementation.)

The Ion Schema Specification does not restrict the possible changes that are allowed in a new major version, so any new major version of Ion Schema SHALL define its own rules regarding the ability to import schemas from prior major versions of Ion Schema. A new major version SHOULD allow importing from the most recent prior major version unless there is a technical reason why it is not possible.

Examples – Ion Schema 1.0

// The first value is not an ISL version marker, so it is implicitly ISL 1.0

$foo_service_interface_version_1 // Open content

$ion_schema_1_0 // Ion Schema 1.0 allows out-of-place version markers 
                // because in Ion Schema 1.0 this is open content.

$ion_schema_2_0 // Ion Schema 1.0 does not allow version 
                // markers that are for a different version

schema_header::{}

// ... other content ...

schema_footer::{}

$ion_schema_1_0  // Explicit version marker
 
$ion_schema_1_0 // Ion Schema 1.0 allows out-of-place version markers 
                // because in Ion Schema 1.0 this is open content.

schema_header::{}

// ... other content ...

schema_footer::{}

Examples – Ion Schema >=2.0

$ion_schema_2_4           // <-- isl version marker

$foo_service_interface_version_1  // <-- allowed open content

$ion_schema_2_4           // <-- ion schema version marker in wrong position
$ion_schema_1_foo         // <-- not allowed as open content

schema_header::{
  imports: [
    { id: fruit, type: apple },
  ]
}

type::{
  name: ion_schema_version_marker,
  type symbol,
  valid_values: [ 
    $ion_schema_1_0,     // <-- all valid; 
    $ion_schema_2_0,     // <   not isl version markers because 
    $ion_schema_2_1,     // <   they are not a top level values.
    $ion_schema_cat_dog, // <
  ]
}

schema_footer::{}

Alternatives Considered

Semantic Versioning

Using Semantic Versioning (SemVer) would require patch versions. The chosen solution has no use for patch versions, so we would need to find something to do with patch versions. We could either leave the patch version to always be 0 or we could use patch versions to version non-behavioural changes in the documentation. If we always keep the patch version at 0, then we have no benefit of using SemVer. If we use patch versions for documentation changes, then we are imposing an unnecessary process on ourselves and future maintainers of the project, and adding complexity that will probably confuse Ion Schema users with no obvious benefit to those users.

Only track major versions of ISL

While this is possible, it does not benefit our customers as much as tracking minor versions. By tracking minor versions, we get the following benefits over tracking only major versions:

Calendar Versioning (CalVer)

“CalVer is a versioning convention based on the project’s release calendar, instead of arbitrary numbers.” It can be used in conjunction with (modified) SemVer or on its own. For example, one could choose to use the year as the major version while following SemVer for the minor and patch versions.

CalVer has the benefit (for developers/maintainers) that you do not need to determine backwards compatibility to decide whether a release should be a new major version—instead you release a new major version with all the feature changes on a pre-determined schedule. However, if we use a calendar-based major version, then we cannot provide consistent guarantees about the compatibility between different versions of the Ion Schema Language, which is not as good for Ion Schema users.

CalVer also has the (general) benefit of making it easy to tie releases to a particular support schedule. This may be useful for a specific software application, but it seems to be irrelevant for a specification.

If we use CalVer, it seems like there would be an implied expectation of a specific release schedule. We do not want the spec to be updated on a specific schedule because we want to be able to react to business needs (for example, release a new major version without having to wait for the next calendar year). On the other hand, we do not want to be making releases solely because it is the predetermined time for a release—we expect the frequency of Ion Schema Specification releases to decrease over time as the specification matures.

Finally, this is a minor consideration, but the Ion Schema 1.0 Specification is already implied to use a versioning scheme (though none is defined) that is not calendar based, so precedent would suggest that we stick to non-calendar-based versioning.

Include a promise about semantic equivalence for major version updates

We could include the following:

Given a valid schema document for a particular major version, when the ISL version marker is changed to the next major version, the schema document SHALL either be (1) a semantically equivalent schema document or (2) not a valid schema document.

In other words:

Changing only the ISL version marker MUST NOT result in a valid but semantically different schema document, even when changing major versions.

This would make it safer for customers to upgrade their schemas because when they update the ISL version marker, it eliminates the possibility of having a valid but semantically different schema document. In addition, it simplifies the implementation of Ion Schema because there would be no need for implementations to push any versioning concerns to the constraint implementations.

However, the implication is that a major version update cannot change the meaning of any existing syntax—it can only remove features from ISL. (Though in the case of a constraint, for example, the constraint may be re-introduced with altered functionality under a different name.) This could add a disproportionately large amount of friction for seemingly small changes that result in a change of meaning, such as changing whether +/-inf are inside the max/min range bounds.

We decided not to do this because upgrading major versions should be an infrequent occurrence. If a change is significant enough to require a new major version, then it will probably be non-trivial for users to update their schemas to a new version. Since it would be non-trivial, we would provide a tool to help Ion Schema users upgrade to the latest version. Furthermore, this promise would make it more difficult for us to introduce changes to ISL.

Frequently Asked Questions

Does Ion Schema provide any features for versioning of users’ schemas or types?

ISL 1.0 does not provide any special functionality for schema versions. That is up to the discretion of the end user. However, the Ion Schema cookbook should provide suggestions.

Why does ISL not have schema versioning? Customers have asked about schema versioning, but they have described multiple different versioning strategies, and Ion Schema is not at a point right now where we can choose one to be the blessed way of versioning schemas/types.

However, Ion Schema will need to choose a versioning strategy for Ion Schema Schemas, because there will need to be multiple versions of the schema schemas. This will be a forcing function for Ion Schema to document schema versioning approaches for the Ion Schema cookbook.

When an implementation adds support for a new ISL major version, must an implementation that follows SemVer also bump its major version?

No. Adding support for a new ISL version never requires the implementation to increase its major version because it is strictly adding functionality. However, if an implementation (that follows SemVer) ever drops support for an older version of ISL, that would require a new major version under the SemVer rules.