OPTIMADE API specification v1.2.0-rc.2

2.1 Data types

An API implementation handles data types and their representations in three different contexts:

• In the HTTP URL query filter, see section API Filtering Format Specification.

• In the HTTP response. The default response format is JSON-based and thus uses JSON data types. However, other response formats can use different data types. For more info, see section Responses.

• The underlying database backend(s) from which the implementation serves data.

Hence, entry properties are described in this proposal using context-independent types that are assumed to have some form of representation in all contexts. They are as follows:

• Basic types: string, integer, float, boolean, timestamp.

• list: an ordered collection of items, where all items are of the same type, unless they are unknown. A list can be empty, i.e., contain no items.

• dictionary: an associative array of keys and values, where keys are pre-determined strings, i.e., for the same entry property, the keys remain the same among different entries whereas the values change. The values of a dictionary can be any basic type, list, dictionary, or unknown.

An entry property value that is not present in the database is unknown. This is equivalently expressed by the statement that the value of that entry property is null. For more information see section Properties with an unknown value

The definition of a property of an entry type specifies a type. The value of that property MUST either have a value of that type, or be unknown.

3.1 Versioning of this standard

This standard describes a communication protocol that, when implemented by a server, provides clients with an API for data access.

Released versions of the standard are versioned using semantic versioning v2 in reference to changes in that API (i.e., not in the server-side implementation of the protocol).

To clarify: semantic versioning mandates version numbers of the form MAJOR.MINOR.PATCH, where a "backwards incompatible API change" requires incrementing the MAJOR version number. A future version of the OPTIMADE standard can mandate servers to change their behavior to be compliant with the newer version. However, such changes are only considered "backwards incompatible API changes" if they have the potential to break clients that correctly use the API according to the earlier version.

Furthermore, the addition of new keys in key-value-formatted responses of the OPTIMADE API are not regarded as "backwards incompatible API changes." Hence, a client MUST disregard unrecognized keys when interpreting responses (but MAY issue warnings about them). On the other hand, a change of the OPTIMADE standard that fundamentally alters the interpretation of a response due to the presence of a new key will be regarded as a "backwards incompatible API change" since a client interpreting the response according to a prior version of the standard would misinterpret that response.

Working copies distributed as part of the development of the standard are marked with the version number for the release they are based on with an additional "~develop" suffix. These "versions" do not refer to a single specific instance of the text (i.e., the same "~develop" version string is retained until a release), nor is it clear to what degree they contain backwards incompatible API changes. Hence, the suffix is intentionally designed to make these version strings not to conform with semantic versioning to prevent incorrect comparisons to released versions using the scheme prescribed by semantic versioning. Version strings with a "~develop" suffix MAY be used by implementations during testing. However, a client that encounters them unexpectedly SHOULD NOT make any assumptions about the level of API compatibility.

In conclusion, the versioning policy of this standard is designed to allow clients using the OPTIMADE API according to a specific version of the standard to assume compatibility with servers implementing any future (non-development) version of the standard sharing the same MAJOR version number.

3.2 Base URL

Each database provider will publish one or more base URLs that serve the API, for example: http://example.com/optimade/. Every URL path segment that follows the base URL MUST behave as standardized in this API specification.

3.3 Version Negotiation

The OPTIMADE API provides three concurrent mechanisms for version negotiation between client and server.

1. The versions endpoint served directly under the unversioned base URL allows a client to discover all major API versions supported by a server in the order of preference (see section Versions Endpoint).

2. A client can access the API under versioned base URLs. In this case, the server MUST respond according to the specified version or return an error if the version is not supported (see section Versioned Base URLs).

3. When accessing the API under the unversioned base URL, clients are encouraged to append the OPTIONAL query parameter api_hint to hint the server about a preferred API version for the request. This parameter is described in more detail below.

The api_hint query parameter MUST be accepted by all API endpoints. However, for endpoints under a versioned base URL the request MUST be served as usual according to the version specified in the URL path segment regardless of the value of api_hint. In this case, the server MAY issue a warning if the value of api_hint suggests that the query may not be properly supported. If the client provides the parameter, the value SHOULD have the format vMAJOR or vMAJOR.MINOR, where MAJOR is a major version and MINOR is a minor version of the API. For example, if a client appends api_hint=v1.0 to the query string, the hint provided is for major version 1 and minor version 0.

If the server supports the major version indicated by the api_hint parameter at the same or a higher minor version (if provided), it SHOULD serve the request using this version. If the server does not support the major version hinted, or if it supports the major version but only at a minor version below the one hinted, it MAY use the provided values to make a best-effort attempt at still serving the request, e.g., by invoking the closest supported version of the API. If the hinted version is not supported by the server and the request is not served using an alternative version, the server SHOULD respond with the custom HTTP server error status code 553 Version Not Supported. Note that the above protocol means that clients MUST NOT expect that a returned response is served according to the version that is hinted.

For end users: Users are strongly encouraged to include the api_hint query parameter for URLs in, e.g., journal publications for queries on endpoints under the unversioned base URL. The version hint will make it possible to serve such queries in a reasonable way even after the server changes the major API version used for requests without version hints.

3.4 Index Meta-Database

A database provider MAY publish a special Index Meta-Database base URL. The main purpose of this base URL is to allow for automatic discoverability of all databases of the provider. Thus, it acts as a meta-database for the database provider's implementation(s).

The index meta-database MUST only provide the info and links endpoints, see sections Info Endpoints and Links Endpoint. It MUST NOT expose any entry listing endpoints (e.g., structures).

These endpoints do not need to be queryable, i.e., they MAY be provided as static JSON files. However, they MUST return the correct and updated information on all currently provided implementations.

The is_index field under attributes as well as the relationships field, MUST be included in the info endpoint for the index meta-database (see section Base Info Endpoint). The value for is_index MUST be true.

A few suggestions and mandatory requirements of the OPTIMADE specification are specifically relaxed only for index meta-databases to make it possible to serve them in the form of static files on restricted third-party hosting platforms:

• When serving an index meta-database in the form of static files, it is RECOMMENDED that the response excludes the subfields in the top-level meta field that would need to be dynamically generated (as described in the section JSON Response Schema: Common Fields.) The motivation is that static files cannot keep dynamic fields such as time_stamp updated.

• The JSON:API specification requirements on content negotiation using the HTTP headers Content-Type and Accept are NOT mandatory for index meta-databases. Hence, API Implementations MAY ignore the content of these headers and respond to all requests. The motivation is that static file hosting is typically not flexible enough to support these requirements on HTTP headers.

• API implementations SHOULD serve JSON content with either the JSON:API mandated HTTP header Content-Type: application/vnd.api+json or Content-Type: application/json. However, if the hosting platform does not allow this, JSON content MAY be served with Content-Type: text/plain.

Note: A list of database and definition providers acknowledged by the Open Databases Integration for Materials Design consortium is maintained externally from this specification and can be retrieved as described in section Namespace Prefixes. This list is also machine-readable, enabling the automatic discoverability of OPTIMADE API services.

3.5 Namespace Prefixes

There are two mechanisms by which a provider can serve properties that are not standardized by the OPTIMADE specification.

1. By serving properties under a database-provider-specific namespace prefix. This is the preferred mechanism for serving properties that are specific to a particular database provider.

2. By adopting a property definition external to the specification by a definition provider. This is the preferred mechanism in cases where a database-specific field aligns with a field that is already defined by a definition provider, and can be used to enable aggregated filtering over all OPTIMADE APIs that support this property.

A list of known database and definition providers and their assigned prefixes is published in the form of an OPTIMADE Index Meta-Database with base URL https://providers.optimade.org. Visiting this URL in a web browser gives a human-readable description of how to retrieve the information in the form of a JSON file, and specifies the procedure for registration of new prefixes. A human-readable dashboard is also hosted at https://www.optimade.org/providers-dashboard.

API implementations SHOULD NOT make up and use new prefixes without first getting them registered in the official list.

Examples:

• A database-provider-specific prefix: exmpl. Used as a field name in a response: _exmpl_custom_field.

• A definition-provider prefix: dft. Used as a field name in a response by multiple different providers: _dft_cell_volume (note: this is a hypothetical example).

The initial underscore indicates an identifier that is under a separate namespace under the ownership of that organization or definition provider. Identifiers prefixed with underscores will not be used for standardized names.

3.6 URL Encoding

Clients SHOULD encode URLs according to RFC 3986. API implementations MUST decode URLs according to RFC 3986.

3.7 Relationships

The API implementation MAY describe many-to-many relationships between entries along with OPTIONAL human-readable descriptions that describe each relationship. These relationships can be to the same, or to different, entry types. Response formats have to encode these relationships in ways appropriate for each format.

In the default response format, relationships are encoded as JSON:API Relationships, see section Entry Listing JSON Response Schema.

For implementers: For database-specific response formats without a dedicated mechanism to indicate relationships, it is suggested that they are encoded alongside the entry properties. For each entry type, the relationships with entries of that type can then be encoded in a field with the name of the entry type, which are to contain a list of the IDs of the referenced entries alongside the respective human-readable description of the relationships. It is the intent that future versions of this standard uphold the viability of this encoding by not standardizing property names that overlap with the entry type names.

3.8 Properties with an unknown value

Many databases allow specific data values to exist for some of the entries, whereas for others, no data value is present. This is referred to as the property having an unknown value, or equivalently, that the property value is null.

The text in this section describes how the API handles properties with the value null. The use of null values inside nested property values (such as, e.g., lists or dictionaries) are described in the definitions of those data structures elsewhere in the specification, see section Entry List. For these properties, null MAY carry a special meaning.

REQUIRED properties with an unknown value MUST be included and returned in the response with the value null.

OPTIONAL properties with an unknown value, if requested explicitly via the response_fields query parameter, MUST be included and returned in the response with the value null. (For more info on the response_fields query parameter, see section Entry Listing URL Query Parameters.)

The interaction of properties with an unknown value with query filters is described in the section Filtering on Properties with an unknown value. In particular, filters with IS UNKNOWN and IS KNOWN can be used to match entries with values that are, or are not, unknown for some property, respectively.

3.9 Handling unknown property names

When an implementation receives a request with a query filter that refers to an unknown property name it is handled differently depending on the database-specific prefix:

• If the property name has no database-specific prefix, or if it has the database-specific prefix that belongs to the implementation itself, the error 400 Bad Request MUST be returned with a message indicating the offending property name.

• If the property name has a database-specific prefix that does not belong to the implementation itself, it MUST NOT treat this as an error, but rather MUST evaluate the query with the property treated as unknown, i.e., comparisons are evaluated as if the property has the value null.

  • Furthermore, if the implementation does not recognize the prefix at all, it SHOULD return a warning that indicates that the property has been handled as unknown.

  • On the other hand, if the prefix is recognized, i.e., as belonging to a known database provider, the implementation SHOULD NOT issue a warning but MAY issue diagnostic output with a note explaining how the request was handled.

The rationale for treating properties from other databases as unknown rather than triggering an error is for OPTIMADE to support queries using database-specific properties that can be sent to multiple databases.

For example, the following query can be sent to API implementations exmpl1 and exmpl2 without generating any errors:

filter=_exmpl1_band_gap<2.0 OR _exmpl2_band_gap<2.5

3.10 Transmission of large property values

A property value may be too large to fit in a single response. OPTIMADE provides a mechanism for a client to handle such properties by fetching them in separate series of requests. It is up to the implementation to decide which values are too large to represent in a single response, and this decision MAY change between responses.

In this case, the response to the initial query gives the value null for the property. A list of one or more data URLs together with their respective partial data formats are given in the response. How this list is provided is response format-dependent. For the JSON response format, see the description of the partial_data_links field, nested under data and then meta, in the section JSON Response Schema: Common Fields.

The default partial data format is named "jsonlines" and is described in the Appendix OPTIMADE JSON lines partial data format. An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix.

Below follows an example of the data and meta parts of a response using the JSON response format that communicates that the property value has been omitted from the response, with three different links for different partial data formats provided.

{
  // ...
  "data": {
    "type": "structures",
    "id": "2345678",
    "attributes": {
        "a": null
    }
    "meta": {
      "partial_data_links": {
        "a": [
          {
            "format": "jsonlines",
            "link": "https://example.org/optimade/v1.2/extensions/partial_data/structures/2345678/a/default_format"
          },
          {
            "format": "_exmpl_bzip2_jsonlines",
            "link": "https://db.example.org/assets/partial_values/structures/2345678/a/bzip2_format"
          },
          {
            "format": "_exmpl_hdf5",
            "link": "https://cloud.example.org/ACCHSORJGIHWOSJZG"
          }
        ]
      }
    }
  }
  // ...
}

3.11 Metadata properties

A metadata property represents entry and property-specific metadata for a given entry. How these are communicated in the response depends on the response format. For the JSON response format, the metadata properties are stored in the resource object metadata field, meta in a dictionary field property_metadata with the keys equal to the names of the respective properties for which metadata is available, see JSON Response Schema: Common Fields.

The format of the metadata property is specified by the field x-optimade-metadata-definition in the Property Definition of the field, see Property Definitions. Database providers are allowed to define their own metadata properties in x-optimade-metadata-definition, but they MUST use the database-provider-specific prefix even for metadata of database-specific fields. For example, the metadata property definition of the field _exmpl_example_field MUST NOT define a metadata field named, e.g., accuracy; the field rather needs to be named, e.g., _exmpl_accuracy. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields.

Implementation of the meta field is OPTIONAL. However, when an implementation supports the property_metadata field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response.

Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field elements_ratios and the database-specific per entry metadata field _exmpl_originates_from_project:

{
  "data": [
    {
      "type": "structures",
      "id": "example.db:structs:0001",
      "attributes": {
        "elements_ratios":[0.33336, 0.22229, 0.44425]
      },
      "meta": {
        "property_metadata": {
          "elements_ratios": {
            "_exmpl_originates_from_project": "piezoelectic_perovskites"
          }
        }
      }
    },
    {
      "type": "structures",
      "id": "example.db:structs:1234",
      "attributes": {
        "elements_ratios":[0.5, 0.5]
      },
      "meta": {
        "property_metadata":{
          "elements_ratios": {
            "_exmpl_originates_from_project": "ferroelectric_binaries"
          }
        }
      }
    }
    //...
  ]
  // ...
}

Example of the corresponding metadata property definition contained in the field x-optimade-metadata-definition which is placed in the property definition of elements_ratios:

// ...
"x-optimade-metadata-definition": {
  "title": "Metadata for the elements_ratios field",
  "description": "This field contains the per-entry metadata for the elements_ratios field.",
  "x-optimade-type": "dictionary",
  "x-optimade-unit": "inapplicable",
  "type": ["object", "null"],
  "properties" : {
    "_exmpl_originates_from_project": {
      "$id": "https://properties.example.com/v1.2.0/elements_ratios_meta/_exmpl_originates_from_project",
      "description" : "A string naming the internal example.com project id where this property was added to the database.",
      "x-optimade-type": "string",
      "x-optimade-unit" : "inapplicable",
      "type": ["string", "null"]
    }
  }
}
// ...

4.1 Response Format

This section defines a JSON response format that complies with the JSON:API v1.1 specification. All endpoints of an API implementation MUST be able to provide responses in the JSON format specified below and MUST respond in this format by default.

Each endpoint MAY support additional formats, and SHOULD declare these formats under the endpoint /info/<entry type> (see section Entry Listing Info Endpoints). Clients can request these formats using the response_format URL query parameter. Specifying a response_format different from json (e.g. response_format=xml) allows the API to break conformance not only with the JSON response format specification, but also, e.g., in terms of how content negotiation is implemented.

Database-provider-specific and definition-provider-specific response_format identifiers MUST include the corresponding prefix (see section Namespace Prefixes).

4.2 JSON Response Schema: Common Fields

In the JSON response format, property types translate as follows:

• string, boolean, list are represented by their similarly named counterparts in JSON.

• integer, float are represented as the JSON number type.

• timestamp uses a string representation of date and time as defined in RFC 3339 Internet Date/Time Format.

• dictionary is represented by the JSON object type.

• unknown properties are represented by either omitting the property or by a JSON null value.

Every response SHOULD contain the following fields, and MUST contain at least meta:

• meta: a JSON:API meta member that contains JSON:API meta objects of non-standard meta-information. It MUST be a dictionary with these fields:

  • api_version: a string containing the full version of the API implementation. The version number string MUST NOT be prefixed by, e.g., "v". Examples: 1.0.0, 1.0.0-rc.2.

  • query: information on the query that was requested. It MUST be a dictionary with this field:

    • representation: a string with the part of the URL following the versioned or unversioned base URL that serves the API. Query parameters that have not been used in processing the request MAY be omitted. In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. Example: /structures?filter=nelements=2.

  • more_data_available: false if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a filter query at the last page of a paginated response) and true if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page).

  meta SHOULD also include these fields:

  • time_stamp: a timestamp containing the date and time at which the query was executed.

  • data_returned: an integer containing the total number of data resource objects returned for the current filter query, independent of pagination.

  • provider: information on the database provider of the implementation. It MUST be a dictionary with these fields:

    • name: a short name for the database provider.

    • description: a longer description of the database provider.

    • prefix: database-provider-specific prefix (see section Database-Provider-Specific Namespace Prefixes).

    provider MAY include these fields:

    • homepage: a JSON API link, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields:

      • href: a string containing the homepage URL.

      • meta: a meta object containing non-standard meta-information about the database provider's homepage.

  meta MAY also include these fields:

  • data_available: an integer containing the total number of data resource objects available in the database for the endpoint.

  • last_id: a string containing the last ID returned.

  • response_message: response string from the server.

  • request_delay: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request.

    Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error 429 Too Many Requests and the Retry-After HTTP header. Implementations are suggested to provide consistent handling of request overload through both mechanisms.

  • database: a dictionary describing the specific database accessible at this OPTIMADE API. If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in Links Endpoint JSON Response Schema. The dictionary can contain the following OPTIONAL fields:

    • id: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database.

    • name: a human-readable name for the database, e.g., for use in clients.

    • version: a string describing the version of the database.

    • description: a human-readable description of the database, e.g., for use in clients.

    • homepage: a JSON API link, pointing to a homepage for the particular database.

    • maintainer: a dictionary providing details about the maintainer of the database, which MUST contain the single field:

      • email with the maintainer's email address.

  • implementation: a dictionary describing the server implementation, containing the OPTIONAL fields:

    • name: name of the implementation.

    • version: version string of the current implementation.

    • homepage: a JSON API link, pointing to the homepage of the implementation.

    • source_url: a JSON API link pointing to the implementation source, either downloadable archive or version control system.

    • maintainer: a dictionary providing details about the maintainer of the implementation, MUST contain the single field:

      • email with the maintainer's email address.

    • issue_tracker: a JSON API link pointing to the implementation's issue tracker.

  • warnings: a list of warning resource objects representing non-critical errors or warnings. A warning resource object is defined similarly to a JSON:API error object, but MUST also include the field type, which MUST have the value "warning". The field detail MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. The field status, representing an HTTP response status code, MUST NOT be present for a warning resource object. This is an exclusive field for error resource objects.

    Example for a deprecation warning:

    {
      "id": "dep_chemical_formula_01",
      "type": "warning",
      "code": "_exmpl_dep_chemical_formula",
      "title": "Deprecation Warning",
      "detail": "chemical_formula is deprecated, use instead chemical_formula_hill"
    }

    Note: warning ids MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field code for this. Warning ids can only be trusted to be unique in the list of warning resource objects, i.e., together with the type.

    General OPTIMADE warning codes are specified in section Warnings.

  • Other OPTIONAL additional information global to the query that is not specified in this document, MUST start with a database-provider-specific prefix (see section Database-Provider-Specific Namespace Prefixes).

  • Example for a request made to http://example.com/optimade/v1/structures/?filter=a=1 AND b=2:

    {
      "meta": {
        "query": {
          "representation": "/structures/?filter=a=1 AND b=2"
        },
        "api_version": "1.0.0",
        "schema": "http://schemas.optimade.org/openapi/v1/optimade.json",
        "time_stamp": "2007-04-05T14:30:20Z",
        "data_returned": 10,
        "data_available": 10,
        "more_data_available": false,
        "provider": {
          "name": "Example provider",
          "description": "Provider used for examples, not to be assigned to a real database",
          "prefix": "exmpl",
          "homepage": "http://example.com"
        },
        "implementation": {
          "name": "exmpl-optimade",
          "version": "0.1.0",
          "source_url": "http://git.example.com/exmpl-optimade",
          "maintainer": {
            "email": "admin@example.com"
          },
          "issue_tracker": "http://tracker.example.com/exmpl-optimade"
        },
        "database": {
          "id": "example_db",
          "name": "Example database 1 (of many)",
          "description": "The first example database in a series hosted by the Example Provider.",
          "homepage": "http://database_one.example.com",
          "maintainer": {
            "email": "science_lead@example.com"
          }
        }
      }
      // ...
    }

  • schema: a JSON:API links object that points to a schema for the response. If it is a string, or a dictionary containing no meta field, the provided URL MUST point at an OpenAPI schema. It is possible that future versions of this specification allow for alternative schema types. Hence, if the meta field of the JSON:API links object is provided and contains a field schema_type that is not equal to the string OpenAPI the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors.

    Note: The schema field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the describedby subfield of the top-level links field.

• data: The schema of this value varies by endpoint, it can be either a single JSON:API resource object or a list of JSON:API resource objects. Every resource object needs the type and id fields, and its attributes (described in section API Endpoints) need to be in a dictionary corresponding to the attributes field.

  Every resource object MAY also contain a meta field which MAY contain the following keys:

  • property_metadata: an object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also Metadata properties

  • partial_data_links: an object used to list links which can be used to fetch data that has been omitted from the data part of the response. The keys are the names of the fields in attributes for which partial data links are available. Each value is a list of objects that MUST have the following keys:

    • format: String. The name of the format provided via this link. For one of the objects this format field SHOULD have the value "jsonlines", which refers to the format in OPTIMADE JSON lines partial data format.

    • link: String. A JSON API link that points to a location from which the omitted data can be fetched. There is no requirement on the syntax or format for the link URL.

    For more information about the mechanism to transmit large property values, including an example of the format of partial_data_links, see Transmission of large property values.

The response MAY also return resources related to the primary data in the field:

• links: a JSON API links object is REQUIRED for implementing pagination. (see section Entry Listing URL Query Parameters.) Each field of a links object, i.e., a "link", MUST be one of:

  • null

  • a string representing a URI, or

  • a dictionary ("link object") with fields

    • href: a string representing a URI

    • meta: (OPTIONAL) a meta object containing non-standard meta-information about the link

  Example links objects:

  • base_url: a links object representing the base URL of the implementation. Example:

    {
      "links": {
        "base_url": {
          "href": "http://example.com/optimade",
          "meta": {
            "_exmpl_db_version": "3.2.1"
          }
        }
        // ...
      }
      // ...
    }

  The links field SHOULD include the following links objects:

  • describedby: a links object giving the URL for a schema that describes the response. The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level $schema and/or $id fields that can be used by the client to identify the schema format.

    Note: This field is the standard facility in JSON:API to communicate a response schema. It overlaps in function with the field schema in the top level meta field.

  The following fields are REQUIRED for implementing pagination:

  • next: represents a link to fetch the next set of results. When the current response is the last page of data, this field MUST be either omitted or null-valued.

  An implementation MAY also use the following reserved fields for pagination. They represent links in a similar way as for next.

  • prev: the previous page of data. null or omitted when the current response is the first page of data.

  • last: the last page of data.

  • first: the first page of data.

  Finally, the links field MAY also include the following links object:

  • self: a links object giving the URL from which the response was obtained.

• included: a list of JSON:API resource objects related to the primary data contained in data. Responses that contain related resources under included are known as compound documents in the JSON:API.

  The definition of this field is found in the JSON:API specification. Specifically, if the query parameter include is included in the request, included MUST NOT include unrequested resource objects. For further information on the parameter include, see section Entry Listing URL Query Parameters.

  This value MUST be either an empty array or an array of related resource objects.

If there were errors in producing the response all other fields MAY be present, but the top-level data field MUST be skipped, and the following field MUST be present:

• errors: a list of JSON:API error objects, where the field detail MUST be present. All other fields are OPTIONAL.

An example of a full response:

{
  "links": {
    "next": null,
    "base_url": {
      "href": "http://example.com/optimade",
      "meta": {
         "_exmpl_db_version": "3.2.1"
      }
    }
  },
  "meta": {
    "query": {
      "representation": "/structures?filter=a=1 AND b=2"
    },
    "api_version": "1.0.0",
    "time_stamp": "2007-04-05T14:30:20Z",
    "data_returned": 10,
    "data_available": 10,
    "last_id": "xy10",
    "more_data_available": false,
    "provider": {
      "name": "Example provider",
      "description": "Provider used for examples, not to be assigned to a real database",
      "prefix": "exmpl",
      "homepage": {
        "href": "http://example.com",
        "meta": {
          "_exmpl_title": "This is an example site"
        }
      }
    },
    // <OPTIONAL implementation- or database-provider-specific metadata, global to the query>
  },
  "data": [
    // ...
  ],
  "included": [
    // ...
  ]
}

• @context: A JSON-LD context that enables interpretation of data in the response as linked data. If provided, it SHOULD be one of the following:

  • An object conforming to a JSON-LD standard, which includes a @version field specifying the version of the standard.

  • A string containing a URL that resolves to such an object.

• jsonapi: A JSON:API object. The version subfield SHOULD be "1.1". The meta subfield SHOULD be included and contain the following subfields:

  • api: A string with the value "OPTIMADE".

  • api-version: A string with the full version of the OPTIMADE standard that the processing and response adheres to. This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter api_hint to request processing according to another version.

  If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the jsonapi as the first top level member to simplify identification of the response.

4.3 HTTP Response Status Codes

All HTTP response status codes MUST conform to RFC 7231: HTTP Semantics. The code registry is maintained by IANA and can be found here.

See also the JSON:API definitions of responses when fetching data, i.e., sending an HTTP GET request.

Important: If a client receives an unexpected 404 error when making a query to a base URL, and is aware of the index meta-database that belongs to the database provider (as described in section Index Meta-Database), the next course of action SHOULD be to fetch the resource objects under the links endpoint of the index meta-database and redirect the original query to the corresponding database ID that was originally queried, using the object's base_url value.

4.4 HTTP Response Headers

There are relevant use-cases for allowing data served via OPTIMADE to be accessed from in-browser JavaScript, e.g. to enable server-less data aggregation. For such use, many browsers need the server to include the header Access-Control-Allow-Origin: * in its responses, which indicates that in-browser JavaScript access is allowed from any site.

4.5 Warnings

Non-critical exceptional situations occurring in the implementation SHOULD be reported to the referrer as warnings. Warnings MUST be expressed as a human-readable message, OPTIONALLY coupled with a warning code.

Warning codes starting with an alphanumeric character are reserved for general OPTIMADE error codes (currently, none are specified). For implementation-specific warnings, they MUST start with _ and the database-provider-specific prefix of the implementation (see section Database-Provider-Specific Namespace Prefixes).

5.1 Query parameters

Query parameters to the endpoints are documented in the respective subsections below. However, in addition, all API endpoints MUST accept the api_hint parameter described under Version Negotiation.

5.2 Versions Endpoint

The versions endpoint aims at providing a stable and future-proof way for a client to discover the major versions of the API that the implementation provides. This endpoint is special in that it MUST be provided directly on the unversioned base URL at /versions and MUST NOT be provided under the versioned base URLs.

The response to a query to this endpoint is in a restricted subset of the RFC 4180 CSV (text/csv; header=present) format. The restrictions are: (i) field values and header names MUST NOT contain commas, newlines, or double quote characters; (ii) Field values and header names MUST NOT be enclosed by double quotes; (iii) The first line MUST be a header line. These restrictions allow clients to parse the file line-by-line, where each line can be split on all occurrences of the comma ',' character to obtain the head names and field values.

In the present version of the API, the response contains only a single field that is used to list the major versions of the API that the implementation supports. The CSV format header line MUST specify version as the name for this field. However, clients MUST accept responses that include other fields that follow the version.

The major API versions in the response are to be ordered according to the preference of the API implementation. If a version of the API is served on the unversioned base URL as described in the section Base URL, that version MUST be the first value in the response (i.e., it MUST be on the second line of the response directly following the required CSV header).

It is the intent that all future versions of this specification retain this endpoint, its restricted CSV response format, and the meaning of the first field of the response.

Example response:

version
1
0

The above response means that the API versions 1 and 0 are served under the versioned base URLs /v1 and /v0, respectively. The order of the versions indicates that the API implementation regards version 1 as preferred over version 0. If the API implementation allows access to the API on the unversioned base URL, this access has to be to version 1, since the number 1 appears in the first (non-header) line.

5.3 Entry Listing Endpoints

Entry listing endpoints return a list of resource objects representing entries of a specific type. For example, a list of structures, or a list of calculations.

Each entry in the list includes a set of properties and their corresponding values. The section Entry list specifies properties as belonging to one of three categories:

1. Properties marked as REQUIRED in the response. These properties MUST always be present for all entries in the response.

2. Properties marked as REQUIRED only if the query parameter response_fields is not part of the request, or if they are explicitly requested in response_fields. Otherwise they MUST NOT be included. One can think of these properties as constituting a default value for response_fields when that parameter is omitted.

3. Properties not marked as REQUIRED in any case, MUST be included only if explicitly requested in the query parameter response_fields. Otherwise they SHOULD NOT be included.

Examples of valid entry listing endpoint URLs:

• http://example.com/optimade/v1/structures

• http://example.com/optimade/v1/calculations

There MAY be multiple entry listing endpoints, depending on how many types of entries an implementation provides. Specific standard entry types are specified in section Entry list.

The API implementation MAY provide other entry types than the ones standardized in this specification. Such entry types MUST be prefixed by a database-provider-specific prefix (i.e., the resource objects' type value should start with the database-provider-specific prefix, e.g., type = _exmpl_workflows). Each custom entry type SHOULD be served at a corresponding entry listing endpoint under the versioned or unversioned base URL that serves the API with the same name (i.e., equal to the resource objects' type value, e.g., /_exmpl_workflows). It is RECOMMENDED to align with the OPTIMADE API specification practice of using a plural for entry resource types and entry type endpoints. Any custom entry listing endpoint MUST also be added to the available_endpoints and entry_types_by_format attributes of the Base Info Endpoint.

For more on custom endpoints, see Custom Extension Endpoints.

{
  "data": [
    {
      "type": "structures",
      "id": "example.db:structs:0001",
      "attributes": {
        "chemical_formula_descriptive": "Es2 O3",
        "url": "http://example.db/structs/0001",
        "immutable_id": "http://example.db/structs/0001@123",
        "last_modified": "2007-04-05T14:30:20Z"
      }
    },
    {
      "type": "structures",
      "id": "example.db:structs:1234",
      "attributes": {
        "chemical_formula_descriptive": "Es2",
        "url": "http://example.db/structs/1234",
        "immutable_id": "http://example.db/structs/1234@123",
        "last_modified": "2007-04-07T12:02:20Z"
      }
    }
    // ...
  ]
  // ...
}

5.4 Single Entry Endpoints

A client can request a specific entry by appending a URL-encoded ID path segment to the URL of an entry listing endpoint. This will return properties for the entry with that ID.

In the default JSON response format, the ID component MUST be the content of the id field.

Examples:

• http://example.com/optimade/v1/structures/exmpl%3Astruct_3232823

• http://example.com/optimade/v1/calculations/232132

The rules for which properties are to be present for an entry in the response are the same as defined in section Entry Listing Endpoints.

{
  "data": {
    "type": "structures",
    "id": "example.db:structs:1234",
    "attributes": {
      "chemical_formula_descriptive": "Es2",
      "url": "http://example.db/structs/1234",
      "immutable_id": "http://example.db/structs/1234@123",
      "last_modified": "2007-04-07T12:02:20Z"
    }
  },
  "meta": {
    "query": {
      "representation": "/structures/example.db:structs:1234?"
    }
    // ...
  }
  // ...
}

5.5 Info Endpoints

Info endpoints provide introspective information, either about the API implementation itself, or about specific entry types.

There are two types of info endpoints:

1. Base info endpoints: placed directly under the versioned or unversioned base URL that serves the API (e.g., http://example.com/optimade/v1/info or http://example.com/optimade/info)

2. Entry listing info endpoints: placed under the endpoints belonging to specific entry types (e.g., http://example.com/optimade/v1/info/structures or http://example.com/optimade/info/structures)

The types and output content of these info endpoints are described in more detail in the subsections below. Common for them all are that the data field SHOULD return only a single resource object. If no resource object is provided, the value of the data field MUST be null.

{
  "data": {
    "type": "info",
    "id": "/",
    "attributes": {
      "api_version": "1.0.0",
      "available_api_versions": [
        {"url": "http://db.example.com/optimade/v0/", "version": "0.9.5"},
        {"url": "http://db.example.com/optimade/v0.9/", "version": "0.9.5"},
        {"url": "http://db.example.com/optimade/v0.9.2/", "version": "0.9.2"},
        {"url": "http://db.example.com/optimade/v0.9.5/", "version": "0.9.5"},
        {"url": "http://db.example.com/optimade/v1/", "version": "1.0.0"},
        {"url": "http://db.example.com/optimade/v1.0/", "version": "1.0.0"}
      ],
      "formats": [
        "json",
        "xml"
      ],
      "entry_types_by_format": {
        "json": [
          "structures",
          "calculations"
        ],
        "xml": [
          "structures"
        ]
      },
      "available_endpoints": [
        "structures",
        "calculations",
        "info",
        "links"
      ],
      "is_index": false
    }
  }
  // ...
}

{
  "data": {
    "type": "info",
    "id": "/",
    "attributes": {
      "api_version": "1.0.0",
      "available_api_versions": [
        {"url": "http://db.example.com/optimade/v0/", "version": "0.9.5"},
        {"url": "http://db.example.com/optimade/v0.9/", "version": "0.9.5"},
        {"url": "http://db.example.com/optimade/v0.9.2/", "version": "0.9.2"},
        {"url": "http://db.example.com/optimade/v1/", "version": "1.0.0"},
        {"url": "http://db.example.com/optimade/v1.0/", "version": "1.0.0"}
        ],
      "formats": [
        "json",
        "xml"
      ],
      "entry_types_by_format": {
        "json": [],
        "xml": []
      },
      "available_endpoints": [
        "info",
        "links"
      ],
      "is_index": true
    },
    "relationships": {
      "default": {
        "data": { "type": "links", "id": "perovskites" }
      }
    }
  }
  // ...
}

{
  "data": {
    "type": "info",
    "id": "structures",
    "description": "a structures entry",
    "properties": {
      "nelements": {
        "$id": "urn:uuid:10a05e55-0c20-4f68-89ad-35a18eb7076f",
        "title": "Number of elements",
        "x-optimade-type": "integer",
        "type": ["integer", "null"],
        "description": "Number of different elements in the structure as an integer.\n
         \n
         -  Note: queries on this property can equivalently be formulated using `elements LENGTH`.\n
         -  A filter that matches structures that have exactly 4 elements: `nelements=4`.\n
         -  A filter that matches structures that have between 2 and 7 elements: `nelements>=2 AND nelements<=7`.",
        "examples": [
          3
        ],
        "x-optimade-property": {
           "property-format": "1.2"
        },
        "x-optimade-unit": "dimensionless",
        "x-optimade-implementation": {
          "sortable": true,
          "query-support": "all mandatory"
        },
        "x-optimade-requirements": {
          "support": "should",
          "sortable": false,
          "query-support": "all mandatory"
        }
      },
      "lattice_vectors": {
        "$id": "urn:uuid:81edf372-7b1b-4518-9c14-7d482bd67834",
        "title": "Lattice vectors",
        "x-optimade-definition": {
          "label": "lattice_vectors_optimade_structures",
          "kind": "property",
          "format": "1.2",
          "version": "1.2.0",
          "name": "lattice_vectors"
        },
        "x-optimade-type": "list",
        "x-optimade-dimensions": {
           "names": ["dim_lattice", "dim_spatial"],
           "lengths": [3, 3]
        },
        "x-optimade-unit-definitions": [
            {
              "symbol": "angstrom",
              "title": "ångström",
              "description": "The ångström unit of length.",
              "standard": {
                "name": "gnu units",
                "version": "3.09",
                "symbol": "angstrom"
              }
            }
          ],
        "x-optimade-unit": "inapplicable",
        "x-optimade-implementation": {
          "sortable": false,
          "query-support": "none"
        },
        "x-optimade-requirements": {
          "support": "should",
          "sortable": false,
          "query-support": "none"
        },
        "type": ["array", "null"],
        "description": "The three lattice vectors in Cartesian coordinates, in ångström (Å).\n
        \n
        - MUST be a list of three vectors *a*, *b*, and *c*, where each of the vectors MUST BE a
          list of the vector's coordinates along the x, y, and z Cartesian coordinates.
        ",
        "examples": [
          [[4.0, 0.0, 0.0], [0.0, 4.0, 0.0], [0.0, 1.0, 4.0]]
        ],
        "items": {
          "type": "array",
          "x-optimade-type": "list",
          "x-optimade-unit": "inapplicable",
          "x-optimade-dimensions": {
            "names": ["dim_spatial"],
            "lengths": [3]
          },
          "items": {
            "type": "number",
            "x-optimade-type": "float",
            "x-optimade-unit": "angstrom",
            "x-optimade-implementation": {
              "sortable": true,
              "query-support": "none"
            },
            "x-optimade-requirements": {
              "sortable": false,
              "query-support": "none"
            }
          }
        }
      }
      // ... <other property descriptions>
    },
    "formats": ["json", "xml"],
    "output_fields_by_format": {
      "json": [
        "nelements",
        "lattice_vectors",
        // ...
      ],
      "xml": ["nelements"]
    }
  }
  // ...
}

5.6 Links Endpoint

This endpoint exposes information on other OPTIMADE API implementations that are related to the current implementation. The links endpoint MUST be provided under the versioned or unversioned base URL serving the API at /links.

{
  "data": [
    {
      "type": "links",
      "id": "index",
      "attributes": {
        "name": "Index",
        "description": "Index for example's OPTIMADE databases",
        "base_url": "http://example.com/optimade",
        "homepage": "http://example.com",
        "link_type": "root"
      }
    },
    {
      "type": "links",
      "id": "cat_zeo",
      "attributes": {
        "name": "Catalytic Zeolites",
        "description": "Zeolites for deNOx catalysis",
        "base_url": {
          "href": "http://example.com/optimade/denox/zeolites",
          "meta": {
            "_exmpl_catalyst_group": "denox"
          }
        },
        "homepage": "http://example.com",
        "link_type": "child"
      }
    },
    {
      "type": "links",
      "id": "frameworks",
      "attributes": {
        "name": "Zeolitic Frameworks",
        "description": "",
        "base_url": "http://example.com/zeo_frameworks/optimade",
        "homepage": "http://example.com",
        "link_type": "child"
      }
    },
    {
      "type": "links",
      "id": "testdb",
      "attributes": {
        "name": "Test database",
        "description": "A test database",
        "base_url": "http://example.com/testdb/optimade",
        "homepage": "http://example.com",
        "link_type": "child",
        "aggregate": "test"
      }
    },
    {
      "type": "links",
      "id": "internaldb",
      "attributes": {
        "name": "Database for internal use",
        "description": "An internal database",
        "base_url": "http://example.com/internaldb/optimade",
        "homepage": "http://example.com",
        "link_type": "child",
        "aggregate": "no",
        "no_aggregate_reason": "This is a database for internal use and might contain nonsensical data"
      }
    },
    {
      "type": "links",
      "id": "frameworks",
      "attributes": {
        "name": "Some other DB",
        "description": "A DB by the example2 provider",
        "base_url": "http://example2.com/some_db/optimade",
        "homepage": "http://example2.com",
        "link_type": "external"
      }
    },
    {
      "type": "links",
      "id": "optimade",
      "attributes": {
        "name": "Materials Consortia",
        "description": "List of OPTIMADE providers maintained by the Materials Consortia organisation",
        "base_url": "https://providers.optimade.org",
        "homepage": "https://optimade.org",
        "link_type": "providers"
      }
    }
  ]
}

5.7 Custom Extension Endpoints

API implementations MAY provide custom endpoints under the Extensions endpoint. Custom extension endpoints MUST be placed under the versioned or unversioned base URL serving the API at /extensions. The API implementation is free to define roles of further URL path segments under this URL.

6.1 Lexical Tokens

The following tokens are used in the filter query component:

• Property names: the first character MUST be a lowercase letter, the subsequent symbols MUST be composed of lowercase letters or digits; the underscore ("_", ASCII 95 dec (0x5F)) is considered to be a lower-case letter when defining identifiers. The length of the identifiers is not limited, except that when passed as a URL query parameter the whole query SHOULD NOT be longer than the limits imposed by the URI specification. This definition is similar to one used in most widespread programming languages, except that OPTIMADE limits allowed letter set to lowercase letters only. This allows to tell OPTIMADE identifiers and operator keywords apart unambiguously without consulting a reserved word table and to encode this distinction concisely in the EBNF Filter Language grammar.

  Examples of valid property names:

  • band_gap

  • cell_length_a

  • cell_volume

  Examples of incorrect property names:

  • 0_kvak (starts with a number);

  • "foo bar" (contains space; contains quotes)

  • BadLuck (contains upper-case letters)

  Identifiers that start with an underscore are specific to a database or definition provider, and MUST be on the format of a namespace prefix (see section Namespace Prefixes).

  Examples:

  • _exmpl_formula_sum (a property specific to that database)

  • _exmpl_band_gap

  • _exmpl_supercell

  • _exmpl_trajectory

  • _exmpl_workflow_id

• Nested property names A nested property name is composed of at least two identifiers separated by periods (.).

• String values MUST be surrounded by double quote characters (", ASCII symbol 34 dec, 0x22 hex). A double quote that is a part of the value, not a delimiter, MUST be escaped by prepending it with a backslash character (\\, ASCII symbol 92 dec, 0x5C hex). A backslash character that is part of the value (i.e., not used to escape a double quote) MUST be escaped by prepending it with another backslash. An example of an escaped string value, including the enclosing double quotes, is given below:

  • "A double quote character (\", ASCII symbol 34 dec) MUST be prepended by a backslash (\\, ASCII symbol 92 dec) when it is a part of the value and not a delimiter; the backslash character \"\\\" itself MUST be preceded by another backslash, forming a double backslash: \\\\"

  (Note that at the end of the string value above the four final backslashes represent the two terminal backslashes in the value, and the final double quote is a terminator, it is not escaped.)

  String value tokens are also used to represent timestamps in form of the RFC 3339 Internet Date/Time Format.

• Numeric values are represented as decimal integers or in scientific notation, using the usual programming language conventions. A regular expression giving the number syntax is given below as a POSIX Extended Regular Expression (ERE) or as a Perl-Compatible Regular Expression (PCRE):

  • ERE: [-+]?([0-9]+(.[0-9]*)?|.[0-9]+)([eE][-+]?[0-9]+)?

  • PCRE: [-+]?(?:d+(.d*)?|.d+)(?:[eE][-+]?d+)?

An implementation of the search filter MAY reject numbers that are outside the machine representation of the underlying hardware; in such case it MUST return the error 501 Not Implemented with an appropriate error message that indicates the cause of the error and an acceptable number range.

• Examples of valid numbers:

  • 12345, +12, -34, 1.2, .2E7, -.2E+7, +10.01E-10, 6.03e23, .1E1, -.1e1, 1.e-12, -.1e-12, 1000000000.E1000000000, 1., .1

• Examples of invalid numbers (although they MAY contain correct numbers as substrings):

  • 1.234D12, .e1, -.E1, +.E2, 1.23E+++, +-123

• Note: this number representation is more general than the number representation in JSON (for instance, 1. is a valid numeric value for the filtering language specified here, but is not a valid float number in JSON, where the correct format is 1.0 instead).

While the filtering language supports tests for equality between properties of floating point type and decimal numbers given in the filter string, such comparisons come with the usual caveats for testing for equality of floating point numbers. Normally, a client cannot rely on that a floating point number stored in a database takes on a representation that exactly matches the one obtained for a number given in the filtering string as a decimal number or as an integer. However, testing for equality to zero MUST be supported.

More examples of the number tokens and machine-readable definitions and tests can be found in the Materials-Consortia API Git repository (files integers.lst, not-numbers.lst, numbers.lst, and reals.lst).

• Boolean values are represented with the tokens TRUE and FALSE.

• Operator tokens are represented by usual mathematical relation symbols or by case-sensitive keywords. Currently the following operators are supported: =, !=, <=, >=, <, > for tests of number, string (lexicographical) or timestamp (temporal) equality, inequality, less-than, more-than, less, and more relations; AND, OR, NOT for logical conjunctions, and a number of keyword operators discussed in the next section.

  In future extensions, operator tokens that are words MUST contain only upper-case letters. This requirement guarantees that no operator token will ever clash with a property name.

6.2 The Filter Language Syntax

All filtering expressions MUST follow the EBNF grammar of appendix The Filter Language EBNF Grammar of this specification. The appendix contains a complete machine-readable EBNF, including the definition of the lexical tokens described above in section Lexical Tokens. The EBNF is enclosed in special strings constructed as BEGIN and END, both followed by EBNF GRAMMAR Filter, to enable automatic extraction.

identifier <operator> constant
constant <operator> identifier

identifier <operator> identifier
constant <operator> constant

identifier IS KNOWN
identifier IS UNKNOWN

7.1 Property Definition keys from JSON Schema

In addition to the requirements on the format of a Property Definition in the previous section, it MUST also adhere to the OPTIONAL and REQUIRED keys described in this subsection. The format described in this subsection forms a subset of the JSON Schema Validation Draft 2020-12 and JSON Schema Core Draft 2020-12 standards.

REQUIRED keys

• type: List. Specifies the corresponding JSON type for this level of the defined property and whether the property can be null or not. The value is directly correlated with x-optimade-type (cf. the definition of the x-optimade-type field).

  It MUST be a list of one or two elements where the first element is a string correlated with x-optimade-type as follows; if x-optimade-type is:

  • "boolean", "string", or "integer" then type is the same string.

  • "dictionary" then type is "object".

  • "list" then type is "array".

  • "float" then type is "number".

  • "timestamp" then type is "string".

  If the second element is included, it MUST be the string "null". This two element form specifies that the defined property can be null.

  The inclusion or not of "null" in the field type for a subfield defined at a nested level by a Property Definition declares if that subfield is nullable. Property Definitions for which the nullability of a subfield differs MUST NOT share the same $id. However, the nullability of the subfield SHOULD NOT be taken into account when comparing the nested Property Definition for that subfield with other definitions, i.e., a nullable and non-nullable subfield that are otherwise defined the same SHOULD share the same $id. Hence, formally OPTIMADE Property Definitions regard nullability of a subfield to belong one level above where it appears in the JSON Schema definition.

  Implementation notes:

  • The field type can be derived from the field x-optimade-type and its role is only to provide the JSON type names corresponding to x-optimade-type. The motivation to include these type names is that it makes the JSON representation of a Property Definition a fully valid standard JSON Schema. Nevertheless, for consistency across formats, these JSON type names MUST still be included when a property definition is represented in other output formats (i.e., the JSON names MUST NOT be translated into the type names of that output format).

  • The allowed values of the type field are highly restricted compared to what is permitted using the full JSON Schema standard. Values can only be defined to be a single OPTIMADE data type or, optionally, null. This restriction is intended to reduce the complexity of possible data types that implementations have to handle in different formats and database backends.

Keys that are REQUIRED on the outermost level of a Property Definition, but otherwise OPTIONAL:

• $schema: String. A URL for a meta schema that describes the Property Definitions format. For Property Definitions adhering to the format described in this document, it should be set to: https://schemas.optimade.org/meta/v1.2/optimade/property_definition.json.

• $id: String. A static IRI identifier that is a URN or URL representing the specific version of this level of the defined property. (If it is a URL, clients SHOULD NOT assign any interpretation to the response when resolving that URL.) It SHOULD NOT be changed as long as the property definition remains the same, and MUST be changed when the property definition changes.

• title: String. A short single-line human-readable explanation of the defined property appropriate to show as part of a user interface.

• description: String. A human-readable multi-line description that explains the purpose, requirements, and conventions of the defined property. The format SHOULD be a one-line description, followed by a new paragraph (two newlines), followed by a more detailed description of all the requirements and conventions of the defined property. Formatting in the text SHOULD use Markdown in the CommonMark v0.3 format format, with mathematical expressions written to render correctly with the LaTeX mode of Mathjax 3.2. When possible, it is preferable for mathematical expressions to use as straightforward notation as possible to make them readable also when not rendered.

OPTIONAL keys

• $comment: String. A human-readable comment relevant in the context of the raw definition data. These comments should normally not be shown to the end users. Comments pertaining to the Property Definition that are relevant to end users should go into the field description. Formatting in the text SHOULD use Markdown using the format described in the definition of the description field.

• deprecated: Boolean. If TRUE, implementations SHOULD not use the defined property, and it MAY be removed in the future. If FALSE, the defined property is not deprecated. The field not being present means FALSE. A Property Definition marked as deprecated is generally considered to be the same as its non-deprecated counterpart, i.e., it SHOULD retain its $id.

• examples: List. A list of example values that the defined property can have. These examples MUST all be of a data type that matches the type field and otherwise adhere to the rest of the Property Definition.

• enum: List. The defined property MUST take one of the values given in the provided list. The items in the list MUST all be of a data type that matches the type field and otherwise adhere to the rest of the Property Definition. If this key is given, for null to be a valid value of the defined property, the list MUST contain a null value and the type MUST be a list where the second value is the string "null".

Furthermore, depending on what string the type is equal to, or contains as first element, the following additional requirements also apply:

• "object":

  REQUIRED

  • properties: Dictionary. Gives key-value pairs where each value is an inner Property Definition. The defined property is a dictionary that can only contain keys present in this dictionary, and, if so, the corresponding value is described by the respective inner Property Definition. (Or, if the type field is the list "object" and "null", it can also be null.)

  OPTIONAL

  • required: List. The list MUST only contain strings. The defined property MUST have keys that match all the strings in this list. Other keys present in the properties field are OPTIONAL in the defined property. If not present or empty, all keys in properties are regarded as OPTIONAL.

  • maxProperties: Integer. The defined property is a dictionary where the number of keys MUST be less than or equal to the number given.

  • minProperties: Integer. The defined property is a dictionary where the number of keys MUST be greater than or equal to the number given.

  • dependentRequired: Dictionary. The dictionary keys are strings and the values are lists of unique strings. If the defined property has a key that is equal to a key in the given dictionary, the defined property MUST also have keys that match each of the corresponding values. No restriction is inferred from this field for keys in the defined property that do not match any key in the given dictionary.

• "array":

  REQUIRED

  • items: Dictionary. Specifies an inner Property Definition. The defined property is a list where each item MUST match this inner Property Definition.

  OPTIONAL

  • uniqueItems: Boolean. If TRUE, the defined property is an array that MUST only contain unique items. If FALSE, this field sets no limitation on the defined property.

  Furthermore, despite being defined in the JSON Schema standard, the fields minItems and maxItems MUST NOT be used to indicate limits of the number of items of a list. See the definition of the x-optimade-dimensions field for more information.

• "integer":

  OPTIONAL

  • multipleOf: Integer. An integer is strictly greater than 0. The defined property MUST have an integer value that when divided by the given integer results in an integer (i.e., it must be even divisible by this integer without a fractional part).

  • maximum: Integer. The defined property is an integer that MUST be less than or equal to this number.

  • exclusiveMaximum: Integer. The defined property is an integer that MUST be strictly less than this number; it cannot be equal to the number.

  • minimum: Integer. The defined property is an integer that MUST be greater than or equal to this number.

  • exclusiveMinimum: Integer. The defined property is an integer that MUST be strictly greater than this number; it cannot be equal to the number.

• "number":

  OPTIONAL

  • multipleOf: Float. An integer is strictly greater than 0. The defined property MUST have an integer value that when divided by the given integer results in an integer (i.e., it must be even divisible by this integer without a fractional part).

  • maximum: Float. The defined property is a float that MUST be less than or equal to this number.

  • exclusiveMaximum: Float. The defined property is a float that MUST be strictly less than this number; it cannot be equal to the number.

  • minimum: Float. The defined property is a float that MUST be greater than or equal to this number.

  • exclusiveMinimum: Float. The defined property is a float that MUST be strictly greater than this number; it cannot be equal to the number.

• "string":

  OPTIONAL

  • maxLength: Integer. A non-negative integer. The defined property is a string that MUST have a length that is less than or equal to the given integer. (The length of the string is the number of individual Unicode characters it is composed of.)

  • minLength: Integer. A non-negative integer. The defined property is a string that MUST have a length that is less than or equal to the given integer. (The definition of the length of a string is the same as in the field maxLength.)

  • format: String. Choose one of the following values to indicate that the defined property is a string that MUST adhere to the specified format:

    • "date-time": the date-time production in RFC 3339 section 5.6.

    • "date": the full-date production in RFC 3339 section 5.6.

    • "time": the full-time production in RFC 3339 section 5.6.

    • "duration": the duration production in RFC 3339 Appendix A.

    • "email": the "Mailbox" ABNF rule in RFC 5321 section 4.1.2.

    • "uri": a string instance is valid against this attribute if it is a valid URI according to RFC 3986.

    • "iri": a string instance is valid against this attribute if it is a valid IRI according to RFC 3987.

  • pattern: String. This string SHOULD be a valid regular expression, according to the ECMA-262 regular expression dialect. A string instance is considered valid if the regular expression matches the instance successfully. The regular expression is not implicitly anchored, i.e., it can match the string at any position unless the expression contains a leading '^' or a trailing '$'.

A complete example of a Property Definition is found in the appendix Property Definition Example.

7.2 Physical Units in Property Definitions

In OPTIMADE, there is no facility to allow a property to be represented in a choice of units, e.g., either ångström (Å) or meter (m). The unit is always permanently fixed by the Property Definition. Clients and servers that use other units internally thus have to do unit conversions as part of preparing and processing OPTIMADE responses.

The physical unit of a property, the embedded items of a list, or values of a dictionary, are defined with the field x-optimade-unit with the following requirements:

• The field MUST be given with a non-null value both at the highest level in the OPTIMADE Property Definition and all inner Property Definitions.

• If the property refers to a physical quantity that is dimensionless and unitless (often also referred to as having the dimension 1) or refers to a dimensionless and unitless count of something (e.g., the number of protons in a nucleus) the field MUST have the value dimensionless. However, quantities that use counting units, e.g., the mole, or quantities that use dimensionless units, e.g., the radian MUST NOT set the field to dimensionless.

• If the property refers to an entity for which the assignment of a unit would not make sense, e.g., a string representing a chemical formula or a serial number the field MUST have the value inapplicable.

• If the field does not take the value dimensionless or inapplicable, it MUST be set to a single unit symbol or a Compound Unit Expressions from a set of unit symbols using the format described in Compound Unit Expressions.

• All unit symbols used in x-optimade-unit fields at any level in a Property Definition MUST be defined in the units field inside the x-optimade-property field in the outermost level of the Property Definition, or in the units field in the Entry info endpoint (the latter is only possible for Property Definitions embedded in such a response).

• The units MUST be a list of dictionaries using the format for OPTIMADE Physical Unit Definitions described in Physical Unit Definitions.

7.3 Compound Unit Expressions

A Compound Unit Expression is formed by a sequence of symbols for units or constants separated by a single multiplication * character. Each symbol can also be suffixed by a single ^ character followed by a positive or negative integer to indicate the power of the preceding symbol, e.g., m^3 for cubic meter, m^-3 for inverse cubic meter. (Positive integers MUST NOT be preceded by a plus sign.) Each unit or constant symbol MAY be directly prefixed by a prefix symbol. A prefix symbol MUST be directly followed by a unit symbol, i.e., it MUST NOT be used on its own, and MUST NOT be followed by ^ to indicate a power. When defining prefix symbols it is important to ensure that they do not introduce ambiguity. If there are ambiguous interpretations of a symbol as either having or not having a prefix, it MUST be interpreted as a unit without a prefix.

Furthermore:

• No whitespace, parentheses, or other symbols than specified above are permitted.

• The (prefixed) unit and constant symbols MUST appear in alphabetical order.

7.4 Physical Unit Definitions

An OPTIMADE Physical Unit Definition is a dictionary adhering to the following format:

REQUIRED keys:

• $schema: String. A URL for a meta schema that describes the Physical Unit Definitions format. For Property Definitions adhering to the format described in this document, it should be set to: https://schemas.optimade.org/meta/v1.2/optimade/physical_unit_definition.json.

• x-optimade-definition: Dictionary. The same field as defined in the definition of the x-optimade-definition field for Property Definitions but where the kind subfield MUST be unit.

• $id: String. A static IRI identifier that is a URN or URL representing the specific version of the Physical Unit Definition. (If it is a URL, clients SHOULD NOT assign any interpretation to the response when resolving that URL.) It SHOULD NOT be changed as long as the Physical Unit Definition remains the same, and SHOULD be changed when the definition changes. Physical Unit Definitions SHOULD be regarded as the same if they only differ by:

  • Additions of annotating notes to end of the description field.

  • Changes to the following specific fields at any level: deprecated and $comment.

• symbol: String. Specifies the symbol to be used in x-optimade-unit to reference this unit.

• title: String. A human-readable single-line string name for the unit.

• description: String. A human-readable multiple-line detailed description of the unit.

  Additions appended to the end of the description field that are clearly marked as notes that clarify the definition without changing it are viewed as annotations to the Physical Unit Definition rather than an integral part of it. Such annotations SHOULD only be added to the end of an otherwise unmodified description and MUST NOT change the meaning or interpretation of the text above them. The purpose is to provide a way to add explanations and clarifications to a definition without having to regard it as a new definition. For example, these annotations to the description MAY be used to explain why a definition has been deprecated.

OPTIONAL keys:

• standard: Dictionary. This field is used to express that the unit is part of a preexisting standard. The dictionary has the following format:

  REQUIRED keys:

  • name: String. The abbreviated name of the standard being referenced. One of the following:

    • "si": the symbol is defined as part of the SI standard of unit symbols and prefixes.

    • "codata": the symbol is defined as part of one of the CODATA series of publications.

    • "iso-iec-80000": the symbol is defined in the iso-iec-80000 standard.

    • "gnu units": the symbol is a (compound) unit expression based on the symbols in the file definitions.units distributed with GNU Units software.

      A standard set of symbols for units and prefixes for OPTIMADE is taken from version 3.15 of the (separately versioned) unit database definitions.units included with the source distribution of GNU Units version 2.22. A prefix is indicated in the file by a trailing -, but that trailing character MUST NOT be included when using it as a prefix. If the unit is available in this database, or if it can be expressed as a Compound Unit Expression using these units and prefixes, the value of x-optimade-unit SHOULD use the (compound) string symbol. If there are multiple prefixes in the file with the same meaning, an implementation SHOULD use the shortest one consisting of only lowercase letters a-z and underscores, but no other symbols. If there are multiple ones with the same shortest length, then the first one of those SHOULD be used. For example, the GNU Units database defines the symbol "km" for kilometers by a combination of the k- SI kilo prefix and the m symbol for the SI meter unit.

    • "ucum": the symbol is defined in The Unified Code for Units of Measure (UCUM) standard.

    • "qudt": the symbol is defined in the QUDT standard. Not only symbols strictly defined within the standard are allowed, but also other compound unit expressions created according to the scheme for how new such symbols are formed in this standard.

  • symbol: String. The symbol to use from the referenced standard, expressed according to that standard. The field MAY use mathematical expressions written the same way as described in the definition of the description field. This field MAY be different from the symbol being defined via the definition if the unit will be referenced in x-optimade-unit field using a different symbol than the one used in the standard or if the symbol is expressed in the standard in a way that requires mathematical notation. However, if possible, the symbol field SHOULD be the same.

  OPTIONAL keys:

  • version: String. The version string of the referenced standard.

  • year: Integer. The year that the standard adopted the definition.

  • category The category of the definition in case the standard uses categories to organize definitions.

• alternate-symbols: List of String. A list of other symbols that are commonly associated with the unit. The stings MAY use mathematical expressions written the same way as described in the definition of the description field.

• property-format: String. Specifies the minor version of the Property Definitions format that the Physical Units Definition is expressed in. (The Physical Units Definition format is not versioned independently.) The format is the same as described above for the definition of the property-format field in Property Definitions. This field MUST be included when Physical Unit Definitions are used standalone, i.e., when they are not embedded inside a Property Definition that already declares a property-format at the top level.

• version: String. This string indicates the version of the Physical Unit Definition. The string SHOULD be in the format described by the semantic versioning v2 standard.

• resources: List of Dictionaries. A list of dictionaries that reference remote resources that describe the unit. The format of each dictionary is:

  REQUIRED keys:

  • relation: String. A human-readable description of the relationship between the unit and the remote resource, e.g., a "natural language description".

  • resource-id: String. An IRI of the external resource (which MAY be a resolvable URL).

• defining-relation: Dictionary. A dictionary that encodes a defining relation to another unit or set of units, with the primary intended use of relating a unit to its definition in SI units, if such a relationship exists. Some units, e.g., the atomic mass unit (also known as dalton, commonly denoted u), only has an approximate relationship to SI units, in which case the defining-relation MUST be omitted or null. The dictionary MUST adhere to the following format:

  OPTIONAL keys:

  • base-units: List of Dictionaries. A list specifying the base IRIs and unit symbols for the units in which the dimensional formula for the defining relation is expressed. Each item MUST be a dictionary that adheres to the following format:

    REQUIRED keys:

    • symbol: String. The symbol used to reference this unit in the dimensional formula.

    • id: String. The IRI of one of the units referenced in the dimensional formula for the defining relation.

  • base-units-expression: String. A string expressing the base units part of the defining relation for the unit being defined. It MUST adhere to the format for compound unit expression described in Physical Units in Property Definitions. If the field is missing or null the base-units-expression is taken to be equal to 1, i.e., the defining relation is dimensionless.

  • scale: Dictionary. A dictionary specifying the scale in the defining relation, adhering to the following format:

    OPTIONAL keys:

    • numerator: Integer.

    • denominator: Integer.

    • base: Integer.

    • exponent: Integer.

    These four fields specify the value as the rational number numerator / denominator, multiplied by base to the power of exponent. If omitted or null, the defaults for the numerator, denominator, base, and exponent are respectively 1, 1, 10, and 0.

    • standard_uncertainty: Float. The standard uncertainty of the value used in the defining relation. Some definitions define an entity (e.g. a constant) to a specific value along with an uncertainty of that value.

  • offset: Dictionary. A dictionary specifying the offset value, adhering to the same format as scale. If omitted or null, the defaults for the numerator, denominator, and exponent are respectively 0, 1, and 0.

  If the fields in scale are designated as sn, sd, and se; and the fields in offset are designated as on, od, and oe; and base-units-expression is designated as b, these fields state the following defining relation: a value v multiplied by the unit being defined is equal to the following expression (v * (sn/sd) * 10**se + (on/od) * 10**oe)*b, where * designates multiplication and ** designates exponentiation. For example, the defining relation of the temperature unit Fahrenheit F in Celsius C, that says that x F = (x - 32) * (5/9) C = 5/9 x + (-160/9) C could be expressed as follows:

  "defining-relation": {
    "base-units": [
      {
        "symbol": "C",
        "id": "https://units.example.com/celsius"
      }
    ],
    "base-units-expression": "C",
    "scale": {
      "numerator": 5,
      "denominator": 9
    },
    "offset": {
      "numerator": -160,
      "denominator": 9
    }
  }

• approximate-relations: List of Dictionary. A list of dictionaries that encode approximate relations to another unit or set of units. The intended use is to express one or a few approximate relationships from the unit being defined to other unit systems (primarily intended to be SI). This field is useful for units not defined by such a relationship, in which case the defining-relation field would be used. For example, the atomic mass unit (also known as dalton, commonly denoted u) is defined as one twelfth of the mass of a free carbon-12 atom at rest and only has an approximate relationship to the SI kilogram. While this field allows expressing multiple relationships, the intent is only to provide the most relevant relationships (e.g., to an SI base unit) from which other relationships can be derived.

  Each element in the list MUST be a dictionary adhering to the following format:

  OPTIONAL keys:

  • base-units: List of Dictionaries, and base-units-expression: String. These fields take the same format and roles as in the definition of defining-relation

  • scale: Dictionary. A dictionary specifying the scale in the approximate relation. It MUST adhere to the following format:

    REQUIRED keys:

    • value: Float. The value of the scale in the approximate relation.

    OPTIONAL keys:

    • standard_uncertainty: Float. The standard uncertainty of the value in the approximate relation.

    • relative_standard_uncertainty: Float. The relative standard uncertainty of the value in the approximate relation.

  • offset: Dictionary. A dictionary specifying the offset in the approximate relation. It MUST adhere to the same format as the scale field above.

  The values for scale and offset take the same meaning as in the definition of defining-relation to express a relationship between the unit being defined and the compound unit expression in base-units-expression.

• deprecated: Boolean. If TRUE, implementations SHOULD not use the unit defined in this Physical Unit Definition. If FALSE, the unit defined in this Physical Unit Definition is not deprecated. The field not being present means FALSE.

• $comment: String. A human-readable comment relevant in the context of the raw definition data. These comments should normally not be shown to the end users. Comments pertaining to the Property Definition that are relevant to end users should go into the field description. Formatting in the text SHOULD use Markdown using the format described in the definition of the description field of Property Definitions.

An example of a Physical Unit Definition, including a defining relation that involves more than one other unit, is embedded in the example of a Property Definition in the appendix Property Definition Example.

7.5 Prefixes and constants

Prefixes and constants are defined in OPTIMADE using nearly identical schemas as the one for units in Physical Unit Definitions. The only difference is that for prefixes:

• The $schema SHOULD be set to: "https://schemas.optimade.org/meta/v1.2/optimade/prefix_definition.json".

• The subfield kind of the field x-optimade-definition MUST be prefix.

And for Constants:

• The $schema SHOULD be set to: "https://schemas.optimade.org/meta/v1.2/optimade/constant_definition.json".

• The subfield kind of the field x-optimade-definition MUST be constant.

7.6 Unrecognized keys in property definitions

Implementations MAY add their own keys in Property Definitions, both inside and outside of the fields x-optimade-property, x-optimade-implementation, and x-optimade-requirements in the form of x-exmpl-name where exmpl is the database-specific prefix (without underscore characters) and name is the part of the key chosen by the implementation. Implementations MUST NOT add keys to property definitions on other formats.

Client and server implementations that interpret an OPTIMADE Property Definition and encounter unrecognized keys starting with x-exmpl- where exmpl is a recognized database prefix MAY issue errors or warnings. Other unrecognized keys starting with x- MUST NOT issue errors, SHOULD NOT issue warnings, and MUST otherwise be ignored.

To allow forward compatibility with future versions of both OPTIMADE and the JSON Schema standards, unrecognized keys that do not start with x- SHOULD issue a warning but MUST otherwise be ignored.

8.1 Properties Used by Multiple Entry Types

8.2 Structures Entries

structures entries (or objects) have the properties described above in section Properties Used by Multiple Entry Types, as well as the following properties:

8.3 Calculations Entries

The calculations entries have the properties described above in section Properties Used by Multiple Entry Types.

8.4 References Entries

The references entries describe bibliographic references. The following properties are used to provide the bibliographic details:

• address, annote, booktitle, chapter, crossref, edition, howpublished, institution, journal, key, month, note, number, organization, pages, publisher, school, series, title, volume, year: meanings of these properties match the BibTeX specification, values are strings;

• bib_type: type of the reference, corresponding to type property in the BibTeX specification, value is string;

• authors and editors: lists of person objects which are dictionaries with the following keys:

  • name: Full name of the person, REQUIRED.

  • firstname, lastname: Parts of the person's name, OPTIONAL.

• doi and url: values are strings.

• Requirements/Conventions:

  • Support: OPTIONAL support in implementations, i.e., any of the properties MAY be null.

  • Query: Support for queries on any of these properties is OPTIONAL. If supported, filters MAY support only a subset of comparison operators.

  • Every references entry MUST contain at least one of the properties.

Example:

{
  "data": {
    "type": "references",
    "id": "Dijkstra1968",
    "attributes": {
      "authors": [
        {
          "name": "Edsger Dijkstra",
          "firstname": "Edsger",
          "lastname": "Dijkstra"
        }
      ],
      "year": "1968",
      "title": "Go To Statement Considered Harmful",
      "journal": "Communications of the ACM",
      "doi": "10.1145/362929.362947"
    }
  }
}

8.5 Files Entries

The files entries describe files. The following properties are used to do so:

8.6 Custom Entry Types

Database and definition providers can define custom entry types. The names of such entry types MUST start with corresponding namespace prefix (see appendix Namespace Prefixes). Custom entry types MUST have all properties described above in section Properties Used by Multiple Entry Types.

• Requirements/Conventions for properties in custom entry types:

  • Support: Support for any properties in database-provider-specific or definition-provider-specific entry types is fully OPTIONAL.

  • Query: Support for queries on these properties are OPTIONAL. If supported, only a subset of the filter features MAY be supported.

8.7 Relationships Used by Multiple Entry Types

In accordance with section Relationships, all entry types MAY use relationships to describe relations to other entries.

{
  "data": {
    "type": "structures",
    "id": "example.db:structs:1234",
    "attributes": {
      "formula": "Es2",
      "url": "http://example.db/structs/1234",
      "immutable_id": "http://example.db/structs/1234@123",
      "last_modified": "2007-04-07T12:02:20Z"
    },
    "relationships": {
      "references": {
        "data": [
          { "type": "references", "id": "Dijkstra1968" },
          {
            "type": "references",
            "id": "1234",
            "meta": {
              "description": "Reference for the general crystal prototype."
            }
          }
        ]
      }
    }
  },
  "included": [
    {
      "type": "references",
      "id": "Dijkstra1968",
      "attributes": {
        "authors": [
          {
            "name": "Edsger Dijkstra",
            "firstname": "Edsger",
            "lastname": "Dijkstra"
          }
        ],
        "year": "1968",
        "title": "Go To Statement Considered Harmful",
        "journal": "Communications of the ACM",
        "doi": "10.1145/362929.362947"
      }
    },
    {
      "type": "references",
      "id": "1234",
      "attributes": {
        "doi": "10.1234/1234"
      }
    }
  ]
}

{
  "data": {
    "type": "structures",
    "id": "example.db:structs:1234",
    "attributes": {
      "chemical_formula_descriptive": "H2O"
    },
    "relationships": {
      "files": {
        "data": [
          { "type": "files", "id": "example.db:files:1234" }
        ]
      }
    }
  },
  "included": [
    {
      "type": "files",
      "id": "example.db:files:1234",
      "attributes": {
        "media_type": "chemical/x-cif",
        "url": "https://example.org/files/cifs/1234.cif"
      }
    }
  ]
}

9.1 The Filter Language EBNF Grammar

(* BEGIN EBNF GRAMMAR Filter *)
(* The top-level 'filter' rule: *)

Filter = [Spaces], Expression ;

(* Values *)

OrderedConstant = String | Number ;
UnorderedConstant = ( TRUE | FALSE ) ;

Value = ( UnorderedConstant | OrderedValue ) ;

OrderedValue = ( OrderedConstant | Property ) ;
(* Note: support for Property in OrderedValue is OPTIONAL *)

ValueListEntry = ( Value | ValueEqRhs | ValueRelCompRhs | FuzzyStringOpRhs ) ;
(* Note: support for ValueEqRhs, ValueRelCompRhs and FuzzyStringOpRhs in ValueListEntry are OPTIONAL *)

ValueList = ValueListEntry, { Comma, ValueListEntry } ;
ValueZip = ValueListEntry, Colon, ValueListEntry, { Colon, ValueListEntry } ;

ValueZipList = ValueZip, { Comma, ValueZip } ;

(* Expressions *)

Expression = ExpressionClause, [ OR, Expression ] ;

ExpressionClause = ExpressionPhrase, [ AND, ExpressionClause ] ;

ExpressionPhrase = [ NOT ], ( Comparison | OpeningBrace, Expression, ClosingBrace ) ;

Comparison = ConstantFirstComparison
           | PropertyFirstComparison ;
(* Note: support for ConstantFirstComparison is OPTIONAL *)

ConstantFirstComparison = ( OrderedConstant, ValueOpRhs
                          | UnorderedConstant, ValueEqRhs ) ;

PropertyFirstComparison = Property, [ ValueOpRhs
                                    | KnownOpRhs
                                    | FuzzyStringOpRhs
                                    | SetOpRhs
                                    | SetZipOpRhs
                                    | LengthOpRhs ] ;
(* Note: support for SetZipOpRhs in Comparison is OPTIONAL *)

ValueOpRhs = ( ValueEqRhs | ValueRelCompRhs ) ;

ValueEqRhs = EqualityOperator, Value ;

ValueRelCompRhs = RelativeComparisonOperator, OrderedValue ;

KnownOpRhs = IS, ( KNOWN | UNKNOWN ) ;

FuzzyStringOpRhs = CONTAINS, Value
                 | STARTS, [ WITH ], Value
                 | ENDS, [ WITH ], Value ;

SetOpRhs = HAS, ( ( Value | EqualityOperator, Value | RelativeComparisonOperator, OrderedValue | FuzzyStringOpRhs ) | ALL, ValueList | ANY, ValueList | ONLY, ValueList ) ;
(* Note: support for the alternatives with EqualityOperator, RelativeComparisonOperator, FuzzyStringOpRhs, and ONLY in SetOpRhs are OPTIONAL *)

SetZipOpRhs = PropertyZipAddon, HAS, ( ValueZip | ONLY, ValueZipList | ALL, ValueZipList | ANY, ValueZipList ) ;

PropertyZipAddon = Colon, Property, { Colon, Property } ;

LengthOpRhs = LENGTH, [ Operator ], Value ;
(* Note: support for [ Operator ] in LengthOpRhs is OPTIONAL *)

(* Property *)

Property = Identifier, { Dot, Identifier } ;

(* TOKENS *)

(* Separators: *)

OpeningBrace = '(', [Spaces] ;
ClosingBrace = ')', [Spaces] ;

Dot = '.', [Spaces] ;
Comma = ',', [Spaces] ;
Colon = ':', [Spaces] ;

(* Boolean relations: *)

AND = 'AND', [Spaces] ;
NOT = 'NOT', [Spaces] ;
OR = 'OR', [Spaces] ;

IS = 'IS', [Spaces] ;
KNOWN = 'KNOWN', [Spaces] ;
UNKNOWN = 'UNKNOWN', [Spaces] ;

CONTAINS = 'CONTAINS', [Spaces] ;
STARTS = 'STARTS', [Spaces] ;
ENDS = 'ENDS', [Spaces] ;
WITH = 'WITH', [Spaces] ;

LENGTH = 'LENGTH', [Spaces] ;
HAS = 'HAS', [Spaces] ;
ALL = 'ALL', [Spaces] ;
ONLY = 'ONLY', [Spaces] ;
ANY = 'ANY', [Spaces] ;

(* Comparison operator tokens: *)

Operator = ( EqualityOperator | RelativeComparisonOperator ) ;
EqualityOperator = [ '!' ], '=' , [Spaces] ;
RelativeComparisonOperator = ( '<' | '>' ), [ '=' ], [Spaces] ;

(* Boolean values *)

TRUE = 'TRUE', [Spaces] ;
FALSE = 'FALSE', [Spaces] ;

(* Property syntax *)

Identifier = LowercaseLetter, { LowercaseLetter | Digit }, [Spaces] ;

Letter = UppercaseLetter | LowercaseLetter ;

UppercaseLetter = 'A' | 'B' | 'C' | 'D' | 'E' | 'F' | 'G' | 'H' | 'I'
                | 'J' | 'K' | 'L' | 'M' | 'N' | 'O' | 'P' | 'Q' | 'R'
                | 'S' | 'T' | 'U' | 'V' | 'W' | 'X' | 'Y' | 'Z' ;

LowercaseLetter = 'a' | 'b' | 'c' | 'd' | 'e' | 'f' | 'g' | 'h' | 'i'
                | 'j' | 'k' | 'l' | 'm' | 'n' | 'o' | 'p' | 'q' | 'r'
                | 's' | 't' | 'u' | 'v' | 'w' | 'x' | 'y' | 'z' | '_' ;

(* Strings: *)

String = '"', { EscapedChar }, '"', [Spaces] ;

EscapedChar = UnescapedChar | '\', '"' | '\', '\' ;

UnescapedChar = Letter | Digit | Space | Punctuator | UnicodeHighChar ;

Punctuator = '!' | '#' | '$' | '%' | '&' | "'" | '(' | ')' | '*' | '+'
           | ',' | '-' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?'
           | '@' | '[' | ']' | '^' | '`' | '{' | '|' | '}' | '~' ;

(* BEGIN EBNF GRAMMAR Number *)
(* Number token syntax: *)

Number = [ Sign ] ,
         ( Digits, [ '.', [ Digits ] ] | '.' , Digits ),
         [ Exponent ], [Spaces] ;

Exponent =  ( 'e' | 'E' ) , [ Sign ] , Digits ;

Sign = '+' | '-' ;

Digits =  Digit, { Digit } ;

Digit = '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' ;

(* White-space: *)

(* Special character tokens: *)

tab = ? \t ?;
nl  = ? \n ?;
cr  = ? \r ?;
vt  = ? \v ?;
ff  = ? \f ?;

Space = ' ' | tab | nl | cr | vt | ff ;

Spaces = Space, { Space } ;

(* The 'UnicodeHighChar' specifies any Unicode character above 0x7F.
   It is specified in this grammar by an extension to EBNF that allows a
   regular expression to specify terminal symbol ranges. *)

UnicodeHighChar = ? [^\x00-\x7F] ? ;

(* END EBNF GRAMMAR Number *)
(* END EBNF GRAMMAR Filter *)

Note: when implementing a parser according this grammar, the implementers MAY choose to construct a lexer that ignores all whitespace (spaces, tabs, newlines, vertical tabulation and form feed characters, as described in the grammar 'Space' definition), and use such a lexer to recognize language elements that are described in the (* TOKENS *) section of the grammar. In that case, it can be beneficial to remove the '[Spaces]' element from the Filter = [Spaces], Expression definition as well and use the remaining grammar rules as a parser generator input (e.g., for yacc, bison, antlr).

9.2 Regular Expressions for OPTIMADE Filter Tokens

The string below contains Perl-Compatible Regular Expressions to recognize identifiers, number, and string values as specified in this specification.

#BEGIN PCRE identifiers
[a-z_][a-z_0-9]*
#END PCRE identifiers

#BEGIN PCRE numbers
[-+]?(?:\d+(\.\d*)?|\.\d+)(?:[eE][-+]?\d+)?
#END PCRE numbers

#BEGIN PCRE strings
"([^\\"]|\\.)*"
#END PCRE strings

The strings below contain Extended Regular Expressions (EREs) to recognize identifiers, number, and string values as specified in this specification.

#BEGIN ERE identifiers
[a-z_][a-z_0-9]*
#END ERE identifiers

#BEGIN ERE numbers
[-+]?([0-9]+(\.[0-9]*)?|\.[0-9]+)([eE][-+]?[0-9]+)?
#END ERE numbers

#BEGIN ERE strings
"([^\"]|\\.)*"
#END ERE strings

9.3 The Symmetry Operation String Regular Expressions

Symmetry operation strings that comprise the space_group_symmetry_operations_xyz property MUST conform to the following regular expressions. The regular expressions are recorded below in two forms, one in a more readable form using variables and the other as an explicit pattern compatible with the OPTIMADE Regular Expression Format.

• Perl Compatible Regular Expression (PCRE) syntax, with Perl extensions used for readability and expressivity. The symop_definitions section defines several variables in Perl syntax that capture common parts of the regular expressions (REs) and need to be interpolated into the final REs used for matching. The symops section contains the REs themselves. The whitespace characters in these definitions are not significant; if used in Perl programs, these expressions MUST be processed with the /x RE modifier. A working example of these REs in action can be found in the tests/cases/pcre_symops_001.sh and other test cases.

  #BEGIN PCRE symop_definitions
  
  $translations = '1\/2|[12]\/3|[1-3]\/4|[1-5]\/6';
  
  $symop_translation_appended = "[-+]? [xyz] ([-+][xyz])? ([-+] ($translations) )?";
  $symop_translation_prepended = "[-+]? ($translations) ([-+] [xyz] ([-+][xyz])? )?";
  
  $symop_re = "($symop_translation_appended|$symop_translation_prepended)";
  
  #END PCRE symop_definitions

  #BEGIN PCRE symops
  
  ^ # From the beginning of the string...
  ($symop_re)(,$symop_re){2}
  $ # ... match to the very end of the string
  
  #END PCRE symops

• The regular expression is also provided in an expanded form as an OPTIMADE regex:

  #BEGIN ECMA symops
  
  ^([-+]?[xyz]([-+][xyz])?([-+](1/2|[12]/3|[1-3]/4|[1-5]/6))?|[-+]?(1/2|[12]/3|[1-3]/4|[1-5]/6)([-+][xyz]([-+][xyz])?)?),([-+]?[xyz]([-+][xyz])?([-+](1/2|[12]/3|[1-3]/4|[1-5]/6))?|[-+]?(1/2|[12]/3|[1-3]/4|[1-5]/6)([-+][xyz]([-+][xyz])?)?),([-+]?[xyz]([-+][xyz])?([-+](1/2|[12]/3|[1-3]/4|[1-5]/6))?|[-+]?(1/2|[12]/3|[1-3]/4|[1-5]/6)([-+][xyz]([-+][xyz])?)?)$
  
  #END ECMA symops

9.4 OPTIMADE JSON lines partial data format

The OPTIMADE JSON lines partial data format is a lightweight format for transmitting property data that are too large to fit in a single OPTIMADE response. The format is based on JSON Lines, which enables streaming of JSON data. Note: since the below definition references both JSON fields and OPTIMADE properties, the data type names depend on context: for JSON they are, e.g., "array" and "object" and for OPTIMADE properties they are, e.g., "list" and "dictionary".

To aid the definition of the format below, we first define a "slice object" to be a JSON object describing slices of arrays. The dictionary has the following OPTIONAL fields:

• "start": Integer. The slice starts at the value with the given index (inclusive). The default is 0, i.e., the value at the start of the array.

• "stop": Integer. The slice ends at the value with the given index (inclusive). If omitted, the end of the slice is the last index of the array.

• "step": Integer. The absolute difference in index between two subsequent values that are included in the slice. The default is 1, i.e., every value in the range indicated by start and stop is included in the slice. Hence, a value of 2 denotes a slice of every second value in the array.

For example, for the array ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"] the slice object {"start": 1, "end": 7, "step": 3} refers to the items ["b", "e", "h"].

Furthermore, we also define the following special markers:

• The end-of-data-marker is this exact JSON: ["PARTIAL-DATA-END", [""]].

• A reference-marker is this exact JSON: ["PARTIAL-DATA-REF", ["<url>"]], where "<url>" is to be replaced with a URL being referenced. A reference-marker MUST only occur in a place where the property being communicated could have an embedded list.

• A next-marker is this exact JSON: ["PARTIAL-DATA-NEXT", ["<url>"]], where "<url>" is to be replaced with the target URL for the next link.

There is no requirement on the syntax or format of the URLs provided in these markers. When data is fetched from these URLs the response MUST use the JSON lines partial data format, i.e., the markers cannot be used to link to partial data provided in other formats. The markers have been deliberately designed to be valid JSON objects but not valid OPTIMADE property values. Since the OPTIMADE list data type is defined as a list of values of the same data type or null, the above markers cannot be encountered inside the actual data of an OPTIMADE property.

Implementation note: the recognizable string values for the markers should make it possible to prescreen the raw text of the JSON data lines for the reference-marker string to determine which are the lines that one can exclude from further processing to resolve references (alternatively, this screening can be done by the string parser used by the JSON parser). The underlying design idea is that for lines that have reference-markers, the time it takes to process the data structure to locate the markers should be negligible compared to the time it takes to resolve and handle the large data they reference. Hence, the most relevant optimization is to avoid spending time processing data structures to find markers for lines where there are none.

The full response MUST be valid JSON Lines that adheres to the following format:

• The first line is a header object (defined below).

• The following lines are data lines adhering to the formats described below.

• The final line is either an end-of-data-marker (indicating that there is no more data to be given), or a next-marker indicating that more data is available, which can be obtained by retrieving data from the provided URL.

The first line MUST be a JSON object providing header information. The header object MUST contain the keys:

• "optimade-partial-data": Object. An object identifying the response as being on OPTIMADE partial data format.

  It MUST contain the following key:

  • "format": String. Specifies the minor version of the partial data format used. The string MUST be of the format "MAJOR.MINOR", referring to the version of the OPTIMADE standard that describes the format. The version number string MUST NOT be prefixed by, e.g., "v". In implementations of the present version of the standard, the value MUST be exactly 1.2. A client MUST NOT expect to be able to parse the format value if the field is not a string of the format MAJOR.MINOR or if the MAJOR version number is unrecognized.

• "layout": String. A string either equal to "dense" or "sparse" to indicate whether the returned format uses a dense or sparse layout.

The following key is RECOMMENDED in the header object:

• "returned_ranges": Array of Object. For dense layout, and sparse layout of one dimensional list properties, the array contains a single element which is a slice object representing the range of data present in the response. In the specific case of a hierarchy of list properties represented as a sparse multidimensional array, if the field "returned_ranges" is given, it MUST contain one slice object per dimension of the multidimensional array, representing slices for each dimension that cover the data given in the response.

The header object MAY also contain the keys:

• "property_name": String. The name of the property being provided.

• "entry": Object. An object that MUST have the following two keys:

  • "id": String. The id of the entry of the property being provided.

  • "type": String. The type of the entry of the property being provided.

• "has_references": Boolean. An optional boolean to indicate whether any of the data lines in the response contains a reference marker. A value of false means that the client does not have to process any of the lines to detect reference markers, which may speed up the parsing.

• "item_schema": Object. An object that represents a JSON Schema that validates the data lines of the response. The format SHOULD be the relevant partial extract of a valid property definition as described in Property Definitions. If a schema is provided, it MUST be a valid JSON schema using the same version of JSON schema as described in that section.

• "links": Object. An object to provide relevant links for the property being provided. It MAY contain the following key:

  • "base_url": String. The base URL of the implementation serving the database to which this property belongs.

  • "item_describedby": String. A URL to an external JSON Schema that validates the data lines of the response. The format and requirements on this schema are the same as for the inline schema field item_schema.

The format of data lines of the response (i.e., all lines except the first and the last) depends on whether the header object specifies the layout as "dense" or "sparse".

• Dense layout: In the dense partial data layout, each data line reproduces one item from the OPTIMADE list property being transmitted in the JSON format. If OPTIMADE list properties are embedded inside the item, they can either be included in full or replaced with a reference-marker. If a list is replaced by a reference marker, the client MAY use the provided URL to obtain the list items. If the field "returned_ranges" is omitted, then the client MUST assume that the data is a continuous range of data from the start of the array up to the number of elements given until reaching the end-of-data-marker or next-marker.

• Sparse layout for one-dimensional list: When the response sparsely communicates items for a one-dimensional OPTIMADE list property, each data line contains a JSON array of the format:

  • The first item of the array is the zero-based index of list property item being provided by this line.

  • The second item of the array is the list property item located at the indicated index, represented using the same format as each line in the dense layout. In the same way as for the dense layout, reference-markers are allowed inside the item data for embedded lists that do not fit in the response (see example below).

• Sparse layout for multidimensional lists: the server MAY use a specific sparse layout for the case that the OPTIMADE property represents a series of directly hierarchically embedded lists (i.e., a multidimensional sparse array). In this case, each data line contains a JSON array of the format:

  • All array items except the last one are integer zero-based indices of the list property item being provided by this line; these indices refer to the aggregated dimensions in the order of outermost to innermost.

  • The last item of the array is the list property item located at the indicated coordinates, represented using the same format as each line in the dense layout. In the same way as for the dense layout, reference-markers are allowed inside the item data for embedded lists that do not fit in the response (see example below).

If the final line of the response is a next-marker, the client MAY continue fetching the data for the property by retrieving another partial data response from the provided URL. If the final line is an end-of-data-marker, any data not covered by any of the responses are to be assigned the value null.

If "returned_ranges" is included in the response and the client encounters a next-marker before receiving all lines indicated by the slice, it should proceed by not assigning any values to the corresponding items, i.e., this is not an error. Since the remaining values are not assigned a value, they will be null if they are not assigned values by another response retrieved via a next link encountered before the final end-of-data-marker. (Since there is no requirement that values are assigned in a specific order between responses, it is possible that the omitted values are already assigned. In that case the values shall remain as assigned, i.e., they are not overwritten by null in this situation.)

{"optimade-partial-data": {"format": "1.2.0"}, "layout": "dense", "returned_ranges": [{"start": 10, "stop": 20, "step": 2}]}
123
345
-12.6
["PARTIAL-DATA-NEXT", ["https://example.db.org/value4"]]

{"optimade-partial-data": {"format": "1.2.0"}, "layout": "dense", "returned_ranges": [{"start": 10, "stop": 20, "step": 2}]}
[[10,20,21], [30,40,50]]
["PARTIAL-DATA-REF", ["https://example.db.org/value2"]]
[[11, 110], ["PARTIAL-DATA-REF", ["https://example.db.org/value3"]], [550, 333]]
["PARTIAL-DATA-NEXT", ["https://example.db.org/value4"]]

{"optimade-partial-data": {"format": "1.2.0"}, "layout": "sparse"}
[3,5,19,  [10,20,21,30]]
[30,15,9, ["PARTIAL-DATA-REF", ["https://example.db.org/value1"]]]
["PARTIAL-DATA-NEXT", ["https://example.db.org/"]]

{"optimade-partial-data": {"format": "1.2.0"}, "layout": "sparse"}
[3,5,19,  10]
[30,15,9, 31]
["PARTIAL-DATA-NEXT", ["https://example.db.org/"]]

{"optimade-partial-data": {"format": "1.2.0"}, "layout": "sparse"}
[3,5,19, [ [10,20,21], [30,40,50] ] ]
[3,7,19, ["PARTIAL-DATA-REF", ["https://example.db.org/value2"]]]
[4,5,19, [ [11, 110], ["PARTIAL-DATA-REF", ["https://example.db.org/value3"]], [550, 333]]]
["PARTIAL-DATA-END", [""]]

9.5 Property Definition Example

This appendix provides a more complete example of a Property Definition in the format defined in Property Definitions. (Note: the description strings have been wrapped for readability only.)

{
  "title": "Forces and atomic masses list",
  "$id": "https://properties.example.com/v1.2.0/forces_and_masses",
  "x-optimade-type": "list",
  "x-optimade-property": {
    "version": "1.2.0",
    "property-format": "1.2",
    "units": [
      {
        "title": "Newton",
        "symbol": "N",
        "$id": "https://units.example.com/v1.2.0/N",
        "description": "The newton SI unit of force, defined as 1 kg m/s^2
                        using the 2019 redefinition of the SI base units.",
        "standard": {
          "name": "gnu units",
          "version": "3.15",
          "symbol": "newton"
        },
        "defining-relation": {
          "base-units": [
            {
              "symbol": "kg",
              "id": "https://units.example.com/v1.2.0/kg"
            },
            {
              "symbol": "m",
              "id": "https://units.example.com/v1.2.0/m"
            },
            {
              "symbol": "s",
              "id": "https://units.example.com/v1.2.0/s"
            }
          ],
          "base-units-expression": "kg*m*s^-2"
        }
      },
      {
        "title": "Dalton mass unit",
        "symbol": "u",
        "$id": "https://units.example.com/v1.2.0/u",
        "description": "The dalton mass unit defined as 1/12 of the mass of an
                        unbound neutral atom of carbon-12 in its nuclear and
                        electronic ground state and at rest. Approximately
                        equal to $1.66053906660(50)*10^{-27}$ kg",
        "standard": {
          "name": "gnu units",
          "version": "3.15",
          "symbol": "atomicmassunit"
        }
      }
    ]
  },
  "type": ["array", "null"],
  "description": "A list of forces and atomic masses",
  "examples": [
    [{"force": 42.0, "mass": 28.0855}, {"force": 44.2, "mass": 15.9994}],
    [{"force": 12.0, "mass": 24.3050}]
  ],
  "x-optimade-unit": "inapplicable",
  "x-optimade-requirements": {
    "support": "should",
    "sortable": false,
    "query-support": "none"
  },
  "items": {
    "title": "Force and atomic mass pair",
    "x-optimade-type": "dictionary",
    "description": "A dictionary containing a force and mass value",
    "x-optimade-unit": "inapplicable",
    "type": ["object"],
    "properties": {
      "force": {
        "title": "Force",
        "description": "A force value",
        "x-optimade-type": "float",
        "x-optimade-unit": "N",
        "type": ["number"],
        "examples": [42.0]
      },
      "mass": {
        "title": "Mass",
        "description": "An atomic mass",
        "x-optimade-type": "float",
        "x-optimade-unit": "u",
        "type": ["number"],
        "examples": [15.9994]
      }
    }
  }
}

9.6 OPTIMADE Regular Expression Format

This section defines a Unicode string representation of regular expressions (regexes) to be referenced from other parts of the specification. The format will be referred to as an "OPTIMADE regex".

Regexes are commonly embedded in a context where they need to be enclosed by delimiters (e.g., double quotes or slash characters). If this is the case, some outer-level escape rules likely apply to allow the end delimiter to appear within the regex. Such delimiters and escape rules are not included in the definition of the OPTIMADE regex format itself and need to be clarified when this format is referenced. The format defined in this section applies after such outer escape rules have been applied (e.g., when all occurrences of \/ have been translated into / for a format where an unescaped slash character is the end delimiter). Likewise, if an OPTIMADE regex is embedded in a serialized data format (e.g., JSON), this section documents the format of the Unicode string resulting from the deserialization of that format.

An OPTIMADE regex is a regular expression that adheres to ECMA-262, section 21.2.1 with additional restrictions described below which define a subset of the ECMA-262 format chosen to match features commonly available in different database backends. The regex is interpreted according to the ECMA-262 processing rules that apply for an expression where only the Unicode variable is set to true of all variables set by the RegExp internal slot described by ECMA-262, section 21.2.2.1.

The subset includes only the following tokens and features:

• Individual Unicode characters matching themselves, as defined by the JSON specification (RFC 8259).

• The . character to match any one Unicode character except the line break characters LINE FEED (LF) (U+000A), CARRAGE RETURN (U+000D), LINE SEPARATOR (U+2028), PARAGRAPH SEPARATOR (U+2029) (see ECMA-262 section 2.2.2.7).

• A literal escape of one of the characters defined as syntax characters in the ECMA-262 standard, i.e., the escape character (\) followed by one of the following characters ^ $ \ . * + ? ( ) [ ] { } | to represent that literal character. No other characters can be escaped. (This rule prevents other escapes that are interpreted differently depending on regex flavor.)

• Simple character classes (e.g., [abc]), complemented character classes (e.g. [^abc]), and their ranged versions (e.g., [a-z], [^a-z]) with the following constraints:

  • The character - designates ranges, unless it is the first or last character of the class in which case it represents a literal - character.

  • If the first character is ^ then the expression matches all characters except the ones specified by the class as defined by the characters that follows.

  • The characters \ [ ] can only appear escaped with a preceding backslash, e.g. \\ designates that the class includes a literal \ character. The other syntax characters may appear either escaped or unescaped to designate that the class includes them. (This rule prevents other escapes inside classes that are not the same across regex flavors and expressions that, in some flavors, are interpreted as nested classes.)

  • Except as specified above, all characters represent themselves literally (including syntax characters).

  • Characters that represent themselves literally can only appear at most once. (This rule prevents various kinds of extended character class syntax that differs between regex formats that assigns special meaning to duplicated characters such as POSIX character classes, e.g., [:alpha:], equivalence classes, e.g., [=a=], set constructs, e.g. [A--B], [A&&B], etc.).

• Simple quantifiers: + (one or more), * (zero or more), ? (zero or one) that appear directly after a character, group, or character class. (This rule prevents expressions with special meaning in some regex flavors, e.g., +? and (?...).)

• The beginning-of-input (^) and end-of-input ($) anchors.

• Simple grouping ((...)) and alternation (|).

Note that lazy quantifiers (+?, *?, ??) are not included, nor are range quantifiers ({x}, {x,y}, {x,}). Furthermore, there is no support for escapes designating shorthand character classes as \ and a letter or number, nor is there any way to represent a Unicode character by specifying a code point as a number, only via the Unicode character itself. (However, the regex can be embedded in a context that defines such escapes, e.g., in serialized JSON a string containing the character \u followed by four hexadecimal digits is deserialized into the corresponding Unicode character.)

An OPTIMADE regex matches the string at any position unless it contains a leading beginning-of-input (^) or trailing end-of-input ($) anchor listed above, i.e., the anchors are not implicitly assumed. For example, the OPTIMADE regex "es" matches "expression".

Regexes that utilize tokens and features beyond the designated subset are allowed to have an undefined behavior, i.e., they MAY match or not match any string or MAY produce an error. Implementations that do not produce errors in this situation are RECOMMENDED to generate warnings if possible.

Compatibility notes:

• The subset is intended to be compatible with, but even further restricted than, the subset recommended in the JSON Schema standard, see JSON Schema: A Media Type for Describing JSON Documents 2020-12, section 6.4. The compatibility with the JSON Schema standard is expressed here as "intended" since there is some room for interpretation of the precise features included in the recommendation given in that standard.

• The definition tolerates (with undefined behavior) regexes that use tokens and features beyond the defined subset. Hence, a regex can be directly handed over to a backend implementation compatible with the subset without needing validation or translation.

• Additional consideration of how the . character operates in relation to line breaks may be required for multiline text. If the regex is applied to strings containing only the LINE FEED (U+000A) character and none of the other Unicode line break characters, most regex backend implementations are compatible with the defined behavior. If the regex is applied to string data containing arbitrary combinations of Unicode line break characters and the right behavior cannot be achieved via environmental settings and regex options, implementations can consider a translation step where other line break characters are translated into LINE FEED in the text operated on.

• Compatibility with different regex implementations may change depending on the environment, implementation programming language versions, and options and has to be verified by implementations. However, as a general guide, we have used third-party sources, e.g., the Regular Expression Engine Comparison Chart to collect the following information for compatibility when operating on text using LINE FEED as the line break character:

  • ECMAScript (also known as javascript) and version 1 and 2 of PCRE are meant to be compatible by design when used with appropriate options.

  • The following regex formats appear generally compatible when operating in Unicode mode: Perl, Python, Ruby, Rust, Java, .NET, MySQL 8, MongoDB, Oracle, IBM Db2, Elasticsearch, DuckDB (which uses the re2 library).

  • SQLite supports regexes via libraries and thus can use a compatible format (e.g., PCRE2).

  • XML Schema appears to use a compatible regex format, except it is implicitly anchored: i.e., the beginning-of-input ^ and end-of-input $ anchors must be removed, and missing anchors replaced by .*.

  • POSIX Extended regexes (and their extended GNU implementations) are incompatible because \ is not a special character in character classes. POSIX Basic regexes also have further differences, e.g., the meaning of some escaped syntax characters is reversed.