Creating or Modifying an OpenZL Codec

Workflow for Creating a New Codec

If the user provides an input to this skill, interpret it as an informal specification of the codec they want to create. Before writing any code:

1. Ask clarifying questions about the format. Resolve ambiguities around input/output types, element widths, codec header layout, edge cases, error conditions, and whether the codec is public or private.
2. Write spec.md first. ALWAYS generate the decoder wire format specification (spec.md) before any other file. Follow the conventions of existing specs in src/openzl/codecs/ (inputs, codec header, decoding algorithm, outputs).
3. Ask the user to verify the spec. Do NOT proceed with implementation until the user has reviewed and approved the spec.md. The spec is the contract — all code flows from it.

Security

Decoder MUST Be Safe to Malicious Inputs

The decoder processes untrusted data from compressed frames. All assumptions MUST be validated, especially:

• Bounds on sizes, counts, and offsets read from the frame header
• Element widths and stream types
• Buffer sizes before writing

Never trust values from the compressed stream without verification. A malicious frame must not cause crashes, out-of-bounds access, or undefined behavior.

Encoder MUST Be Safe to Malicious Inputs

The encoder is processing data that the user provides. Unless explicitly stated otherwise all assumptions MUST be validated. Typcially, this is less of an issue on the encoder side, but any assumptions the encoder makes about the input data (e.g. it doesn't contain the value 0) must be validated, otherwise the data could be corrupted.

Requirements

General

• Codecs must work on big & little endian machines, and must not depend on endian-ness.
• Prefer to use helpers that already exist in src/openzl/shared/, rather than re-implementing them.
  • Especially openzl/shared/mem.h, openzl/shared/bits.h, and openzl/shared/utils.h.
• All code MUST be portable across platforms. E.g. both ARM and x86-64, 32- and 64-bit systems, and both little- and big-endian.

Bumping ZL_MAX_FORMAT_VERSION

When adding a new codec, or making a breaking change to a codec that requires bumping the format version, we need to make sure the development branch bumps the format version. If not adding a codec or making a format breaking change to a codec, then you can skip this section.

Determine the production max format version from the ZL_MAX_FORMAT_VERSION macro in fbcode/openzl/prod/include/openzl/zl_version.h, call it $prod_max_format_version. Determine the development max format version from the ZL_MAX_FORMAT_VERSION macro in fbcode/openzl/dev/include/openzl/zl_version.h, call it $dev_max_format_version. If $dev_max_format_version == $prod_max_format_version, then the development ZL_MAX_FORMAT_VERSION in fbcode/openzl/dev/include/openzl/zl_version.h must be bumped. This needs to be done before hooking up the encoder and decoder registry so that the new max format version is used during registration.

Format Versioning

Codecs MUST preserve forward and backward compatibility with all supported format versions from ZL_MIN_FORMAT_VERSION to ZL_MAX_FORMAT_VERSION. Codecs MAY change their format, with very careful consideration, but they MUST do it in way that preserves compatibility:

• ZL_MAX_FORMAT_VERSION must be bumped in the dev branch (see above)
• The spec.md file MUST be updated to reflect the variation based on the format version
• The encoder MUST check the format version with ZL_Encoder_getCParam(eictx, ZL_CParam_formatVersion) and only emit the new format for the latest format version
• The encoder MUST maintain the ability to emit the older format versions down to the minimum format version the codec supports
• The decoder MUST check the format version with DI_getFrameFormatVersion(dictx) and correctly interpret the encoded data based on the format version

…