Skip to content

Stdlib · extensions

compress

compression (streaming codecs)

extensions.md · 80 lines · 2 min read

use compress

Compression codecs as streams: each one satisfies Readable/Writeable (io), so it composes with io.copy and buffering without materializing everything. DEFLATE/gzip/zlib are frozen formats (decades-old RFCs), hence tier 0; brotli/zstd are newer, hence tier 1 (an addition to the same package). The immediate consumer is http.auto_decompress (which only orchestrates them, the way net/TLS does with crypto).

Compressing is wrapping a destination Writeable; decompressing is wrapping a source Readable. It is the same pattern as the io adapters (BufWriter.over, tee):

// gzip (tier 0)
fn (GzipWriter) over(dst: Writeable) -> GzipWriter // writing to it compresses into 'dst'
fn (w: *GzipWriter) write(data: []byte) -> error{Io}
fn (w: *GzipWriter) flush() -> error{Io}
fn (w: *GzipWriter) close() -> error{Io} // emits the trailer; does NOT close 'dst'
fn (GzipReader) over(src: Readable) -> Result[GzipReader, error{Malformed}] // reading from it decompresses 'src'
fn (r: *GzipReader) read(buf: mut []byte) -> Result[usize, error{Malformed, Io}] // Ok(0) = end
decl GzipWriter { ... } // Writeable + Closeable
decl GzipReader { ... } // Readable

deflate (raw, no header) and zlib (deflate + header/adler32) have the same form: DeflateWriter/ DeflateReader, ZlibWriter/ZlibReader. The compression level is an option at construction:

fn (GzipWriter) with_level(dst: Writeable, level: Level) -> GzipWriter
decl Level { Fast; Default; Best; None } // None = store (no compression, just wraps)

One-shot (buffer to buffer) for the small case, on top of the streaming codecs:

fn gzip(data: []byte) -> []byte // compresses everything (allocates)
fn gunzip(data: []byte) -> Result[[]byte, error{Malformed}] // decompresses everything
fn deflate(data: []byte) -> []byte
fn inflate(data: []byte) -> Result[[]byte, error{Malformed}]
use compress
// streaming: file to file, compressing, without loading into memory
src := fs.open("dump.sql") catch |e| { return e }
out := fs.create("dump.sql.gz") catch |e| { return e }
gz := compress.GzipWriter.over(out)
defer gz.close()
io.copy(gz, src) catch |e| { return e } // pumps while compressing

Same form (BrotliReader/BrotliWriter, ZstdReader/ZstdWriter, one-shots brotli/unbrotli/ zstd/unzstd), as a tier 1 addition to the package, since they churn more than the classic formats. zstd adds dictionaries (training for small repetitive payloads), the feature that justifies having it:

fn (ZstdWriter) with_dict(dst: Writeable, dict: []byte) -> ZstdWriter // dictionary compression
fn zstd_train(samples: []const []byte, dict_size: usize) -> []byte // trains a dictionary
  • Its own package, not inside encoding. encoding is structural representation (to/from) (Serializable, JSON, base64); compress is opaque byte transformation. Distinct concepts, distinct homes.
  • Tier per format: DEFLATE/gzip/zlib (frozen, RFC 1950 to 1952) tier 0; brotli/zstd tier 1. http only needs the classics.
  • Codecs are streams (Readable/Writeable): they compose with io.copy/buffering, streaming by default; one-shots are convenience on top.
  • close() does not close the destination: the wrapper emits its trailer and returns; whoever owns dst/src closes it (composition, like the rest of io). No allocator in the signatures (colorless).