Cloudflare tests Zstandard to cut CDN storage by two-thirds

Cloudflare is evaluating a new approach to reduce storage and bandwidth costs across its global content delivery network. The company has built a prototype called Cache Transcoding that compresses eligible cache assets using the Zstandard (zstd) algorithm before writing them to disk, aiming to expand effective cache capacity without hardware upgrades.

How the prototype works

The system integrates with Cloudflare’s Pingora proxy architecture. When an eligible asset enters the cache, it is encoded with zstd and stored in compressed form. This compressed representation remains on disk and is transferred between data centers via Tiered Cache. The asset is only decoded when served to the client, preserving the original content while reducing storage and backbone traffic.

Initial testing showed that eligible assets shrank to roughly one-third of their original size on average. The prototype uses zstd level 3, balancing compression ratio and speed to avoid CPU bottlenecks. Encoding occurs once per cache fill, while the storage and bandwidth savings persist for every subsequent request.

What qualifies for transcoding

Not all assets are eligible for compression. The prototype targets uncompressed text assets—such as HTML, JSON, CSS, and JavaScript—with a minimum size of 4 KiB. These assets represented 67.3% of requests but only 22.3% of bytes in Cloudflare’s traffic sample. Pre-compressed content like images, video, and fonts is excluded, as recompressing them would waste CPU without meaningful size reduction.

The 4 KiB threshold was chosen to avoid per-object overhead for tiny requests while capturing most compressible bytes. In testing, this threshold excluded only about 1% of otherwise eligible bytes. The prototype also skips responses with active upstream compression, range requests, and unknown-length bodies.

Performance trade-offs

The system introduces a small CPU cost for encoding and decoding, but Cloudflare’s modeling suggests the trade-off is favorable. Encoding is more expensive per byte but occurs only once per cache fill, while decoding happens on every serve. Under the traffic and reuse patterns tested, the extra CPU cost remained within a few percent of baseline.

Storage savings are significant. Smaller on-disk representations allow each server to retain more objects, increasing cache density and reducing evictions. The compressed form also reduces data transferred between data centers, improving backbone efficiency. Cloudflare estimates the approach could save petabytes of storage across its network.

Next steps and limitations

The prototype is still in testing. Cloudflare plans to evaluate higher zstd compression levels, broader content types, and additional parameters like eligibility criteria. Future work may explore range requests, pre-compressed origin responses, and passing compressed objects directly to downstream components that support them.

The current implementation does not represent a fleet-wide rollout. Testing was conducted on a controlled corpus of compressible assets, and broader deployment would require validation across more diverse traffic patterns. The measured 2.8× compression ratio is not guaranteed for all content types.