Compress to impress
Boost your web application performance with the latest compression techniques
In the last two newsletters, I covered latency. First, I explained latency, its importance, and its impacts. Second, I explained how to bring the client and server closer, which is the approach that usually has the highest impact. Another approach you should combine with geography proximity is reducing the number of roundtrips between the client and server.
The easiest way to reduce the number of roundtrips, almost without involving any code/flow change, is to implement compression over the data. In a web environment, compression can reduce the size of pages, scripts, and style sheets by up to 70%. The cons of applying compression is the time it takes to compress/decompress. But in general, the bigger the file, the higher the benefits of compression.
The HTTP standard suggests that browsers and servers implement seven registered algorithms, but the most common compression schemes are Gzip, Brotli, and, in recent years, zstd. Which one should you use?
Generally speaking, Brotli should currently be the "go-to. " But Zstd is just around the corner. Starting two weeks ago, Chrome supports Zstd by default.
Brotli is the default compression algorithm used by several significant cloud services, including AWS, GCP, Azure, and Cloudflare.
The exception is if you are heavily on mobile or working with low-end devices on a large scale; Zstd should be your go-to; if it works for Facebook on mobile, it should also work for you.
What changed since last year [2018] is that we introduced zstd compression for mobile traffic, and it's working great. Compared to gzip, it offers improvement on all key metrics simultaneously, saving bandwidth, server load, and even improving client responsiveness, so much that it's visible in our interaction metrics. As a consequence, nowadays, http traffic for Facebook mobile applications is being delivered zstd-compressed. (source)
Both Brotli and Zstd can compress with a static pre-made dictionary. The dictionary helps the algorithm by defining repetitive phrases in advance, which creates a better compression ratio and speeds the compression/decompression. Brotli, by default, contains a static dictionary of 13K web-related phrases.
In a web application example, the static dictionary can be the current version of your JS/CSS. Once you update the js/css file, compress it with the prior version.
Another example of a dictionary could be all your log's strings and patterns in your code.
"10 different product listing pages from Etsy were used with a 1mb dictionary (132kb compressed). Most of the gains could probably have been had with a much smaller dictionary but this was useful for exploring the upper bound in savings.
Using a custom dictionary resulted in pages that were 84-90% smaller to transfer than using brotli alone (98% smaller than the uncompressed size). For example, a 539kb uncompressed HTML page was 84kb using brotli and 10kb using brotli with a custom dictionary.
(source)"
A new initiative launched last October (2023), supported by Chrome and Edge, defines the protocol for sharing the static dictionary between the browser and the client: adding a new header and "link" HTML tags.
If you transfer data between microservices instead of to the browser, you have more options to consider, and which tactic to use depends on your needs, for example:
When working with textual data, applying higher quality levels of Brotli/Zstd can maximize the compression ratio and increase the compression time, so keep it in mind.
Use other compression algorithms like Snappy or LZ4 (snappy is the default in Spark, for example)
As you can see, compression, which is one of the oldest strategies, is evolving with time and can be customized to achieve better performance and latency.