Increase gzip compression level to 9 (BestCompression)

[lyoung-confluent]

Currently when building an image apko uses pgzip.NewWriter which will use the DefaultCompression constant (-1) which is becomes a compression level of 5. We can decrease the final image/layer size at the cost of small amount of additional CPU during build time by instead opting into the BestCompression level (9) via pgzip.NewWriterLevel instead.

This won't necessarily result in a huge change in image size for most images, for example using examples/alpine-base.yaml it decreases from 27691520 bytes to 26865152 bytes (~3%).

[jonjohnsonjr]

I think this would be great to add as an option/flag, but I'm reluctant to make it the default. We want building (and rebuilding) images to be fast during development so we can get a fast feedback loop. Since melange uses apko to produce the build environment for our package builds, the vast majority of apko-produced images are effectively "single use" so it would be wasteful to burn the additional CPU to make them smaller, but it's definitely unfortunate that we're leaving some gains on the table for the final release images.

(Arguably, it might make more sense to have 9 be the default and have a flag to go faster during development...)

I think klauspost's choice for default strikes a good balance (some older discussion here), but I also spent a month last year chasing performance as much as possible, so I might have gone a little too far :)

Would you be okay with this as a flag?

[xnox]

To win on both speed, and image size.... We should consider Zstd compression. But I have no idea if the ecosystem of where images are executed can support that.

[jonjohnsonjr]

But I have no idea if the ecosystem of where images are executed can support that.

I feel like the major runtimes (containerd, mostly) probably already support zstd, but I'd be a little worried about things like Cloud Run. Would be worth experimenting with.