[CSDb] - User Forums - Release id #232862 : BitPickler 1.0.0

You are not logged in - nap

CSDb User Forums

Forums > CSDb Entries > Release id #232862 : BitPickler 1.0.0

2023-05-31 16:55

ChristopherJam

Registered: Aug 2004
Posts: 1411

Release id #232862 : BitPickler 1.0.0

BitPickler is a c64 cruncher optimised for the compression of small demos for 4k competitions and the like. It's very heavy on RAM usage during decompression (by default it uses all of $7c00-$ffff as working memory), but has an excellent compression ratio, and outputs around 800 or so bytes per second.

I'd been playing with some of the components for a while, but basically pulled it together to try and help get Onscreen 5k as small as Thundax and I could manage (while simultaneously being the musician who was one of the reasons we ended up having to work quite hard to get it below 5k and keep it there, nevermind hitting the 4k we'd pondered early in development :) )

At it's core it's a range encoder (tANS with an implicit table) with context mixing, combined with an optional ALZ64 style LZ77 encoder - amusingly, for Onscreen 5k the extra code size for the LZ77 layer almost exactly counterbalanced the savings from the improved ratio. It also searches for an area that benefits from a delta encoding preprocess (so please do group your sine tables etc together into a single block), and can deinterleave instructions from their first and second operands for an initial code block, with the length specified by you as a commandline option.

It is not designed for crunching files bigger than 10kb or so, or for trackmos, or for level crunching. It performs poorly on large empty areas, not so much because it doesn't squash them, but because those areas throw off the predictive encoder despite adaptation. I'm not planning on fixing any of those things.

For most small demos though, it has a significantly better ratio that ALZ64, while decrunching up to 1.5x faster.

Here's how BitPickler 1.0 performs on Artefacts, Immortality, 4kRastersIntro, and a prerelease version of Onscreen 5k,
compared with ALZ64, Exomizer V3.1.0, NuCrunch V1.0.0, TinyCrunch V1.2 and Bitbreaker's port of ZX0 1.5 with Soci's optimisations. (I grabbed the latter from 'somewhere' before the various Dali releases here).

For each test demo, the compressed file size in bytes and decompression time in seconds:

+-----------------------------+  +-----------------------------+
| Onscreen 5k                 |  | Artefacts                   |
+------------+-------+--------+  +------------+-------+--------+
| BitPickler | 4879b |  8.78s |  | BitPickler | 3963b |  5.93s |
| ALZ64      | 5396b | 12.21s |  | ALZ64      | 4024b |  9.21s |
| Exomizer   | 5434b |  0.99s |  | Exomizer(2)| 4095b |  1.93s |
| ZX0        | 5456b |  0.54s |  | ZX0        | 4228b |  0.37s |
| NuCrunch   | 5862b |  0.49s |  | NuCrunch   | 4539b |  0.67s |
| TinyCrunch | 6219b |  0.20s |  | TinyCrunch | 4639b |  0.47s |
+------------+-------+--------+  +------------+-------+--------+
                                 
+-----------------------------+  +-----------------------------+
| Immortality                 |  | 4k Rasters                  |
+------------+-------+--------+  +------------+-------+--------+
| BitPickler |  911b |  2.18s |  | BitPickler | 2363b |  4.43s |
| ALZ64      |  987b |  1.63s |  | ALZ64      | 2438b |  5.14s |
| Exomizer   | 1038b |  0.19s |  | Exomizer   | 2504b |  0.46s |
| ZX0        | 1015b |  0.09s |  | ZX0        | 2545b |  0.23s |
| NuCrunch   | 1269b |  0.10s |  | NuCrunch   | 2788b |  0.23s |
| TinyCrunch | 1221b |  0.07s |  | TinyCrunch | 2868b |  0.11s |
+------------+-------+--------+  +------------+-------+--------+

(note 1, these were with BitPickler's activity indicator enabled - disabling that would save four bytes and improve speed by about 1%)
(note 2, the measurements for Artefacts+exo above used Exo4K-Decruncher, as per the original demo release; it saves 90 bytes compared to exo 3.1.0's usual decruncher, while only running 2.7x slower)

2023-05-31 21:32

Krill

Registered: Apr 2002
Posts: 2989

Pretty great! =)

3963 bytes for repacked Artefacts doesn't leave enough room for the director's cut in 4K, but then i did not expect that anyways.

Will surely use this for upcoming small onefiled demos! :)

2023-06-02 10:56

Frostbyte

Registered: Aug 2003
Posts: 183

Seriously impressive! If I ever get my wannabe-4k intro into a state of nearly-completeness, I'll certainly give BitPickler a go.

As a side note, I'm also very impressed about the speed vs compression ratio of both Exomizer and ZX0. I knew they are good, but didn't expect such solid compression performances in most cases.

Just out of curiosity, what were the starting points, i.e. the uncompressed file sizes?

2023-06-02 16:20

ChristopherJam

Registered: Aug 2004
Posts: 1411

Thanks Frostbyte! I look forward to seeing what you can do with it.

And yes, Exomizer and ZX0 have been seriously pushing the envelope at the midrange. Exo's doubled its decompression speed in the past few years, and its ratio's improved a bit too. Not sure what I'm going to do with my as yet unreleased exo-killer side project now; its a tad slower than Exomizer and ZX0, and the ratio gains aren't as compelling as they once were...

Similarly, Antonio Savona's been doing great work at the high speed/low ratio end of the spectrum - tscrunch is setting a high bar there!

Here are the starting points for the benchmarks above:

+----------------+-------+---------+----------+
| Demo           | Raw   | Pickled | % of Raw |
+----------------+-------+---------+----------+
| Onscreen 5k    | 7423b |  4879b  |   65.7%  |
| Artefacts      | 5224b |  3963b  |   75.9%  |
| 4k Rasters     | 4096b |  2363b  |   57.7%  |
| Immortality    | 2561b |   911b  |   35.6%  |
+----------------+-------+---------+----------+

2023-06-02 17:44

Frostbyte

Registered: Aug 2003
Posts: 183

Thanks CJam! The compression ratios give me hope. ;)

2023-06-06 11:38

Bacchus

Registered: Jan 2002
Posts: 156

Great work. It will make it possible to add even more impossible magic to 4k :-)

/Bacchus

Refresh

Subscribe to this thread: