[CSDb] - User Forums - Release id #214940 : TSCrunch

You are not logged in - nap

CSDb User Forums

Forums > CSDb Entries > Release id #214940 : TSCrunch

2022-02-28 10:11

Krill

Registered: Apr 2002
Posts: 2854

Release id #214940 : TSCrunch

Quoting tonysavon

"TSCrunch is an optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs, while achieving decent compression ratio (for a bytecruncher, that is). It crunches as well as other popular bytecrunchers, while being considerably faster at decrunching."
[...]
TSCrunch is a bytepacker, with 2byte RLE, 1-2byte LZ tokens and a 512 bytes search window. In this "space" it provides the optimal solution to the puzzle. Exomizer is s different beast, being a bit-cruncher.

According to these specs, i'd expect it to fall somewhere into the 60% cluster in the graph below, from https://codebase64.org/doku.php?id=base:compression_benchmarks.

If it is made for in-memory decompression of data, can it also work with prefix data for better compression, i.e., back-referencing to data already in memory (either a static dictionary or, e.g., the previous level)?

And i can't quite follow what "bit-cruncher" vs "byte-cruncher" means. :)

IIRC, Exomizer works on byte-aligned source data as well, also with LZ and RLE.
It produces a bit-stream on the control symbols, though, which is interleaved with byte-aligned literals.

A "bit-cruncher" might maybe be something LZMA- or ANS-like (think ALZ64, or Shrinkler on Amiga), but not Exomizer.

... 31 posts hidden. Click here to view all posts....

2022-02-28 23:10

Dano

Registered: Jul 2004
Posts: 227

Gave it a spin on one of my unreleased anim parts.

For 64 frames of screen and colorram it packs $200 bytes than TinyCrunch. Nice one!

And looking at the mem in C64Debugger it looks noticeably faster.

Great release!

2022-02-28 23:15

Krill

Registered: Apr 2002
Posts: 2854

Quoting Dano

Gave it a spin on one of my unreleased anim parts.

Sorry for slight off-topic-ness, but... i always wonder if using a non-lossy general-purpose compressor is the right choice for media content. :)

2022-02-28 23:26

Dano

Registered: Jul 2004
Posts: 227

Depends on what you feed it with and things. Sometimes you want things to be lossless. Sometimes you just want to see where you get with a shovel and hammer.

2022-03-01 09:56

ChristopherJam

Registered: Aug 2004
Posts: 1381

Great work Tony, it looks like it fits your use case really well.

As you suspected, TinyCrunch has no explicit support for RLE, and performs pretty abysmally on any data with large empty spaces - the maximum match length is 17 bytes, and it uses a two byte token for any matches longer than two bytes, so the compression ratio is capped to 8.5. My own speed demon's on hold for the moment, but as I mentioned on FB, yours will be an interesting one to try and best :)

2022-03-01 11:06

Krill

Registered: Apr 2002
Posts: 2854

Quoting tonysavon

For my purposes I needed to stay below 20 cycles per output byte, in the general case

Could you give a figure for average cycles per input byte? :)

(This is the relevant number to estimate whether a block can be fully decrunched until the next block is received.)

2022-03-01 11:37

tonysavon

Registered: Apr 2014
Posts: 24

Quote: Quoting tonysavon
For my purposes I needed to stay below 20 cycles per output byte, in the general case
Could you give a figure for average cycles per input byte? :)

(This is the relevant number to estimate whether a block can be fully decrunched until the next block is received.)

Sure.
In this metric, if you crunch better you do worse, I guess, because there are fewer bytes to read at the denominator, but this is what I get for Chopper Command amyway:

Total decrunch cycles:

TSCrunch: 799826
TC: 1133039
Dali: 1436401

Crunched file size:

TSCrunch: 13321
TC: 15419
Dali: 10478

Cycles per output byte:

TSCrunch: 17.0
TC: 24.1
Dali: 36.1

Cycles per input byte:

TSCrunch: 60.0
TC: 73.5
Dali: 137.1

2022-03-01 11:41

tonysavon

Registered: Apr 2014
Posts: 24

Quote: Great work Tony, it looks like it fits your use case really well.

As you suspected, TinyCrunch has no explicit support for RLE, and performs pretty abysmally on any data with large empty spaces - the maximum match length is 17 bytes, and it uses a two byte token for any matches longer than two bytes, so the compression ratio is capped to 8.5. My own speed demon's on hold for the moment, but as I mentioned on FB, yours will be an interesting one to try and best :)

Well, I'm definitely looking forward to whatever you'll release, especially with that amazing name you chose! :-)
TC is such a nice packer btw, now that I'm looking more and more at it. Lots of clever solutions, and I especially love the two-parts sfx decdunching. Plus all the options it comes with...
Keep up the good working!

2022-03-08 12:57

tonysavon

Registered: Apr 2014
Posts: 24

All right, I added inplace decrunching, so now it's up to you, @Krill :-)

I'll wait a bit before I bump the version here on CSDb as well, but in the meantime you can download version 1.1 from here:

https://github.com/tonysavon/TSCrunch

2022-03-09 14:20

Krill

Registered: Apr 2002
Posts: 2854

Quoting tonysavon

All right, I added inplace decrunching, so now it's up to you, @Krill :-)

Alright, a few first numbers from the WIP next loader version. =)

CPU% ZX0  TS   WIN
100  9953 9744 ZX0
 90  9086 9437 TS
 80  7147 7054 ZX0
 70  6222 6328 TS
 60  5538 5670 TS
 50  4706 4685 ZX0
 40  4017 4359 TS
 30  3000 3432 TS
 20  2030 2404 TS
 10   929 1126 TS

Bitfire benchmark, numbers are throughput in decrunched bytes per second, higher is better.

Both TSCrunch and ZX0 are quite close to each other, but at just under 50% or less CPU for loading/decrunching, TSCrunch shows its strengths over ZX0.
This matters a lot in a real-world scenario where things are going on while loading in the background.

2022-03-09 16:33

tonysavon

Registered: Apr 2014
Posts: 24

Quote: Quoting tonysavon
All right, I added inplace decrunching, so now it's up to you, @Krill :-)
Alright, a few first numbers from the WIP next loader version. =)
CPU% ZX0 TS WIN 100 9953 9744 ZX0 90 9086 9437 TS 80 7147 7054 ZX0 70 6222 6328 TS 60 5538 5670 TS 50 4706 4685 ZX0 40 4017 4359 TS 30 3000 3432 TS 20 2030 2404 TS 10 929 1126 TS
Bitfire benchmark, numbers are throughput in decrunched bytes per second, higher is better.

Both TSCrunch and ZX0 are quite close to each other, but at just under 50% or less CPU for loading/decrunching, TSCrunch shows its strengths over ZX0.
This matters a lot in a real-world scenario where things are going on while loading in the background.

That's really cool, Krill. Thanks for integrating it so quickly.
These figures seems to validate the hypothesis that it's not just about extreme speed and extreme compression, but mostly about finding the sweet spot. When the loader/decruncher has all the CPU time for itself, then better compression has an impact, but as you start doing something at the same time than speed helps.
It's really fascinating from an analytical standpoint, as there are so many variables involved and it'll be really difficult to find a cruncher (or even a benchmark) to rule them all.

Previous - 1 | 2 | 3 | 4 | 5 - Next

Refresh

Subscribe to this thread: