[CSDb] - User Forums - Release id #214940 : TSCrunch

You are not logged in - nap

CSDb User Forums

Forums > CSDb Entries > Release id #214940 : TSCrunch

2022-02-28 10:11

Krill

Registered: Apr 2002
Posts: 2854

Release id #214940 : TSCrunch

Quoting tonysavon

"TSCrunch is an optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs, while achieving decent compression ratio (for a bytecruncher, that is). It crunches as well as other popular bytecrunchers, while being considerably faster at decrunching."
[...]
TSCrunch is a bytepacker, with 2byte RLE, 1-2byte LZ tokens and a 512 bytes search window. In this "space" it provides the optimal solution to the puzzle. Exomizer is s different beast, being a bit-cruncher.

According to these specs, i'd expect it to fall somewhere into the 60% cluster in the graph below, from https://codebase64.org/doku.php?id=base:compression_benchmarks.

If it is made for in-memory decompression of data, can it also work with prefix data for better compression, i.e., back-referencing to data already in memory (either a static dictionary or, e.g., the previous level)?

And i can't quite follow what "bit-cruncher" vs "byte-cruncher" means. :)

IIRC, Exomizer works on byte-aligned source data as well, also with LZ and RLE.
It produces a bit-stream on the control symbols, though, which is interleaved with byte-aligned literals.

A "bit-cruncher" might maybe be something LZMA- or ANS-like (think ALZ64, or Shrinkler on Amiga), but not Exomizer.

2022-02-28 13:18

tonysavon

Registered: Apr 2014
Posts: 24

Hi Krill,
Thanks for giving TSCrunch a go! I'm flattered :-)
I think it's my bad in rushing out the post and the terminology not being so clear as a result. By byte-cruncher I mean crunchers that have fixed-size tokens, multiple of a byte, while other crunchers might rely on (elias) gamma coding to work with integers whose range is not known a-priory. The latter approach yields better crunching but of course the bitstream becomes more difficult to manage, hence the slow decompression.

So, looking at your graph, you are right: TSCrunch belongs to the "family" of TC, WVL and LZW, but it should be faster at decrunching in general.
Whether it crunches better than TC depends on the type of data that you are crunching: TC has a wider lookback window, but it doesn't explicitly manage RLE (I think). So this usually translates to TC crunching a bit better if the repeated patterns are quite far apart and there are not many same byte sequences, while TSCrunch should crunch better for the remaining cases. I don't know what data is used in that benchmark, but I ran many experiments myself (plus TSCrunch has been quietly used by few beta testers in their productions) using only TSCrunch, B2 and TC (for my needs speed is the most important aspect), and for real case scenarios on large files it seems that TSCrunch behaves better also from the point of view of crunching %.
This graph shows figures for memcrunching a game (chopper command). The data is in the readme.txt, but you can take a look at this picture for performance at a glance.

2022-02-28 13:22

Krill

Registered: Apr 2002
Posts: 2854

Can you add ZX0 (aka Salvador aka Dali) for reference? :)
(Bitbreaker's variant with a 6502-optimised bitstream would probably perform best.)

It's a strong contender to Exomizer in terms of crunchiness, but decompresses so much faster.

2022-02-28 13:33

tonysavon

Registered: Apr 2014
Posts: 24

Quote: Can you add ZX0 (aka Salvador aka Dali) for reference? :)
(Bitbreaker's variant with a 6502-optimised bitstream would probably perform best.)

It's a strong contender to Exomizer in terms of crunchiness, but decompresses so much faster.

Yes! I'm a big fan of zx0/Dali and I already "ported" bitbreaker's optimized version to Kickassembler. I'll post results later today when off work.

2022-02-28 16:55

tonysavon

Registered: Apr 2014
Posts: 24

I added dali data. It's really a fantastic cruncher, definitely the most balanced one: you kinda get the best of the two worlds!

BTW, In my game I'm using dali for code crunching (where I need the best compression), and TSCrunch for level crunching, because the faster the better in that case, and I can tolerate some % crunching penalty if I can decrunch twice as fast than with Dali. The sample file I'm using for this benchmark (the prg for the game Chopper Command) has a bit of everything (code, music, gfx, a KLA, tables etc.). So I think it represents a nice testbed.

I tried to post raw numbers here, but the preview messes up the spacing in the tables, so the data becomes unreadable.
The graph should tell enough though.

2022-02-28 18:11

Krill

Registered: Apr 2002
Posts: 2854

I think i should add TSCrunch to the loader, too, in the "recommended for demos" section along ZX0, tinycrunch, Bitnax and LZSA2. :)

But for that, in-place decompression is required.
(It's not required for the loadcompd call per se, but for the "demo-friendly" label.)

Basically, this just means stopping decompression a little before the end of data, and having the final incompressible piece of data already there.
(Along with forward-decrunching and slapping on the correct load address for the compressed data, so both incarnations end at the same address.)

It's on your to-do list as per readme, when do you reckon could you spare the time to add it? :)

Oh and for future graphs, consider inverting the "% of original (lower is better)" to "% of original saved (higher is better)" so higher is better on all of them (saves one brain-indirection when looking at those graphs). =)

2022-02-28 18:24

Burglar

Registered: Dec 2004
Posts: 1047

yay, loader benchmarks \o/

@tonysavon: can you use a smaller diagram size? this does not fit the screen (both in width and height). 40% of the current size would be good :)

2022-02-28 18:43

Krill

Registered: Apr 2002
Posts: 2854

Quoting Burglar

yay, loader benchmarks \o/

The graphs are for pure de/crunch performance without loading, though. :)

But TSCrunch seems to go towards memcpy throughput even more than tinycrunch, and thus is a good candidate to completely decrunch a loaded block before the next one rolls in.

2022-02-28 20:25

tonysavon

Registered: Apr 2014
Posts: 24

@Burglar I've made the picture smaller, hopefully readable.
@Krill, indeed, that's exactly the purpose of this cruncher. It was designed to stream-uncrunch data as quickly as possible, be it from a disk (once you add it to Krill, which would be a-m-a-z-i-n-g), or cartridge which is how I currently use it. For my purposes I needed to stay below 20 cycles per output byte, in the general case (then of course some nasty sequences might clock in at a bit more than that, but usually it averages at 18 cycles per byte), and no cruncher other than simple RLE would give me that performance, so I had to get my hands dirty. It'd be really nice if you could test it with Krill loader already and see how loadcmpd performs. I'll be adding inplace decrunching anyway over the weekend, hopefully.
Thanks everyone for the feedback.

2022-02-28 20:37

Krill

Registered: Apr 2002
Posts: 2854

Quoting tonysavon

and see how loadcmpd performs. I'll be adding inplace decrunching anyway over the weekend, hopefully.

Great!

I've been heavily refactoring and cleaning up everything on the host side, and adding a new decruncher is now done in an hour or two. =)

Still need to solve some other puzzles until the next release, but it'll be exciting to see how TSCrunch performs with the Bitfire and Spindle benchmarks. (Now also a part of the source package, and ZX0 comes out on top.)

2022-02-28 22:17

Burglar

Registered: Dec 2004
Posts: 1047

Quoting tonysavon

@Burglar I've made the picture smaller, hopefully readable.

thank you! :) also good you have a throughput graph, when crunchers are this close in compression, throughput is what matters most.

... 31 posts hidden. Click here to view all posts....

Previous - 1 | 2 | 3 | 4 | 5 - Next

Refresh

Subscribe to this thread:

You need to be logged in to post in the forum.

Search the forum:
Search for in
All times are CET.

Search CSDb

Advanced

Users Online

iAN CooG/HVSC
Loloke
Yogibear/Protovision
iceout/Avatar/HF
Didi/Laxity
Hani
Hok/Remember
Hoild/Ultimate Newco..
Guests online: 144

Top Demos

1 Next Level  (9.8)
2 13:37  (9.7)
3 Mojo  (9.7)
4 Coma Light 13  (9.7)
5 Edge of Disgrace  (9.6)
6 Comaland 100%  (9.6)
7 Uncensored  (9.6)
8 No Bounds  (9.6)
9 Wonderland XIV  (9.6)
10 Bromance  (9.5)

Top onefile Demos

1 Layers  (9.7)
2 It's More Fun to Com..  (9.6)
3 Cubic Dream  (9.6)
4 Party Elk 2  (9.6)
5 Copper Booze  (9.6)
6 TRSAC, Gabber & Pebe..  (9.5)
7 Rainbow Connection  (9.5)
8 Dawnfall V1.1  (9.5)
9 Quadrants  (9.5)
10 Daah, Those Acid Pil..  (9.5)

Top Groups

1 Oxyron  (9.3)
2 Booze Design  (9.3)
3 Censor Design  (9.3)
4 Crest  (9.3)
5 Performers  (9.3)

Top Diskmag Editors

1 Jazzcat  (9.4)
2 Magic  (9.4)
3 hedning  (9.2)
4 Elwix  (9.1)
5 A Life in Hell  (9.1)

Page generated in: 0.047 sec.