| |
Silver Dream !
Registered: Nov 2005
Posts: 107 |
Exomizer question(s)
What is the "circular buffer"
https://bitbucket.org/magli143/exomizer/src/b817e089425104856ca..
here? |
|
| |
Krill
Registered: Apr 2002
Posts: 2839 |
It's the ring buffer at the heart of a streaming decruncher, allowing you to feed and retrieve data that is bigger than RAM.
This is an internal data structure where data is unpacked to initially, and from and to which back-referenced unpacked data is copied. Back-references are limited to the size of this buffer. |
| |
Silver Dream !
Registered: Nov 2005
Posts: 107 |
Right - the best way to get rid of a temporary brain eclipse is to post a public question ;-)
IOW - yeah, I figured that one out and I provided the "circular_buffer" the size of the -m parameter (16 KiB) but I am still unable to decrunch my data correctly. There shouldn't be any magic to this, should it? Need to do some debugging. |
| |
Silver Dream !
Registered: Nov 2005
Posts: 107 |
OK - found the problem - works! :-) Now need to get the ZX0 decruncher to work and do decrunch speed comparison with my data |
| |
Krill
Registered: Apr 2002
Posts: 2839 |
Do you really need the streaming decruncher? The regular one is surely slightly faster and has no back-reference restrictions. |
| |
Silver Dream !
Registered: Nov 2005
Posts: 107 |
In general case it seems better suited as I am not decrunching from one memory place to another (or the same) but to an I/O port. And the compressed data _might_ be larger than available memory. In particular cases, the compressed data can be fully loaded into a buffer and decompressed from there but the decruncher would still need to be modified to either return single bytes of decrunched data or to store them directly to the port by itself.
With Exomizer there seem to be incompatibilities between major versions, which kept me scratching my head for some time but what I eventually got with it (call `get_decrunched_byte()` until you're done) looks and works elegantly. The downside is that the decrunching is admittedly quite slow with this approach. I am trying to understand how I can achieve something similar with the ZX0 decruncher provided along `bitfire`. The expectation is that it will be substantially faster, even if compression rate may suffer in comparison. |
| |
MagerValp
Registered: Dec 2001
Posts: 1055 |
Stream decrunchers will always be rather slow as you need to keep a ring buffer, ruling out several optimizations that you can't do since you have to handle wrapping. An algorithm that is slower at in memory decrunching but compresses better might win over a theoretically faster algorithm. The only way to know is benchmarking, curious to see how exomizer compares if you get ZX0 to work. |
| |
Krill
Registered: Apr 2002
Posts: 2839 |
For your use-case (with unpacked data fitting in memory), i'd use the regular decrunchers and just put a write to the IO port after every write to output memory.
The regular decrunchers are the most optimised, and comparing those would probably give you the most robust figures. |
| |
Silver Dream !
Registered: Nov 2005
Posts: 107 |
Quoting KrillFor your use-case (with unpacked data fitting in memory) Decrunched data won't fit w/o sharding. Right now I get some 60+ KiB worth of well compressible data down to about five KiB. And I'll need a few of such chunks. I could do the sharding but then the compression will suffer a lot. I tried the same data in small chunks and the compressed total was about two and a half times the size of single compressed chunk. If reusing compression data/tables/whatever the crunchers use, between chunks was possible that could potentially change the picture. |
| |
Krill
Registered: Apr 2002
Posts: 2839 |
Quoting Silver Dream !If reusing compression data/tables/whatever the crunchers use, between chunks was possible that could potentially change the picture. With ZX0, there is such a thing. You can give the cruncher a chunk of (previously) uncompressed data which the decruncher will then expect before the uncompressed address of the chunk to be decompressed. It's used for back-references to copy.
So you could have some kind of buffer ping-pong to approximate streaming. You'll possibly have to copy a large chunk of uncompressed data from a high address to a low address between decompressing chunks, though. |
