| |
chatGPZ
Registered: Dec 2001
Posts: 11281 |
WANTED: Packer for "Solid Archive" Memory Depacking
I was just looking for this and apparently it doesnt exist, a packer that covers the following usecase:
- pack many small files into one chunk of packed data
- uses dictionary and whatever other lookup data for all files
- depacker allows to depack individual files of that chunk into memory (randomly)
Think of eg a music collection - there you have many relatively small files which are very similar (or even identical) in large parts (the replayer code). With this kind of data you'd get a much better ratio when building a dictionary across all the files instead of packing each file separately.
I know one of you packer dudes are bored at easter.... :) |
|
... 8 posts hidden. Click here to view all posts....
|
| |
Martin Piper
Registered: Nov 2007
Posts: 697 |
The common data for these music files seems to be the note hi/lo table values and some position independent code. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11281 |
Don't get too focussed on that example - there are many other usecases :) |
| |
Martin Piper
Registered: Nov 2007
Posts: 697 |
I'm just using it as easily accessible data files with some commonality. It's not tweaked for music specifically. It would work with graphics data too. |
| |
Martin Piper
Registered: Nov 2007
Posts: 697 |
6502 code works. I'll put a demo on my github in a bit... |
| |
Krill
Registered: Apr 2002
Posts: 2940 |
How do you decide which sequences to put into the dictionary vs. which to keep as local back-references, ideally with some optimality? |
| |
Martin Piper
Registered: Nov 2007
Posts: 697 |
Frequency of use combined with encoding of offset efficiency. |
| |
Oswald
Registered: Apr 2002
Posts: 5074 |
"The 1024 byte dictionary was filled after the second file, which is why the first file benefits most from the dictionary"
doesnt sound optimal, dictionary should contain most used strings across all files regardless order. |
| |
Martin Piper
Registered: Nov 2007
Posts: 697 |
That was from an earlier test before I added the dictionary optimisation pass.
There is a configurable limit to the dictionary size as well. |
| |
Martin Piper
Registered: Nov 2007
Posts: 697 |
Readme: https://github.com/martinpiper/C64Public/blob/master/Dictionary..
Demo: https://github.com/martinpiper/C64Public/raw/master/DictionaryC..
The decompression source is a readable reference implementation. There is plenty of opportunity for optimisation.
I've got some ideas for more compression optimisations, but it will do for now. |
| |
Martin Piper
Registered: Nov 2007
Posts: 697 |
PS. The compression has been optimised again and the tool/example updated.
Original data size: 12710 bytes
Previous compressed size: 7864 bytes
New compressed size: 7574 bytes
Basically it's trying different compression options (and using multiple threads) to hunt for the best savings. |
| Previous - 1 | 2 - Next |
