Log inRegister an accountBrowse CSDbHelp & documentationFacts & StatisticsThe forumsAvailable RSS-feeds on CSDbSupport CSDb Commodore 64 Scene Database
You are not logged in - nap
CSDb User Forums


Forums > C64 Coding > WANTED: Packer for "Solid Archive" Memory Depacking
2022-04-13 00:07
chatGPZ

Registered: Dec 2001
Posts: 11386
WANTED: Packer for "Solid Archive" Memory Depacking

I was just looking for this and apparently it doesnt exist, a packer that covers the following usecase:

- pack many small files into one chunk of packed data
- uses dictionary and whatever other lookup data for all files
- depacker allows to depack individual files of that chunk into memory (randomly)

Think of eg a music collection - there you have many relatively small files which are very similar (or even identical) in large parts (the replayer code). With this kind of data you'd get a much better ratio when building a dictionary across all the files instead of packing each file separately.

I know one of you packer dudes are bored at easter.... :)
 
... 8 posts hidden. Click here to view all posts....
 
2022-04-29 17:24
Martin Piper

Registered: Nov 2007
Posts: 722
The common data for these music files seems to be the note hi/lo table values and some position independent code.
2022-04-29 17:44
chatGPZ

Registered: Dec 2001
Posts: 11386
Don't get too focussed on that example - there are many other usecases :)
2022-04-30 03:12
Martin Piper

Registered: Nov 2007
Posts: 722
I'm just using it as easily accessible data files with some commonality. It's not tweaked for music specifically. It would work with graphics data too.
2022-05-02 13:19
Martin Piper

Registered: Nov 2007
Posts: 722
6502 code works. I'll put a demo on my github in a bit...
2022-05-02 14:25
Krill

Registered: Apr 2002
Posts: 2980
How do you decide which sequences to put into the dictionary vs. which to keep as local back-references, ideally with some optimality?
2022-05-02 14:37
Martin Piper

Registered: Nov 2007
Posts: 722
Frequency of use combined with encoding of offset efficiency.
2022-05-02 16:21
Oswald

Registered: Apr 2002
Posts: 5094
"The 1024 byte dictionary was filled after the second file, which is why the first file benefits most from the dictionary"

doesnt sound optimal, dictionary should contain most used strings across all files regardless order.
2022-05-02 17:34
Martin Piper

Registered: Nov 2007
Posts: 722
That was from an earlier test before I added the dictionary optimisation pass.

There is a configurable limit to the dictionary size as well.
2022-05-02 17:39
Martin Piper

Registered: Nov 2007
Posts: 722
Readme: https://github.com/martinpiper/C64Public/blob/master/Dictionary..

Demo: https://github.com/martinpiper/C64Public/raw/master/DictionaryC..


The decompression source is a readable reference implementation. There is plenty of opportunity for optimisation.

I've got some ideas for more compression optimisations, but it will do for now.
2022-05-05 07:18
Martin Piper

Registered: Nov 2007
Posts: 722
PS. The compression has been optimised again and the tool/example updated.
Original data size: 12710 bytes
Previous compressed size: 7864 bytes
New compressed size: 7574 bytes

Basically it's trying different compression options (and using multiple threads) to hunt for the best savings.
Previous - 1 | 2 - Next
RefreshSubscribe to this thread:

You need to be logged in to post in the forum.

Search the forum:
Search   for   in  
All times are CET.
Search CSDb
Advanced
Users Online
MWR/Visdom
Broti/DT/KRN
Scrap/Genesis Project
rambo/Therapy/ Resou..
The MeatBall
rtiainen
Dan
megasoftargentina
Steve/Laser, Zenith,..
rexbeng
Murphy/Exceed
Guests online: 108
Top Demos
1 Next Level  (9.7)
2 13:37  (9.7)
3 Coma Light 13  (9.6)
4 Mojo  (9.6)
5 Edge of Disgrace  (9.6)
6 The Demo Coder  (9.6)
7 What Is The Matrix 2  (9.6)
8 Uncensored  (9.6)
9 Comaland 100%  (9.6)
10 Wonderland XIV  (9.6)
Top onefile Demos
1 Layers  (9.6)
2 No Listen  (9.6)
3 Party Elk 2  (9.6)
4 Cubic Dream  (9.6)
5 Copper Booze  (9.6)
6 Rainbow Connection  (9.5)
7 Dawnfall V1.1  (9.5)
8 Onscreen 5k  (9.5)
9 Morph  (9.5)
10 Libertongo  (9.5)
Top Groups
1 Performers  (9.3)
2 Booze Design  (9.3)
3 Oxyron  (9.3)
4 Triad  (9.3)
5 Censor Design  (9.3)
Top Fullscreen Graphicians
1 Joe  (9.7)
2 Sulevi  (9.6)
3 The Sarge  (9.6)
4 Veto  (9.6)
5 Facet  (9.6)

Home - Disclaimer
Copyright © No Name 2001-2024
Page generated in: 0.064 sec.