Log inRegister an accountBrowse CSDbHelp & documentationFacts & StatisticsThe forumsAvailable RSS-feeds on CSDbSupport CSDb Commodore 64 Scene Database
You are not logged in - nap
CSDb User Forums


Forums > C64 Coding > Doynamite 1.x
2014-03-07 11:45
Bitbreaker

Registered: Oct 2002
Posts: 469
Doynamite 1.x

Hi Folx,

after doynamite was used in some recent productions and people often stumbled over the .prg/.bin pitfall i decided to make some improvements to the packer, it can now spit out a sfx, level-packed data including a valid load-address and depack-address, as well as forward literals to keep the safety margin low. Raw data can still be loaded and output without any bytes added. Also the optimal bitlengths can be iterated now and the optimal table be glued to the output file.
I also happend to make a leaner version that lets the files get slightly bigger, but shrinks the depacker to $e0 bytes and makes depacking 5-10% faster. This might be of good use for demo systems where size matters a lot.
Any more things one could wish?
2014-03-07 14:44
Oswald

Registered: Apr 2002
Posts: 4815
that a very nice job you've done there. I'd only ask for a win exe, 64tass src depackers & a readme.txt :)
2014-03-07 16:13
Groepaz

Registered: Dec 2001
Posts: 10486
acme and source only it is right now - which is perfect =) and friggin fast <3
2014-03-07 19:43
soci

Registered: Sep 2003
Posts: 461
Quote: that a very nice job you've done there. I'd only ask for a win exe, 64tass src depackers & a readme.txt :)

All of these were there, or did I look at the wrong release?
2014-03-07 19:54
Groepaz

Registered: Dec 2001
Posts: 10486
its not yet released :)
2014-03-07 19:57
Burglar

Registered: Dec 2004
Posts: 939
some benchmarks would be nice (in terms of pack ratio and depack time), especially compared to others.
2014-03-08 08:38
Bitbreaker

Registered: Oct 2002
Posts: 469
The .asm source is done in a format that is very easy to adopt to other assemblers i'd say, if people hurt themselves on that, it is not my fault :-) I'll not include any .exe as the source is plattform-independent (got to test that with those changes however) anyway.
I should do some extra documentation on the bunch of switches being added though.
As for benchmarks, i'll dig some numbers out on monday, Axis did a comparison with at least ByteBoozer on C13 (benchmark to load/depack the whole demo). I know that doynamite beats ByteBoozer in speed and pack ratio, but can't remember details. I'll tweak that benchmark to have also numbers at hand for the new features.
Are switches like $01-value sei/cli for the sfx still necessary in our nowadays world? And is it okay to destroy the zeropage to some extent with the depacker by placing code there? Never gained any experience on sfx-depackers yet.
2014-03-08 10:34
WVL

Registered: Mar 2002
Posts: 869
@Enno :

I did just that for the original Doynamite packer. I copied this from my comments for LZWVL :



As you can see I was too lazy to measure how much time Pucrunch took to decrunch, but I'm betting it would be slower than Byteboozer any day ;)

Here's some more hard data :

filesizes

file bin rle lzwvl-f lzwvl-s bb pu doynax
1 11008 8020 4529 4151 3383 3410 3265
2 4973 4314 3532 3309 2648 2687 2512
3 3949 3498 2991 2617 2187 2226 2108
4 7016 6456 4242 4085 3681 3595 3617
5 34760 27647 25781 24895 21306 20887 20405
6 31605 12511 11283 10923 9194 8877 8904
7 20392 17295 12108 11285 9627 9460 9289
8 5713 5407 4179 3916 3251 3314 3132
9 8960 7986 6914 6896 5586 5651 5430

filesize in %

1 100% 73% 41% 38% 31% 31% 30%
2 100% 87% 71% 67% 53% 54% 51%
3 100% 89% 76% 66% 55% 56% 53%
4 100% 92% 60% 58% 52% 51% 52%
5 100% 80% 74% 72% 61% 60% 59%
6 100% 40% 36% 35% 29% 28% 28%
7 100% 85% 59% 55% 47% 46% 46%
8 100% 95% 73% 69% 57% 58% 55%
9 100% 89% 77% 77% 62% 63% 61%

#frames to depack

1 11 13 14 58 27
2 5 7 7 38 17
3 4 6 6 28 12
4 8 9 9 43 20
5 36 39 42 300 119
6 20 25 25 126 49
7 22 25 26 138 60
8 6 8 8 43 18
9 9 12 12 73 32
2014-03-08 10:37
Groepaz

Registered: Dec 2001
Posts: 10486
you guys should spend more time coding demoparts than bothering about depack times, its friggin fast, what else do you need to know? =P
2014-03-08 11:50
Bitbreaker

Registered: Oct 2002
Posts: 469
Going by the numbers BB is faster than doynax? But going by the graph doynax is faster than BB? So what's correct now? :-) And can i have the testfiles so i have compareable results in future? :-) Thx.
Edit: ah, the table is missing the doynax row! :-)
2014-03-08 12:34
Burglar

Registered: Dec 2004
Posts: 939
and wvl's test is using a pretty old doynax version. so what does the current version look like? ;)
2014-03-08 13:47
WVL

Registered: Mar 2002
Posts: 869
the last table is missing the pu column.(didnt waste time testing pu, since we all know it is slow depacking and byteboozer and doynax had better or equal compression on all the files anyway).

So for the last file (file 9), rle takes 9 frames to depack, lzwvl takes 9 (fast mode) or 12 (slow mode), byteboozer takes 73 and doynax took 32 frames.

Anyway, based on the old version : compression was always better than byteboozer and always faster to decompress than byteboozer.

-> when you need small files, go for doynamite or exomizer (but be prepared for really slow decrunch). You can forget about pu or byteboozer.
-> if you need faster depack than doynamite, then go for LZWVL in slow mode or fast mode.
2014-03-08 16:22
Oswald

Registered: Apr 2002
Posts: 4815
this all should be examined combined with loading speed, wether the faster depack worths the bigger file or not ;)
2014-03-08 17:07
Testa
Account closed

Registered: Oct 2004
Posts: 197
thanks wvl!!!, very informative..
2014-03-08 18:45
QuasaR

Registered: Dec 2001
Posts: 144
Thanx WVL for all the work. What I'm missing is Exomizer or is it even more slower than Pucrunch? In my testings it was nearly as good as Pucrunch at compressing but much faster.
2014-03-08 20:10
Burglar

Registered: Dec 2004
Posts: 939
exomizer will beat all in compression (except maybe on tiny files), it will also beat all depacking time, it's the slowest of all.
2014-03-08 22:36
Urban Space Cowboy

Registered: Nov 2004
Posts: 44
At least with Exomizer you can set custom decrunch effects to help pass the time. Cruncher AB nostalgists try this: -s"lsr $d011" -x"and #$05 sta $d020" -f"rol $d011"

Here's WVL's results table again, this time readable:
filesizes
#   bin   rle wvl-f wvl-s    bb    pu doyna
- ----- ----- ----- ----- ----- ----- -----
1 11008  8020  4529  4151  3383  3410  3265
2  4973  4314  3532  3309  2648  2687  2512
3  3949  3498  2991  2617  2187  2226  2108
4  7016  6456  4242  4085  3681  3595  3617
5 34760 27647 25781 24895 21306 20887 20405
6 31605 12511 11283 10923  9194  8877  8904
7 20392 17295 12108 11285  9627  9460  9289
8  5713  5407  4179  3916  3251  3314  3132
9  8960  7986  6914  6896  5586  5651  5430

filesize in %
#   bin   rle wvl-f wvl-s    bb    pu doyna
- ----- ----- ----- ----- ----- ----- -----
1  100%   73%   41%   38%   31%   31%   30%
2  100%   87%   71%   67%   53%   54%   51%
3  100%   89%   76%   66%   55%   56%   53%
4  100%   92%   60%   58%   52%   51%   52%
5  100%   80%   74%   72%   61%   60%   59%
6  100%   40%   36%   35%   29%   28%   28%
7  100%   85%   59%   55%   47%   46%   46%
8  100%   95%   73%   69%   57%   58%   55%
9  100%   89%   77%   77%   62%   63%   61%

number of frames to depack
#   bin   rle wvl-f wvl-s    bb    pu doyna
- ----- ----- ----- ----- ----- ----- -----
1          11    13    14    58          27
2           5     7     7    38          17
3           4     6     6    28          12
4           8     9     9    43          20
5          36    39    42   300         119
6          20    25    25   126          49
7          22    25    26   138          60
8           6     8     8    43          18
9           9    12    12    73          32
2014-03-09 17:50
ChristopherJam

Registered: Aug 2004
Posts: 1248
Is the Pearls for Pigs corpus downloadable anywhere?

I'd quite like to see how the simplistic packer I put together for Jam Ball 2 fares with it. Compression factor's not the best, but the depacker's less than $80 bytes long; no idea how speed compares.
2014-03-09 21:59
WVL

Registered: Mar 2002
Posts: 869
I'll try to dig up the files I used and upload them. They're not really what you would call normal testfiles, but I think they're a pretty good average of stuff you'd find in a demo (graphics, tables, code, etc..)
2014-03-10 07:43
HCL

Registered: Feb 2003
Posts: 704
Hmm.. are you guys saying that byte-boozer decrunching is slow? Perhaps i could do something about it. I assume these values are already using the "optimized" version of the decruncher that has inlined getter of the code-bytes..

Apart from that, i think you guys should spend a minute or two on linking instead. It's quite easy to make 5-10 seconds of black empty screen feel like nothing if you just have something on the screen moving or doing some kind of easy effect.
2014-03-10 12:40
Bitbreaker

Registered: Oct 2002
Posts: 469
Doynamite 1.1

Here you can go with new tests. I'd be interested in suggestions on the simple/ folder. Any further ideas on how to make this smaller and faster? The shifting shit is still eating so much cycles :-(
2014-03-10 18:05
Groepaz

Registered: Dec 2001
Posts: 10486
just remove the zeros!
2014-03-10 18:12
ruk

Registered: Jan 2012
Posts: 43
Quote: Hmm.. are you guys saying that byte-boozer decrunching is slow? Perhaps i could do something about it. I assume these values are already using the "optimized" version of the decruncher that has inlined getter of the code-bytes..

Apart from that, i think you guys should spend a minute or two on linking instead. It's quite easy to make 5-10 seconds of black empty screen feel like nothing if you just have something on the screen moving or doing some kind of easy effect.


Yes, what HCL says. "Dead air" is really boring, but easily remedied. And while a fast depacker surely helps, it is not the only way.

We've successfully used the "oh-no-so-dawg-slow" Exomizer for all our parts in Revolved, Solaris and Continuum without any noticeable pauses. In those cases where we couldn't load and unpack while a part was running, some small rasterbar effect or similar filler-part was inserted.
2014-03-10 18:21
algorithm

Registered: May 2002
Posts: 700
Yes, even if exomizer is a lot slower, usually, it only requires a few extra seconds of 'demo' effect before its done its job. Main use for the faster depacking would probably be if requiring fast load and depack (some type of chunk based streaming) that is compressed finally with a packed that depacks fast
2014-03-10 18:25
Bitbreaker

Registered: Oct 2002
Posts: 469
It is also useful if you want to achieve a higher pace than offence :-) But one could of course say it is art and intended to be slow :-)
2014-03-12 13:52
Urban Space Cowboy

Registered: Nov 2004
Posts: 44
Quoting ChristopherJam
Is the Pearls for Pigs corpus downloadable anywhere?
It's included in LZWVL, the file "bin.rar".
2014-03-12 14:13
Oswald

Registered: Apr 2002
Posts: 4815
its easy to talk about it, but in reality a fast paced demo needs blood,tears, sweat and a human sacrifice
2014-03-12 17:43
WVL

Registered: Mar 2002
Posts: 869
Quote: Quoting ChristopherJam
Is the Pearls for Pigs corpus downloadable anywhere?
It's included in LZWVL, the file "bin.rar".


W00t! I totally forgot that :-D

Those files are nice collection of 'real' data. Some music, some code, some graphics, tables, etc etc :-)
2014-03-13 05:05
ChristopherJam

Registered: Aug 2004
Posts: 1248
Thank you. And yes, it sounds like a pretty representative dataset for what packers face in practice.
2014-03-13 05:52
ChristopherJam

Registered: Aug 2004
Posts: 1248
..and as for things to do while performing a slow decrunch, loader games anyone? ;)
2014-03-13 09:35
WVL

Registered: Mar 2002
Posts: 869
So.. how did your packed perform then? :-)
2014-03-13 11:53
ChristopherJam

Registered: Aug 2004
Posts: 1248
Not quite as good as WVL-S! Here are the results with tinycrunch added:

filesizes
#   bin   rle wvl-f wvl-s    tc    bb    pu doyna
- ----- ----- ----- ----- ----- ----- ----- -----
1 11008  8020  4529  4151  4329  3383  3410  3265
2  4973  4314  3532  3309  3423  2648  2687  2512
3  3949  3498  2991  2617  2972  2187  2226  2108
4  7016  6456  4242  4085  4225  3681  3595  3617
5 34760 27647 25781 24895 25210 21306 20887 20405
6 31605 12511 11283 10923 11614  9194  8877  8904
7 20392 17295 12108 11285 11445  9627  9460  9289
8  5713  5407  4179  3916  3936  3251  3314  3132
9  8960  7986  6914  6896  6572  5586  5651  5430

filesize in %
#   bin   rle wvl-f wvl-s   tc    bb    pu doyna
- ----- ----- ----- -----  --- ----- ----- -----
1  100%   73%   41%   38%  39%   31%   31%   30%
2  100%   87%   71%   67%  69%   53%   54%   51%
3  100%   89%   76%   66%  75%   55%   56%   53%
4  100%   92%   60%   58%  60%   52%   51%   52%
5  100%   80%   74%   72%  73%   61%   60%   59%
6  100%   40%   36%   35%  37%   29%   28%   28%
7  100%   85%   59%   55%  56%   47%   46%   46%
8  100%   95%   73%   69%  69%   57%   58%   55%
9  100%   89%   77%   77%  73%   62%   63%   61%

number of frames to depack
#   bin   rle wvl-f wvl-s    tc    bb    pu doyna
- ----- ----- ----- ----- ----- ----- ----- -----
1          11    13    14    15    58          27
2           5     7     7     9    38          17
3           4     6     6     7    28          12
4           8     9     9    10    43          20
5          36    39    42    59   300         119
6          20    25    25    37   126          49
7          22    25    26    32   138          60
8           6     8     8    10    43          18
9           9    12    12    16    73          32


As you can see, my sizes are always bracketed by WVL-F and WVL-S, and my decompression speed is two thirds of yours for 6.bin.

(crunching the entire corpus took 7 seconds on a single core of a 3GHz i7. It's a fairly slack python script, I put all recent substrings in a dict. The parameters are tuned for the lower entropy components of JamBall2, I might be able to improve the ratio if I play with it a bit).
2014-03-13 12:15
WVL

Registered: Mar 2002
Posts: 869
Score \o/
2014-03-13 12:26
ChristopherJam

Registered: Aug 2004
Posts: 1248
Yes, well done!

All I can say in my defence is my decoder's even smaller than yours ;)
2014-03-14 08:16
Bitbreaker

Registered: Oct 2002
Posts: 469
A bad ratio still spoils the fun, no matter how fast and tiny the depackers get. Here's some benchmark results for loading + depacking the first side (well most of, not those 2 going under IO) of CL13:

bb hclfix                       $0ac4
lzwvl                           $0a08
doynax                          $08a9
doynax_small                    $08e8
doynax_small loaddecomp         $0749


So the additional loading overhead kills all the speed advantage. Loading and decompressing in one go gives the best results, but bloats the code a lot. The spindle system suffers from the same problem, a bad ratio due to having references in only one block. It feels still fast though and i get testfiles loaded around the same speed as with loaddecomp (is there a framecounter available in spindle to proof the feeling?)
2014-03-14 09:06
HCL

Registered: Feb 2003
Posts: 704
Thanx, now i will not waste my time ;). *this* will not keep me from winning any compo in the future, though perhaps other things will :P.
2014-03-14 09:20
enthusi

Registered: May 2004
Posts: 668
I get the feeling that it really doesnt matter much :)
Rather for crackers and onefilers or one-siders maybe? :)
2014-03-14 09:23
ChristopherJam

Registered: Aug 2004
Posts: 1248
Well, the main reason I was optimising for decoder size with tinycrunch was because that was all I had room for once the demo was decoded. It's admittedly a fairly special case, it's not often I'm scraping for every last fraction of a page.

Even the music data was interleaved into unused fragments of the character definitions (only 5 bytes of every 8 were visible).

Agreed that total time for load+decrunch is usually more significant, except in cases where you can background load into some free space, but then need to quickly decrunch between ending one part and starting the next.
RefreshSubscribe to this thread:

You need to be logged in to post in the forum.

Search the forum:
Search   for   in  
All times are CET.
Search CSDb
Advanced
Users Online
Aomeba/Artline Desig..
Didi/Laxity
Peacemaker/CENSOR D...
eryngi
Guests online: 92
Top Demos
1 Edge of Disgrace  (9.6)
2 Coma Light 13  (9.6)
3 Uncensored  (9.6)
4 Still Rising  (9.6)
5 Bromance  (9.6)
6 E2IRA  (9.5)
7 Unboxed  (9.5)
8 Comaland 100%  (9.5)
9 Memento Mori  (9.5)
10 The Shores of Reflec..  (9.5)
Top onefile Demos
1 Raising Snakes  (9.8)
2 Hellofiends  (9.6)
3 Copper Booze  (9.6)
4 Barry Boomer - Trapp..  (9.5)
5 Dawnfall V1.1  (9.5)
6 Daah, Those Acid Pil..  (9.5)
7 Tribute to Ben - Las..  (9.5)
8 Pandemoniac Part 1 o..  (9.5)
9 To Norah  (9.5)
10 Lovecats  (9.4)
Top Groups
1 Booze Design  (9.4)
2 Censor Design  (9.3)
3 Maniacs of Noise  (9.3)
4 Crest  (9.3)
5 1001 Crew  (9.2)
Top Cover Designers
1 Duce  (9.8)
2 Electric  (9.8)
3 Junkie  (9.7)
4 The Elegance  (9.5)
5 Mermaid  (9.3)

Home - Disclaimer
Copyright © No Name 2001-2022
Page generated in: 0.073 sec.