| |
Shadow
Account closed
Registered: Apr 2002
Posts: 355 |
Load first decrunch later or stream decrunch?
Work on my first attempt on an IRQ-loading demo continues. I am using Dreamload, and everything is working great.
However, some parts tend to be rather big and eat both diskspace and load time.
Obvious solution would of course be to compress the data.
As I see it, there should be two options.
1. Load entire compressed part, then decompress.
2. Load byte-by-byte and stream-decompress
At the moment I am leaning towards solution 1. To get this working, the unpacking must allow for overlapping between packed and unpacked data, since I don't have space for both obviously. But I guess a smart decompressor works 'in reverse' so to speak, so overlapping should not be a problem as long as unpacked data is larger than packed...
I have looked at Exomizer, and it seems like it does things that way, and the decompressor code is fairly compact, so it could be a way to go.
Option 2 I have not looked into as much. Dreamload does support a callback on byte-for-byte basis, so it should be possible I guess.
So, I ask all veterans in this field - how is it usually done? Any tips and general good ideas? |
|
... 59 posts hidden. Click here to view all posts....
|
| |
Ninja
Registered: Jan 2002
Posts: 411 |
Well, you won't come around experimenting a bit and develop a feeling for an apropriate solution.
Depacking on the fly can be quite faster, if you know what you are doing. If not, there is hardly a difference.
Exomizer can be slower when depacking, but maybe it pays off as you might have to transfer one or a couple of sectors less? And what if you rewrite the decompressor just a bit to speed it up (replacing JSR getbit with a macro already helps)?
So many options and we haven't talked about interleave yet ;)
@Shadow: If you don't need maximum speed, take an easily understandable approach. Doing the first trackmo is usually complicated enough. |
| |
HCL
Registered: Feb 2003
Posts: 728 |
Didn't i just post on this therad!? WTF where is? |
| |
HCL
Registered: Feb 2003
Posts: 728 |
Yeye.. really phun to rewrite the whole post again :(. Well, to keep it short..
I of course (also) have the solution to these problems. My loader system has ByteBoozer-decrunch built in, so you can either load+decrunch at the same time, or load/decrunch separately. You have to use ByteBoozer then of course, but there is both c64-and PC-version of it + source, so that should be bearable.
Looking at performance, it's really the fastest way to load and decrunch at the same time. The byte-per-byte approach sounds cute, but is really inefficient if you look at the properties of crunched data. You will have chunks of data that should be copied into memory, and doing one JSR for each byte there will have an impact on the performance for sure. I had an idea of a hybrid solution there also, using a cyclic buffer, but let's just forget that for the moment. |
| |
Stryyker
Registered: Dec 2001
Posts: 468 |
I know Jolz used a 1 sector buffer in C64 memory in Vandalism News loader. Load a sector to RAM, process that while the 1541 moves to next sector. |
| |
HCL
Registered: Feb 2003
Posts: 728 |
Yep, something like that. There is some room for tweaking there i guess on the buffer size, in some situations i think that a buffer of 2 sectors would work better. ..But i haven't tried it out, so i really shouldn't come with wise comments. |
| |
Shadow
Account closed
Registered: Apr 2002
Posts: 355 |
Alright, thanks for the input all. To get going, I did some preliminary tests:
Loading a 27.4kb unpacked data took 12.4 seconds.
The same data packed with exomizer ended up at 12.5kb.
With the exomizer self-extractor (I assume this uses the same code as the depacker source included) it took 4.3 seconds to unpack. I guess it will take a bit longer in the demo though, since I will have music IRQ still going stealing a bit of time. Still, should be below 5 seconds.
If we assume that loading 12.5kb instead of 27.4kb takes 12.5*(12.4/27.4)=5.6 seconds, the total for loading and unpacking would be less than the time required for just loading the unpacked data.
However, I suspect that this reasoning might not be correct, that there is some 'init'-time to the loading (finding right track etc.) meaning that it probably will take more than 5.6 seconds to load the 12.5kb.
Still, looks like it could be a way to go. Now I need to find some space to put the exomizer depacker in though. Perhaps $0200-$03ff, that is just about the only space I have left. Wonder if the exomizer depacker will fit in there... |
| |
HCL
Registered: Feb 2003
Posts: 728 |
The thing you're not considering is that you can actually do some depacking *at the same time* while loading. The loader spends some time just waiting for the disk drive to find the next sector and reading it. This time can be used for decrunching instead.
Then there's also difference in performance on different decrunchers :P.
ByteBoozer decruncher is easily less than $100 bytes. Integrated with loader it ends up on ~$220 bytes. Your choice ;).
|
| |
Danzig
Registered: Jun 2002
Posts: 440 |
@hcl: sound like you try to sell it, but you forgot the pricing ;) |
| |
Shadow
Account closed
Registered: Apr 2002
Posts: 355 |
Haha, yeah, I was also started contemplating if HCL perhaps has a background in the sales department... ;)
Well, it was a nice pitch, and I would consider it perhaps if I had more time, but switching loader etc. at this stage is not something I would venture into (must finish demo in time for LCP08!)
Anyway, I did implement the exomize-after-load variant, and the result was very satisfying:
The time for data loading + decrunching for my test part ended up at 11 sec, so a 1.4 second win in total time compared to the unpacked data, as well as saving more than 50% diskspace! |
| |
HCL
Registered: Feb 2003
Posts: 728 |
Hehe :D, well i don't know if i could sell vacuumers when i can't even sell this, which is for free! I'll even buy you a beer if you use it ;).
Horrible to hear that you're ok with such an un-optimized solution. But in the age of warp-speed, who cares anymore :(. Just for teh phun Shadow, plz send me your test philez and i'll see if my shit is really that much faster. |
| Previous - 1 | 2 | 3 | 4 | 5 | 6 | 7 - Next |
