| |
Shadow
Account closed
Registered: Apr 2002
Posts: 355 |
Load first decrunch later or stream decrunch?
Work on my first attempt on an IRQ-loading demo continues. I am using Dreamload, and everything is working great.
However, some parts tend to be rather big and eat both diskspace and load time.
Obvious solution would of course be to compress the data.
As I see it, there should be two options.
1. Load entire compressed part, then decompress.
2. Load byte-by-byte and stream-decompress
At the moment I am leaning towards solution 1. To get this working, the unpacking must allow for overlapping between packed and unpacked data, since I don't have space for both obviously. But I guess a smart decompressor works 'in reverse' so to speak, so overlapping should not be a problem as long as unpacked data is larger than packed...
I have looked at Exomizer, and it seems like it does things that way, and the decompressor code is fairly compact, so it could be a way to go.
Option 2 I have not looked into as much. Dreamload does support a callback on byte-for-byte basis, so it should be possible I guess.
So, I ask all veterans in this field - how is it usually done? Any tips and general good ideas? |
|
... 59 posts hidden. Click here to view all posts....
|
| |
Krill
Registered: Apr 2002
Posts: 2980 |
Raven: Please check out my loader to see that decrunching while loading is faster than first loading and then decrunching, plus interleave is something you can forget now. :D |
| |
raven
Account closed
Registered: Jan 2002
Posts: 137 |
@HCL: well, you need to compromise a little if you aim
for the highest possible speed with a given loader.
I used to concentrate files with equal interleaves in
"zones", thus minimizing the gaps.
Also, in situations in the demo where there is plenty
of time to load, a not-so-optimal interleave can be used,
one that fills the gaps :)
@Krill: Well, when my new loader is finally finished (been
way too long in development, on & off) we can do some
tests.
I still think there's no substitute for properly optimizing
the disk: saving files sequentially, starting from track1,
with the best interleave for the loader.
This also assures the head only moves 1 track at a time,
which greatly minimizes seek times between tracks.
Maybe this all sounds like lots of work, but if you need
to milk every little bit of speed it might be worth it. |
| |
HCL
Registered: Feb 2003
Posts: 728 |
@Raven: Yes, of course it's still worth it to shove all the philez in order starting from track#1. That will always minimize the searching time for each new file, plus it makes the poor 1541 STFU AMAP.
Then I second Krill: You can (almost) forget about the interleave when you're done with your loader, just set it to 8 or something and you'll be fine(st) in more or less all situations. |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
Ack. Interleave optimizing can still be done in very narrow limititations, but with doubtable gain. The drive has varying rotation speed plus your irq handler runs sometimes longer, sometimes shorter, and interrupts the loader at pretty much random points.. So that is, you can't really optimize this like a raster routine. Same goes for other parts of the loader, but i guess you know this after the experience with Insomnia. |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
Oh and by the way: With a slight modification of the file format (you'll lose one or two bytes of data per block depending on certain options) you can have a speed equivalent to the per-load (!) optimum sector interleave. That is, even with varying spinning speeds, irq hander run-times, &c., you'll have the best virtual interleave for any instance of loading.
My implementation is somewhere in the far future, but i can disclose my ideas to anyone interested and willing to implement it. |
| |
raven
Account closed
Registered: Jan 2002
Posts: 137 |
@HCL: interleave 8 is slooow :)
I mostly used 5 & 4 in Insomnia & even that wasnt enough
at times.
Ofcourse I couldnt use that interleave when a "full" effect
was running, so i usually disable some of the routines
for a short time (start/end the part in stages) to free
cycles for loading.
@Krill: I also fooled around with various ideas involving
file-system changes, but didnt implement any of them.
Decided to first simply try to get the most of what I
already have & see how far it gets me. |
| |
HCL
Registered: Feb 2003
Posts: 728 |
@Raven: Quote:interleave 8 is slooow :)
Common, get at grip! You can forget the interleave. It doesn't matter if it's 4 or 8, it will be faster than your 4 + decrunch afterwards anyway. That't the whole point. |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
*mutters something about out-of-order loading and re-ordering on the c64 side* |
| |
HCL
Registered: Feb 2003
Posts: 728 |
Yeah, that sounds kinda funny, JackAsser was babbeling about it also (non-soberly ;). But how do you avoid loading stuff that might belong to another phile!? You're wasting data for knowing that!?
Dunno, since we have all forgot about the interleave by now, what's the use of catching sectors!? ;). |
| |
JackAsser
Registered: Jun 2002
Posts: 2014 |
Quote: Yeah, that sounds kinda funny, JackAsser was babbeling about it also (non-soberly ;). But how do you avoid loading stuff that might belong to another phile!? You're wasting data for knowing that!?
Dunno, since we have all forgot about the interleave by now, what's the use of catching sectors!? ;).
Nevermind me dear chap! ;D |
| Previous - 1 | 2 | 3 | 4 | 5 | 6 | 7 - Next |
