| |
Mr. Mouse
Registered: Dec 2001
Posts: 235 |
Warp compression method (AR)
Does anyone know the compression scheme used in Action Replay's "warp" disksaving method? I notice the sectors look a little different, where's the information about which next sector to go to while decompressing?
|
|
| |
chatGPZ
Registered: Dec 2001
Posts: 11386 |
warp*25 uses a non standard sector format indeed... but no idea where to find info on it, could be interesting :) |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
Which "compression" are you talking about? It's merely a custom disk format. As far as i remember, there is no sector link information - the sector layout is fixed. That is the reason why you can't have more than one warp file per track. |
| |
Mr. Mouse
Registered: Dec 2001
Posts: 235 |
What do you mean, no more than one file per track ? I've got several small ones that are on the same track. Okay, so I'd seen that one file was smaller after a warp save. Also the block contents are quite different compared to the non-warped saves. So what format is it in then? What is the order? I've noticed that on an empty disk the first normal files are started at track 17, the warped files start at track 1, sector 0. Suppose my file is smaller than oh 200 bytes. I won't need more than one sector. If I save several warped files of that small size, they each start in order, skipping 1 sector. So the first warped at sector 0, the next sector 2, then sector 4 etc.
Still, that leaves to find out for me how to read those sectors. Where do they start and in which direction does the reader go? And what process is used to alter the way the original bytes are represented. |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
It looks to me like a warp saved track has the same layout as a normal dos track except for the contents of the data block.
Maybe the "compression" you see is that the t/s link is implied, thus giving two more bytes per sector free?
|
| |
rattus
Account closed
Registered: Oct 2002
Posts: 14 |
This might be actually quite interesting... Loading time for a warp file
was unbelievable in those days... Maybe usable in demos or so... |
| |
Mr. Mouse
Registered: Dec 2001
Posts: 235 |
The T/S link is not implied at all, in my view. Take a look at the first sector of a warped file. There are no immediate references to tracks or sectors. And the contents are really quite different from normal dos blocks. Try saving a small file on an empty disk, then warp-save the file and take a look at the BAM and sectors. You'll find the bytes in the warped sectors match nothing in the normal dos version. And that part interests me. What is happening, and how can I dissolve this. |
| |
Mr. Mouse
Registered: Dec 2001
Posts: 235 |
Here's an example file.
http://c64.xentax.com/downloads/warp_test.zip
There is an 8 byte file of values 01, 02, 04, 08, 10, 20 , 40, 80 (hex) stored at $1000.
And it's warped counterpart. |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Implied as in: the order is implicit i.e 0, 2, 4, 6, 8, etc...
I haven't checked the BAM yet.
Here's your first sector:
--- sync -----------------------
0000: ff ff ff ff ff
--- gcr ------------------------
0005: 52 54 b5 29 4b d2 b4 a5 55 55 55 55 55 55 55
--- decoded --------------------
0005: 08 01 00 01 a0 a0 0f 0f 0f 0f 0f 0f
--- interpretation -------------
sector header, t=1, s=0 (ID='$a0,$a0') chk=$01 (calc=$01)
--------------------------------
--- sync -----------------------
0014: 55 ff ff ff ff ff ff
--- gcr ------------------------
001b: 55 d4 ad a6 a9 69 49 d5 db 76 ab ca 52 95 a6 a6 a6 5d 76 ce 49 49 49 72
0033: 79 7b aa ab 6e 4e 53 99 4b 9d 4e 9a 96 9e a9 d6 ca b2 d2 b5 ae db c9 a6
004b: b3 d6 9b db 52 9a 72 bd 53 6b 73 4e 53 b5 bd d9 d9 5e 96 ad 53 52 a6 6d
0063: ad d3 7a a6 d6 6a ca 56 5d d6 9b 49 c9 66 ca 9e 96 ca aa cd 9d b3 ae 5e
007b: 9a db 5a c9 5b 93 d2 55 53 49 73 bb 6a b9 53 9b ba 6d 5b ad d6 4d 7b 4d
0093: 4a 4d 56 da ce 66 ca c9 4a 5d 9b b3 72 75 4d 6d da b2 66 d5 ce ae 72 9e
00ab: 5b 79 ce d5 d9 a6 75 49 d2 a5 6e 4e 52 9b d6 d2 5a b2 ba 55 ab aa b3 69
00c3: 5a d6 66 ad 7a b2 ab 4b 65 a9 95 69 56 52 96 4d 79 d6 d5 4e 5a 9b bb 73
00db: b2 49 5a 7b 7a b2 ab 4b 65 cd a6 ce 56 52 96 db 4e 9e ba 79 53 da 73 d2
00f3: cb 9e 53 66 66 52 ca 9d 4b 75 d6 d2 5d a6 7b 6e c9 d2 d6 b6 5a 79 b5 9e
010b: 93 56 ba 79 7a b2 ab 4b 65 cb 9a da 56 52 96 ce 4d 9e d9 5a 7a b2 ab 4b
0123: 65 a6 d2 cb 56 52 96 52 da 9e 4a ab 53 da 73 d2 6b 56 65 a9 a9 52 ca 9d
013b: 4b 75 d6 d2 5d a6 7b 6e c9 d2 d6 b6 5b 53 b9 9e 93 56 ba 79 7a b2 ab 4b
0153: 65 cb 9a da 56 52 96 ce 4d 9e 5e 9d 9d ff 4a 55 55 55 55 55
--- decoded --------------------
001b: 07 00 b8 f8 xx xx 16 b6 f5 xx xx xx xx xx ad 64
002f: xx xx xx 39 54 f0 6b 24 04 xx xx 04 30 18 d0 46
0043: 90 9c xx c4 b5 xx xx e6 35 cf xx 32 7f 86 xx a4
0057: 04 ab b6 49 1a 10 xx a2 xx 6a xx ba xx xx f2 26
006b: 17 18 62 48 xx xx xx 60 f1 xx xx d4 1a cc 66 68
007f: 14 8d xx 03 xx 9b 6a f9 04 cb xx 0b f7 xx ae ac
0093: 88 xx c6 64 xx xx 22 2d 34 97 xx 0c c7 c1 xx 6f
00a7: 9a xx xx 2b xx xx 16 xx 4f xx xx 14 89 88 7f 62
00bb: 10 91 xx c1 f0 96 26 66 xx 67 xx f1 8c 2a xx 18
00cf: 09 88 xx 19 ab xx xx xx 7c 9b xx 0a 5c d1 xx a1
00e3: xx xx c3 26 00 1c 63 xx 78 xx xx 33 a1 xx xx xx
00f7: xx xx xx 01 47 1c xx xx 5c xx xx 66 69 xx 3c xx
010b: 2c 11 xx 1a 60 xx 69 41 31 xx xx xx 99 xx d6 0a
011f: 50 90 62 xx xx xx 6f 22 29 8c xx 20 fc 8d xx e2
0133: cc xx 10 c8 01 xx a2 bf ab xx b8 3b cb xx xx f6
0147: 1c 8b xx xx 0a xx 2e f2 fc xx 12 xx b8 xx xx 64
015b: 86 xx xx xx xx xx 0f 0f 0f 0f
As you can see the header block is normal, but the data block uses a custom encoding. |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Quote: Here's an example file.
http://c64.xentax.com/downloads/warp_test.zip
There is an 8 byte file of values 01, 02, 04, 08, 10, 20 , 40, 80 (hex) stored at $1000.
And it's warped counterpart.
Warped files aren't storable on a d64 as it doesn't support bit level encodings. Please use g64 instead. |
| |
Mr. Mouse
Registered: Dec 2001
Posts: 235 |
http://c64.xentax.com/downloads/warptest_g64.zip
Okay, used g64. What tools do you use to view those, other than hex editors?
|
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
You should use a disassembler and check the code reading and writing the warp files.
http://ftp.pokefinder.org/BANK_2.txt
Line 3035++
Enjoy.
/cz |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Quote: http://c64.xentax.com/downloads/warptest_g64.zip
Okay, used g64. What tools do you use to view those, other than hex editors?
I used a rather crude analyzer I made a couple of years ago: g64scan-0.4.tar.gz
Run it with -t <track> to get detailed output, if not you'll get a full disk error scan.
Other than that I agree with cz that you should analyze the code.
I would go for the loader part I think. Run it in vice, Halt during loading and pop into the monitor. |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
Quote: This might be actually quite interesting... Loading time for a warp file
was unbelievable in those days... Maybe usable in demos or so...
Possibly not so interesting. Somebody on Forum64.de has come up with a way to read a standard formatted track in two revolutions, which is warp speed. Actually i'll add this to my loader some day, but it's not that useable for demos. May be nice for booting games into ram extensions and flash cartridges, though :) |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
Btw., this is not the first time i have the suspicion that there are some differences between Action Replay versions concerning warp. Which version are we talking about? |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
So it seems like you CAN have several warp files on a track, but still with implied sector linkage - the next sector in the chain is always implicitly known, no matter on which sector the file starts.
The rest seems fairly easy: Some custom encoding which can be decoded and checksummed while reading the raw data, at the expense of more format loss, which means you can store less useable data on a warp track compared to a normal one. I don't think any warp-saved file gets smaller compared to its KERNAL-formatted counterpart. |
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
AFAIK there is just Warp25 at all. There should be no differences on saving the same file with Warp25 on several versions. It may be that the drivecode was slightly changed between 4.x and 5.x Action Replays but I am not fully aware of that. The Bank2-link above is 4.x code resourced by Crisp and MWS.
/cz |
| |
TNT
Account closed
Registered: Oct 2004
Posts: 189 |
I have somewhere a comparison of several *25 loaders (Warp*25, VorpaLoad, Heureka and at least one more I can't remember just now - Super Snapsot?) and not so surprisingly most of them seem to be copies of each other. Decoding routines may have index registers swapped or RAW -> decoded tables altered, but they all use eight raw bits as an index to six decoded bits (extra bits set/reset depending on the index value to keep hardware happy). That makes decoding much faster than GCR, but limits bytes per sector to 240. |
| |
Mr. Mouse
Registered: Dec 2001
Posts: 235 |
If I use vice to plug in an AR cart and warp save a test file on a d64 image, and transfer it to a c64 it works and loads, so the info on the d64 file is accurate enough to work with. Call me stupid, and I probably am, but using g64 then should not make a difference. [EDIT] I am stupid, it doesn't work with D64, never mind. :)
I've read in the original manual of the AR that warp*25 files will always be 6% larger than the original, because only 240 bytes of the 255 in the sectors contain file-data, so more sectors are needed. That rules out compression, but introduces some kind of encoding.
In the d64 image you can see that a warped file of only 8 bytes long originally has data that is about 9 or 10 bytes long. But this data is encoded somehow, perhaps as presented above. It will be interesting to find out how.
I'm wanting to find out, because I have a corrupted image of a warped file that used to work, and I want to find out where exactly things go awry when decoding those bytes, perhaps I can repair it. The BAM points to the start of these files and indeed the next sectors are probably arranged in a fixed order. I wonder if the code also checked for occupied sectors or would simply overwrite existing. If the latter is the case, then warp saving multiple large files may corrupt others. |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
@mr.mouse: so you are saying is ...
warp save (GCR, not valid block data) -> d64 -> GCR -> warp load
... doesn't lose data in the general case.
I'd say it's unlikely, but not impossible.
(it also depends on what software interprets the invalid encodings into .d64)
I think you'll find it a hard time trying to find out how .d64 data gets mapped from the custom encoding by analysing the code though.
Better start by understanding the gcr encoding, and maybe then you can prove the next step... |
| |
Mr. Mouse
Registered: Dec 2001
Posts: 235 |
Yeah, you are right, I've tried it on other warped files, and it failed. So back to reading up on GCR and the source code. |
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
Quote: I have somewhere a comparison of several *25 loaders (Warp*25, VorpaLoad, Heureka and at least one more I can't remember just now - Super Snapsot?) and not so surprisingly most of them seem to be copies of each other. Decoding routines may have index registers swapped or RAW -> decoded tables altered, but they all use eight raw bits as an index to six decoded bits (extra bits set/reset depending on the index value to keep hardware happy). That makes decoding much faster than GCR, but limits bytes per sector to 240.
Dig it up and pass the comparison over please!
/cz |
| |
TNT
Account closed
Registered: Oct 2004
Posts: 189 |
I found some early notes comparing Epyx Vorpal and Datel Warp*25 drive code, here they are side by side.
First code snipped is disk reading part and related table, the second one is serial transfer and related table. Tables have unused parts blanked so you can compare them...
Sorry for missing memory locations for Vorpal code in these notes.
Vorpal Action Replay Warp*25
BVC * 05e0 50 FE BVC $05E0
CLV 05e2 B8 CLV
LDX $1C01 05e3 AE 01 1C LDX $1C01
EOR $04D1,X 05e6 5D E4 04 EOR $04E4,X
STA $0146,Y 05e9 99 46 01 STA $0146,Y
>8:04d1 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 04e4 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:04e1 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 04f4 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:04f1 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 0504 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:0501 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 0514 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:0511 .. .. .. .. .. .. .. .. .. 00 01 02 .. 03 04 .. 0524 .. .. .. .. .. .. .. .. .. 00 01 02 .. 03 04 ..
>8:0521 .. .. 05 06 .. 07 08 .. .. 09 0a 0b .. 0c 0d .. 0534 .. .. 05 06 .. 07 08 .. .. 09 0a 0b .. 0c 0d ..
>8:0531 .. .. .. .. .. 0e 0f .. .. 20 21 22 .. 23 24 .. 0544 .. .. .. .. .. 0e 0f .. .. 20 21 22 .. 23 24 ..
>8:0541 .. .. 25 26 .. 27 28 .. .. 29 2a 2b .. .. .. .. 0554 .. .. 25 26 .. 27 28 .. .. 29 2a 2b .. .. .. ..
>8:0551 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 0564 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:0561 .. .. 2c 2d .. 2e 2f .. .. 40 41 42 .. 43 44 .. 0574 .. .. 2c 2d .. 2e 2f .. .. 40 41 42 .. 43 44 ..
>8:0571 .. .. .. .. .. 45 46 .. .. 47 48 49 .. 4a 4b .. 0584 .. .. .. .. .. 45 46 .. .. 47 48 49 .. 4a 4b ..
>8:0581 .. .. 4c 4d .. 4e 4f .. .. 60 61 62 .. 63 .. .. 0594 .. .. 4c 4d .. 4e 4f .. .. 60 61 62 .. 63 .. ..
>8:0591 .. .. .. .. .. .. .. .. .. 64 65 66 .. 67 68 .. 05a4 .. .. .. .. .. .. .. .. .. 64 65 66 .. 67 68 ..
>8:05a1 .. .. 69 6a .. 6b 6c .. .. 6d 6e 6f .. .. .. .. 05b4 .. .. 69 6a .. 6b 6c .. .. 6d 6e 6f .. .. .. ..
>8:05b1 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 05c4 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:05c1 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 05d4 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
LDX $0146,Y 047c BE 46 01 LDX $0146,Y
STX $1800 047f 8E 00 18 STX $1800
LDA $0690,X 0482 BD 74 06 LDA $0674,X
STA $1800 0485 8D 00 18 STA $1800
ASL A 0488 0A ASL A
AND #$0F 0489 29 0F AND #$0F
STA $1800 048b 8D 00 18 STA $1800
>8:0690 00 01 00 01 02 03 02 03 00 01 00 01 02 03 02 03 0674 00 01 00 01 02 03 02 03 00 01 00 01 02 03 02 03
>8:06a0 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 0684 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:06b0 04 05 04 05 06 07 06 07 04 05 04 05 06 07 06 07 0694 04 05 04 05 06 07 06 07 04 05 04 05 06 07 06 07
>8:06c0 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 06a4 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:06d0 08 09 08 09 0a 0b 0a 0b 08 09 08 09 0a 0b 0a 0b 06b4 08 09 08 09 0a 0b 0a 0b 08 09 08 09 0a 0b 0a 0b
>8:06e0 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 06c4 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
>8:06f0 0c 0d 0c 0d 0e 0f 0e 0f 0c 0d 0c 0d 0e 0f 0e 0f 06d4 0c 0d 0c 0d 0e 0f 0e 0f 0c 0d 0c 0d 0e 0f 0e 0f
>8:0700 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 06e4 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
|
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
TNT: Very nice - thanx a lot!
/cz |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
That leaves the question of what being first.. :) Of course, there are just so many ways to do warp-speed loading, but the similarities here are glaring. |
| |
TNT
Account closed
Registered: Oct 2004
Posts: 189 |
Super Snapshot v5 has identical tables, and they are at exactly same memory locations as Vorpal. Vorpal was written by Scott Nelson and Stephen Landrum, SSv5 manual credits Richard Bond for Turbo*25. I wonder who ripped who...
I need to get my hands on SSv3 & SSv4 images to check them too. |
| |
TNT
Account closed
Registered: Oct 2004
Posts: 189 |
Vorpal was released in 1986, SSv4 is from 1987/1988 according to manual. I don't know if SSv3 had turbo*25 already. |
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
Absolutely definately I'd bet on Vorpal. Warp25 became available with the Action Replay 3 according to adverts. Interesting part here: early adverts by Datel from April (AR3 release) to June 1987 do not mention Warp25 while any later advert often has an extra bright box for the feature.
Going to put more online sometime soon.
/cz |
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
Snapshot64 released November 1985
Super Snapshot 2 released January 1987 (two versions that year)
Super Snapshot 3 released February 1988
Super Snapshot 4 released November 1988
Super Snapshot 3 released December 1989
And now get me ROM dumps of SSv2 and SSv3!
(And while we are at that - Freeze Frame above MK1, FC2, Formel 64 and Game Killer I am missing out as well - anyone? :) )
@TNT: Take a look at Heureka Sprint while you are at it - would be interesting to know wether it uses the same tables as well. :)
/cz |
| |
raven
Account closed
Registered: Jan 2002
Posts: 137 |
This GCR replacement is actually great for demos, with one small change: next track/sector is better put back in place, since its very useful to change interleave in a demo, according to free rastertime while loading.
A demo using this will have one disadvantage though - it wont be possible to spread it using D64. |
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
Just came across this - adding to this "warp info dump thread" - as I see it - for later reference :)
http://www.faime.demon.co.uk/retro/loader.html
/cz |
| |
Mr. Mouse
Registered: Dec 2001
Posts: 235 |
Nice find, Count! |
| |
Krill
Registered: Apr 2002
Posts: 2980 |
Quote: This GCR replacement is actually great for demos, with one small change: next track/sector is better put back in place, since its very useful to change interleave in a demo, according to free rastertime while loading.
A demo using this will have one disadvantage though - it wont be possible to spread it using D64.
I don't think so.
The custom encoding will speed things up a little, and i mean just a little. Keep in mind you want to do IRQ loading, and that slows down actual transfer quite a bit compared to switching off the screen and all interrupt sources and having 100% CPU for the loader.
Not to mention that demos which need maximum loading speed at all costs to keep up the flow are doubtfully crafted IMO.
The 1571 is capable of decoding a complete standard format block while reading it, just like the 1541 is with this encoding (minus a few bytes of format overhead), and really, it's not that much faster than a 1541 (like 4 kB/s instead of 3.5 kB/s with my loader).
And interleave is something no sane IRQ loader would need to worry about. Also there are scenarios when you have quite some varying CPU time free for the loader, which makes any fixed interleave a bad idea. Yes, i'm advocating out-of-order loading again. The extra revolution it takes to find the block order on a given track can be removed by using a special _high level_ format which would encode the block and file indices needed in the cooked data of every block, which is like 2 bytes lost for a plain 256-byte block. (And with warp encoding, you neither need these meta data nor sector linkage information, since the sector order is fixed, so you can do out-of-order loading easily without interleave worries and a scanning pass. This would work for plainly encoded files, too, and yield acceptable limitations, like e.g. only one file per track.)
And finally, a demo which cannot be spread on d64 is bad. Even when it comes with special tools for transfer, it's a pain in the arse to handle. |
| |
WVL
Registered: Mar 2002
Posts: 902 |
I'm still wondering why there's no format that spreads the content of a page over all the sectors in a track. Basically that would allow the loader to always immediately start reading bytes and never have to wait for the correct sector to pass under the head.
The data content of a sector would look something like this:
(let's describe it compared to a normal disk and call what's a sector on the original disk an osector).
so on sector 1 of the new format :
byte 0 of osector 1, byte 0 of osector 2, byte 0 of oscector 3, etc, byte 1 of osector 1, byte 1 of osector 2, etc, etc, until the sector is filled.
sector 2 :
byte 20 of osector 1, byte 20 of osector 2, byte 20 of oscector 3, etc, byte 21 of osector 1, byte 21 of osector 2, etc, etc, until the sector is filled.
This does mean that more bytes are wasted though, but max bytes wasted is the number of sectors for that track.
The problem is ofcourse that the out-of-orderness is even more mayhem, because bytes that belong to a page are distributed over the whole track.
|
| |
Krill
Registered: Apr 2002
Posts: 2980 |
I'm not sure if i understood you correctly, but this looks like it would be super-slow and/or super-complicated.
You can never read right away because you first need to know which sector you're dealing with, i.e., wait for and read the sector header, which means the time needed to read half a block is wasted, on average (without considering correlations arising from reading previous blocks on the same track).
And i don't see the advantages, since the loader does not have to wait for the one and only next block in the chain using out-of-order loading anyways. :) |
