| | lft
Registered: Jul 2007
Posts: 369 |
Fastloaders & serial transfer timing
This is a continuation of an off-topic discussion in X2016 Competitions: Your Input Wanted!.
There was some concern about serial transfer routines that might fail due to small margins.
I tried adding one safety cycle per bit pair in the transfer routine of Spindle. This caused an average slowdown of 1.5% for my benchmark scenario.
Crosses are bitfire 0.3, just to get some perspective.
Hollow circles are the latest unreleased Spindle with the original (fast) transfer routine.
Green circles are the latest unreleased Spindle with the new (possibly more robust) transfer routine.
I'm thinking that perhaps the robust version should be default, and the fast version available as an option. But on the other hand, nobody has reported any actual issues with the fast code on real drives, so this decision would be based on fear, more or less. Spindle users, what do you say? |
|
| | ChristopherJam
Registered: Aug 2004
Posts: 1378 |
Quoting lftWe can has loadercompo?
Yassss!
Standard corpus, all segments loaded to $8000 to $ffff (writing under IO if needed), tested while displaying a FLI image (preloaded to bank 2) and calling a ten raster dummy music routine, count cycles using VICE, separate league tables depending on whether or not it fails more than one time in ten when loading from a real 1541 from a disk written with a different 1541. |
| | mankeli
Registered: Oct 2010
Posts: 110 |
It doesn't really matter to me which one is the default, I'll probably use "robust mode" anyway and change to the "fast mode" only if absolutely necessary.
I haven't had any problems with the current mode, but I like compatibility. :)
Other feature suggestions: Support for kernel vectors (IDE64 & Vice without TDE) and API call for "seek to the beginning of file #". (disk drive could then perform the seeking while demo part is still running) |
| | Fungus
Registered: Sep 2002
Posts: 616 |
I think groepaz has a drive with the weird CIA revision that can act up.
Maybe try a soak test for 24 hours just to be sure. |
| | doynax
Account closed
Registered: Oct 2004
Posts: 212 |
I, too, would be most interested to know how big of an issue this is in practice and the percentage of real drives affected.
In any event isn't this effectively the same timing bound on most modern IRQ loaders? That is to say the host waits 14-cycles from toggling ATN to reading the result, while the drive-side waiting loop with 13-cycles of worst-case latency. Or is more subtle than that?
I gather that it is well-known that the 1571DCR can't quite manage this in 1 MHz mode. Presumably the same applies to some other set-ups as well though.
Of course the timing on the open-drain bus isn't symmetrical so perhaps it might be sufficient to delay only the sampling after the rising ATN flanks.
There is also the additional headache of supporting multiple drives on the bus where the additional capacitance surely can't help, unless perhaps balanced out by the stronger pull-ups.
Currently I attempt to detect the reliability of the faster timing at start-up but that's mostly a workaround to a CCS64 bug and seems unlikely to prove reliable given temperature variations and whatever other surprises the real world may have to offer. |
| | soci
Registered: Sep 2003
Posts: 473 |
These three timings are from Bitfire:
Bitfire 0.5 is just as tight as Spindle. Does it fail on Chameleon in standalone mode too? |
| | Hoogo
Registered: Jun 2002
Posts: 102 |
I guess it will be difficult to find such a "bad" drive and someone willing to perform all those tests at the same time.
What about these ideas?
-Increase the length of the serial cable until it fails.
-Create a test program that measures the ratio of the clock speeds of C64 and drive? Not sure if there is really much variation in it. |
| | lft
Registered: Jul 2007
Posts: 369 |
Quoting doynaxCurrently I attempt to detect the reliability of the faster timing at start-up
...
I considered that too, but I think it could lead coders astray, especially since Spindle is aimed specifically at trackmos. There is a risk that the coder is timing the demo under perfect conditions, but with a real drive the loader suddenly decides to degrade and run at a lower speed throughout the entire demo because of a single bit error at the wrong moment.
I think it's a better idea to use fixed timing, and give the coder some influence over the risk/speed tradeoff. |
| | doynax
Account closed
Registered: Oct 2004
Posts: 212 |
Quoting HoogoI guess it will be difficult to find such a "bad" drive and someone willing to perform all those tests at the same time. That may admittedly be asking a bit much but hooking up a logic analyzer+oscilloscope to the C64 CIA and 1541 VIA along side the IEC bus on _one_ system while loading down the signal with a decade capacitor/inductor, and playing with a hot air gun/freezer spray and the like to determine the margins ought to be doable. Not that I'm volunteering mind you ;)
Quoting lftI considered that too, but I think it could lead coders astray, especially since Spindle is aimed specifically at trackmos. Plus probably get lynched for wasting all of those precious bytes in the resident code.. |
| | Fungus
Registered: Sep 2002
Posts: 616 |
When we had this issue with n0sd0s it seemed to be limited to 1571 drives, just to note. |
| | Bitbreaker
Registered: Oct 2002
Posts: 500 |
Quoting sociThese three timings are from Bitfire:
Bitfire 0.5 is just as tight as Spindle. Does it fail on Chameleon in standalone mode too?
Thanks for the test and numbers! So far i have not noticed any drive having problems in the transfer, but due to too strict timing when reading a sector or sector header. This is finally fixed in the next version, even THCM's floppy that is known to be very very picky managed to do a 24h test without any error. I suspect, that this drive is changing clock speed when getting warm in a noticeable way, while rotation speed stays stable. With the three floppys i have at home i never experienced any problems, no matter what i did, so it is about finding those not so in spec drives.
As for chamaeleon i don't know, Groepaz did not rant so far, so guess it is fine :-D |
| | Bitbreaker
Registered: Oct 2002
Posts: 500 |
Quote: When we had this issue with n0sd0s it seemed to be limited to 1571 drives, just to note.
Means, a drive detection is enough and manipulating the code in case to slow things down, just as one does with NTSC. |
| | doynax
Account closed
Registered: Oct 2004
Posts: 212 |
Quoting BitbreakerMeans, a drive detection is enough and manipulating the code in case to slow things down, just as one does with NTSC. Alternatively:lda #%00100000
sta $1801
;IRQ transfer loop..
lda #%00000000
sta $1801 |
| | Bitbreaker
Registered: Oct 2002
Posts: 500 |
In fact 2 MHz mode can be switched on all over, except for the sector reading. Seems to speed up things by ~4% then.
That said, who is going to try it out on his 1571? .d64 is available for testing :-) |
| | doynax
Account closed
Registered: Oct 2004
Posts: 212 |
Quoting BitbreakerIn fact 2 MHz mode can be switched on all over, except for the sector reading. Seems to speed up things by ~4% then. As long as the track stepping is VIA timed I suppose. Of course in Lft's case sector reading and IEC transmission is basically _all_ there is.
Quoting BitbreakerThat said, who is going to try it out on his 1571? .d64 is available for testing :-) I'll give it a quick whirl for you on my C128D (non-CR) which has survived $1801 pokes in the past. It doesn't suffer from sensitive IEC timing though. |
| | Flavioweb
Registered: Nov 2011
Posts: 447 |
All this theory could mean that if a real hw drive is connected through another drive (eg running on drive 9 with 8 connected), we can have timing troubles? |
| | doynax
Account closed
Registered: Oct 2004
Posts: 212 |
Quoting FlaviowebAll this theory could mean that if a real hw drive is connected through another drive (eg running on drive 9 with 8 connected), we can have timing troubles? It will certainly _affect_ the timing. Whether it's a help or a hindrance is difficult to guess at without getting out the oscilloscope. |
| | chatGPZ
Registered: Dec 2001
Posts: 11114 |
yes, connecting more drives generates more trouble :) some guy reported the weirdest things for chameleon...and then it turned out he had 4 drives connected. doh! |
| | Bitbreaker
Registered: Oct 2002
Posts: 500 |
A lot of tests and pain was gained on tests on real hardware and it turned out that serial is not the biggest problem to cope with. A loader should also sustain some variation in rotation speed. Also in very rare cases it is not enough to rely on the checksum, neither of the header nor the payload. When spinning up and starting directly to load, it can happen that one still passes all tests and load chunk. Been there, seen that. Same goes if you ommit a check on right track during header sanity checks. It might happen that for a short while you are reading from a neighbour track after stepping, with valid checksums of course. All problems that do not occur on emulated hardware but might let you fail after a bunch of successful runs over a whole disk.
Another though: if you populate the gaps in specific gcr_decode tables with other variables/data to make optimum use of your floppy ram, there is another potential risk:
When reading chunk while spinning up/stepping one could also read invalid gcr codes and make looukups in the gaps. If we have zeroes there, it is an easier to get a false positive checksum. As we read an arbitrary amount of zeroes, we will match with a zero checksum as well, though we have just read nothing. But i have no yet checked, thought of what the floppy is reading if there's no change in polarization, maybe 00000 all over? |
| | doynax
Account closed
Registered: Oct 2004
Posts: 212 |
Quoting BitbreakerA lot of tests and pain was gained on tests on real hardware and it turned out that serial is not the biggest problem to cope with. A loader should also sustain some variation in rotation speed. Also in very rare cases it is not enough to rely on the checksum, neither of the header nor the payload, ... It seems to me that what is really called for is a stronger checksum. More carefully tuned drivecode is going to keep failing regardless, during disk swaps and mechanical problems if nothing else.
I don't know quite what though. A one's complement sum would be a start though, with the sector header and data block sums linked in.
I suppose anything fancier is tricky on a single-accumulator machine though but fletcher-16 is doable, and CRC-8 is only 6 extra cycles per byte if you can afford the 256-byte table. Probably an 8-bit sum in parallel with the standard parity byte would be quite sufficient though.
As has been noted a clearer error model to work with would certainly also help in evaluating the candidates.
Side-note: Things get rather interesting when combined with Kabuto's arithmetic coding, which sort-of makes errors cascade by default. Combined with a header swizzled into the middle of the payload I figure an 8-bit one's complement sum is farily decent. Of course having been to lazy to bothered doing the analysis and it is quite possible that the two arithmetic codes interact badly ;) |
| | Bitbreaker
Registered: Oct 2002
Posts: 500 |
Just let such an error happen again on a real drive (thanks THCM), when i turn on the motor and start reading without writing.
Some parts of the broken sector is the same as from the orig, but the last three bytes would be the checksum of the orig sector + the two zero bytes. Funny enough the checksum of the broken sector would be $23, so i would need to transfer one byte further to see if that is true. Actually i should run into the next sync mark there?
If one waits 0,5s until the motor is at a stable rotation speed, those problems vanish.
broken
!byte $d2,$14,$18,$15,$6b,$15,$70,$70,$00,$06,$d0,$d1,$90,$be,$0c,$75
!byte $1f,$00,$fe,$fe,$85,$d5,$70,$75,$b3,$e2,$c7,$b0,$cb,$53,$fd,$07
!byte $cd,$ba,$e9,$e1,$bd,$e5,$fd,$f5,$57,$ad,$01,$c0,$bf,$1b,$0d,$f8
!byte $f0,$df,$6d,$f4,$3e,$7f,$6a,$05,$66,$2a,$dc,$ad,$33,$ad,$e0,$1d
!byte $d0,$27,$29,$25,$22,$37,$e0,$2b,$f0,$53,$00,$50,$33,$de,$2f,$f1
!byte $e6,$46,$02,$c0,$0a,$76,$28,$25,$1b,$37,$41,$95,$0a,$36,$4c,$77
!byte $17,$4c,$7b,$3f,$98,$10,$17,$a0,$98,$00,$9d,$68,$1a,$98,$9d,$40
!byte $aa,$bd,$da,$0e,$22,$9d,$4c,$1a,$0a,$a0,$00,$85,$44,$17,$8d,$b1
!byte $77,$60,$8d,$55,$a0,$8d,$63,$c3,$93,$4c,$1e,$1b,$cc,$85,$ff,$bd
!byte $bb,$fc,$30,$09,$d9,$f2,$1b,$90,$05,$f0,$02,$49,$ff,$18,$69,$02
!byte $cd,$af,$15,$4a,$90,$0f,$b0,$10,$98,$85,$11,$30,$c9,$c9,$02,$90
!byte $02,$f0,$10,$bd,$6b,$1a,$65,$fe,$9d,$5e,$11,$bd,$6c,$a5,$ff,$4c
!byte $db,$19,$02,$e5,$98,$02,$1c,$0c,$8d,$7e,$18,$a8,$a2,$00,$35,$9d
!byte $43,$30,$2e,$8a,$a2,$1b,$9d,$3e,$1a,$ca,$10,$fa,$8d,$12,$d4,$8d
!byte $15,$6e,$fe,$8c,$94,$8e,$3a,$c3,$aa,$20,$9f,$17,$a2,$07,$a9,$05
!byte $9d,$82,$41,$a9,$01,$9d,$5d,$6d,$b2,$5f,$1a,$4c,$f4,$6b,$00,$00
orig
!byte $d2,$14,$18,$15,$1b,$15,$3e,$1e,$7e,$41,$46,$02,$07,$21,$6c,$e8
!byte $af,$b1,$e9,$1d,$7f,$52,$19,$1d,$af,$b1,$3b,$3e,$2c,$3d,$25,$c7
!byte $9a,$2a,$0c,$34,$e7,$68,$30,$35,$3d,$5c,$1d,$ce,$9a,$46,$a1,$70
!byte $76,$01,$02,$04,$05,$c9,$f4,$3e,$7f,$6a,$05,$56,$2a,$dc,$ad,$c3
!byte $ad,$e0,$1e,$22,$27,$29,$2c,$32,$37,$e0,$29,$3f,$53,$00,$58,$43
!byte $de,$2f,$f8,$a6,$46,$02,$c0,$20,$76,$28,$2c,$31,$37,$41,$95,$b0
!byte $36,$4c,$77,$17,$4c,$7b,$37,$d9,$10,$17,$a8,$a9,$00,$9d,$68,$1a
!byte $98,$9d,$4d,$1a,$bd,$da,$0e,$e2,$9d,$4c,$1a,$60,$a0,$00,$8c,$b4
!byte $17,$8d,$b0,$17,$60,$8d,$5c,$1a,$8d,$63,$c3,$93,$4c,$1e,$19,$bc
!byte $85,$ff,$bd,$9b,$fc,$30,$09,$d9,$f2,$1b,$90,$05,$f0,$02,$49,$ff
!byte $18,$69,$02,$ac,$af,$15,$4a,$90,$0f,$b0,$1d,$b9,$85,$11,$3d,$e8
!byte $c9,$02,$90,$02,$f0,$10,$bd,$6b,$1a,$65,$fe,$9d,$5e,$11,$bd,$6c
!byte $a5,$ff,$4c,$1b,$19,$02,$e5,$98,$02,$1c,$0c,$8d,$7e,$18,$a8,$a2
!byte $00,$35,$9d,$43,$30,$2e,$8a,$a2,$1b,$9d,$3e,$1a,$ca,$10,$fa,$8d
!byte $12,$d4,$8d,$15,$6e,$fe,$8c,$94,$8e,$3a,$c3,$aa,$20,$9f,$17,$a2
!byte $07,$a9,$05,$9d,$82,$41,$a9,$01,$9d,$5d,$6d,$b2,$5f,$1a,$4c,$f4
|
| | ChristopherJam
Registered: Aug 2004
Posts: 1378 |
Intriguing. Are you able to obtain the 320 byte raw datastream from such an event? |
| | Bitbreaker
Registered: Oct 2002
Posts: 500 |
I'm afraid, not yet, but it seems like the gcr decode/read misses some bytes or at least reads some with wrong value. Added the possibility to receive the checksum at least, c64 currently running and i am waiting for another fail :-) |
| | chatGPZ
Registered: Dec 2001
Posts: 11114 |
Quote:But i have no yet checked, thought of what the floppy is reading if there's no change in polarization, maybe 00000 all over?
it will randomly insert ones every 3 or 4 bits or so (the so called "weakbit" case) |
| | Bitbreaker
Registered: Oct 2002
Posts: 500 |
Okay. seems like also spindle is stumbling over that problem. Krill seems to be safe. My fair guess, it is due to the extra revolution for scanning that it does, and that there, the checksum is going over 4 bytes only, thus less error prone. After that process the drive is at a good speed most likely. Working on a solution for that, that is a bit more intelligent than just dumb waiting :-) Tests currently running... |
| | lft
Registered: Jul 2007
Posts: 369 |
Thanks for finding and reporting this! I'll fix it, but I'll try to come up with a better way than waiting for half a second on every track change. There may be just 34 of them, but that's still a whopping 17 seconds. |
| | Fungus
Registered: Sep 2002
Posts: 616 |
Someone with an old white or longboard 1541 with an older alps mechanism that only goes to track 38 should test stepping, if it's done too fast on the older drives it will lock up without fail.
I'd be interested in a table of safe stepping delays for these old ones myself. |
| | lft
Registered: Jul 2007
Posts: 369 |
I think those are separate issues. Minimum stepping delay is to prevent the stepper from missing a signal, and stopping half a track too early. This is about stopping at the right location, but then wobbling around due to inertia and reading parts of the sector from outside the track. Both are important to get right. |
| | Kabuto
Account closed
Registered: Sep 2004
Posts: 58 |
Randomly occurring read errors should come in 2 different flavours: those that desynchronise (i.e. the drive reading more/less bits than were written) and those that don't (and just report one or more adjacent/nearby bits incorrectly).
If there's just a single non-desyncing read error then the XOR checksum used is almost guaranteed to catch it, but anything that desynchronises has odds of 1/256 to incorrectly report the sector as valid. OTOH, rejecting invalid GCR codes (anything not in the table of commodore's 16 codes) will catch many of these errors, but I doubt that timing of fast loaders permits for that.
(And I'm wondering how many of each occur in practice.) |
| | Bitbreaker
Registered: Oct 2002
Posts: 500 |
Okay, i let my test scrub for 474 times over night and it didn't have a single corrupt file after loading (i checksum stuff on c64 side again) It seem like it is enough to successfully read 2 arbitrary sectors while spinning up. One was still too less and caused sporadic errors after like 100 full disk reads. So no plain waiting needed as it seems :-)
One could of course take care for illegal gcr-codes upon checksumming, at least in my case, but that would give a heavy speed penalty. However, illegal gcr codes also lead to further problems when sending, as we would then also lookup illegal values during the gcr2ser lookup, what could lead to a ATN hi for both bit pairs and corrupt data further.
As for spindle it is a bit difficult to build a checksum over whole files, as the depacker would possibly choke on broken blocks. But one could compare the packed results when checksumming over a depack buffer.
Else things are really getting rock solid, due to endless checking (stopping upon errors does not make sense on a loader for demos) it survives removing of the disk and reinserting or also slowing it down by squeezing the disk by hand to slow it down, or pushing it around in the drive while reading where it would skew to other tracks. This makes turn disk so easy, by just letting it read the new dir sector with a magic number until there's success. |
|
