Maybe, but

FDCB  24 4D    BIT $4D       ;new interrupt ?
FDCD  10 FC    BPL $FDCB     ;no,
FDCF  A5 41    LDA $41       ;read PORT A: data input
FDD1  91 16    STA ($16),Y   ;store into assigned buffer
FDD3  45 08    EOR $08       ;calculate new checksum
FDD5  85 08    STA $08       ;store new checksum result
FDD7  C8       INY           ;all bytes read ?
FDD8  D0 F1    BNE $FDCB     ;no,

getting readily decoded bytes right from the hardware, with IO registers in zeropage, too, feels like quite the luxury. =)

The GCR encode/decode hardware seems rather compact, only a prom and a few TTL chips. No real custom formats possible though, except changing the general layout.

The later 8050 drive has a 6502 instead of the 6504 and uses the SO pin, but that's higher density so I guess it might be pushing it a bit, even with hw GCR decoding.

Yes, but then there is not much of a need for custom formats with that speed provided by hardware already. Plus Commodore GCR density was outstanding for its time (better than most (all?) competitors). Things like Kabuto's encoding weren't even considered, i guess. =) (And then it's "only" 9% denser.

Very likely. From https://github.com/mist64/cbmsrc/blob/master/CONTROLLER_8250/r_w :

L100   BVC L100        ; NO TIME FOR JSR
       CLV             ; CLEAR BYTE READY (V)
;
       LDA DIN         ; LOAD BYTE
       STA (BUFPT)Y    ; STORE IN BUFFER
       INY
       BNE L100        ;LOOP FOR 256 BYTES

They also moved the checksumming to a separate loop after reading.

Quoting Krill

Things like Kabuto's encoding weren't even considered, i guess. =) (And then it's "only" 9% denser.)

The first papers on arithmetic coding had just been published when the first Commodore drives were designed, so that had been very cutting edge. :)

Haven't looked very much at those higher density drives. Are they just a higher bitrate with the same GCR + 80 tracks?

Quoting tlr

Quoting Krill

Things like Kabuto's encoding weren't even considered, i guess. =) (And then it's "only" 9% denser.)

The first papers on arithmetic coding had just been published when the first Commodore drives were designed, so that had been very cutting edge. :)

Yeah. :) Plus that scheme is not just arithmetic coding, but something else based on the ideas of arithmetic coding.
So this might have been original research even, back then. (It may have very well been back in 2012, too, though. :D)

I'm pretty sure I saw a paper on this (which I didn't understand fully) after searching around RLL a couple of years back. Will see if I can find it back.