Looking at the schematics I haven't seen a way to bypass the c64's RAM so VIC & Cartridge could communicate directly
also it would be like an a1200 with 060 strapped on: a very fast cpu trying to stuff gfx into a very slow display system.
The communication between the two ARM chips could be done at hi-speed by SPI.