| |
AmiDog
Registered: Mar 2003
Posts: 97 |
Dealing with (and distributing) large files
Are there any good solutions for dealing with large files, i.e. files of more than 1MB?
The least annoying I could come up with in an emulated environment was to put everything in a large (16MB) REU-file, and DMA the data. I know the 1541U can emulate a large REU, so that might be a solution.
There is the IDE64, but there doesn't seem to be any command line tools to create and manage drive images, making it somewhat annoying to use when crossdeveloping.
Then we have multiple network adapters, which I assume might be a solution, assuming there actually are some ready to use, fast enough, loaders available somewhere?
And last but not least, spreading the data over multiple floppies might work, but would be very annoying unless one has access to a FD-2000/4000 and can use disks of a decent size. I assume it would be very slow though.
Any thoughts? |
|
| |
chatGPZ
Registered: Dec 2001
Posts: 11135 |
considering that most people would probably not download these files using their c64.... why not just put them into a .zip and let the user decide where/how to put it on the c64? |
| |
AmiDog
Registered: Mar 2003
Posts: 97 |
Still, the code to access the large file(s) would have to be written. Some devices may have direct support for seeking in files, some (most?) do not.
So there are really two problems:
1. Getting the files to some kind of media or storage the C64 can access one way or another.
2. Accessing the files, which may or may not be split into smaller chunks in order to fit on some kind of floppy-disks. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11135 |
uhm. of course. what are you trying to achieve anyway, and who is your target audience? a generic solution would have to support most (preferably all) of what you listed basically :) |
| |
soci
Registered: Sep 2003
Posts: 474 |
Quoting AmiDogThere is the IDE64, but there doesn't seem to be any command line tools to create and manage drive images, making it somewhat annoying to use when crossdeveloping.
I tend to disagree. There's fusecfs which lets you mount CFS disk images on Linux (and BSDs). The same image can be used in VICE at the same time if you're careful not to write to it. It's also handy for transferring gigabytes of stuff, especially since that a nice quadratic performance bug is gone ;)
Quoting AmiDogThen we have multiple network adapters, which I assume might be a solution, assuming there actually are some ready to use, fast enough, loaders available somewhere?
There's PCLink on IDEDOS. I use it whenever I have real hardware with me. Works over various physical connections, but I use USB/Ethernet most of the time.
I don't even copy stuff over, as it's completely transparent for file operations. Even seeking works the same way as with a filesystem on CF card (using 0.91 beta). Did I copied all those MP3s to test the MP3 player's tag parsing to a CF card or into VICE? Of course not.
I hacked VICE to emulate USB PCLink for IDE64, but I was lazy to convert it to use sockets instead of pair of pipes, therefore it's not available. Last time I've checked the CS8900 emulation was not accurate enough for Ethernet PCLink without workarounds which degraded performance on real hardware. Yes, I know I'll get a comment soon that maybe I should fix it sometime ;) |
| |
Count Zero
Registered: Jan 2003
Posts: 1825 |
What groepaz said - there may also be the question about the data being static or in some way externally modified. If it is static and a game or so you might want to consider a more or less static solution such as an easyflash as a target.
Whenever it comes to speed on an emulated or original system hardly anything can beat a REU anyhow and "more than 1MB" is also a border you may want to tweak on depending on the targetted hardware. So IMHO such decisions depend on more fine grained parameters here :) |
| |
AmiDog
Registered: Mar 2003
Posts: 97 |
Quoting sociI tend to disagree. There's fusecfs which lets you mount CFS disk images on Linux (and BSDs). The same image can be used in VICE at the same time if you're careful not to write to it. It's also handy for transferring gigabytes of stuff, especially since that a nice quadratic performance bug is gone ;)
Yes, for Linux that's a solution. However, I do most of my developing using Windows and Cygwin (which I've put on a USB stick so I can use more or less any PC I have access to without the need to install anything).
I've considered trying to hack that FUSE driver into a command line tool, but not had the time to look into it yet. |
| |
AmiDog
Registered: Mar 2003
Posts: 97 |
Quoting Groepazuhm. of course. what are you trying to achieve anyway, and who is your target audience? a generic solution would have to support most (preferably all) of what you listed basically :)
Well, while this isn't meant to be taken seriously, it made it quite obvious that some kind of solution is needed to deal with large files. And while the REU hack does work in this case, atleast in en emulated environment, it's limited to 16MB which may be too little... |
| |
chatGPZ
Registered: Dec 2001
Posts: 11135 |
i'd just use SD-card then |
| |
soci
Registered: Sep 2003
Posts: 474 |
Quoting AmiDogI've considered trying to hack that FUSE driver into a command line tool, but not had the time to look into it yet.
Sounds interesting. Could be useful for other FUSE filesystems as well.
What about mkisofs or genisoimage? This is part of cygwin.
You can make an iso image with "mkisofs -J directory >file.iso". Feed this to VICE as one of the IDE64 disk images, and it'll emulate a DVD drive. IDEDOS can use it with long filenames, proper directory structure and seekable files. |
| |
Oswald
Registered: Apr 2002
Posts: 5022 |
I'd love to see a vid of that scpu-doom, no matter how slow :)insta nerdgasm |
| |
chatGPZ
Registered: Dec 2001
Posts: 11135 |
too hard to run it in the emu? =P |
| |
Oswald
Registered: Apr 2002
Posts: 5022 |
indeed. I could find the scpu vice hack, and add the reu file, but drive 8 emulation is disabled, and I find no way to start io.prg
edit: okay, so the code is compiled from a doom code compiled for another cpu, doesnt sound too promising regarding speed. :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11135 |
what about trying the command line given in the readme? ^_^ |
| |
Oswald
Registered: Apr 2002
Posts: 5022 |
ah okay, didnt notice it, guess it has linux linebreaks and its all in one long line.
still no luck tho. command line attaches the reu image, but io.prg seems not to be loaded. autostart option seems have no effect, list doesnt shows anything, run is not typed out upon start. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11135 |
works just fine here... are you using a recent snapshot? |
| |
Oswald
Registered: Apr 2002
Posts: 5022 |
okay, found the latest version, runs fine :D
boy this is awesome no matter how slow and unplayable it is. never thought I will play DOOM on a c64!! (in emu:)
I wonder how fast it could be if directly compiled for scpu, and optimized.
wish there was some dithering it could help visibility a lot. |
| |
Count Zero
Registered: Jan 2003
Posts: 1825 |
So, thats a nice use case! :)
xscpu64 -config nonexistfiletoresetconf +cart -reu -reusize 16384 -reuimage doom.reu io.prg
Nice one AmiDog - boo, Oswald! :)
Such a distribution is perfectly fine with me - maybe Oswald & friends need help files pointing out details.
Just release with some fun pron WADs people watching nufli vids will rush to play SCPU doom clones :)
But what you are actually asking for is an xscpu64 option which pre-loads the up to 16MB RAM of the REU as well, to speed up the whole process, no?
Soci surely can help out here, I think.
For real hardware some generic loader thing indeed is not too trivial. Bitbanging the serial bus is no fun, network adapters not well spread and I wouldn't bet dimes on 1541U working well with SCPU on real HW either :( |
| |
Axis/Oxyron
Registered: Apr 2007
Posts: 91 |
This doom port is surely ugly, but really impressive. Never thought this game could run on that platform. Also speed is not 'that' bad, compared to AGA Amiga ports that are not so much faster.
If this is just the compiled and ported Linux Doom. I would suggest to do the port based on the work of BadMooD, a very impressive Doom port for Atari Falcon.
http://devilsdoorbell.com
The guy behind it did a lot of optimizations on things like AI and collision detection, which took more than 50% of the CPU time. |
| |
AmiDog
Registered: Mar 2003
Posts: 97 |
Quoting Oswaldedit: okay, so the code is compiled from a doom code compiled for another cpu, doesnt sound too promising regarding speed. :)
I agree, but it probably isn't as bad as you might think, at least not for 32bit applications. The MIPS is a really simple 32bit RISC CPU. There are no CPU flags to keep track of, it has 32 general purpose registers, and few and simple instructions.
Sure, it is a load/store architecture, which means any memory accesses first goes to an emulated register in the direct page before any processing can be done. On the other hand, a good 65816 compiler would have to cache data in the direct page to achieve good performance. As an example, a 32bit add:
clc
lda <low>
adc <low>
sta <low>
lda <high>
adc <high>
sta <high>
When all addresses are in the direct page, it takes (2+6*4) 26 cycles to complete. If all addresses are in the "far" segment and require long addressing, each memory access take 6 instead of 4 cycles. However, that is only true if the address is in fast SRAM. If it's in DRAM (i.e. SuperRAM) the situation is a lot worse.
Since code and data are in different sections, they will be located far apart. A read from a different column results in a 1 cycle additional delay, and since a column is only 4 bytes, that is always going to be true. If the read is from a different row, which is also very likely since a row is 2-8KB and code and data for a large project will be far from each other, you instead get a 3.5 cycle penalty.
But that's only half of the story, after the memory has been read, the CPU needs to fetch the next opcode, which will be far away from the data, causing another penalty. So all in all, that 6 cycle load will actually take 6 + 3.5 + 3.5 = 13 cycles. Writes are somewhat faster. There is no column penalty and the row penalty is 3.0 cycles. All in all, that 32bit add does take (2+13*4+12.5*2) 79 cycles if the operands are in SuperRAM.
Just image what would happen if your compiler isn't clever enough to cache global data in the direct page before a loop, causing the same value to be read over and over again, taking (2*13) 26 cycles instead of (2*4) 8 cycles each iteration... |
| |
AmiDog
Registered: Mar 2003
Posts: 97 |
Quoting Axis/OxyronThis doom port is surely ugly
And now it can be made to look even worse :-) New release uploaded which fixes low detail mode and map rendering. There's also a 16 color FLI mode now, which does look a little less bad but slows things down even more...
edit: The FLI mode probably doesn't work on the real hardware due to timing differences between VICE and real hardware (bug #543). Also, I haven't bothered to fix the leftmost three columns ("Fli-bug" area) since SuperCPU-style fullscreen FLI doesn't work in VICE currently (bug #545). |
