| |
Digger
Registered: Mar 2005
Posts: 422 |
Release id #232976 : Next Level
Well, well. I think that such demo deserves a thread here.
I already spoke with BitBreaker about some things, and what puzzles me is the process behind capturing 100 games by Pex. This part has absolutely crushed me during the compo – great idea and execution there!
However, I must say some game screens felt pretty static (I guess due to amount of VIC/SID data needed to be captured and played back) and some classic (e.g. Turrican) games were missing.
I was wondering if it crossed your mind to compromise on the amount of games but keep 'em on screen for twice as long.
And of course if @Mahoney would share some secrets about the capturing process it would be awesome. Thx. |
|
| |
wacek
Registered: Nov 2007
Posts: 505 |
He actually described it quite detailed in a facebook post in one of the C64 groups... |
| |
Burglar
Registered: Dec 2004
Posts: 1051 |
not everybody is on facebook :)
but this is what I thought:
- charmode only for background (hence missing games like last ninja)
- capture charset + sprites via vice
- capture many io locations each frame (sprite pointers, locations, colors, sid registers, etc, etc)
- fancy multiplexer (the texts were probably the most challenging:)
- custom packer/unpacker
- buffered loading (i dont think every game-capture is same size and some seem too big to load <3 sec;)
I probably missed a bunch, feel free to correct :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11154 |
I think he just visited crossbow and said "you cant do this!" |
| |
PAL
Registered: Mar 2009
Posts: 270 |
I actually went to Mahony after the demoshow at X and asked straight out if he had been insane enough to create a sort of recording routine and actually recorded all the sprite movement, charsets and screen setups and he looked (a bit surprised) at me and said: Yes PAL that is what I did. 4mb of data from the memory of the machine from his clever alterations and then put back into a playback engine... he then said... how you know? I think that is what I would have done in a way if I could code, but Mahony how did you make all them music play so fast also... he said, I just played them games PAL and I also recorded all the sidwrites.
So it must have been almost like a timeline editing thing in the end as I guess and we all know he is so clever and all was in sync from the very first recording of anything in anything.
What a great way of doing this. INSANE OUR OWN MAHONY! |
| |
Digger
Registered: Mar 2005
Posts: 422 |
Quote: He actually described it quite detailed in a facebook post in one of the C64 groups...
Great! Do you think you could find the link Wacek? |
| |
Digger
Registered: Mar 2005
Posts: 422 |
Here's what I got first hand from Mahoney and BB if anyone is insterested:
I rewrote the Vice emulator to dump not only the whole state of the emulator at a certain point in time as a snapshot, but then also every single read and write to memory with a timestamp. Recording a 5-second snippet of a game resulted in a 4MB file on disk.
Then, I wrote a python script to extract the colours on the screen, the chars on the screen, the charset, the sprites. Every game is 2 seconds, which means 100 frames of updates needs to be tracked, so I gather a list of every single memory location that needs to be changed to update the graphics, by fast forwarding the state of the C64 SID and VIC-II chips some 19000 clock cycles to get to the next frame.
This leads to a 100 to 200 lists of memory locations with 100 values that needs to be written to replay the graphics and sound. I need to compress these, but more importantly, I need to decompress these lists in real-time with as few CPU cycles as possible, since I only have some 19000 clock cycles until the next frame needs to be shown.
And then I added a sprite multiplexer to show the texts on top of that. It's easy when you look at one single part of this at a time, but really messy when you combine all needed parts - I need to load the data for the next game while running the current one, for instance.
Some interesting stuff from BitBreaker:
The 100 games would not have fit if I hadn't brought bitfire to next level, by sharing sectors with the fileend and start of the upcoming file. This saved ~50 blocks, the zx0/salvador/dali packer did another good job in saving. It is 2 bytes left on that diskside, while not using 40 tracks, as we felt it is not as vanilla as we want to be. |
| |
Pex Mahoney Tufvesson
Registered: Sep 2003
Posts: 50 |
Question from Digger: And was the whole process iterative, i.e. did you have to capture different game moments with trial and error to get it all fit?
Lots of iterations, of course! I started with capturing 10 games to find out how difficult it would be to extract the graphics. I quickly found out that I needed a compression scheme that would be really efficient for the most common event: counting down a number of frames until the next change of a memory location is to be done. It realise I could probably write a whole book about this demo! Maybe I should? :) |
| |
Boogaloo
Registered: Aug 2019
Posts: 21 |
Quote: Question from Digger: And was the whole process iterative, i.e. did you have to capture different game moments with trial and error to get it all fit?
Lots of iterations, of course! I started with capturing 10 games to find out how difficult it would be to extract the graphics. I quickly found out that I needed a compression scheme that would be really efficient for the most common event: counting down a number of frames until the next change of a memory location is to be done. It realise I could probably write a whole book about this demo! Maybe I should? :)
As interesting as that would be, your time is probably better spent making the next awesome demo. :-) |
| |
Raistlin
Registered: Mar 2007
Posts: 578 |
Quoting bitbreakernot using 40 tracks, as we felt it is not as vanilla as we want to be
I can agree with this .. we could've done 40-track demos back in the day .. but we never did because we'd be making it more complicated for people to spread our demos... most people back then couldn't easily copy a 40-track disk.
We talked about 40-tracks for No Bounds .. but opted against it in the end.
Nowadays of course it's no problem for most to use 40 track disks .. most people just copy a D64 file onto a USB stick for their UII+ .. but, still, I agree with BB here that it makes the final product something other than vanilla. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11154 |
You can always provide an Installer (like Crossbow did).
Backdraw: you'll have to make a Slideshow
*ducks and runs* |
| |
Burglar
Registered: Dec 2004
Posts: 1051 |
Quote:The 100 games would not have fit if I hadn't brought bitfire to next level, by sharing sectors with the fileend and start of the upcoming file. This saved ~50 blocks, the zx0/salvador/dali packer did another good job in saving. IFFL Will Never Die!
(yes, this is not iffl, but still) |
| |
wacek
Registered: Nov 2007
Posts: 505 |
Quote: Great! Do you think you could find the link Wacek?
What you quoted was exactly what was posted on fb :) |
| |
Raistlin
Registered: Mar 2007
Posts: 578 |
Quote: Great! Do you think you could find the link Wacek?
https://www.facebook.com/groups/2503729026388481/permalink/6497..
Pex's answer:-
"I know how it was done. I rewrote the Vice emulator to dump not only the whole state of the emulator at a certain point in time as a snapshot, but then also every single read and write to memory with a timestamp. Recording a 5-second snippet of a game resulted in a 4MB file on disk. Then, I wrote a python script to extract, as Robert correctly stated, the colours on the screen, the chars on the screen, the charset, the sprites. Every game is 2 seconds, which means 100 frames of updates needs to be tracked, so I gather a list of every single memory location that needs to be changed to update the graphics, by fast forwarding the state of the C64 SID and VIC-II chips some 19000 clock cycles to get to the next frame. This leads to a 100 to 200 lists of memory locations with 100 values that needs to be written to replay the graphics and sound. I need to compress these, but more importantly, I need to decompress these lists in real-time with as few CPU cycles as possible, since I only have some 19000 clock cycles until the next frame needs to be shown. And then I added a sprite multiplexer to show the texts on top of that. It's easy when you look at one single part of this at a time, but really messy when you combine all needed parts - I need to load the data for the next game while running the current one, for instance."
So, yeah, as Wacek mentions, it's word for word what you posted above. Nothing sinister implied, if Pex sent you the above then he's either copy/pasted from his FB post, or he's copy/pasted -to- his FB post. |
| |
Pex Mahoney Tufvesson
Registered: Sep 2003
Posts: 50 |
The largest games on disk are Bombjack (3167 bytes), Giana Sisters and Revenge of the mutant camels, all larger than 3kB.
The most difficult game to get working was Thing on a spring, which contains a lot of animated graphics and I need to update 138 memory addresses continuously. So, during Lazy Jones, I need to load 2.5kB from disk and unpack to 12kB. I was forced to place Panic 64, one of the smallest games, directly after Thing on a spring. (636 bytes to load, 5kB to unpack).
Bitbreaker's optimization on his bitfire loader helped a lot and let me choose even the wildest part of 2 seconds in the games.
But, as Burglar states, I did not implement any support for large area scrolling, so the games could not be full screen scrollers.
Also, bitmap mode graphics would require too much space on disk, so I skipped those.
And, for games with builtin sprite multiplexers, I would need to choose the position on the screen where the most interesting sprite graphics were. I had to give up on Green Beret and one Commando scene due to this.
I do not support graphics mode switching or raster interrupts or bank switching, so the upper half of the Ghettoblaster screen had to be empty.
In some games I had to remove some of the rows of the graphics to save diskspace. Bitbreaker's Geos-part that ends the disk uses 7,5kB, so some static texts with score had to go. Also, the "Another visitor. Stay a while. Stay forever"-sample is actually loaded before the noisefader part starts on disk #1, and is kept in memory and used five minutes later in the demo. |
| |
Bansai
Registered: Feb 2023
Posts: 36 |
Quoting RaistlinPex's answer:-
This leads to a 100 to 200 lists of memory locations with 100 values that needs to be written to replay the graphics and sound. For those who deal with Verilog/VHDL simulator design, that's exactly how many modern sim dump formats store the staggering amounts of signal data generated in sim, yet make it compressible and quickly accessible for waveform viewer tools. As time goes on, I notice more and more parallels between 8-bit assembly language programming and logic design work. |
| |
Bitbreaker
Registered: Oct 2002
Posts: 501 |
Quoting Pex Mahoney Tufvesson
The largest games on disk are Bombjack (3167 bytes), Giana Sisters and Revenge of the mutant camels, all larger than 3kB...
In sum there's 807kb of data prior to being packed with dali. The packed data then pretty much fills up the whole disk. So also zx0 compression helps a lot and that would not have fit with the old Doynax/Bitnax or sectorwise compression mechanisms. |
| |
Krill
Registered: Apr 2002
Posts: 2856 |
Quoting BitbreakerIn sum there's 807kb of data prior to being packed with dali. The packed data then pretty much fills up the whole disk. So also zx0 compression helps a lot and that would not have fit with the old Doynax/Bitnax or sectorwise compression mechanisms. So, 100 tiny files totalling to 807 KB are compressed to about 170 KB on disk, without the block allocation overhead of 127 bytes per file on average.
Out of some interest, how much smaller is the packed data if you compress all 100 tiny files as a single blob? =)
(Obviously, i'm aiming at stream decrunching as a size optimisation opportunity.) |
| |
Bitbreaker
Registered: Oct 2002
Posts: 501 |
The 100 games pack to 149799 bytes, the raw data glued all together to a single blob pack to 145904 bytes.
How are you resolving large offsets of up to 15 bit with streaming the data? You still need old data to be there in RAM to be referenced?
Can provide the raw data if wanted. |
| |
Krill
Registered: Apr 2002
Posts: 2856 |
Quoting BitbreakerThe 100 games pack to 149799 bytes, the raw data glued all together to a single blob pack to 145904 bytes. Okay, that's about 16 blocks saved. Not much. =| But about one track on the inner rim. =D
Quoting BitbreakerHow are you resolving large offsets of up to 15 bit with streaming the data? You still need old data to be there in RAM to be referenced? You limit the offsets to however big the streaming ringbuffer is - backrefs never fall out of the sliding window. Have a look at Exomizer's exostreamdecr.s for an example. :)
For practical purposes in a demo, though, you'd just split up the blob to approximately equal-size packed chunks, then load into alternating slots.
It's not as fancy as true stream decrunching, though. |
| |
Bitbreaker
Registered: Oct 2002
Posts: 501 |
Sure, that sliding window thing also was implemented in doynax packer back then, it limits the maximum offset, depending on how big the buffer with sliding window will be. So you loose again some pack ratio by that, as soon as you go below 32kb. Worst case is a 256 byte sector buffer. And then again, you melt it down to two buffers to depack to alternatingly, if done right, one can at least reference one of the preceeding data as dict if it is within offset range. So overall it is mostly about convenience that is traded with memory footprint for a bigger buffer compared to just loading alternatingly, what is not much effort codingwise as well. Another advantage of loading alternatingly: You can directly load/depack to the right destination and have the initial charset ready without any additional copy action (and memory) from a buffer. |
| |
Krill
Registered: Apr 2002
Posts: 2856 |
Quoting BitbreakerAnother advantage of loading alternatingly: You can directly load/depack to the right destination and have the initial charset ready without any additional copy action (and memory) from a buffer. True stream decrunching doesn't make so much sense when you need to copy from the stream buffer.
If i understood Pex's approach right, the stream's poke-data could be read directly and applied, without any additional copying from the stream buffer. Initial charsets etc. would still need to be copied around, though, unless you format the depacked data just right so the charsets sit at the correct position. Very hacky, though. =)
But yeah, depending on how big or not so big the streaming buffer is, pack ratio suffers, eating into the gain of crunching the entire blob. |
| |
ws
Registered: Apr 2012
Posts: 235 |
Unpopular opinion:
1. i loved Next Level (and all that happened at X and that X happened). The 100 Games Part really was what i wanted to see on a c64 and all the other fx roxxored
2. but ...
is it really noteworthy to do stuff with superior systems and then boil it down to a common denominator platform? like the c64 and its restrictions are just a filter mechanism? just asking. just some doubts. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11154 |
Sounds like you have been living under a rock the past decade(s) |
| |
ws
Registered: Apr 2012
Posts: 235 |
Ok, i'm sorry i asked. Please excuse. Oof. ¯\_(ツ)_/¯ |
| |
Krill
Registered: Apr 2002
Posts: 2856 |
Quoting wsis it really noteworthy to do stuff with superior systems and then boil it down to a common denominator platform?
🤔 |
| |
ws
Registered: Apr 2012
Posts: 235 |
@ Krill - as i said i am sorry. Kein Grund nachzutreten. Its okay. Done. I lose. |
| |
Mixer
Registered: Apr 2008
Posts: 422 |
The old school programmer: "We choose to code demos and other things, not because they are easy, but because they are hard." |
| |
Bitbreaker
Registered: Oct 2002
Posts: 501 |
Quoting wsUnpopular opinion:
is it really noteworthy to do stuff with superior systems and then boil it down to a common denominator platform? like the c64 and its restrictions are just a filter mechanism? just asking. just some doubts.
That might be the perspective of some. I see it the other way round: How can i make an idea happen on a machine with the following restrictions and how do i exploit a machine totake advantage in that certain task. If you think it is all easy because there's superior machines that do most of the work, you haven't understood much. It still implies to do all kind of tools, to know all kind of techniques to achieve that goal. I am using more illegal opcodes and coding tricks than ever before on a 6510. Superior machines open up a wider range of possibilities and save time when all is set up, that doesn't however make it easier, but allows for even more complex and difficult stuff. I observe people love to question the way things are achieved if they can't keep track on their own.
So things appear to be easy, as the makers speak with ease about the process of creation, in fact it needs a huge knowledge as background to do so. Seeing how many already fail in building tools or having proper build-processes, it should be obvious that many hurdles and obstacles need to be taken to create such demos. And it is fun, just like climbing up mountains, although it is way more exhausting than a little stroll. Things evolve and that means changes, that is what scares people most, as they are scared to not keep up anymore. Understandable! But holding back for others to feel comfortable? No! |
| |
Krill
Registered: Apr 2002
Posts: 2856 |
Quoting BitbreakerIf you think it is all easy because there's superior machines that do most of the work It's just another of the numerous asymmetries in the computing world. =)
Division is more complex than multiplication.
Encoding is more complex than decoding.
Solving a problem is more complex than verifying the solution.
And so on.
Numbercrunching for a demo is more complex than executing it.
So better do this with more computing power at development time than at run-time.
This is also somewhat universal and a thing on modern platforms, too.
And it was a thing for C-64 development already back in the 1980's. =) |
| |
Raistlin
Registered: Mar 2007
Posts: 578 |
Yeah, with modern demos, there’s often a huge amount of work that goes into the data generation that is easy to overlook. A lot of the effects that I make involve writing 1000s of lines of C++ that does data analysis, compression, reordering and spewing out ASM and BIN files that are then “just” compiled and linked into a C64 prg. The actual handwritten ASM might be simple… but the amount of work, and the difficulty, is still very much there.
Performers’ demo at X was a real standout. A deserved winner for sure for me. And that side 4 on Censor’s demo, also, wow (that one’s hard for me to quantify because I just never coded such a synth/digi part before - but reading Algorithm’s analysis of exactly what’s going on made me realise that, yes, it’s something very special).
Story demos are a lot of work too, a hell of a lot goes into them. Just try coding up a single, simple bitmap animation and you soon realise - it’s a lot of hard work and effort. C64 isn’t PC, you can’t just plug in a HTML5 renderer and move some PNGs around the screen.
So, yeah, I think all deserve praise. Everyone who entered at X does. |
| |
Burglar
Registered: Dec 2004
Posts: 1051 |
Quoting RaistlinJust try coding up a single, simple bitmap animation and you soon realise - it’s a lot of hard work and effort. Actually, png2prg 1.2 supports animations out of the box :)
but I get your point and it is true, I just never let an opportunity slide to shamelessly plug png2prg :) |
