| |
Bacchus
Registered: Jan 2002
Posts: 156 |
Replacing games loader ...
OK, this on cracking but still highly related to coding.
Most games I fiddled with over the year call the loader using a parameter that was the index of the file. Using the same parameter as index for my IFFL or converting it to a two byte string starting the file name has worked well for me in most cases.
I am now facing two games that are not distributed yet (old, but no scene version is out) where there is a lot of data stored directly on the disk and the game then loads it using direct track and sector. Think of it like action adventures. The game loads strings or other really small things by loading T/S and then exctacting the needed part. It's hence not really "files" most of it and there are so many that a file per string is plainly not within reach.
I can think of a few approaches;
1) Keep it as data on disk. Allocate the sectors used and then store the files on the unallocated sectors. You can't compress it - it does take a full disk side any way you look at it. It does work, looks rather neat but cannot be counted as a firstie.
2) Make a big chunk of the data to a file and push it to a REU the first thing you do. The game become ever so much more playable and fast. And the file can be compressed efficiently. You do need a REU (or simply enable it in your emulator or Ultimate Cart) but it's also still not counted as a firstie.
3) Make a big file which you then need to scan as the original 256 byte sectors are now 254, so a sector that was a full page is by necessity spread over two sectors in a file based option. I guess you can also compress the sectors individually and think of the sectors as files in an IFFL. One IFFL file equals a sector. This is an ugly bitch but could be counted as a firstie.
Any other thought on this technical challenge? I must admit I am growing fond of the REU option, and the firstie restriction is the only thing that holds me back. The Tink games we just released had been perfect in REU version. Would have saved SO much work, loading would have been near instant and it would have been a release of two neat files.
Am I missing any options or can someone provide some lateral thinking, that opens up new options by finding approaches I have missed? |
|
| |
Maxlide
Registered: Apr 2003
Posts: 31 |
When the things you need to load are primary text than I would go for an IFFL firstie. RLE pack the text or convert the text from 8-bit text to 7-bit or 6-bit text and there you go. |
| |
Martin Piper
Registered: Nov 2007
Posts: 722 |
Compress each block, store all the data without gaps and have an index.
Then when you want a block, decompress it. |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1409 |
I quite like Martin's plan. You could do quite a specialised cruncher for single blocks, too, as offsets are never more than 256 bytes.
Alternately, break the data into tracks rather than blocks, and decrunch a track at a time for better ratio.
How does it know how many bytes to read each time? And can you extract all the possible start locations/chunks, or is that impossible to find without playing through every possible game response? |
| |
Martin Piper
Registered: Nov 2007
Posts: 722 |
A Huffman algorithm would work quite well. The tree can be optimised generally across all the data and stored in code. Then the blocks can be compressed with that tree. Resulting in consistently smaller blocks. |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1409 |
I'm always itching to see the data when I read posts like this, or at least a sample thereof. So many different potential strategies depending on the characteristics. |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
@maxlide: Compression is sort of second here. Key is the question of accessing 256 byte chunks.
Data can be text and gfx and the games extracts the stuff it needs from the sector it just loaded.
@martin: This is very much IFFL. If I think of sectors as files, then the IFFL strategy is only viable if the tables can fit in the RAM I can find for tables. If I use drive RAM, very few IFFL routines will support more than 128 files/sectors (scanner will generate track/sector/offset table). An using multiple IFFL files is a REAL nuance (you need RAM for the scanner and you need to do rescanning). So this is ONLY an option for cases where you can isolate less than 128 "files". But if so, it's for sure valid ...
@chris: well, merging the sectors to tracks means that when I want to load a sector in a higher sector of a track, then I need to start loading+depacking, and then just ditch the data. If compressed, I never know the offset. That is subject to the compression. Compression makes it so that you cannot compute the offset - you need to scan and store a table and you need RAM for the table. |
| |
MagerValp
Registered: Dec 2001
Posts: 1078 |
This is exactly the problem I faced with Ultima IV Remastered where there are 688 raw sectors spread out over three disk sides. Source and docs on GitHub: https://github.com/MagerValp/u4remastered/blob/master/docs/ULoa..
TL;DR: Each sector individually crunched and then packed into IFFL archives that are scanned on startup. Track/Sector/Offset saved under I/O. |
| |
Knight Rider
Registered: Mar 2005
Posts: 131 |
Did you consider an EasyFlash release ? Much wider target audience than REU
Red Storm Rising for example also uses track/sector loading too. This was simply changed to X/Y (tr/se) file base loading and then build as CRT image.
Open the .CRT in a hex editor (from $e1f), and you will even see this
01 00
01 01
01 02
01 03
01 04
01 05
01 06
|
| |
MagerValp
Registered: Dec 2001
Posts: 1078 |
Alternate strategy that was considered: Pack up four sectors into a 1024 byte file that is then crunched. When loading you need a temporary 1024 byte unpack buffer, but it reduces the number of "files" to a quarter. You also get slightly better compression ratio. |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1409 |
Quoting Bacchus
@chris: well, merging the sectors to tracks means that when I want to load a sector in a higher sector of a track, then I need to start loading+depacking, and then just ditch the data. If compressed, I never know the offset. That is subject to the compression. COmpression makes it so that you cannot compute the offset - you need to scan and store a table.
Oh I'm aware of that - but I've no idea of how much free space or free CPU you have. For some games decrunching a track into a buffer would be affordable, for others it would not. Then there are compromises, like the 1k buffer MagerValp suggested above.
-cjam |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
Thanks all! I REALLY appreciate your input.
@Knight Rider: EasyFlash or REU is sort of the same. It's a release that doesn't count as a firstie. We need a new point system that encourages the production of quality releases, also supporting alternative platforms that can support making the games more playable. Today we compete over firsties. First is one merit but I really want back the spirit of fighting over BEST version. There we could see a trend of people investing time in doing REU or EasyFlash versions, rather than shit like NTSC fixing which has a value to exactly noone!
@Magervalp - thanks Per. You have the full insight of what I want to accomplish. I do agree there are compromises you can do here. Very fruitful insights!
@Magervalp and @chris: Cluster sectors, to have fewer files - but trading that for a bit slower loading once you are after the last segments of a cluster. The room for tradeoffs is limited by the available RAM for tables. Per had plenty of space for tables in his Ultima. In most cases you don't really have that. I would say that in the normal case, you need to then aggregate enough to get few enough files to fit your IFFL restriction. Nut I do acknowledge that this is an important strategy for moving yourself in a position where the IFFL solution works.
The dilemma here is still why I make significant efforts to replace a working mechanism, just to obtain a result that can be file copied. The merit is to win the accolade of a firstie. That's a gain for me. You are making tradeoffs and make the game less playable as a cost for the file based structure. From the gamer's perspective, you are adding an inferior experience, while playing. Is the file based structure really something worth striving for, worth the cost both in effort for me doing it and for the gamer playing it at a user experience inferior to the original game?
My conclusion is still the same - the fight for firsties is counter productive. We need to add a fight for besties! |
| |
Danzig
Registered: Jun 2002
Posts: 440 |
Do it filecopyable or don't do it at all... |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
@Danzig - Who are we doing it for? For ourselves and the fun of it, for the first release points or for the people interested in playing it?
The focus on firsties and the associated rules undermine the strive for quality. |
| |
MagerValp
Registered: Dec 2001
Posts: 1078 |
If you like points hunting, digging up unreleased oldies, racing the boards, then the first release scene is for you. If you want to take your time and make the release the best it can be, it's not going to make much sense being in the first release race. Sometimes these paths intersect, sometimes they go in opposite directions. |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
@MagerValp: My conflicting interest is that I try to do both. Dig up the old stuff and making really good work on them, and still capture the points. Given that the points are only awarded for the firsties today, the rules for this have a reach for anyone wanting to be doing competitive work on games. This is where an alternative mechanism for point scoring (like Gamer's Guide once was) is much needed. |
| |
Martin Piper
Registered: Nov 2007
Posts: 722 |
@Bacchus I would not rely on pre-written IFFL routines. You'd need to find around <2K for the lookup table for all sectors. |
| |
Knight Rider
Registered: Mar 2005
Posts: 131 |
@Bacchus Regarding points : when I was in the "scene" in the 80's, I was too young to know about this. Now I am simply too old to care.
But this is my recommendation:
1) Make a .d64 with KERNAL loader and have files names tr-se to get your points.
in parallel
2) Make the best release (instant loading CRT/REU) and get satisfaction and appreciation from other old gits like us |
| |
Raistlin
Registered: Mar 2007
Posts: 680 |
Re: "Firsties" / "Besties".. why not release a title twice? If it's an old title that it's unlikely for someone to release while you're working away at it, wait and release both at the same time ... if it's a new title, race for a firstie and then race for a bestie right after?
G*P in the 80s/90s would often release a game twice .. cracking it once so that people can play the new game ASAP ... and releasing it later with cheats, improved load times etc etc. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11386 |
giving any useful answer really needs some hard numbers here... depending on the number of "files" and their size... the solution is quite obvious :) so what is the problem exactly that needs to be solved? |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
@Knight rider: That's how I did the Tink series. But that also only works up until 144 files (where the boot and docs file would take from that same pool). The Tink series loading was rather painful. Dir scanning to the last sector of track 18 and then load a file that was under a block. Painful!
@groepaz I do well understand that there are different scenarios. Under 128 files IFFLs and file loaders are OK, under 142 files IFFLS are sort of our unless you do funky stuff clustering and you can only do files. After 142 you need to do funky stuff.
The discussion gets really interesting only while the IFFL and standard file loaders are sort of out of scope.
For the sake of argument; let's assume five files of 20 blocks and then 400 "256 byte pages". That makes a total of 500 blocks. |
| |
Knight Rider
Registered: Mar 2005
Posts: 131 |
is .d81 out of scope for points ?? |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
Points are only awarded for stuff that runs on stock 64. |
| |
Tim
Account closed
Registered: Mar 2002
Posts: 467 |
Bacchus,
I'm not qualified to answer your question, but I do acknowledge the problem.. in some cases it's actually the reason why some releases are not done yet.
Good luck in Fairlight's attempt to combine points and quality, it is noticed, and it is respected.
Side note;
Having to make similar choices in our crew frequently, it is no secret that if you (or anyone for that matter) would start a quality rating system, it would certainly mean we shift even more resources in that direction. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11386 |
Quote:For the sake of argument; let's assume five files of 20 blocks and then 400 "256 byte pages". That makes a total of 500 blocks.
you are missing "how much memory is there to use for it".
eg: your example is pretty trivial if you have 800 bytes for an index table |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
You rarely have memory - you claim it ;)
Still, IMHO tables for over 256 is less trivial. Forget drive RAM. 400 files means 1200 bytes tables. Track, sector and offset. Times 400. That sort of memory is not common in any game. |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1409 |
…so then perhaps for each group of 4 source blocks, have the track/sector/offset of a record in the IFFL that itself contains four lightly compressed blocks, preceded by a header giving the compressed length of each of the four?
Then you only have 300 bytes of tables, 200 if you pad each group of four to a block boundary.
Of course, then you might have to read as many as three extra blocks, but they should all be on the same track most of the time.
(apologies if my terminology is off; I have zero experience in cracking, obviously) |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
If the compressed sectors fit on the same physical sector, the loading wouldn't be too delayed.
I agree this is a validated way forward. It's sort of a more detailed line of thoughts as per what was already provided above. |
| |
Mason
Registered: Dec 2001
Posts: 461 |
Well depending on the number of files on the tracks
You can find out how much room you got for buffer in memory - if it's $0800 and you can compress the files into $80 bytes then you load in chucks in $0800 - that would be 16 files
16 x 128 in the IFFL would make it possible to store 768 packed $80 files in $0800 chucks. To unpack use the method like old Level Crueler where you load the packed files into memory and depacked it afterwards |
| |
JackAsser
Registered: Jun 2002
Posts: 2014 |
If you have >256 files then you already reference the file table via 16-bit value in the code. Why is a table needed at all? Why not provide the t/s/off/size directly in the code when needed? |
| |
Martin Piper
Registered: Nov 2007
Posts: 722 |
Quote: If you have >256 files then you already reference the file table via 16-bit value in the code. Why is a table needed at all? Why not provide the t/s/off/size directly in the code when needed?
Depends on the level of the code hook I suppose. Conversion from the old track/sector to the new track/sector/pos table could change less in the calling code. |
| |
MagerValp
Registered: Dec 2001
Posts: 1078 |
Quoting Bacchus400 files means 1200 bytes tables. Track, sector and offset.
Strictly speaking you need less than 10 bits to store the track and sector, if you can spare a few bytes for a small div routine. Even so, that's 900 bytes for the tables, which is a lot.
I'm really liking ChristopherJam's idea of using 1k clusters and three bytes of offsets before the first compressed sector. You still need to read the full compressed cluster, but you don't need more than a single 256 byte buffer in RAM since you can skip the data you don't want. Performance wise I don't think there would be a large hit from using 1k clusters, since it's a relatively small time compared to drive seeking and decrunching the data. |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
Arhgh. Big reply didn't stick. Recreation of the core parts: :-(
@tim I hear you - our front end guy got snowed over and can't do it now. It's still our ambition to do it...
@jackasser
# Files vis direct T/S
If data is stored in T/S format, you have a number of advantages: You have the full 256 bytes and the data is where you left it. Using files, data "moves around". Where on the disk the data file ends up is different per disk depending in the order you copy the files. Add a little note to a disk before you copy your game then the T/S used is totally different than if you copy it directly to the disk without the note. Also, interleaving is different between drive kernals. ProfessionalDOS doesn't use 10. I also think Jiffy goes for less. Again - with files, you can make zero assumptions on where data is.
You hence need to scan the IFFL file and make tables, as there is no way to programatically tell where the stuff is. That luxury is reserved to the static T/S environment. You typically latch the scanner before the intro so it can do it's job before while you watch the intro.
# Available memory
Your loader + depacker takes memory. If you're lucky, it fits where the old loader was. Otherwise you need to find RAM in the game which is most often a real challenge.
In the ideal IFFL case, the loader and tables fit in drive ram. No need to fiddle with computer ram. Works fine for scenarios where the 2k of drive ram is good for tables, code, sector buffer and all else you need there. Typically you hit a brick wall at around 128 files depending on how compact the loader is. @MagerValp migrated the tables to the computer in his Uload. Works fine if you have the RAM in the computer.
It all boils down to tradeoffs, and my main question was around the scenario of 256+ files and quite restricted memory.
Clustering files gives you the benefit of smaller tables at the cost of slightly deteriorated average loading times (small cost!) |
| |
Perplex
Registered: Feb 2009
Posts: 255 |
When the files are stored on disk, tracks and sectors are determined by some kind of logic, not chosen randomly. If you implement this same logic in native code, all you need is table with sizes for each file, and you can calculate t/s and offset from that, right? |
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
No.
That "logic" is too easily influenced by file copiers, modified drive ROMs (speeders), etc. |
| |
Perplex
Registered: Feb 2009
Posts: 255 |
Won't those break your tables whether they are precalculated or not? |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1409 |
Quote: Won't those break your tables whether they are precalculated or not?
Not if you build them by scanning the file when the game first loads, as Bacchus mentioned above.
Interesting challenge, actually - read all the track & sector links in a file in as little time as possible and store a compact representation in drive ram that can be used for fast random seeking… Probably safe to assume there's usually only a new track every ~20 blocks. |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
So, scanning is needed.
Storing tables is needed.
And preserving memory is also needed. |
| |
Martin Piper
Registered: Nov 2007
Posts: 722 |
It's these little puzzles that keep programming fun. :) |
| |
Maxlide
Registered: Apr 2003
Posts: 31 |
Quote: It's these little puzzles that keep programming fun. :)
...or let you bite into your table :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11386 |
Quote:Strictly speaking you need less than 10 bits to store the track and sector, if you can spare a few bytes for a small div routine. Even so, that's 900 bytes for the tables, which is a lot.
when you are dealing with 400 chunks of 256 bytes you dont need to put the offset into the table (you can calculate it easily and fast) |
| |
Oswald
Registered: Apr 2002
Posts: 5094 |
Quote: ...or let you bite into your table :)
you will not enjoy it finally working, not having had eatan half of the table :) |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1409 |
Oh! The piece of furniture your computer sits on!
I was trying to parse Maxlide's comment as being something about code overwriting the offset table or something, and utterly failing to make sense of it. Thanks Oswald.
*stops eating table* |
| |
MagerValp
Registered: Dec 2001
Posts: 1078 |
Quote: Quote:Strictly speaking you need less than 10 bits to store the track and sector, if you can spare a few bytes for a small div routine. Even so, that's 900 bytes for the tables, which is a lot.
when you are dealing with 400 chunks of 256 bytes you dont need to put the offset into the table (you can calculate it easily and fast)
I'm assuming that each block is crunched individually, and thus the size varies.
Another idea that I don't think I've ever seen done is to keep the T/S/O in an on disk directory file. It would have to be written after scanning on first launch, but the game could launch without scanning after that. No RAM required for the tables, but just like with regular file loads you need to seek to the directory before each file loads. Probably not great if you have 400 of them... |
| |
Bacchus
Registered: Jan 2002
Posts: 156 |
@MagerValp: It has been done. I know for a fact - guilty as charged ... ;-)
We used that to swap out stuff in order to have the space to implement save in Herakles. |
| |
MagerValp
Registered: Dec 2001
Posts: 1078 |
Oh neat! |
| |
Jazzcat
Registered: Feb 2002
Posts: 1044 |
i would support and assist a gamers guide online for sure! |
