| |
Oswald
Registered: Apr 2002
Posts: 5086 |
koala otpimizing
Hi Everyone,
I know there's a c64 tool out there that helps with optimizing koala pictures for packing, but no idea what it is called. Anyone knows?:) Timanthes would do the job for me aswell with its nibble swapper tool, but is there a way to load/save a native c64 koala format picture with it ? *.prg doesnt works, what format does it expect to be .prg anyway ?:) |
|
| |
Ninja
Registered: Jan 2002
Posts: 411 |
On the way soon... |
| |
JackAsser
Registered: Jun 2002
Posts: 2014 |
@oswald: Normally when Lars have sent me graphics it's usually layouted like this (Koala) in the prg:
bitmap $2000-$3f3f
charmem $3f40-$4327
colormem $4328-
/Andreas
|
| |
CenTraX
Account closed
Registered: Jan 2002
Posts: 117 |
Quote: Hi Everyone,
I know there's a c64 tool out there that helps with optimizing koala pictures for packing, but no idea what it is called. Anyone knows?:) Timanthes would do the job for me aswell with its nibble swapper tool, but is there a way to load/save a native c64 koala format picture with it ? *.prg doesnt works, what format does it expect to be .prg anyway ?:)
Try: Advanced Art Studio Cleaner |
| |
WVL
Registered: Mar 2002
Posts: 896 |
hires or multicolor?
I've got a pretty nice tool to optimize Hires bitmaps (commandline tho).
BTW, Timanthes can also optimize by itself, there's an auto-option built in if you know where to look ;) |
| |
Mirage
Registered: Jan 2003
Posts: 113 |
Timanthes can't load .prg, it's one of the many flaws :)
I suggest you use WVL's little tool (or wait for ninja's, as he seems to have something aswell... it will probably use lots of illegal opcodes, though ;) |
| |
WVL
Registered: Mar 2002
Posts: 896 |
Quote: Timanthes can't load .prg, it's one of the many flaws :)
I suggest you use WVL's little tool (or wait for ninja's, as he seems to have something aswell... it will probably use lots of illegal opcodes, though ;)
but my little tool only works for hires, since I made it especially for Trans*Form.
I think it's a must for Timanthes to load .prg's, or at least be able to convert VICE-screenshots to restricted mode, without having to recolor all the colors. (just a new c64-colorramp option, to convert pixels to the nearest c64 color will be enough, me thinks)
I think I'm getting RSI just from doing all that recoloring.
Now that I think of it.. I also have a tool to optimize multicolor bitmaps :D It's part of the PDtool, to optimize the backgrounds of Pinball dreams :) |
| |
Compyx
Registered: Jan 2005
Posts: 631 |
@Mirage:
If you need something to load/save Koala pictures (and Amica Paint while you're at it) take a look at my gfxpack functions: gfxpack-20070308.tar.bz2
There's a function `read_file()` which will load a file and (re)allocate memory dynamically while loading. It also has the option to skip the load-address of a .prg file.
Not finished at all, many more file formats should be added. I'm actually writing this so hopefully Tao's cbmplugs can load/save Amica Paint and other packed files ;)
Ofcourse it's all written in ANSI-C, not C#, but I think it should be easy enough to write a wrapper for it.
|
| |
algorithm
Registered: May 2002
Posts: 705 |
to make the data more compressible but keeping the data structure intact, some type of routine can be used to swap col nybbles, 2bit mcol pixel pairs, etc until thedata compresses to a smaller size than usual
|
| |
Graham
Account closed
Registered: Dec 2002
Posts: 990 |
Try this one:
GFX Transformer 1.0
Just load & save a koala. Every image will be cleaned after loading. |
| |
Conrad
Registered: Nov 2006
Posts: 847 |
Hey neat! Never knew there was koala packer avaiable. I'll start using this too. :-) |
| |
Ninja
Registered: Jan 2002
Posts: 411 |
algorithm: What I mostly did is, rearrange the colors sorted by brightness. This is useful for fading, and often helps with compression. Still, if one is aiming for max. compression, a lot more analysis of the data is needed. A bitmap with lots of repeatings patterns should be created then, I guess. One of those interesting problems, which can easily grow pretty complex :D |
| |
Mace
Registered: May 2002
Posts: 1799 |
The thing with the way files are archived in CSDb is that it's virtually impossible to find a tool of your desire, without knowing the name.
@NinjaDRM: just sorting won't work, imho. You should also put equal nybbles of concurrent positions in the same place. Like, when you have colour 1,2,3 and 3,4,5 in neighbouring positions, you might want to have colour 3 in $d8xx, alhough this might not result in cascading brightness. (If you catch my drift) |
| |
WVL
Registered: Mar 2002
Posts: 896 |
My tool does exactly this.
first it checks what colors are _really used_ in the current char, and then checks if the same colors are used in the previous one, and rearranges to colors so the nybbles match
(if there's an unused color, it tries to fill the value up so it matches with the previous color! coolness! ie, char 0 color is $12, char 1 color is $x1 (x can be anything but isnt used), it rearranges char 1 to $12 too!)
However, this does not always result in a better packing.. since you're only looking at the colors, and not at the compressibility of the bitmap itself. Actually, i think you should combine the optimizer with the packer, and try out all color combinations for a char and see how well it packs.
so :
1)start with char 0
2)try all color permutations, and send this into the packer, select char with lowest # of bytes
3)go to the next char, and repeat.
However, even when this will be a lot better, it will still not give you optimal packing.. you'd have to consided all color-permutations of all chars at the same time. Horrifying.
I guess the best would be some kind of heuristic algorithm, deciding what is the best way to permutate the colors in a char.
|
| |
Ninja
Registered: Jan 2002
Posts: 411 |
Mace: As I wrote, the sorting was intended for fading mainly, and it just turns out that it often helps with compression, too. I think for real hardcore compression, it would be useful to rearrange the bitmap to have lots of repeating patterns and not just concentrate on RLE in the color data (cause of 8K vs. 1.5K). Or maybe to skip this bitmap+colordata-idea and use something abstract. Lots of possibilities here! |
| |
Ninja
Registered: Jan 2002
Posts: 411 |
WVL: It seems, we not only coded quite similar tools, we also came to quite similar conclusions :D |
| |
Mace
Registered: May 2002
Posts: 1799 |
Quote:Or maybe to skip this bitmap+colordata-idea and use something abstract. I bet there are a million ways to store the same picture in the memory and use some exotic routine to convert the information back to colourmaps and bitmap. But in the end you just want an optimized picture that has the original format. Then, there's not much else to do but sort nybbles and change the bitmap, I guess...
Hehe... we could do a contest: let's all make a file that holds the same picture and let's see who gets the smallest result after using the same cruncher :-)
No need to stick to a file-format: the only thing that counts is that it shows the original multicolour hires pic. |
| |
WVL
Registered: Mar 2002
Posts: 896 |
Quote: Quote:Or maybe to skip this bitmap+colordata-idea and use something abstract. I bet there are a million ways to store the same picture in the memory and use some exotic routine to convert the information back to colourmaps and bitmap. But in the end you just want an optimized picture that has the original format. Then, there's not much else to do but sort nybbles and change the bitmap, I guess...
Hehe... we could do a contest: let's all make a file that holds the same picture and let's see who gets the smallest result after using the same cruncher :-)
No need to stick to a file-format: the only thing that counts is that it shows the original multicolour hires pic.
hehe.. sounds like a contest I could fancy ;) |
| |
enthusi
Registered: May 2004
Posts: 677 |
just post a pic and the cruncher top be used (I'd prefer exomizer (latest version)) :) |
| |
Ninja
Registered: Jan 2002
Posts: 411 |
I'd suggest using at least 2 (better 3) quite different pictures and add up the sizes. Otherwise we have lots of finetuned algorithms for just 1 picture ;) Latest exo would be a good choice, I agree. |
| |
Mace
Registered: May 2002
Posts: 1799 |
Ok, let me suggest the first picture.
The title picture of Short Circuit:
(this is a screenshot from a demo, you should use the original picture, of course) |
| |
WVL
Registered: Mar 2002
Posts: 896 |
 |
| |
Ninja
Registered: Jan 2002
Posts: 411 |
Song of the Sunset
:) |
| |
Mace
Registered: May 2002
Posts: 1799 |
Great :-)
So, let's see who creates the smallest file that displays these three pics (press space to swap ;) ).
Rules:
1) It has to be one file containing all three pictures
2) ANY form of optimizing is allowed
3) Final file crunched with Exomizer
Shall we set a date on which the file must be ready? |
| |
WVL
Registered: Mar 2002
Posts: 896 |
I think it's better to come up with 3 exo-data files, that de-exo'd give the bitmap-data,colors and d800 color in the memory.
As simple as that.
Also, Im not sure I have time for this :) |
| |
Mace
Registered: May 2002
Posts: 1799 |
I don't have time either, so let's set the date for Xmas 2007 :-)
Also, just exo'd data isn't interesting.
For optimizing sake you might be right, but this doesn't leave room for exotic ways of storing a picture in the memory.
If this is the plan, I'm out, as it's no fun ;-)
I might just give it a go... some time.
First some other 'projects'. |
| |
algorithm
Registered: May 2002
Posts: 705 |
pre processing the file and then using custom compression would give optimum sults, but the lz based compresion isgood enough for most purposes |
| |
tlr
Registered: Sep 2003
Posts: 1787 |
I probably won't have time either, but why restrict it to exomizer?
Any runnable program that shows each of those 3 pictures should be allowed.
Some of those will be _slow_! ;)
Also I think it should be just one picture per binary.
|
| |
Mace
Registered: May 2002
Posts: 1799 |
@ TLR: be my guest ;-) |
| |
MagerValp
Registered: Dec 2001
Posts: 1074 |
Maybe it's time someone did arithmetic (de)coding on the 6502?
|
| |
Mace
Registered: May 2002
Posts: 1799 |
I always liked data (en)(de)coding, although it's not quite my field of expertice.
One of my experiments was plain Huffman encoding, but I quite when I had to find a way to store the binary 'tree' in such a way that the crunching actually produced a smaller file :)
Also, my knowledge of existing mathemetic algorithms is fairly non-existant. That's not handy, when doing stuff like this.
Just a brain fart: what experiments could you do with rotating data?
If you have data like:
.byte $00,$ff,$00,$ff,$00,$ff,$00,$ff, Rotating that 90 degrees would result in 8 bytes with the same value.
Easier to pack than a pattern... sometimes :) |
| |
WVL
Registered: Mar 2002
Posts: 896 |
another brainfart :
interleaving the bytes of the bitmap.
Instead of storing
byte 0,byte 1, byte 2, byte 3
you store
byte 0, byte 2, byte 1, byte 3, to try and undo the effect of pixelgrids.
Maybe shifting every odd byte 2 pixels would be even better. |
| |
Mace
Registered: May 2002
Posts: 1799 |
Since we're having a farting party anyway, some other thought:
When looking at the colour bytes, you have 'only' 15*14*13 = 2730 (which happens to be $AAA) combinations. The 16th colour is the background and each colour will only occur once in the colour triplet (= 3 colour nybbles).
Unused colours can be optimized to match a triplet that is already in use.
The bitmap can be optimized to create the largest amount of equal triplets.
The triplets can be sorted to see how many combinations are really used. These can then be represented by the shortest combination of bits.
[edit, afterfart]
Ooh, another one!
You can know how many nybbles of the triplets are used, if you have the bitmap. So when the bitmap says 'only one nybble in use', you can use the other bits of the triplet to store info for the next position. |
| |
algorithm
Registered: May 2002
Posts: 705 |
dynamic compression algorithms can be applied to specific areas of the bitmap. etc |
| |
Mace
Registered: May 2002
Posts: 1799 |
Hey, this is not fair!
If Algorithm is in, I'm out!
He can't be beaten by us mortals when it comes to graphics encoding. ;-) |
| |
Mace
Registered: May 2002
Posts: 1799 |
However, before we forget: the original question was how to optimize the content of a Koala picture for crunching.
This is still an interesting question. |
| |
tlr
Registered: Sep 2003
Posts: 1787 |
Quote: However, before we forget: the original question was how to optimize the content of a Koala picture for crunching.
This is still an interesting question.
I completely missed that. :) I just read the later posts. |
| |
algorithm
Registered: May 2002
Posts: 705 |
Its a whole new ball game if you wish to keep the file format intact. As mentioned before somewhere in this subject, swapping over color data between nybbles, etc or/and changing bit sequence order of twobit pixels so that there will be more identical sections - which in turn would allow a compressor such as exomizer to compress the data more tighter..
I still prefer the entire encoding of data and then custom compression - which would give tighter pack rates.. and its all about size.. |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1408 |
I thought this 'otpimizing' sounded like an entertaining challenge, so I've given it a go.
No rules seem to have been particularly agreed upon, but the one I considered most entertaining was attempting to get all three images in the smallest executable file possible, vis my 19141 byte entry here:
http://jaruth.com/c64/0704/3pics.prg
It deinterleaves the colour data before using a simple run length compression scheme (the nybbles have been chosen to maximise the run lengths), then applies LZ77 with a fixed huffman code for the bitmap. The third image in the corpus is pixel shuffled to improve the packing. (saves over 200 bytes on the compressed data, but then spends 91 on the deshuffle code). I'm not familiar with Exomiser, but this does beat pucrunch for this particular data set.
FWIW, I got Song of the Sunset down to 5436 bytes - I could shave off another six or seven bytes if I hard coded some decompression parameters, but that would be in breach of ninjadrm's anti-finetuning rule. I didn't consider the optional pixel shuffle to be an issue, as whether or not to use it is specified in the compressed data, not the decoder (kind of like PNG's alternate pixel interleaves).
http://jaruth.com/c64/0704/song.prg
|
| |
tlr
Registered: Sep 2003
Posts: 1787 |
Fine tuning is tricky to restrict. You compressor could just try a set of heuristics and choose the best one. Is that fine tuning?
Anyway, this got me interested! :)
Could you post the three source pictures you used so we don't work with different ripps?
|
| |
ChristopherJam
Registered: Aug 2004
Posts: 1408 |
Good point about the tuning - it's a fairly arbitrary restriction, and difficult to quantify. TBH I was also thinking it was time I downed tools and uploaded something, as I have other work I should be doing :)
Perhaps 'smallest executable that displays the three images in sequence, with space to advance from first to second and from second to third' should be the only rule.
Glad to see the interest, anyway! I've uploaded three PNGs and the common palette used as http://jaruth.com/c64/0704/srcs.tar.gz
but it may be simpler for you to just rip the data from the 3pics.prg. Bitmap is always displayed at $6000, with the video matrix at $5c00.
I just noticed that I failed to acquire the original shortcircuit loading screen - I vote we continue with the intro version at this point.
Where did WVL's contribution come from?
Happy coding! |
| |
tlr
Registered: Sep 2003
Posts: 1787 |
It's here: Foris - Untitled
The others: Short Circuit & Song of the Sunset
I ripped the short circuit picture from the original game and the others from the releases. Available as prg's: test_corpus-20070401.zip
Bitmap: $2000-$3f3f
Screen: $3f40-$4327
Color: $4327-$470f
$d021: $4710
|
| |
WVL
Registered: Mar 2002
Posts: 896 |
Christopher : is your pixelshuffler the same kind like the brainfart i had to shift every 2nd byte by pixels to deinterleave? |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1408 |
@tlr - Thanks for the pointers, and the c64 friendly corpus :)
@WVL - similar.
There are two levels of interleaving; the first takes each pair of bytes in the source and swaps the low nibble of the first with the high nibble of the second, so each byte then represents a 2x2 pixel block. The second shuffle places every second byte into a group of four, that hence represent a 2x8 pixel group (half char)
*rereads your comment*
"byte 0, byte 2, byte 1, byte 3, to try and undo the effect of pixelgrids."
Yup, that's pretty much my step two (though I take bytes 0,2,4,6,1,3,5,7), so a somewhat different effect given that I'm operating on already rearranged data. I was mostly attempting to increase the locality of the pixels, and experimented with a couple of different shuffles to see what combination made the most compressible image. Sadly, the first two pics both get slightly larger (around 30-40 bytes) if they are shuffled. |
| |
wil
Registered: Jan 2019
Posts: 51 |
Inspired by this thread, I made a tool that tries out different variants of rearranging the pixels in a Koala pic. In addition, it removes color information where less than 3 forground colors are used in a 8x8 area (or 8x4 if we count fat pixels) and sorts the colors accroding to a palette. All these features together help in compressing the Koala pic without changing the appearance.
The rearrranging requires a different display routine, since the bytes need to be unshuffled before displaying.
For packing I used exomizer.
All in all I got a filesize of 19045 byte including load address, decompression routine, unshuffling routine and display routine. I'm quite sure there is still plenty of room to optimze it.
The tool (it can be also used to optimize Koala pics without changing the format) and the packed example of the 3 file are here:
Pixelshuffler V0.4 |
| |
Zyron
Registered: Jan 2002
Posts: 2381 |
I normally use Pico v1.3. How good is that compared to the other options mentioned in this thread? |
| |
wil
Registered: Jan 2019
Posts: 51 |
Quote: I normally use Pico v1.3. How good is that compared to the other options mentioned in this thread?
I tried out Pico, it looks like a great tool, but what I understood is that you have to make some decisions to make it sorting? I only tried out the kill color function (CBM+K) which didn't give a good result yet. |
| |
soci
Registered: Sep 2003
Posts: 479 |
Compress all 3 together as I could easily get 18580 bytes without any tricks:
https://singularcrew.hu/temp/3koala-soci-singular.prg |
| |
wil
Registered: Jan 2019
Posts: 51 |
Quote: Compress all 3 together as I could easily get 18580 bytes without any tricks:
https://singularcrew.hu/temp/3koala-soci-singular.prg
Awesome! I saw that you had optimized Koalas in your solution - which program did you use for Koala cleaning/optimization? |
| |
soci
Registered: Sep 2003
Posts: 479 |
That must be a side effect as I was lazy to rip the pictures and converted from screenshots instead. |
| |
wil
Registered: Jan 2019
Posts: 51 |
Quote: That must be a side effect as I was lazy to rip the pictures and converted from screenshots instead.
A very positive side effect :-) - the resulting version is well suited for packing. What software or cartridge did you use to rip it? |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
Here's an easy one png2prg 0.1. converts 320x200 and vice screenshots to koala. It should optimize a bit automatically, but your miles may vary..
ps: only works if the screenshot contains a multicolor bitmap |
| |
wil
Registered: Jan 2019
Posts: 51 |
Quote: Here's an easy one png2prg 0.1. converts 320x200 and vice screenshots to koala. It should optimize a bit automatically, but your miles may vary..
ps: only works if the screenshot contains a multicolor bitmap
Your png2prg is a great tool! I tested it with PNG version of the three reference images and it got everything right, including that the background in the "Short Circuit N0.5 alive" pic is white.
It seemd, however, that the tool used by Soci was a different one, since the result is less compressable than Soci's version, so I would be still interested in what he used.
Anyway, I bookmarked your png2prg for future use. Also, the golang code looks very tidy - gotta learn this language one day! |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
Quoting wilIt seemd, however, that the tool used by Soci was a different one, since the result is less compressable than Soci's version, so I would be still interested in what he used.
so am I, it could be interesting to see what method it uses, so I can build it into png2prg as well.
so Soci, spill the beans :) |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
Time to un-necro this thread again. I just released png2prg 0.8 which should be quite a bit better on optimizing images, especially compared to png2prg 0.1.
The recently released SPOT - Sparta's Picture Optimizing Tool V1.1 will probably give you the best size-optimized result, while my tool focuses on normalization, ease-of-use for demo-coders and supporting all standard graphics modes. |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1408 |
I got the three images in the corpus along with a slideshow down to 17307 bytes \o/
It's a bit slow to decrunch though - may as well be loading off tape..
Compressed data sizes for the three images:
untitled (floris) - 6755 bytes
song_of_the_sunset - 4594 bytes
short_circuit - 5624 bytes Tinned Koalas
Mostly worked on this in November/December 2023, just didn't get around to releasing until today. It does build on the ANS work for bitpickler of course. |
| |
Jetboy
Registered: Jul 2006
Posts: 292 |
Wouldn't it compress better if the order of bytes was changed? We were doing some research on it back in the days, and storing data in columns was usually giving better compression. For pictures with dithering, storing first odd bytes, then even bytes was giving even better results. |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
I always get best results with pico 1.3 and my brain. Check the short circuit loading pic on the n0s release, it's hand optimized as well with pico and see if it's any better. |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
+-------------+-------------+-------------+
| spot+dali |png2prg+dali | koalacanner |
+-------------+-------------+-------------+
| 7349 | 7546 | 6755 | (Untitled/Floris)
| 5155 | 5464 | 4594 | (Song of the Sunset/Mermaid)
| 5986 | 6155 | 5624 | (Short Circuit/Karen Davies)
+-------------+-------------+-------------+
I think this stat belongs in this thread :)
Using SPOT 1.2, png2prg 1.6 and Tinned Koalas.
<cjam> So, spot-style 'optimizing' for png2prg 1.7? :D
I am not sure, for me cleanliness of the koala is the goal. Also, the better result may be because spot combines hi/lo nibbles in colorram, saving 500 bytes. I have no interest in that, I want to output standard format always. But, I could add a cli-flag for it...
@fungus, can you extract that koala from short circuit and pack with dali? would be fun if ur handiwork beats png2prg or even spot :) |
| |
Krill
Registered: Apr 2002
Posts: 2969 |
Quoting ChristopherJamI got the three images in the corpus along with a slideshow down to 17307 bytes \o/
It's a bit slow to decrunch though - may as well be loading off tape.. This sort of flips the table on the entire premise of optimising C-64 multicolour bitmaps for LZ-style compression.
If it only wasn't THAT slow to decode. =) |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
pico 1.3+dali = 5980 |
| |
soci
Registered: Sep 2003
Posts: 479 |
Packed what I did in the past with dali now and it's 7342, 5156 and 5984 respectively. Almost spot on. |
| |
Sparta
Registered: Feb 2017
Posts: 49 |
Wow, fantastic achievement CJam! Would be interesting to see how differently “otpimized” versions of the same bitmap compress with koalacanner.
Quoting BurglarI am not sure, for me cleanliness of the koala is the goal. Also, the better result may be because spot combines hi/lo nibbles in colorram, saving 500 bytes. I have no interest in that, I want to output standard format always. But, I could add a cli-flag for it...
SPOT doesn’t change the cleanliness of the koala and outputs standard koala but it can also create other formats such as the combined color RAM nibbles if that fits one’s needs better. :)
The goal with SPOT is to get as close to hand optimization as possible in a fraction of the time needed for the latter and in a way that can be added to a democoder’s toolchain. |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
Quoting SpartaSPOT doesn’t change the cleanliness of the koala and outputs standard koala but it can also create other formats such as the combined color RAM nibbles if that fits one’s needs better. :)
Yea, just tested myself as well and indeed, I was just hoping combined nibbles would be the main reason why I'm trailing behind :)
Quoting Funguspico 1.3+dali = 5980
Fungus Handiwork Wins! \o/ |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1408 |
Quoting KrillThis sort of flips the table on the entire premise of optimising C-64 multicolour bitmaps for LZ-style compression.
Yes, it's inspired by the discussion, rather than a direct response to the question. I got a bit distracted :)
Quote:If it only wasn't THAT slow to decode. =)
Yes, it's a bit impractical for 99% of use cases.
Quoting SpartaWow, fantastic achievement CJam! Would be interesting to see how differently “otpimized” versions of the same bitmap compress with koalacanner.
Cheers! FWIW at the moment the first phase of compression just sorts the non-background colours in numerical order and assigns them to bitpairs 1 to n. I could try and allow preserving of input bitpairs, but it would still need to be constrained to using the first m values, where m is the number of distinct colours - the second phase only outputs as many colour nybbles per char as are actually used. Oh, and it'd be a moderate amount of work at this stage too - the current implementation just uses a library call to load a koala into a 160x200 array of four bit values, so it doesn't currently have access to the original bitpairs..
I do need to see if it compresses any better if I sort the colours in luminance order instead - that might be better at dealing with images that use similar shading patterns with different local palettes in different parts of the image.
Nice work on Spot, btw - it actually answers Oswald's original question, and it's the second best option after putting a Fungus in every home :) |
| |
Oswald
Registered: Apr 2002
Posts: 5086 |
I'm coming here for the 4th time, checked CJ's release also a few times, am I the only one not finding the info on what is the compression scheme in there ? :) |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1408 |
I just left a comment on the release with a rough outline. |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
Quote: Quoting SpartaSPOT doesn’t change the cleanliness of the koala and outputs standard koala but it can also create other formats such as the combined color RAM nibbles if that fits one’s needs better. :)
Yea, just tested myself as well and indeed, I was just hoping combined nibbles would be the main reason why I'm trailing behind :)
Quoting Funguspico 1.3+dali = 5980
Fungus Handiwork Wins! \o/
Actually that one was S!R handywork. |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1408 |
Quote:the second best option after putting a FungusS!R in every home |
| |
Oswald
Registered: Apr 2002
Posts: 5086 |
Quote: I just left a comment on the release with a rough outline.
thanks:) |
| |
Sparta
Registered: Feb 2017
Posts: 49 |
SPOT 1.3 WIP + Dali:
Untitled/Floris: 7343
Song of the Sunset/Mermaid: 5145
Short Circuit/Karen Davies: 5974
The total gain is 28 bytes. Not a lot, but at least it beats S!R's handywork. ;P |
| |
ThunderBlade
Registered: Jan 2002
Posts: 77 |
Can you add some info on what SPOT actually does to achieve its great results? Or is there a general article about C64 multicolor optimizing? |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
I don't know of any article, but my work flow is to remove unused bits and colors. Then sort most used colors into the color memory and then try to get the rest of the colors into the other map and then fill the unused with colors to create the longest runs I can.
It might be better to optimize for the longest bitpair runs, but I don't know how you would go about such a thing because of how the bitmap is laid out, perhaps that is what cjam is doing.
People often forget to and the color memory with #$0f. |
| |
Sparta
Registered: Feb 2017
Posts: 49 |
Assuming that people typically use LZ-based compression (except CJam, of course), Koala optimization boils down to the bin packing problem. SPOT rearranges the colors using something that is probably best described as a best-fit-decreasing algorithm. I have a few more optimization steps but as it turns out they still need some fine tuning as they only work in select cases. E.g. I got Mermaid’s pic down to 5137 bytes with Dali but the other two got worse. |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
I suppose you could do some kind of statistical analysis of equal sequences over equal bytes of a given length and try to optimize for that. I would think repeating bytes of 8 would give better results than runs in the color map, if that's possible to do. |
| |
Sparta
Registered: Feb 2017
Posts: 49 |
SPOT 1.3
+--------------+--------------+
| spot1.3+dali | spot1.2+dali |
+--------------+--------------+
| 7332 | 7349 | (Untitled/Floris)
| 5136 | 5155 | (Song of the Sunset/Mermaid)
| 5968 | 5986 | (Short Circuit/Karen Davies)
+--------------+--------------+ |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
png2prg 1.7 dev version
+---------+--------+----------+------------+--------+
| spot1.3 | p2p1.7 | p2p1.7bf | p2p1.7best | p2p1.6 |
+---------+--------+----------+------------+--------+
| 7332 | 7373 | 7325 | bf | 7546 | (Untitled/Floris)
| 5136 | 5246 | 5206 | 5194 | 5464 | (Song of the Sunset/Mermaid)
| 5968 | 5983 | 5988 | 5983 | 6155 | (Short Circuit/Karen Davies)
| 3618 | 3691 | 3591 | bf | 3830 | (Portait L+D/Sander)
| 5094 | 5125 | 5109 | bf | 5320 | (Weee/Mermaid)
| 7497 | 7505 | 7475 | bf | 7612 | (Deadlock/Robin Levy)
| 8068 | 8130 | 8107 | 8087 | 8227 | (Room with a view/Veto)
+---------+--------+----------+------------+--------+
- all resulting koalas are packed with dali
- p2p1.7: default png2prg result w/o options
- p2p1.7bf: -brute-force mode
- p2p1.7best: hand-picked -bitpair-colors
- p2p1.6: default png2prg 1.6 result w/o options
NB: the ones where I beat spot were hard to find ;) |
| |
Sparta
Registered: Feb 2017
Posts: 49 |
Nice Burglar! :) I am looking forward to finding out more about your -brute-force mode (and the hand-picked -bitpair-colors). |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
Quoting SpartaNice Burglar! :) I am looking forward to finding out more about your -brute-force mode (and the hand-picked -bitpair-colors).
cheers :) it's pretty simple:
- iterate over all combinations of the 8 most used colors
- use them as forced/preferred bitpair colors
- crunch with tscrunch
- sort by size
The hand-picking comes from tscrunch being optimized for speed, not size. dali will crunch some bpc combinations better than tscrunch crunches them, so shortest can mean some other combination for either cruncher.
In -verbose mode I print the 10 best combinations based on tscrunched size, so I just try a few of those.
bruteforce code is here: https://github.com/staD020/png2prg/blob/master/bruteforce.go |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
I just released png2prg 1.8, now includes -brute-force mode to often beat SPOT 1.3 in pack ratio :)
+---------+--------+----------+------------+--------+
| spot1.3 | p2p1.8 | p2p1.8bf | p2p1.8best | p2p1.6 |
+---------+--------+----------+------------+--------+
| 7332 | 7372 | 7332 | 7324 | 7546 | Untitled/Floris
| 5136 | 5190 | 5149 | bf | 5464 | Song of the Sunset/Mermaid
| 5968 | 5998 | 5963 | bf | 6155 | Short Circuit/Karen Davies
| 3618 | 3647 | 3616 | 3589 | 3830 | Portrait L+D/Sander
| 5094 | 5080 | 5083 | 5078 | 5320 | Weee/Mermaid
| 7497 | 7471 | 7458 | bf | 7612 | Deadlock/Robin Levy
| 8068 | 8097 | 8046 | 8038 | 8227 | Room with a view/Veto
| 7445 | 7490 | 7432 | bf | 7582 | Vangelis/Talent
| 6759 | 6739 | 6737 | bf | 6963 | Temple of Tears/Hend
| 7859 | 7848 | 7839 | 7821 | 7998 | Thanos/JonEgg
| 4859 | 4849 | 4782 | bf | 4983 | Solar-Sonar/Leon
| 5640 | 5671 | 5613 | bf | 5869 | Cisco Heat/Alan Grier
| 6243 | 6286 | 6228 | bf | 6430 | Daylight/Sulevi
| 2850 | 2884 | 2848 | bf | 3092 | Yie Ar Kung Fu/Steve Wahid
| 6727 | 6721 | 6730 | 6711 | 6901 | Lee/The Sarge
| 7837 | 7828 | 7798 | bf | 7960 | Parrot/Mirage
| 4559 | 4536 | 4494 | bf | 4821 | Dragon's Lair
| 4275 | 4324 | 4292 | 4284 | 4519 | Scorpion/SIR'88
| 5562 | 5558 | 5506 | bf | 5668 | Hatching/Joe
+---------+--------+----------+------------+--------+
| 113328 | 113589 | 112946 | 112853 | 116940 | Total
+---------+--------+----------+------------+--------+
- all resulting koalas are packed with dali
- p2p1.8: default png2prg result w/o options
- p2p1.8bf: -brute-force mode
- p2p1.8best: hand-picked -bitpair-colors, or bruteforced with -npcc and/or -nbc flags
- p2p1.6: default png2prg 1.6 result w/o options |
| |
Sparta
Registered: Feb 2017
Posts: 49 |
Nice improvements Burglar! :)
SPOT 1.4 WIP with the same 19 pics and dali: 112867 bytes
No brute force or hand picking ;) |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
Hey Sparta can you change .scr to .scn because it's a reserved filename in windows.
Burg can you add a split output mode for bitmaps, both MCM and HIRES.
Thank you both, saving me countless hours of manual optimization. |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
Quoting FungusBurg can you add a split output mode for bitmaps, both MCM and HIRES. I'm not sure I want to support alternate outputs. especially when its a oneliner in a normal shell:
dd skip=2 count=8000 if=foo.prg of=foo.bin bs=1
dd skip=8002 count=1000 if=foo.prg of=foo.scn bs=1
dd skip=9002 count=1000 if=foo.prg of=foo.col bs=1 |
| |
Burglar
Registered: Dec 2004
Posts: 1087 |
Quoting SpartaSPOT 1.4 WIP with the same 19 pics and dali: 112867 bytes damn, but dont worry, I still have some tricks up my sleeve :)
looking forward to running a new benchmark myself :) |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
I dealt with it in 64tass, still koala format is not very useful :) |
| |
Jetboy
Registered: Jul 2006
Posts: 292 |
Quote: I dealt with it in 64tass, still koala format is not very useful :)
I find it useful enough :) But seriously, how is koala format not useful for you? |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
Waste of memory to have to move things around into usable locations. |
| |
Raistlin
Registered: Mar 2007
Posts: 657 |
Quote: Waste of memory to have to move things around into usable locations.
When the Koala data is linked, it should be done in 2 or more chunks so that this moving/copying data isn’t needed. |
| |
Fungus
Registered: Sep 2002
Posts: 680 |
Yeah I tend to put the color map at $0800, the color memory at $0c00, the bitmap at $2000 and the code at $1000 so all I have to copy is the color memory and the maximum space is available for linking after it. |
| |
ChristopherJam
Registered: Aug 2004
Posts: 1408 |
Again, that's a thing you do with your assembler or linker, and it's pretty independent of the container format that editors or converters spit the image out in.
Much less directory clutter to have a single file than two or three, and it's no more work in your assembly source to have three lines that grab chunks of one file than three lines that grab three separate files.
Easier to ensure the color and screen attribs are from the same version as the bitmap, too - can't accidentally update one or two of those without the rest. |
| |
Monte Carlos
Registered: Jun 2004
Posts: 358 |
Removing unnecessary information from c64 images is one thing, the memory layout for placing bitmap, screen and colorram the other thing. This thread is mixing up both and it is difficult to follow who talks about what. |
