| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
Track Times in SID files?
I've always wondered this, but why doesn't any of the SID file formats support a track time.
A couple of extra bytes per subtune per SID wouldn't be a huge loss, and it would mean any individual player doesn't need access to the 3MB file from HVSC to correctly know when to move on to the next track.
Given the database exists, it would also be pretty easy to write a program to go through all the files in HVSC, check if they match whichever SID file format supports the tracks times, and updates them with the tune lengths from the text file.
Does that not seem a good idea? |
|
| |
chatGPZ
Registered: Dec 2001 Posts: 11350 |
too easy =) |
| |
SIDWAVE Account closed
Registered: Apr 2002 Posts: 2238 |
lack of coder who spends time to make old format into modern.. |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
I don't see any major gain in having the time of every subtunes inside the sid, instead of having them in a separate file.
If a sidplayer doesn't support yet the sldb (*) it won't support for sure the new format in future. Those sidplayers that already work with the sldb txt, what actual gain would they have by ALSO being able to read that info from inside the sid?
Why not adding the STIL and BUGLIST infos too at this point?
Implementing extra information would mean to use a variable lenght header, this needs to write a whole set of new tools (our toolchain consists of 7+1 programs but all the programs are tailored for sid v2)
SIDedit.exe : sid header editor + disassembler, made in perl2exe,
unmaintained. I personally don't use it but others find it handy.
siddiag.exe : diagnostic of header/code errors, memory usage map, trims
unused memory at end of sid file
sidclean.exe : newer cleaner also trims at start if needed. still in beta but
will replace siddiag and siplay2/hacked
sidknown.exe : identifies duplicates by actually playing the sid and
generating md5 hashes from the sid usage
sidusage.exe : detailed usage checker, generates a map read by siderror
siderror.exe : reads maps generated by sidusage and reports badreads/bad
execution etc
sldetect.exe : songleghts.txt generator, actually plays the sids to detect
when they loop or stop emitting sounds to determine the
lenghts
sldb_merge.exe : accessory to sldetect.exe, to merge partial outputs
eliminating dupes (so sldetect could be run on updates instead
of the whole hvsc)
siddump.exe : hacked to also dump the executed disassembly code mostly used
to compare song output of similar/equal songs but also to spot
execution errors
who will maintain, or rewrite where needed, the new tools?
Anyway a new format will be announced shortly, I hope.
We're about to add a PSID V3 supporting 2 new fields for 2SIDs that won't leave much more space in header.
In this moment there are 60 2SIDS that needed manual hexediting to the new fields, but there aren't many other around for sure, so a specific tool to edit the extra fields is not yet needed.
(*) I can think only of sidplay2/w which I use, infact I don't have any use of the sldb, I decide what to play and how much, not counting that 90% of the times i'm not listening to sids from the actual HVSC dir but from a work dir =) |
| |
chatGPZ
Registered: Dec 2001 Posts: 11350 |
"but there aren't many other around for sure"
until someone makes sids from all those whacky stereoplayer tunes =) |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
if you mean CGSC stereo tunes, it won't happen. |
| |
Radiant
Registered: Sep 2004 Posts: 639 |
Quoting iAN CooGI don't see any major gain in having the time of every subtunes inside the sid, instead of having them in a separate file. If a sidplayer doesn't support yet the sldb (*) it won't support for sure the new format in future. Those sidplayers that already work with the sldb txt, what actual gain would they have by ALSO being able to read that info from inside the sid?
From what I've heard implementing support for Songlengths.txt is really tedious and requires lots of trial and error, though. My guess is more players would support song lengths if it became less convoluted to implement. (I know Sasq fought with getting songlength support into Droidsound for the better part of a day and almost gave up, and he's by no means a bad coder.) |
| |
Frantic
Registered: Mar 2003 Posts: 1646 |
I agree with the initial poster...
I think duration time is a pretty central feature of some (not all) tunes, and as such it should have a pretty obvious place in the sid header. It should not be specified in seconds or such, but in number of frames since almost every sid tune out there is sync'ed to the screen.
Three cases:
- DURATION xxxx frames
- NO END
- NOT YET SPECIFIED (which of the two above)
Fix it! ;) |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
Quote: I don't see any major gain in having the time of every subtunes inside the sid, instead of having them in a separate file.
If a sidplayer doesn't support yet the sldb (*) it won't support for sure the new format in future. Those sidplayers that already work with the sldb txt, what actual gain would they have by ALSO being able to read that info from inside the sid?
Why not adding the STIL and BUGLIST infos too at this point?
Implementing extra information would mean to use a variable lenght header, this needs to write a whole set of new tools (our toolchain consists of 7+1 programs but all the programs are tailored for sid v2)
SIDedit.exe : sid header editor + disassembler, made in perl2exe,
unmaintained. I personally don't use it but others find it handy.
siddiag.exe : diagnostic of header/code errors, memory usage map, trims
unused memory at end of sid file
sidclean.exe : newer cleaner also trims at start if needed. still in beta but
will replace siddiag and siplay2/hacked
sidknown.exe : identifies duplicates by actually playing the sid and
generating md5 hashes from the sid usage
sidusage.exe : detailed usage checker, generates a map read by siderror
siderror.exe : reads maps generated by sidusage and reports badreads/bad
execution etc
sldetect.exe : songleghts.txt generator, actually plays the sids to detect
when they loop or stop emitting sounds to determine the
lenghts
sldb_merge.exe : accessory to sldetect.exe, to merge partial outputs
eliminating dupes (so sldetect could be run on updates instead
of the whole hvsc)
siddump.exe : hacked to also dump the executed disassembly code mostly used
to compare song output of similar/equal songs but also to spot
execution errors
who will maintain, or rewrite where needed, the new tools?
Anyway a new format will be announced shortly, I hope.
We're about to add a PSID V3 supporting 2 new fields for 2SIDs that won't leave much more space in header.
In this moment there are 60 2SIDS that needed manual hexediting to the new fields, but there aren't many other around for sure, so a specific tool to edit the extra fields is not yet needed.
(*) I can think only of sidplay2/w which I use, infact I don't have any use of the sldb, I decide what to play and how much, not counting that 90% of the times i'm not listening to sids from the actual HVSC dir but from a work dir =)
What about devices like SIDStick?
http://www.gadgetgangster.com/find-a-project/56?projectnum=236
Imagine getting a portable media player that you can just copy the SID files onto, without needing the DB too
Okay, you could argue you could put it on the SD card too, but that would probably require many more cycles to seek through the DB each time, than reading the necessary value straight out of the SID you're playing
Regarding STIL and BUGLIST, they don't really affect how a track should be played (well, BUGLIST might if it was converted to some kind of "patch" system to SIDs, but then why not fix the SIDs anyway) |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
"why not fix the SIDs anyway"
yeah sure, pick one from the buglist and fix it. Send anytime!
hint: if it's in the buglist means that no one so far was able to fix it, or it's even impossible due to missing/broken data. Good luck. |
| |
chatGPZ
Registered: Dec 2001 Posts: 11350 |
Quote:From what I've heard implementing support for Songlengths.txt is really tedious and requires lots of trial and error,
my favourite WTF is, that it contains the pointless md5 (wth should one do with this?) as regular data.... and the filename, which is what you really want, is only there as a comment, so everyone who uses the db ignores the md5, and then adds whacky code to seek backwards to last comment and extract the filename. awesome =) the times beeing in min:sec format and not seconds only as you most likely need in your app then is only a small annoyance. somehow looks as whoever invented this format didnt code a thing before =P |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
Quote: "why not fix the SIDs anyway"
yeah sure, pick one from the buglist and fix it. Send anytime!
hint: if it's in the buglist means that no one so far was able to fix it, or it's even impossible due to missing/broken data. Good luck.
Okay, hands up, I'll admit, I only glanced at buglist and didn't read it properly. :-)
But that goes back to my point, tune times have a positive effect on being in the file format. STIL and BUGLIST don't because the player can do nothing with that information. Sure, maybe make them optional in the same way MP3s can have a jpg of the album cover, but not required.
Could you not use
Quote:+7A WORD reserved
in the SID v2 header (according to the file format document in HVSC) to point to a "Optional Data Header", which can then be located anywhere in the file (so not to throw off any players that assume a fixed header size, or just don't want to use it). Point this to a "floating" header record would mean that it wouldn't *break* any of your tools
What languages are the tools written in? |
| |
Steppe
Registered: Jan 2002 Posts: 1510 |
There has been a discussion about this in the HVSC team numerous times. I'd favor putting the playtime into the sid header, but the number of reserved/unused bytes in the header always prevented this (without breaking anything). The idea with the floating header record hasn't been brought up before and to me as a non coder sounds interesting. |
| |
Hermit
Registered: May 2008 Posts: 208 |
A primitive and dirty solution, but if there's no room and there are compatibility issues, why not storing song-length values in the filename's last characters....that could be checked and read easily by a simple text-routine...
|
| |
Mr. SID
Registered: Jan 2003 Posts: 424 |
Quoting Groepaz Quote:
my favourite WTF is, that it contains the pointless md5 (wth should one do with this?) as regular data.... and the filename, which is what you really want, is only there as a comment, so everyone who uses the db ignores the md5, and then adds whacky code to seek backwards to last comment and extract the filename. awesome =) the times beeing in min:sec format and not seconds only as you most likely need in your app then is only a small annoyance. somehow looks as whoever invented this format didnt code a thing before =P
The idea is that the path name often changes. So the MD5 is a more robust way of identifying the file in question. So it's possible to extract the song length even if the file was not loaded from within the HVSC, as long as the MD5 is in the DB.
Btw, the whole system was designed and implemented by Michael Schwendt. |
| |
Mr. SID
Registered: Jan 2003 Posts: 424 |
Quoting HermitA primitive and dirty solution, but if there's no room and there are compatibility issues, why not storing song-length values in the filename's last characters....that could be checked and read easily by a simple text-routine...
Because there's often more than one song in a file. |
| |
Oswald
Registered: Apr 2002 Posts: 5086 |
simply use the bytes left to point to the end of the sid file and stick the extra data there ? anyways I dont think switching to a new format would be the end of the world, why not? |
| |
JCB Account closed
Registered: Jun 2002 Posts: 241 |
The problem with having a "footer" (putting the extra data at the end of the file) is backwards compatibility. If older players that don't know about it read the header, then read the rest of the file into the emulated RAM till eof, it could be nasty.
|
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
Quote: The problem with having a "footer" (putting the extra data at the end of the file) is backwards compatibility. If older players that don't know about it read the header, then read the rest of the file into the emulated RAM till eof, it could be nasty.
I don't think this will actually be an issue for the following reasons
1) The SID file formats do not contain any references to file size, so in order to read the file, it is highly likely that the allocated buffer will be the actual filesize.
2) Say all the data (including the optional footer/header) gets loaded into the C64 memory, the SID code won't access them, so it won't cause any playback issues
I guess the only downside to this is if the LoadAddress field in the header is close to the upper portion of the 64KB limit, and the data including the extra header information could overrun the buffer, but that leads me to...
3) "+04 WORD version" - Each PSID has a version. As this file would be a new version, older players would (should) validate it's a version they support.
2 |
| |
JCB Account closed
Registered: Jun 2002 Posts: 241 |
Yeah, it would only really be an issue if the SID file was loaded towards the end of the emulated RAM. Whether coders of any players do anything other than check existing versions is another matter ;)
If something like this is done then maybe any future HVSC releases with the new data in could come with a "stripper" exe to run to revert the files back to the old format. |
| |
SIDWAVE Account closed
Registered: Apr 2002 Posts: 2238 |
The best is to have all crucial info in 1 header, and
not being afraid to make a new format. just do it.
if people want to support new format, they should do so.
This was designed in 1990 by PHS, and all the optimizations to it since, have been limited by "oh no, i have to not break compatibility"
I dont think that goes anywhere positive with a format that is so old.
Values in header, and STIL etc. in a real database format, and the key to read a tune's info, in the header. done!
If "Commando" has db key 147 for STIL and other stuff info, this key will never change. "Commando" stays at 147 forever.
We have been over this again and again, but when it came to the deciding things, actually making the things, real life came in the way every time, so its stranded.
AFAIK e already know a long time what to do, just e are all lame asses and didnt do it.
Ian: just because im not coding this, and because i have left the team because i have no more time, dont mean i dont have ideas, and have a right to speak them. |
| |
Radiant
Registered: Sep 2004 Posts: 639 |
Quoting Mr. SIDThe idea is that the path name often changes. So the MD5 is a more robust way of identifying the file in question. So it's possible to extract the song length even if the file was not loaded from within the HVSC, as long as the MD5 is in the DB.
But why is calculating the MD5 value such an esoteric task? It should be really simple - right? If people are cheating their way around doing it then clearly something is wrong and the system should be redesigned.
I think the footer idea is good. It won't affect backwards compatibility in most cases, and in those cases where it will and upgrading is not an option, there are workarounds. |
| |
JCB Account closed
Registered: Jun 2002 Posts: 241 |
@rambones, a database would be an ok format for a PC/Mac (plenty of light cross platform options) but then you force slimmer devices (or people who don't require the whole HVSC) to take the entire db just to play a few files..
|
| |
Inge
Registered: Nov 2003 Posts: 144 |
I've made an example using XML, to point out how flexible it all can be done. Here, all STIL info and songlength info are a part of the file itself, making it much more easy to keep track of all info related to the songs.
One negative thing is that each SID-file will be 3-6 KB larger in size, but that should not be any problem unless you absolutely want to load the file directly in to the C64 memory.
<?xml version="1.0" encoding="UTF-8"?>
<SID type="RSID">
<Header>
<LoadAddress="0000">
<InitAddress="4000">
<PlayAddress="0000">
<Songs="20">
<StartSong="1">
<Name="Arkanoid">
<Author="Martin Galway">
<Released="1987 Imagine">
<RelocStart="0000">
<RelocPages="00">
<Flags>
<!-- 0 = built-in music player,
1 = Compute!'s Sidplayer MUS data, music player must be merged. -->
<Format="0">
<!-- 0 = C64 compatible,
1 = PlaySID specific (PSID v2NG)
2 = C64 BASIC flag (RSID) -->
<Player="0">
<!-- 0 = Unknown,
1 = PAL,
2 = NTSC,
3 = PAL and NTSC. -->
<Clock="1">
<!-- 0 = Unknown,
1 = MOS6581,
2 = MOS8580,
3 = MOS6581 and MOS8580 -->
<SidModel="1">
</Flags>
</Header>
<STIL>
<Author>
<Comment="Martin Galway's own comments are denoted (MG).
'I will never do rearrangements of those tunes on other instruments,
the SID is what they're meant to be heard on. It would not be
'updating' them to put them onto a different instrument. If you ever
hear music from me on today's instruments, the tunes will be all-new
(and probably a lot better).' (MG)">
</Comment>
</Author>
<SID>
<Comment="This is the very first published tune on the C64 that used samples.
'I'm glad you spotted 'Cobra' on the Spectrum, whose tune I was in
love with and HAD to use somewhere else...! I figured no-one would
complain if I used it a year later on the C64.' (MG) (It got used in
Arkanoid.)
'Q: How did you get involved with using sampled sounds?
MG: I figured out how samples were played by hacking into someone
else's code... OK, I admit it... It was a drum synthesizer package
called Digidrums, actually, so you could still say I was the first to
include samples in a piece of music. [...] Never would I claim to have
invented that technique, I just got it published first. In fact, I
couldn't really figure out where they got the sample data, just that
they were wiggling the volume register, so I tried to make up my own
drum sample sounds in realtime - which is the flatulence stuff that
shipped in 'Arkanoid'. [...] After the project was in the shops I
gained access to some real drum samples, and I slid those into my own
custom version of the tune. The one that's in the shops is kind of a
collage of farts & burps, don't you think? [...] Later I was able to
acquire some proper drum samples and by 'Game Over' it got quite
sophisticated.' (MG)">
</Comment>
</SID>
</STIL>
<Songs>
<Song1>
<Length="2:22">
<STIL>
<Title="Cobra (Title) [from the Spectrum game]">
<Artist="Martin Galway">
<Comment="'I'm glad you spotted 'Cobra' on the Spectrum, whose tune I was in
love with and HAD to use somewhere else...! I figured no-one would
complain if I used it a year later [for Arkanoid] on the C64.' (MG)">
</STIL>
</Song1>
<Song2>
<Length="1:17">
<STIL>
<Title="Hiscore [from the arcade game Arkanoid - Revenge of Doh]">
<Artist="Hisayoshi Ogura (OGR/Zuntata)">
</STIL>
</Song2>
</Songs>
<Binary>
(Binary code goes here)
</Binary>
</SID>
|
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
<Binary>
(Binary code goes here)
</Binary>
can a binary be included in a plain text xml? Shouldn't it be at least base64 encoded?
Besides converting hvsc to this format will mean to trash *all* existing programs and waiting to be accepted and fully supported. I don't see it forseeable in the near future, if even reading a single sldb txt file is so hard for someone to be implemented....
ps: this is only my opinion, I'm not stopping anyone from proposing ideas, but only ideas are not enough, I'm waiting for someone actually doing some new format with the needed tools to support it, and see if it's usable. |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
TBH I'm not a fan of XML. I think it's overused, and definately not the best approach for this IMO. What with wanting to keep this small for memory constrained devices.
What I'll do then, I'll try and make a modified version of TitchySID and try a converter tool to take a PSIDv2, add the optional header and see what happens :-) |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
make sure to encode it in brainfuck, so hexpattern searching will be REALLY complicated. |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
I don't know, Piet is my preferred language
http://www.dangermouse.net/esoteric/piet/samples.html
Also Whitespace is good
https://secure.wikimedia.org/wikipedia/en/wiki/Whitespace_%28pr..
:-) |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
Quoting pmprogTBH I'm not a fan of XML. I think it's overused, and definately not the best approach for this IMO. What with wanting to keep this small for memory constrained devices.
An IFF style format would work here. It supports most of the nice XML features (flexible, structured, future proof), but is fast and easy to parse. You'd keep the PSID header for backwards compatibility, and just bump the version number and add a pointer to the extended data, which is appended after the music data:
0000 PSID, with ptr to SIFF block, starting at 1100 here
007c data
1100 SIFF
1104 SIFF size
1108 TIME
110c TIME size
1110 4 bytes time info per subtune
and so on.
|
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
MagerValp: Yep, that's the plan, I'm going to have to do some "real" work first, but should be easy enough to create a "converter" tool. I didn't want to call it v3 in the header as iAN said there's already a v3 in the works, so my version field will get set to 0x8002 (v2.5 ;-))
Once the converter is made, I'll create an updated version of TitchySID (just because I understand the source code). Then hopefully look at making an update to libsidplay (which would also let me recompile the Foobar2000 plugin, but after a quick look at the source code, I don't know where to start) |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
Converter - PSID V2.5 Converter
The converter will also let you convert V2.5 PSIDs back to V2
Further down the line I'll add commandline support so it can be batched, but next task is onto a player... |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
1) use a portable language
2) source? |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
Oh yeah, was going to upload the source on the project page too, but I forgot. It's there now
Written in VB.NET, but should be easily run through Mono (though I haven't tried). It was stated on the Summaries page.
If not, I'll make a C command line version in a bit |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
This proposed(?) 2.5 version suffers from the same problems as PSID 1 and 2, as it just adds three variable length fields at the end. A chunk structure, like IFF/PNG/TIFF/AIFF/etc, would be needed for future proofing. |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
iff has variable block lenght fields anyway, what's the difference? You have to state how long is each block anyway.
This format btw clashes with the new v3 format because the pointer at 0x7a is already used to set the 2nd sid address =) |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
@iAN: Well, of course this isn't going to be compatible with v3 because I've never seen the spec. Is it even published? I can't find it on the HVSC site
I was hoping we could find a way to put it in the "next" spec so everybody has access, but by the sounds of it, you aren't very keen on this idea (I can't understand why, as it would save having to maintain a separate bunch of "database" files).
That's fair enough, I suppose, but then it does kinda render my efforts pointless unless everybody has to convert their own files or a duplicate HVSC is created; and that is just absurd. |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
Quote: @iAN: Well, of course this isn't going to be compatible with v3 because I've never seen the spec. Is it even published? I can't find it on the HVSC site
I was hoping we could find a way to put it in the "next" spec so everybody has access, but by the sounds of it, you aren't very keen on this idea (I can't understand why, as it would save having to maintain a separate bunch of "database" files).
That's fair enough, I suppose, but then it does kinda render my efforts pointless unless everybody has to convert their own files or a duplicate HVSC is created; and that is just absurd.
We should maintain a separate database file until the new format is accepted and there are tools to use the new format without having to always convert from/back to old format. The appended block of extra infos is good to me. If there isn't any extra the file it's exactly as the current sid format. |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
Quoting iAN CooGiff has variable block lenght fields anyway, what's the difference?
The difference comes in the form of FORM chunks that contain other chunks. It's as flexible as XML (but orders of magnitude easier to work with), and it's easy to make file formats that are both backwards and forwards compatible. |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
The v3 format is not yet made public, it just contain 2 new fields at 0x76 and 0x7a (sidmodel for 2nd sid and adress of 2nd sid), but at this point I think it should be rediscussed, if in future we want to apply the new extra block with sldb/stil/buglist the 2nd sid address must be put somewhere else to leave space for the extrablock pointer. |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
Quote:This proposed(?) 2.5 version suffers from the same problems as PSID 1 and 2, as it just adds three variable length fields at the end. A chunk structure, like IFF/PNG/TIFF/AIFF/etc, would be needed for future proofing.
Yes, my v2.5 format was supposed to try and maintain compatibility, whilst offering the song lengths. It wasn't a reinvention of the PSID format
Quote:if in future we want to apply the new extra block with sldb/stil/buglist
I find it strange that you consider song lengths not sufficient for inclusion into the file (sorry if I keep repeating myself). Knowing how long a song plays for is just as important as which chip it runs on, and what at speed etc. I only included the STIL/BUGLIST options on my converter to make it seem a little more worth it.
Quote:This format btw clashes with the new v3 format
Actually, whilst I'm thinking... Who cares if you're using some fields in v3 that I'm using in v2.5 for different reasons. If the player bothered reading the file version, it'll know what to do with it. |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
Quote: Quote:This proposed(?) 2.5 version suffers from the same problems as PSID 1 and 2, as it just adds three variable length fields at the end. A chunk structure, like IFF/PNG/TIFF/AIFF/etc, would be needed for future proofing.
Yes, my v2.5 format was supposed to try and maintain compatibility, whilst offering the song lengths. It wasn't a reinvention of the PSID format
Quote:if in future we want to apply the new extra block with sldb/stil/buglist
I find it strange that you consider song lengths not sufficient for inclusion into the file (sorry if I keep repeating myself). Knowing how long a song plays for is just as important as which chip it runs on, and what at speed etc. I only included the STIL/BUGLIST options on my converter to make it seem a little more worth it.
Quote:This format btw clashes with the new v3 format
Actually, whilst I'm thinking... Who cares if you're using some fields in v3 that I'm using in v2.5 for different reasons. If the player bothered reading the file version, it'll know what to do with it.
I personally don't have any use of the sldb, I play the sids whatever I need, I don't let the program to decide how much by itself =) and it's not important as knowing the sidmodel. Playing a sid with wrong sidmodel means that you are not listening it as intended, listening a sid that is meant to loop, but making it break at the loop point to me is wrong.
Sid v3 is meant for stereo sids, but that doesn't mean they don't have a duration, stil and bug infos, so they will need the 2nd sid fields AND the extra block as any other sid file. |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
So how about this: the new format would allow any number of info chunks at the end, after the music data. Each chunk has a 4-byte ASCII name, followed by a 32-bit big endian unsigned integer of the length of its data (optionally: align all chunks, and round all lengths up to the nearest 4 bytes). The benefit of a format like this is that:
* Old players will just ignore the data at the end and can play new files.
* New players can parse and use the chunks they recognize.
* Unknown chunks can be ignored.
* As the format progresses you can add more chunks, and files stay compatible. Players can opt in to new features.
Suggested chunks:
* 2SID: Stereo SID info, two words for model and address
* STIL: STIL data
* TIME: One uint32 for each subtune with the number of frames that it should play. Flag if the tune repeats or stops.
* BUGS: Bug data
|
| |
Mr. SID
Registered: Jan 2003 Posts: 424 |
Quoting MagerValp
* TIME: One uint32 for each subtune with the number of frames that it should play. Flag if the tune repeats or stops.
It really should be a 16bit second count. Frames is not really usable, because in emulator libraries such as libsidplay2 you don't really know how many frames have passed. RSID tunes are basically just programs that run, the emulator does not call INIT/PLAY but just runs a full C64 program (PLAY is usually $0000 in this case). |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
indeed, not counting those sids that have irregular CIA timings (there are plenty) and those with timed play call, one every 5 calls for example. |
| |
Frantic
Registered: Mar 2003 Posts: 1646 |
...but I am quite sure that the majority of tunes in HVSIDS have a duration that aligns evenly with a certain number of frames, but not with an even number of seconds? Easy conclusion: frames is a better choice.
One could also specify various alternative time formats that would fit different kinds of tunes better, I guess, but if one should go for one single duration time format, I cannot see why number of frames is not the best choice. The arguments provided so far did not convince. They sounded more like "hey, there is this exception to what you said, so what you said does not make sense at all." :) |
| |
chatGPZ
Registered: Dec 2001 Posts: 11350 |
Quote:As the format progresses you can add more chunks, and files stay compatible. Players can opt in to new features.
thats the "beauty" of a tag based format. everyone can invent their own new tags. AND THEY WILL. it could grow into something almost as awesome as id3 ! hell yeah! \o/ o_O |
| |
Mr. SID
Registered: Jan 2003 Posts: 424 |
Quoting Frantic...but I am quite sure that the majority of tunes in HVSIDS have a duration that aligns evenly with a certain number of frames, but not with an even number of seconds? Easy conclusion: frames is a better choice.
So how long is a frame?
1/50 seconds?
1/60 seconds?
1/50.12 seconds?
One second precision is more than enough for this purpose, and a 16 bit value would be sufficient then. If you want to waste 32 bits on the song length, you could just as well use microseconds which would be almost cycle accurate. |
| |
Frantic
Registered: Mar 2003 Posts: 1646 |
??
The duration of a frame is not an unknown value, and NTSC versus PAL is specified in the header anyway. The closer the value is to the real duration of a frame the better, of course.
For pal...
985248 / 312 / 63 = 50,1245421....
Use as many decimals as you like. :)
I find it quite disturbing when Deliplayer (on windows) cuts in the middle of a beat and things like that, when playing various amiga formats where the songlength can only be specified in seconds. I mean.. it is not that the unit of a frame is so wonderful as such. It is just that I fail to see why seconds would be better, since there is no inherent relation between "seconds" and the units of time that usually underly SID tunes. To me it is obviously more clean to cut the tune where it is in fact ending, rather than half a second off or so. Half a second is quite a noticeable duration when it comes to sound. |
| |
Sasq
Registered: Apr 2004 Posts: 156 |
CHUNKs is the way to go for sure. But I don't think trying to keep it compatible with the old format is a good idea, it may break things, and what's the point? If you're still using an old player chances are you're still using the old format files. And it's easy enough to add the format to the important players once it gets accepted.
The only problem with putting STIL into the file is the author-info; A bit unfortunate to have to duplicate the same information in every song by the same Author.
Also if we are worried about size, we can always use 16 or 8-bit tags where it fits, and actually end up with smaller files in many cases, like so;
"XSID", $????????
"IN", $0026 // Info (always $60 bytes in PSID)
"A", $04, "Drax" // Author
"N", $0a ,"Test Music" // Name
"C", $15, "198x Maniacs of Noise" // Copyright
"FL", $???? // Flags
...
"DA", $???? // Data
...
But that is probably not worth the extra complexity...
|
| |
Frantic
Registered: Mar 2003 Posts: 1646 |
Couldn't edit the last post when I was supposed to add:
...but I do agree that milli/micro-seconds would be more or less equivalent to frames for most practical purposes. Whole seconds, no... |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
OK, so an uint32 for the number of clock cycles that the tune should play. Divide by 985248 to get the number of seconds for PAL, or 1022727 for NTSC. Songs that cut mid-beat annoy me too. The only drawback I can come up with is that you get a max of 72 minutes.
@groepaz: Social media style tags have nothing to do with this. The format specification would define chunks and their meaning. |
| |
chatGPZ
Registered: Dec 2001 Posts: 11350 |
Quote:@groepaz: Social media style tags have nothing to do with this.
i know exactly what you mean :)
Quote:The format specification would define chunks and their meaning.
which wont stop anyone from inventing their own tags - because they can. just look at id3 for example =) |
| |
WVL
Registered: Mar 2002 Posts: 895 |
Quote: CHUNKs is the way to go for sure. But I don't think trying to keep it compatible with the old format is a good idea, it may break things, and what's the point? If you're still using an old player chances are you're still using the old format files. And it's easy enough to add the format to the important players once it gets accepted.
The only problem with putting STIL into the file is the author-info; A bit unfortunate to have to duplicate the same information in every song by the same Author.
Also if we are worried about size, we can always use 16 or 8-bit tags where it fits, and actually end up with smaller files in many cases, like so;
"XSID", $????????
"IN", $0026 // Info (always $60 bytes in PSID)
"A", $04, "Drax" // Author
"N", $0a ,"Test Music" // Name
"C", $15, "198x Maniacs of Noise" // Copyright
"FL", $???? // Flags
...
"DA", $???? // Data
...
But that is probably not worth the extra complexity...
How to tell the difference from chunk
"A",$04,"Drax" // Author and chunk
"ADDR",$02,$d420 // Address of 2nd sid ?
|
| |
chatGPZ
Registered: Dec 2001 Posts: 11350 |
the quotes are part of the tag in the file? :) |
| |
iAN CooG
Registered: May 2002 Posts: 3186 |
Quote: How to tell the difference from chunk
"A",$04,"Drax" // Author and chunk
"ADDR",$02,$d420 // Address of 2nd sid ?
infact tags should be always 4 bytes
AUTH for author
NAME
RELE as in RELEASED, and not COPYright, but it's true that an optional CREAtion tag could be added, if the exact creation date is known
|
| |
Sasq
Registered: Apr 2004 Posts: 156 |
Quote: How to tell the difference from chunk
"A",$04,"Drax" // Author and chunk
"ADDR",$02,$d420 // Address of 2nd sid ?
In that example we assume that top level chunks are 32bit, 2nd level 16bit and 3rd level 8bit... but like I said, probably not worth it.
... although simply making all tags 16bit could be an option.
|
| |
jssr67 Account closed
Registered: Jan 2011 Posts: 33 |
Quote: OK, so an uint32 for the number of clock cycles that the tune should play. Divide by 985248 to get the number of seconds for PAL, or 1022727 for NTSC. Songs that cut mid-beat annoy me too. The only drawback I can come up with is that you get a max of 72 minutes.
@groepaz: Social media style tags have nothing to do with this. The format specification would define chunks and their meaning.
To make a connection between playing time from clock cycles and NTSC/PAL can be dangerous.
What about SIDs that try to detect NTSC/PAL and try to compensate the clock difference by setting different interrupt dividers, and using variable frequency tables? They do not have a constant number of clock cycles, but a more or less constant playing time.
SIDs that do NOT compensate, may have a constant number of clock cycles, an you may calculate the playing time then, of course. but can we take that for granted?
Could a solution be to have fields for playing time/nr of clock cycles for both "modes"? |
| |
Sasq
Registered: Apr 2004 Posts: 156 |
This is all theoretical anyway, since songlengths.txt only contains seconds, and are those lengths are often off by several seconds.
Length is actual play time when listening, and not number of frames. Why not just use milliseconds, defined when playing with the specified clock, and PAL if not defined.
|
| |
jssr67 Account closed
Registered: Jan 2011 Posts: 33 |
Quote: This is all theoretical anyway, since songlengths.txt only contains seconds, and are those lengths are often off by several seconds.
Length is actual play time when listening, and not number of frames. Why not just use milliseconds, defined when playing with the specified clock, and PAL if not defined.
Just because the current solution is quite imperfect, does not mean it shouldn't be done better.
And there are possibilities to make it precise to less than a frame. And yes, that may matter. For example, if you have collections in which you want to have gapless song switching, because it was meant so by the author.
Using ms or any shorter fixed time unit is one option (frames of less than a ms may be uncommon). But letting the author decide by setting the appropriate tags and leaving the others open (so, he can say ms, frames, clock cycles... whatever suits the SID best) and the player then evaluates what has been given does not add too much complexity. And having the chosen tag for PAL as well as NTSC in the worst case adds one unnecessary tag. Or you can have 3: songlength/PAL, songlength/NTSC, songlength/ANY, given in any of the units stated.
Actually, I think discussing about the feature costs already more time than implementing the most flexible solution that you can think of. Cutting options is ok when it dramatically reduces complexity, I do not see that apply here.
Ah almost forgot, you asked why not just use timing for one clock: because then it is not stated if and how it varies between the standard clocks. It may not matter to you at the moment, but it is plainly incomplete, it MAY matter for some, and why create a new format specification that already has known holes from the start? |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
Well a standard precision float would do the trick as well, but that might be troublesome for embedded players (or the C64 for that matter). Milliseconds is a little too coarse, but how about 16.16 fixed point? Trivial to read out the seconds if that's all you need, 15 µs resolution, and max 18 hours of playing time. Or 9 hours and a loop flag. |
| |
Sasq
Registered: Apr 2004 Posts: 156 |
16.16 is a good idea I think.
So then perhaps we have these basic chunks;
HEAD - mandatory, with addresses and flags
DATA - At least one is mandatory, loaded into C64 mem
INFO - Optional
SLEN - Song lengths, 32 bits per song, optional
The STIL chunks may take a bit more thought...
One question is how the INFO chunk should look; simply the same 48 bytes as PSID, or sub chunks ("TITL", 0x00000010) or more compact (ID byte + length byte) ?
|
| |
Mr. SID
Registered: Jan 2003 Posts: 424 |
Since HEAD would always be the first chunk, and it's useful to have a simple quick way of identifying the format, I would propose to call that chunk XSID.
|
| |
chatGPZ
Registered: Dec 2001 Posts: 11350 |
if you actually want a IFF based format, then using one tag for more than one kind of info is pretty much a nono though. why not have PSID as first tag then, put the entire old file into it, and then put new info into other tags? =) |
| |
Mr. SID
Registered: Jan 2003 Posts: 424 |
Because lots of existing code checks for PSID, and then assumes that the data starts at offset 0x7c (i.e. ignoring the data offset stored in the header). |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
Which is why you'd keep the old header and data to be compatible, and only add chunks of new information at the end. |
| |
pmprog Account closed
Registered: Nov 2005 Posts: 54 |
MagerValp: Your first header wouldn't actually support the IFF format (ie, there's no length of the segment). Also you run the risk like I mentioned ages ago that if the PSID player just loads *all* data after the header (which I would guess was default behaviour), you have the possibility of overrunning the "C64" memory space in the player.
Might as well just wait to see what the HVSC team come up with for the new format now |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
Quoting pmprogMagerValp: Your first header wouldn't actually support the IFF format (ie, there's no length of the segment).
I'm not saying it would. I'm advocating that we keep the old header, and the old payload, and only add IFF-style chunks to the end, not that the new file format should be IFF.
Quote:Also you run the risk like I mentioned ages ago that if the PSID player just loads *all* data after the header (which I would guess was default behaviour), you have the possibility of overrunning the "C64" memory space in the player.
Yep, it's something to watch out for, and it'd be interesting to batch convert HVSC to this new format and see how big they get. I suspect that everything in HVSC would fit though.
|
| |
Sasq
Registered: Apr 2004 Posts: 156 |
I'm betting there are some tunes that end at FFFF which wont work. And if you play with a relocating player on the C64, many tunes with extra info may not work since they can extend into the area marked as free by the reloc-page.
Why not just create a clean, new format instead of another hack. With tools to convert back and forth it should cause problems for very few people.
|
| |
Steppe
Registered: Jan 2002 Posts: 1510 |
For the time being we decided to take a compromise. In the new PSIDv3 format stereo sid support was added, zero termination of header fields was removed (increasing header string size by 1) and the old 28+3 filename limitation was removed. This change should be trivial to implement for sidplayer authors and it shouldn't break backward compatibility. Discussion about a version 4 hasn't really gone anywhere so far, please be patient. So far, thanks a lot for the input everyone! |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
OK, cool.
Just for fun though I spent a lunch break adding chunks to sid files, but I got stuck on the md5 checksum in songlengths.txt. How is that calculated? The checksums in songlengths.txt is different from what I get if I hash the file with /sbin/md5 or python's hashlib.md5(). I also tried hashing just the raw sid data, but that didn't work either. |
| |
Steppe
Registered: Jan 2002 Posts: 1510 |
The whole core of the MD5 code is very small:
// Include C64 data.
myMD5.append(cache.get()+fileOffset,info.c64dataLen);
unsigned char tmp[2];
// Include INIT and PLAY address.
writeLEword(tmp,info.initAddr);
myMD5.append(tmp,sizeof(tmp));
writeLEword(tmp,info.playAddr);
myMD5.append(tmp,sizeof(tmp));
// Include number of songs.
writeLEword(tmp,info.songs);
myMD5.append(tmp,sizeof(tmp));
// Include song speed for each song.
for (unsigned int s = 1; s <= info.songs; s++)
{
selectSong(s);
myMD5.append(&info.songSpeed,sizeof(info.songSpeed));
}
// Deal with PSID v2NG clock speed flags: Let only NTSC
// clock speed change the MD5 fingerprint. That way the
// fingerprint of a PAL-speed sidtune in PSID v1, v2, and
// PSID v2NG format is the same.
if ( info.clockSpeed == SIDTUNE_CLOCK_NTSC )
myMD5.append(&info.clockSpeed,sizeof(info.clockSpeed));
// NB! If the fingerprint is used as an index into a
// song-lengths database or cache, modify above code to
// allow for PSID v2NG files which have clock speed set to
// SIDTUNE_CLOCK_ANY. If the SID player program fully
// supports the SIDTUNE_CLOCK_ANY setting, a sidtune could
// either create two different fingerprints depending on
// the clock speed chosen by the player, or there could be
// two different values stored in the database/cache.
And here's some clarification from Wilfred about this:
But the "selectSong(s)" is the not trivial part of the whole calculation which sets the songSpeed for every song.
If you convert the selectSong method in the loop, for the things needed for the MD5 calculation, you will get:
uint8 u8CiaSpeed = 60;
uint8 u8VblSpeed = 0;
for (i = 0; I < u16NumSongs; i++) {
if (!boPSIDSpecific) {
if (i < 31) // check tune 0-30 (bit 0 - 30)
boVbiSpeed = !(u32Speed & (1 << i));
else
boVbiSpeed = !(u32Speed & (1 << 31));// last bit sets speed for tunes >31
} else {
// old PlaySID implementation
boVbiSpeed = !(u32Speed & (1 << (i % 32)));
}
// speed setting
if (boVbiSpeed)
md5_append(&u8VblSpeed, 1);
else
md5_append(&u8CiaSpeed, 1);
}
Hope that helps, I don't understand a line of what's written up there. :-/ |
| |
Radiant
Registered: Sep 2004 Posts: 639 |
MagerValp: If you can figure it out, pray tell - nobody seems to know for sure. :-) |
| |
Steppe
Registered: Jan 2002 Posts: 1510 |
From what I gathered this is Delphi code. There also exists a C conversion of this. I'm sure Wilfred Bos will be available if you need help. |
| |
Mr. SID
Registered: Jan 2003 Posts: 424 |
It's not pretty code, but it's simple enough once you think about it. The complexity really comes from the way the speed flags work in the PSID header, which is a huge mess and limited to 32 songs too. |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
OK, so here's my attempt at converting it to Python:
m = hashlib.md5()
m.update(sid_data)
m.update(struct.pack("<H", sid_header[HDR_INITADDRESS]))
m.update(struct.pack("<H", sid_header[HDR_PLAYADDRESS]))
m.update(struct.pack("<H", sid_header[HDR_SONGS]))
for i in range(sid_header[HDR_SONGS]):
if sid_header[HDR_SPEED] & (1<<i):
m.update(chr(60))
else:
m.update(chr(0))
Something's not right though, as the checksum still doesn't match (and I'm testing with a PAL tune with less than 32 subtunes). |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
The missing detail was that you need to skip the load address when calculating the md5 sum:
m = hashlib.md5()
m.update(sid_data[2:])
m.update(struct.pack("<H", sid_header[HDR_INITADDRESS]))
m.update(struct.pack("<H", sid_header[HDR_PLAYADDRESS]))
m.update(struct.pack("<H", sid_header[HDR_SONGS]))
for i in range(sid_header[HDR_SONGS]):
if sid_header[HDR_SPEED] & (1<<i):
m.update(chr(60))
else:
m.update(chr(0))
Now for the rest... |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
And here we go:
sid_header = unpack_header(data)
sid_version = sid_header[HDR_VERSION]
sid_data = data[sid_header[HDR_DATAOFFSET]:]
m = hashlib.md5()
m.update(sid_data[2:])
m.update(struct.pack("<H", sid_header[HDR_INITADDRESS]))
m.update(struct.pack("<H", sid_header[HDR_PLAYADDRESS]))
m.update(struct.pack("<H", sid_header[HDR_SONGS]))
for i in range(sid_header[HDR_SONGS]):
if sid_header[HDR_MAGIC] == "RSID":
m.update(chr(60))
else:
if sid_header[HDR_SPEED] & (1<<min(i, 31)):
m.update(chr(60))
else:
m.update(chr(0))
if sid_version == 2:
if (sid_header[HDR_FLAGS] & 0x0c) == 0x08:
m.update(chr(2))
That is one seriously fugly algorithm... |
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
So in case anyone's still playing with this, here's what my tools spits out now:
/MUSICIANS/J/JCH/Tecnetium.sid
{'COMM': 'I always get mixed feelings when I play this
music. The title- tune is\nlong and I put a lot of work
into it in april 1989, but today I can\nhear loads of
harmonic errors here and there. The game was done for
a\nsideways shoot\'em up which a couple of totally
unknown danish guys was\nmaking. They soon lost all
interest in their game, however. The\nhiscore theme was a
conversion of the old Cliff Richard
hit\n"Congratulations" done right out my head. I always
wanted to make such\na hiscore tune, because I always
thought Rob Hubbard did it all wrong\nwhen he made "sad"
hiscore tunes. It is a sad thing when the game\nends, but
getting a high score is supposed to be a GOOD thing! :)',
'TUNS': [{'LENG': 23330816, 'NAME': 'Title Screen'},
{'ARTI': 'Cliff Richard',
'LENG': 5963776,
'TITL': 'Congratulations'},
{'LENG': 262144},
{'LENG': 196608},
{'LENG': 2883584},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 196608, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 196608, 'LOOP': 'G'},
{'LENG': 131072, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'},
{'LENG': 65536, 'LOOP': 'G'}]}
Supported chunks are:
TITL: Title
ARTI: Artist
AUTH: Author
NAME: Name
COMM: Comment
BUGS: Text from buglist
TUNS: List of subtunes
LENG: Song length as 16.16 fp
LOOP: Loop flag
|
| |
MagerValp
Registered: Dec 2001 Posts: 1074 |
Oh, and just to keep the monologue going: attaching chunks to the end of SIDv2 files will break backwards compatibility with songlengths.txt, as the checksum is calculated from all data after the header. That kills the idea, so to extend the file format beyond what was done with v3 (stereo sid and zero termination) we'll need a new file format. |