| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Implement http://ftp.pokefinder.org/ into CSDb
Implement http://ftp.pokefinder.org/ into CSDb for alternative download link search!
Opinions? |
|
| |
cba
Registered: Apr 2002
Posts: 935 |
Quote: Implement http://ftp.pokefinder.org/ into CSDb for alternative download link search!
Opinions?
Very good idea, but Andreas has first to fix his
ftp searcher , right now changes I do to TDD are not listed
Niels |
| |
Mason
Registered: Dec 2001
Posts: 461 |
I cant remember the exact details about pokefinder, but I reckon countzero told me it does indexes now and then - I cant remember how often. |
| |
cba
Registered: Apr 2002
Posts: 935 |
Quote: I cant remember the exact details about pokefinder, but I reckon countzero told me it does indexes now and then - I cant remember how often.
I just checked changes I did 9 / 10 days ago on TDD and
they are not updated yet.
So Mr. Andreas, fix it :)
Niels |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
So let's hope CountZero will fix that :) and Perff put it into CSDb :) |
| |
iopop
Registered: Dec 2001
Posts: 317 |
I suppose you mean like a default button for each entry? Otherwise you could add it yourself like;
http://ftp.pokefinder.org/index.php?s=druid&m=0&h=100 to get all files named "druid".
http://ftp.pokefinder.org/index.php?s=triad&m=2&h=100 to get all dirs named "triad".
But a default button would be a cool feature, for sure. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: I suppose you mean like a default button for each entry? Otherwise you could add it yourself like;
http://ftp.pokefinder.org/index.php?s=druid&m=0&h=100 to get all files named "druid".
http://ftp.pokefinder.org/index.php?s=triad&m=2&h=100 to get all dirs named "triad".
But a default button would be a cool feature, for sure.
Yes! A default button for each entry! Like in pouet.net |
| |
Perff
Administrator
Posts: 1679 |
I'm not 100% sure what you guys want, so perhaps what I just made itsn't exactly it.
What I did, is that I on the edit-page of all releases added a little link that pops up a new window with the searchresults from pokefinder.org.
Then the one who is editing/creating this release, can pick a good link, and add it to CSDb.
I'd still prefer direct links in CSDb, so thats why this new link is only on the edit-page.
If anyone got other similar search-sites, I can easily add another link or two. :)
|
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: I'm not 100% sure what you guys want, so perhaps what I just made itsn't exactly it.
What I did, is that I on the edit-page of all releases added a little link that pops up a new window with the searchresults from pokefinder.org.
Then the one who is editing/creating this release, can pick a good link, and add it to CSDb.
I'd still prefer direct links in CSDb, so thats why this new link is only on the edit-page.
If anyone got other similar search-sites, I can easily add another link or two. :)
:(
This way it's not useful, since a lot of entries are locked! If the link was below the download of the actual product pages, people could download the release even if an entry is locked and it's not possible to correct the link |
| |
Zyron
Registered: Jan 2002
Posts: 2381 |
Perhaps it should at least be possible to add/correct files/links to locked entries? |
| |
Perff
Administrator
Posts: 1679 |
I could also add the link to the front-page. I just don't want that to be an excuse NOT to add links or upload the file.
|
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: I could also add the link to the front-page. I just don't want that to be an excuse NOT to add links or upload the file.
Idea: Write a routine that only lets the "mirror" button to pokefinder's ftp appear if all download links are broken! ;) |
| |
Shake
Posts: 134 |
it would do no harm if users are allowed to update broken links.
This locking is a pain in the ass for this database. What if it's only allowed for people to lock their OWN profile?
People can maintain private information like addresses etc.
|
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
btw yes the search spider is fux0red, cz is working on it :) and i'd like a search button on the release pages too... maybe also on the group page (search for group name and abbrivation) |
| |
Perff
Administrator
Posts: 1679 |
Now you got the links on both the release-page and it's edit page.
About the locking of entries.
If someone locks an entry it should be because to avoid people of messing up the data. At the same time the one locking the entry is now responsible for maintaining the entry with all the info there is. If they don't they shouldn't lock the entry. I think I have written something like this somewhere in the help/docs...
(There can be situations where people on purpose doesn't want some types of info on their entries, and then they of course have that right.) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Now you got the links on both the release-page and it's edit page.
About the locking of entries.
If someone locks an entry it should be because to avoid people of messing up the data. At the same time the one locking the entry is now responsible for maintaining the entry with all the info there is. If they don't they shouldn't lock the entry. I think I have written something like this somewhere in the help/docs...
(There can be situations where people on purpose doesn't want some types of info on their entries, and then they of course have that right.)
Thank you, Klaus! :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
some tips on improving this feature... should be very easy to implement:
1. if the release name contains a groups name or abbrevation, remove it. filenames on ftp sites often dont contain them (especially if they are old releases) it wont give you to many false positives, and the user can always pick the right one himself
2. if the release name contains the word "preview" or "prv" respectivly, replace it by just "pr", this will find both and also add surprisingly few false positives.
3. remove all non-alphanumerical characters in the search query. filenames almost never contain them, and including them in the query will reduce the chance to find something drastically. if the file is a game, you might als want to check certain common suffixes (+4hi,fix,docs etc) since the filenames might not contain them
4. (most important imho) replace all spaces in the query by %2b (+) ... else you will get MANY false positives if a part of the query is a very common word.
|
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: some tips on improving this feature... should be very easy to implement:
1. if the release name contains a groups name or abbrevation, remove it. filenames on ftp sites often dont contain them (especially if they are old releases) it wont give you to many false positives, and the user can always pick the right one himself
2. if the release name contains the word "preview" or "prv" respectivly, replace it by just "pr", this will find both and also add surprisingly few false positives.
3. remove all non-alphanumerical characters in the search query. filenames almost never contain them, and including them in the query will reduce the chance to find something drastically. if the file is a game, you might als want to check certain common suffixes (+4hi,fix,docs etc) since the filenames might not contain them
4. (most important imho) replace all spaces in the query by %2b (+) ... else you will get MANY false positives if a part of the query is a very common word.
For magazines no pound ('#') sign.
Scene+World+14 brings results, Scene+World+#14 does not! |
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
hence "remove all non-alphanumerical characters in the search query." :o) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: hence "remove all non-alphanumerical characters in the search query." :o)
you won :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
i noticed another thing that makes a lot of sense... remove abbrivated short forms like 's (for "is") and 'n (for "and") ... it's magic -> it+magic ... bop'n rumble -> bop+rumble .... etc :) much better search results that way :) |
| |
Perff
Administrator
Posts: 1679 |
Now you people have noticed that it is possible to alter the search string in the new window that pops up????
;P
(I'm not planning to make this link-routine the first true AI.. :) )
|
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
ehrm.... AI ? all i suggested can be done in one line of php (or whatever else you are using) probably. atleast removing non alphnumeric characters is a must imho. |
| |
Perff
Administrator
Posts: 1679 |
he.
I already did that - removing non alphnumeric characters if you havn't noticed. (Perhaps not all of them - had a small problem making a simple php-routine that removes all non alphanumeric chars, but not special letters like æøåôêá and what different contries have thought up. Anyone got a 100% idiot-proff thing to do that?)
I could just see where it was going. Removing simple chars is one thing. Next search for special abrivations and remove them or convert them. Also look for other words like preview, fix etc. and do stuff to them too. The next post would probably have sounded a bit like "Make a routine that understands the meaning of the title, and make a smart search from that understanding." ;)
.. or perhaps it's just me who is lazy.. hmm.. :):) |
| |
CyberBrain
Administrator
Posts: 392 |
Is this pokefinder link really a good idea?
It doesn't give unique results, like download-links does.
So it's likely you won't get the release you were looking at in CSDb.
Wouldn't it perhaps encourage people to not upload files?
Also, if there is a release without a download-link and someone finds the file (f.e. at pokefinder), shouldn't he upload it at the release page, so that other people later will get the right (unique) file? If so, i guess the pokefinder-link not really needed (maybe it could be nice to have at the edit page, but not on the main release-page) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Is this pokefinder link really a good idea?
It doesn't give unique results, like download-links does.
So it's likely you won't get the release you were looking at in CSDb.
Wouldn't it perhaps encourage people to not upload files?
Also, if there is a release without a download-link and someone finds the file (f.e. at pokefinder), shouldn't he upload it at the release page, so that other people later will get the right (unique) file? If so, i guess the pokefinder-link not really needed (maybe it could be nice to have at the edit page, but not on the main release-page)
I just did that for a couple of releases, upload the file for broken downloads with results of the C64 ftp search :)
Since many contain the group name in the file name, uploading the same version as in CSDb is not hard.
So I think it was a good idea :) |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Quote: I just did that for a couple of releases, upload the file for broken downloads with results of the C64 ftp search :)
Since many contain the group name in the file name, uploading the same version as in CSDb is not hard.
So I think it was a good idea :)
I'm not so sure of all those names are correct though. I think you should atleast run it to see if it is the correct one. If the screenshot is missing (or any credits) that can be added at the same time. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: I'm not so sure of all those names are correct though. I think you should atleast run it to see if it is the correct one. If the screenshot is missing (or any credits) that can be added at the same time.
Oh it was just a game, no worries, I am pretty sure I got it correct :) |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Quote: Oh it was just a game, no worries, I am pretty sure I got it correct :)
How can you tell without even looking at it?
You can always run into stuff like these two different releases from Relax: Road Runner +2 and Road Runner +4.
There's little point in csdb if the information here is not validated by the people adding it. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: How can you tell without even looking at it?
You can always run into stuff like these two different releases from Relax: Road Runner +2 and Road Runner +4.
There's little point in csdb if the information here is not validated by the people adding it.
I have more brain mass than you think I have ;)
But thanks for the care! :) |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Quote: I have more brain mass than you think I have ;)
But thanks for the care! :)
It wasn't meant as an insult, no hard feelings? :)
Anyway, there are lots of people reading this thread, so I just wanted to point out the fact that it is quite often that there is incorrect info in the filenames, or atleast incomplete info. (like only one group, when it was an import and bugfix, etc...)
|
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: It wasn't meant as an insult, no hard feelings? :)
Anyway, there are lots of people reading this thread, so I just wanted to point out the fact that it is quite often that there is incorrect info in the filenames, or atleast incomplete info. (like only one group, when it was an import and bugfix, etc...)
If I was taking it personal, I wouldn't have written a ':)' behind the sentence.
Yes, sure, just trust me :) |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Quote: If I was taking it personal, I wouldn't have written a ':)' behind the sentence.
Yes, sure, just trust me :)
I have to, you are a moderator! ;)
No, seriously, I do.
I just want to raise the general awareness about entering correct info.
Anyhow, I think the pokefinder.org-link is quite useful aswell.
|
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: I have to, you are a moderator! ;)
No, seriously, I do.
I just want to raise the general awareness about entering correct info.
Anyhow, I think the pokefinder.org-link is quite useful aswell.
*speechless* :) |
| |
Seven
Registered: Jan 2002
Posts: 202 |
Quote: Is this pokefinder link really a good idea?
It doesn't give unique results, like download-links does.
So it's likely you won't get the release you were looking at in CSDb.
Wouldn't it perhaps encourage people to not upload files?
Also, if there is a release without a download-link and someone finds the file (f.e. at pokefinder), shouldn't he upload it at the release page, so that other people later will get the right (unique) file? If so, i guess the pokefinder-link not really needed (maybe it could be nice to have at the edit page, but not on the main release-page)
My $.05:
$.01) Is it really a good idea to make CSDb the sole resource for C64 files?
My opinion: No. While I appreciate the value this database has, it is no secret that it had its downtime in the past and sure will have some more in the future. That being said I think it's quite dangerous to limit file uploads to this database and forget about the FTP sites that served our purposes well in all these years.
$.02) Is it really a good idea to upload cracks to this database?
My opinion: No. While supplying download links of cracks is a semi-legal thing that, depending on your country, might get you into trouble or not, hosting the files definately IS illegal. And I wouldn't want to see Perff and others being sued and this fine database put offline for that reason.
$.03) Is it a problem that http://ftp.pokefinder.org doesn't supply unique results?
My opinion: Not necessarily. While you might not be able to find Bomb Mania/ONS, you might do just fine with Bomb Mania/SCS*TRC, even though Bomb Mania/SCS*TRC might not be listed on CSDb (yet).
$.04) Would it encourage people to not upload files (to CSDb)?
My opinion: Maybe so, but because of reasons given above I don't necessarily see that as a problem. However, maybe it'd encourage some people to upload to the FTP sites instead. Mass upload to FTP sites is much more convenient than uploading file by file to a webpage anyway.
$.05) Should people use http://ftp.pokefinder.org to find the file, download it and then upload it to CSDb?
http://ftp.pokefinder.org was created to supply a decent way to search multiple FTP sites for a file - if one FTP site doesn't have it, another might. If one FTP site is down, another might not. In the days of point'n'click web-users, http://ftp.pokefinder.org tries to be a convenient web interface for FTP - servers that're available for exactly that: downloading (and uploading) files. It wasn't Count Zero's intention to create a tool to help cannibalize the FTP sites and make them obsolete.
|
| |
tlr
Registered: Sep 2003
Posts: 1790 |
@seven: Maybe there should be a way to mirror csdb? The biggest point of csdb is that there is meta-information about who did what. This will be mostly lost with ftp-only archives. It's ofcourse the meta-info that makes it more work to upload here, but I think it is worth it.
As for the cracks I agree to some extent. The downloads can always be quickly disabled though, they are all flagged as "C64 crack" in the database.
I don't agree with the different releases of cracks being equivalent. As for history, the text in intro scrollers, ratings, etc... is very interresting. If the cracks weren't allowed in here, this could possibly be solved by ripping the intros, or just the whole scroll-text. But that's even more work.
I agree that it would be good to have the stuff on ftp aswell, but couldn't that be taken care of by a partial dump of files from csdb?
Just some thoughts... |
| |
Seven
Registered: Jan 2002
Posts: 202 |
Quote: @seven: Maybe there should be a way to mirror csdb? The biggest point of csdb is that there is meta-information about who did what. This will be mostly lost with ftp-only archives. It's ofcourse the meta-info that makes it more work to upload here, but I think it is worth it.
As for the cracks I agree to some extent. The downloads can always be quickly disabled though, they are all flagged as "C64 crack" in the database.
I don't agree with the different releases of cracks being equivalent. As for history, the text in intro scrollers, ratings, etc... is very interresting. If the cracks weren't allowed in here, this could possibly be solved by ripping the intros, or just the whole scroll-text. But that's even more work.
I agree that it would be good to have the stuff on ftp aswell, but couldn't that be taken care of by a partial dump of files from csdb?
Just some thoughts...
Mirroring CSDb is a tricky thing...
CSDb runs on PostgreSQL, and while PostgreSQL allows master/slave replication it severely limits options of mirrors since most webhosters nowadays will only give you MySQL. With MySQL 5 replication would become a possibilty aswell, but it still leaves us with "just a copy" of the database and in a case of downtime we'd end up with a read-only database.
...unless you want to set up chained replication, but I have to admit that I have my doubts if that'd be advisable over the "public" internet.
A mirror also makes changes to the programming more difficult and opens new doors for abuse (you might recall the MacGyver discussion). Guys supplying a mirror would have to be _really_ reliable.
True, downloads for cracks could quickly be disabled. But then? They wouldn't be available anymore when people just upload to CSDb instead of uploading to the FTP sites... where's the point in that? It stays a fact.. mass-dowloading from an FTP site is way more convenient (and FTP sites are a lot more "mirror-able") than webpage downloads.
As for exchangeability of cracks.. I do realize that if what you were looking for was the intro-scrolltext in the SCS*TRC version of Bomb Mania the Onslaught version won't cut it, but if you want crack comparisons, Mason's project (once he's finished) will be much more suitable for that.
But then again... LOOK, according to CSDb there IS no SCS*TRC version of Bomb Mania... in fact, there is no cracked version at all... wonder how THAT happened *smirks*
|
| |
tlr
Registered: Sep 2003
Posts: 1790 |
I think it would be a good first step if there was read only mirroring of csdb. I understand that read/write access from multiple sites is complex, both technically and securitywise.
I also agree that it is easier to download lots of stuff from ftp servers, and probably many people want to do this. But I also believe that there is quite a large crowd interrested in the scene history. For the latter use it's primarily the meta-information that is interresting, and here csdb is invaluable!
If the files of csdb where exported using an automated naming scheme (<group>_-_<release>.zip for example), we could have the best of both worlds.
|
| |
CyberBrain
Administrator
Posts: 392 |
Quoting Seven
$.03) Is it a problem that http://ftp.pokefinder.org doesn't supply unique results?
My opinion: Not necessarily. While you might not be able to find Bomb Mania/ONS, you might do just fine with Bomb Mania/SCS*TRC, even though Bomb Mania/SCS*TRC might not be listed on CSDb (yet).
$.04) Would it encourage people to not upload files (to CSDb)?
My opinion: Maybe so, but because of reasons given above I don't necessarily see that as a problem. However, maybe it'd encourage some people to upload to the FTP sites instead. Mass upload to FTP sites is much more convenient than uploading file by file to a webpage anyway.
$.05) Should people use http://ftp.pokefinder.org to find the file, download it and then upload it to CSDb?
http://ftp.pokefinder.org was created to supply a decent way to search multiple FTP sites for a file -
if one FTP site doesn't have it, another might. If one FTP site is down, another might not. In the days
of point'n'click web-users, http://ftp.pokefinder.org tries to be a convenient web interface for FTP -
servers that're available for exactly that: downloading (and uploading) files. It wasn't Count Zero's
intention to create a tool to help cannibalize the FTP sites and make them obsolete.
If you were referring to my post here, my point wasn't that CSDb should make FTP-sites obsolete, or people should stop using pokefinder. You can still use pokefinder even if there isn't a link to it from CSDb on the bottom of each release-page, and as you guys pointed out, FTP-sites are useful in another way than CSDb is.
My point is that if there is a release (in CSDb) without download-link, you use the pokefinder-link to search for the file and you actually find the correct file, you should add a download-link (or upload it - doesn't matter) so other people later can get the right file. So the pokefinder-link is only useful for the first person who clicks it, as he "should" add a download-link to the correct file.
What i'm saying is just that i'm not sure that the pokefinder-thing really fits into CSDb's framework.
I'm not saying that pokefinder is useless or something like that.
But if people like it, i guess it's fine :) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Quoting Seven
$.03) Is it a problem that http://ftp.pokefinder.org doesn't supply unique results?
My opinion: Not necessarily. While you might not be able to find Bomb Mania/ONS, you might do just fine with Bomb Mania/SCS*TRC, even though Bomb Mania/SCS*TRC might not be listed on CSDb (yet).
$.04) Would it encourage people to not upload files (to CSDb)?
My opinion: Maybe so, but because of reasons given above I don't necessarily see that as a problem. However, maybe it'd encourage some people to upload to the FTP sites instead. Mass upload to FTP sites is much more convenient than uploading file by file to a webpage anyway.
$.05) Should people use http://ftp.pokefinder.org to find the file, download it and then upload it to CSDb?
http://ftp.pokefinder.org was created to supply a decent way to search multiple FTP sites for a file -
if one FTP site doesn't have it, another might. If one FTP site is down, another might not. In the days
of point'n'click web-users, http://ftp.pokefinder.org tries to be a convenient web interface for FTP -
servers that're available for exactly that: downloading (and uploading) files. It wasn't Count Zero's
intention to create a tool to help cannibalize the FTP sites and make them obsolete.
If you were referring to my post here, my point wasn't that CSDb should make FTP-sites obsolete, or people should stop using pokefinder. You can still use pokefinder even if there isn't a link to it from CSDb on the bottom of each release-page, and as you guys pointed out, FTP-sites are useful in another way than CSDb is.
My point is that if there is a release (in CSDb) without download-link, you use the pokefinder-link to search for the file and you actually find the correct file, you should add a download-link (or upload it - doesn't matter) so other people later can get the right file. So the pokefinder-link is only useful for the first person who clicks it, as he "should" add a download-link to the correct file.
What i'm saying is just that i'm not sure that the pokefinder-thing really fits into CSDb's framework.
I'm not saying that pokefinder is useless or something like that.
But if people like it, i guess it's fine :)
I think it's matter wether to add a link or upload it.
Uploading it will be sure it's there and not lost, like a lot of releases that only got linked often result into a broken download link! :(
That is why I always do both, adding link and uploading the file :) |
| |
Seven
Registered: Jan 2002
Posts: 202 |
@tlr:
I'm all with you about the meta information and I agree that FTP servers are simply not the right platform for those, but they ARE the ideal platform for files and CSDb is not.
Yes, in theory exporting all the files with a halfway decent naming scheme would be a possibility, however, different FTP sites have different naming conventions for their files, and there's alway some kind user (read: admin) interaction required for moving the files to the proper directories.
I guess someone'd have to talk with an FTP admin or two to find out if they're interested or not.
... it'd still leave us with CSDb's "censorship" though ;)
@cyberbrain:
I just don't see people actually looking for every single file to add the download link when most can't even be hassled to upload the file to an FTP site. In a way CSDb is doing exactly that, trying to make FTP sites obsolete by providing a fool-proof upload feature, and in doing so actually somewhat increasing the danger of stuff eventually getting lost by concentrating everything in one place.
In the beginning CSDb was content using links only and all was fine, FTP admins made sure files were properly named and had the correct format (zipped d64, or - back in the day - gzipped prg or lnx/z64). Now, on CSDb, we have files named V2.d64 (anyone wanna guess what that unpacked disk image actually contains?) - and entries that 'forcefully' have their download links removed or are removed entirely.
Noone is forced to use http://ftp.pokefinder.org , if you want, remove it again, I just think it should be up to the user whether he'd like an alternative to the supplied (or in most cases: not supplied) download links on CSDb - with the positive side effect that some newbies might actually discover FTP when most people nowadays think that the World Wide Wait, Email and AOL is all there is to the internet. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
I would suggest to keep it the way it is now and then wether really any of those "laziness and other" fears would happen. |
| |
Perff
Administrator
Posts: 1679 |
a little remark..
Just to make it clear: CSDb is NOT trying to kill off ftp-sites or anything.
In fact, if some ftp-admins out there would want it, I would be happy to discuss with them how to make an export or whatever of the files uploaded to CSDb in any given structure based on the info, to put on the ftp-sites.
(Currently there has been uploaded 10585 unique files) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
now that sounds like a good idea to me... should be easy enough to automatically rename them according to whatever ftp sites standard and upping them there (preferably on more than one ftp site for redundancy) |
| |
Seven
Registered: Jan 2002
Posts: 202 |
Thanks for clearing that up, Perff. The thing is, even though CSDb doesn't aim for it (and I never honestly believed that this might be the case), that's what, to a degree, is happening at the moment.
However, as Groepaz said, exporting the files for an FTP site or two sounds like a good idea.
Any FTP admins around? =D
|
| |
tlr
Registered: Sep 2003
Posts: 1790 |
It would be a cool feature if the file was accompanied by a text file containing a text printout of the meta-info. Maybe just put it all into a zip?
Sure, the meta-info will loose it's links, but it's probably useful to have this if something gets lost and someone needs to re-add it somewhere (here?). |
| |
Count Zero
Registered: Jan 2003
Posts: 1932 |
Well, here we go.
I didn't follow the complete discussion due to lack of motivation, but will, since there seems a real demand, fix a few things and extend ftp.pokefinder.org this weekend.
So my 2 cent is - keep it like it is. :)
Btw, last days of October 119 requests were made and up til now 182 requests in November. If it someday exceeds some 10 or hundred thousand requests coming from csdb we might consider changing the strategy here. :)
l8r
Count Zero/CyberpunX/SCS*TRC |
| |
Seven
Registered: Jan 2002
Posts: 202 |
Quote: It would be a cool feature if the file was accompanied by a text file containing a text printout of the meta-info. Maybe just put it all into a zip?
Sure, the meta-info will loose it's links, but it's probably useful to have this if something gets lost and someone needs to re-add it somewhere (here?).
Errm... but you do realize that once there's updates made to the CSDb entry, that won't be reflected in your "meta-info" file since you won't find a single FTP administrator who will zip up all files with "new" information on a regular basis, yes? =D
besides... re-add? hello? There are BACKUPS of CSDb. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Errm... but you do realize that once there's updates made to the CSDb entry, that won't be reflected in your "meta-info" file since you won't find a single FTP administrator who will zip up all files with "new" information on a regular basis, yes? =D
besides... re-add? hello? There are BACKUPS of CSDb.
As far as I know, there are no backups of the hosted files. |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Quote: Errm... but you do realize that once there's updates made to the CSDb entry, that won't be reflected in your "meta-info" file since you won't find a single FTP administrator who will zip up all files with "new" information on a regular basis, yes? =D
besides... re-add? hello? There are BACKUPS of CSDb.
Not only did I realize it, I also wrote a comment about it in my post. ;)
The text-file would primarily be useful to remedy part of the problems with ftp-archives. If nobody wants it we can just do like it is now on all the current ftp-sites, i.e make the filename from stripped down meta-info.
I'm just providing friendly suggestions here... :) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Not only did I realize it, I also wrote a comment about it in my post. ;)
The text-file would primarily be useful to remedy part of the problems with ftp-archives. If nobody wants it we can just do like it is now on all the current ftp-sites, i.e make the filename from stripped down meta-info.
I'm just providing friendly suggestions here... :)
See it this way; He welcomed you with a friendly "hello" to this round ;) |
| |
Seven
Registered: Jan 2002
Posts: 202 |
I thought tlr was talking about readding the meta-information to the database, not the files. Readding the files would be kind of nonsense since, once the files have been exported for the FTP, it'd be pretty easy to simply add the download links to the FTP with the same naming scheme used for exporting the files in the first place.
And while I don't know about the files, I know for sure that there's backups of the CSDb database and screenshots.
|
| |
Perff
Administrator
Posts: 1679 |
Quote: As far as I know, there are no backups of the hosted files.
I can assure you there are backups of IT ALL.
Both code, database, screenshots and photos AND uploaded files.
The last backup of the uploaded files was about 500mb in tar.gz
The only thing not backed up is whatever external links points to. :) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: I can assure you there are backups of IT ALL.
Both code, database, screenshots and photos AND uploaded files.
The last backup of the uploaded files was about 500mb in tar.gz
The only thing not backed up is whatever external links points to. :)
OK, then my knowledge was old or wrong, sorry O:) |
| |
Perff
Administrator
Posts: 1679 |
Quote: OK, then my knowledge was old or wrong, sorry O:)
Probably old. :)
When the upload file feature was first made, the files was uploaded to an external ftp-site to spare space on the server CSDb runs on. That however often went offline for long periods of time, so I found out that it was ok to use a bit more space on the same server as CSDb runs on, where the files are now stored. That makes it a bit easier to make a backup. :) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Probably old. :)
When the upload file feature was first made, the files was uploaded to an external ftp-site to spare space on the server CSDb runs on. That however often went offline for long periods of time, so I found out that it was ok to use a bit more space on the same server as CSDb runs on, where the files are now stored. That makes it a bit easier to make a backup. :)
Another reason to encourage people to upload the files instead of only linking to external sources! |
| |
Seven
Registered: Jan 2002
Posts: 202 |
Quote: Another reason to encourage people to upload the files instead of only linking to external sources!
Why is that a reason to upload files to CSDb?
The only thing that show is that, if you want an FTP server to host the files, choose a reliable one.
You don't get it, do you? Doing a backup of an FTP site is the easiest you can get.. provided you have an FTP client and don't try downloading file by file with your Internet Exploder. And that way _anybody_ can make a backup, and not just Perff.
Why do I sometimes get the feeling that I'm talking to walls in here... 8r
|
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Why is that a reason to upload files to CSDb?
The only thing that show is that, if you want an FTP server to host the files, choose a reliable one.
You don't get it, do you? Doing a backup of an FTP site is the easiest you can get.. provided you have an FTP client and don't try downloading file by file with your Internet Exploder. And that way _anybody_ can make a backup, and not just Perff.
Why do I sometimes get the feeling that I'm talking to walls in here... 8r
I have a DVD full with backups of FTP servers, just for your information =)
But not everybody has backups of all FTPs and often there are also linked to other sources than FTPs, e.g. http sources. Often they change (perfect example OOO releases) and then they are lost, while files uploaded to CSDb never will be lost because:
a) They are now stored on CSDb server, no longer on another external server.
b) Perff has updates of all such hosted files!
--
I want to thank at this moment, to all the sceners that send me PMs and coorperate to get sources for releases added to CSDb.
I try to reply to all in a timely manner.
Other than that, feel free to email me if you need to send files. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
the point is that file should be stored decentralized and redundant. if everything is at one place there is a good chance that at one point everything will get lost at once.
and ofcourse, downloading stuff from csdb is painful, while downloading from ftp is fast and easy. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: the point is that file should be stored decentralized and redundant. if everything is at one place there is a good chance that at one point everything will get lost at once.
and ofcourse, downloading stuff from csdb is painful, while downloading from ftp is fast and easy.
painful? It's only a few KBs per release and even if CSDb is in its very slow times, downloads anyway take only seconds because of small file sizes.
You must have been sarcastic here! :D
Apart from that, thanks for agreeing on the redundancy point! :) |
| |
cba
Registered: Apr 2002
Posts: 935 |
Quote: painful? It's only a few KBs per release and even if CSDb is in its very slow times, downloads anyway take only seconds because of small file sizes.
You must have been sarcastic here! :D
Apart from that, thanks for agreeing on the redundancy point! :)
He means that downloading for example every demo from
Crest is very slow here !, doing it from a FTP server is
just way easier.
Niels |
| |
Seven
Registered: Jan 2002
Posts: 202 |
@nafcom:
So files uploaded to CSDb will never get lost, because there's one person in the world who has a backup.
If you have the time, please come up with an explanation how that is safe and redundant.
... or better yet, next time think before you post. I'm still waiting for even a vague hint that you actually understood what we're talking about. |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Quote: he.
I already did that - removing non alphnumeric characters if you havn't noticed. (Perhaps not all of them - had a small problem making a simple php-routine that removes all non alphanumeric chars, but not special letters like æøåôêá and what different contries have thought up. Anyone got a 100% idiot-proff thing to do that?)
I could just see where it was going. Removing simple chars is one thing. Next search for special abrivations and remove them or convert them. Also look for other words like preview, fix etc. and do stuff to them too. The next post would probably have sounded a bit like "Make a routine that understands the meaning of the title, and make a smart search from that understanding." ;)
.. or perhaps it's just me who is lazy.. hmm.. :):)
The funny thing is: you're describing one of the biggest problems at the project I'm currently working on (not for c64, but for my jon). I work at a directory service (biggest one in Holland, actually we're related to TDC which should be known to Scandinavian people as well), and there it's very important to somehow transform the user input into a query which will give just the right type of results. For that we use many different techniques, like synonyms, stopwords but also fuzzy search logic etcetera. Some of this we do ourself, for other parts of this system we use the expertise of an American partner.
So based on experience I can tell you: you've got your work cut out for you ;) The bright side is ofcourse that your quest is a lot easier than ours :) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: @nafcom:
So files uploaded to CSDb will never get lost, because there's one person in the world who has a backup.
If you have the time, please come up with an explanation how that is safe and redundant.
... or better yet, next time think before you post. I'm still waiting for even a vague hint that you actually understood what we're talking about.
I don't see a reason why not to trust Perff here |
| |
tlr
Registered: Sep 2003
Posts: 1790 |
Quote: Why is that a reason to upload files to CSDb?
The only thing that show is that, if you want an FTP server to host the files, choose a reliable one.
You don't get it, do you? Doing a backup of an FTP site is the easiest you can get.. provided you have an FTP client and don't try downloading file by file with your Internet Exploder. And that way _anybody_ can make a backup, and not just Perff.
Why do I sometimes get the feeling that I'm talking to walls in here... 8r
@seven: I agree with your point. It would be great to have external backups.
Something might, god forbid, happen to the backups that Perff keeps for instance...
What I don't agree with is how it should be done.
I would find it a bit tedious (and probably others with me) to upload the files to one place (several), then add the entries to another, and add a link to the first. The result would probably be exacly what it is today: some upload to the ftp-servers, and some upload here.
What I have suggested in a previous post is a mirror by exporting all files from csdb to multiple ftp-servers.
I would also like to see the full database mirrored (read-only ofcourse). This could be in the form of database dump in a tar-archive, just so the information won't get lost. Somebody could put up a mirror site later.
Obviously there are a lot of conflicting opinions about this, and Perff is the one who has to implement it, so
it's his call. But I think this would solve many of the problems/risks discussed here. |
| |
Seven
Registered: Jan 2002
Posts: 202 |
@nafcom: You still don't get it, do you? This is not about trust. This is about safety and redundancy, and neither is accomplished by a single person with a backup, no matter how much you trust that person.
@tlr: Perff already said he wouldn't mind exporting the files for an FTP to host them, so I think that's the least of the problems.
The bigger problem will be a copy of the database, especially after the MacGyver incident showed that there's a lot of trust issues to be taken into consideration. How much of the functionality should be left intact with the copy, how much of the database has to be copied for that, and does that include user account information, passwords and (hidden) voting information?
I think there's a lot of work to be done for a "safe" copy of CSDb and even if Perff can be talked into doing that work, I wouldn't count on it anytime soon. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: @nafcom: You still don't get it, do you? This is not about trust. This is about safety and redundancy, and neither is accomplished by a single person with a backup, no matter how much you trust that person.
@tlr: Perff already said he wouldn't mind exporting the files for an FTP to host them, so I think that's the least of the problems.
The bigger problem will be a copy of the database, especially after the MacGyver incident showed that there's a lot of trust issues to be taken into consideration. How much of the functionality should be left intact with the copy, how much of the database has to be copied for that, and does that include user account information, passwords and (hidden) voting information?
I think there's a lot of work to be done for a "safe" copy of CSDb and even if Perff can be talked into doing that work, I wouldn't count on it anytime soon.
No, you just take stuff too personal and too agressive. Cool down.
|
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Okay, let me add a few more cents (..) to this discussion then. I think there's a bigger 'problem' here: CSDB has become too popular for its own good and is in danger of colapsing under its own weight. Perff & co do a fantastic job, came up with a cool idea and actually realised it (which is more than I can say of my own ideas).
However, Perff, as he himself has stated several times, is not a professional software-architect. Hence, there have been problems with trying to make this into something it was never meant to be. One way of handling that is by doing what's done right now: whenever a new problem or situation arises, some kind of hack is performed. Most of the times this works (because Perff is quite talented I might add) but in the long run it will cause new problems.
The best solution would be to simple release a new version, build from the ground up. Perff doesn't have to do that by himself: many of us do this stuff for a living and could help him with it. We have database experts, usability experts, search engine experts in our midst. It would also be very important to actually think things over before implementing them, to come up with a flexible system which at least is prepared to interact & be integrated with all the systems already out there that are worth interacting & integrating with (..). Ofcourse this would bring csdb into a whole new area, that of 'semi-professional' web-application, but I think it would be worth it.
By the way, even though yes, I am one of those professionals, I'm not claiming I should be the one to help him with it, I'm sure there are several others here who are just as qualified as I am. But if you need me Perff, just holla at me ;) |
| |
iopop
Registered: Dec 2001
Posts: 317 |
Why do I always have to agree with TDJ?
No, seriously, the idea is not bad at all and Ive been having the same idea in my mind for a few years. Especially when it comes to the Crack and "umbrella group" situation. Which if I understod correctly is impossible from a db perspective.
And yes, since I whine about it, Im ofcourse obligated to help. |
| |
cba
Registered: Apr 2002
Posts: 935 |
Quote: Why do I always have to agree with TDJ?
No, seriously, the idea is not bad at all and Ive been having the same idea in my mind for a few years. Especially when it comes to the Crack and "umbrella group" situation. Which if I understod correctly is impossible from a db perspective.
And yes, since I whine about it, Im ofcourse obligated to help.
I also would fully support the idea to make a new CSDB,
And as TDJ already says, so many of us are able to help out
in one way or another.
Who is willing to make the first step?
Niels |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: I also would fully support the idea to make a new CSDB,
And as TDJ already says, so many of us are able to help out
in one way or another.
Who is willing to make the first step?
Niels
Err... There is a CSDb version 2 planned, Perff asked several times in the forum for help, got no help. Funny that now people talk here now about a new CSDb! =) |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Quote: Err... There is a CSDb version 2 planned, Perff asked several times in the forum for help, got no help. Funny that now people talk here now about a new CSDb! =)
Where did he ask that? I know I have offered him help several times, but didn't get a reply. So please point me to the posts he made .. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Where did he ask that? I know I have offered him help several times, but didn't get a reply. So please point me to the posts he made ..
Uh, I can't really remember when that was, etc, please forgive me. - But I am sure we spoke about that even on ICQ a couple of times, also with KBS (co-admin of CSDb). |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Quote: Uh, I can't really remember when that was, etc, please forgive me. - But I am sure we spoke about that even on ICQ a couple of times, also with KBS (co-admin of CSDb).
You're forgiven ;) Anyway, who cares if there was no reaction back then - there is now. So hopefully one of you can tell us how far you are in that process, and if you still need help.
Maybe it would be a nice idea to have people willing to help 'state' their qualifications somewhere (in a new thread perhaps)? |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: You're forgiven ;) Anyway, who cares if there was no reaction back then - there is now. So hopefully one of you can tell us how far you are in that process, and if you still need help.
Maybe it would be a nice idea to have people willing to help 'state' their qualifications somewhere (in a new thread perhaps)?
"One of you" excludes me.
CSDb crew is Perff, KBS, Cyberbrain, not me, sorry.
I am not (neither officially nor inofficially) a part of this crew, even when some people think that. :)
So, I suggest you contact one or all 3 of them, or wait till one of them replies to this thread here, again. |
| |
cba
Registered: Apr 2002
Posts: 935 |
Very nice this pokefinder is added but so far I still
see alot of ppl adding entries without links (while the link can be found if you click on the pokefinder)
Is there not a way we can force people only to add entries
that have a file / link ?
Niels |
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
thats not a good idea imho...next someone comes and wants to force ppl entering all credits, and a screenshot etc blabla. imho an entry, even without further link or info, is better than no entry at all. BUT there should be more/better features to aid in maintaining existing entries.... eg something that allows me to quickly go through all entries that match a certain criteria and then let me add those links (or whatever). right now its pretty much a pain in the ass to do stuff like this. |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Quote: Very nice this pokefinder is added but so far I still
see alot of ppl adding entries without links (while the link can be found if you click on the pokefinder)
Is there not a way we can force people only to add entries
that have a file / link ?
Niels
Like Groepaz said, not a good idea.
Because, if you would force it, what about entries you know exist but don't have the file of? Or maybe somebody has a lot of c64 disks with stuff not in here yet, but no way to transfer it? Should those releases be excluded from the database?
IMHO this database is about the *information* on releases first, not the actual releases themselves. |
| |
cba
Registered: Apr 2002
Posts: 935 |
Sure I agree that it should be possible to add entries
where you don't have a file/link from, I've done that
with several SCS*TRC games that are still missing from
my collection.
For me it's weird that if you've already taken the effort in adding something to csdb (in some cases even added credits + screenshot) that you just leave out the file ?
Niels |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Maybe it's easier for somebody to make a screendump from winvice, than to seperate the file from the other files on the d64 image, and create a new image? Dunno, could be a reason .. Although I agree with you people should make that extra effort. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Maybe it's easier for somebody to make a screendump from winvice, than to seperate the file from the other files on the d64 image, and create a new image? Dunno, could be a reason .. Although I agree with you people should make that extra effort.
In such cases, I am sure we could live with PRG files :) |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Quote: In such cases, I am sure we could live with PRG files :)
Which still takes some effort. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Which still takes some effort.
For single file releases just one command (Export as .PRG).
But if here are only lazy butts, why are we here then doing anything at all? ;) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
imho some kind of feature to add a release and automagically upload it to some ftp server (or even better, several ones) is really needed here. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: imho some kind of feature to add a release and automagically upload it to some ftp server (or even better, several ones) is really needed here.
Good idea, but the problem I see is, that all FTP servers first send to /pub dir then move it and rename the file. So the system has to take care that the file gets automatically renamed into the file naming format each single FTP server rquires.
Sounds to me like heavy |
