| |
Style
Registered: Jun 2004
Posts: 498 |
Anyone producing "updates" from CSDB?
Hey all
I have a problem looking for a solution - like many others, I have an 1541U with games and demos (mainly from gamebase and some demo collections I forget) but I have no way of keeping it updated.
Is anyone producing an "offical" CSDB image in some way, if only for demos? Would be really nice maybe to have a read-only SVN server backed onto the CSDB database or similar so we can all resync every now and then?
I could rape the CSDB front end, but Im sure itd violate some user policy somewhere and it's a nasty thing to do.
Thanks! |
|
| |
Shine
Registered: Jul 2012
Posts: 349 |
Quoting Style
...
Would be really nice maybe to have a read-only SVN server backed onto the CSDB database or similar so we can all resync every now and then?
...
That would be a VERY nice idea!!! :) But as i know CSDb so far, that never would be happen. :( *sigh* |
| |
Oswald
Registered: Apr 2002
Posts: 5086 |
jailbird could do it, just look at what an awe gfx site he piggy backed on csdb :) |
| |
Shine
Registered: Jul 2012
Posts: 349 |
<offtopic>
<commercial>
http://www.thepixellingcow.com
</commercial>
</offtopic> |
| |
jailbird
Registered: Dec 2001
Posts: 1578 |
Currently the only option is to parse the site by using its web services. I think C64 Scene for iPad works the same way as well.
http://csdb.dk/help.php?section=usage#10 |
| |
Romppainen
Account closed
Registered: Apr 2008
Posts: 40 |
...and when people would find out such a possibility and start mass sucking the whole database it would be a bandwidth nightmare. |
| |
cba
Registered: Apr 2002
Posts: 935 |
Demo wise just download everything from my FTP site : ftp.scs-trc.net/pub/c64/Demos , almost 100% synced with CSDB. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11359 |
if you really want it, pull news via RSS, download new files. couple of lines of script. (do the same with pokefinder if you also want to get whats not added to csdb).
that said, the backend actually uses ftp for the files - but i am not sure if it could take the additional bandwidth |
| |
Style
Registered: Jun 2004
Posts: 498 |
Ahhh thanks CBA, Id actually forgotten where Id gotten the original demo tree ages ago, but it was obviously your FTP. Ill just sync your ftp site :) Glad you're keeping it up to date!
thanks bud |
| |
Sasq
Registered: Apr 2004
Posts: 156 |
http://swimsuitboys.com/csdb/
Although it looks like it's time for an update... |
| |
Feltzkrone
Registered: Oct 2013
Posts: 4 |
Unluckily the web service still produces problematic XML when unsafe characters are present in the content:
http://csdb.dk/webservice/?type=release&id=97105&depth=1
Innstead of having several users making their own syncs by parsing the web service output or the HTML pages and sucking dry server bandwidth + producing lots of database accesses... Why not consider providing backups of the whole database (I guess there's a MySQL, PostgreSQL or some other open source DBMS used as the backend) and torrents for the uploaded files on a snapshot basis? Apart from that: It's always good to have backups in more than one place only.
There might be reasons against my suggestion which I currently don't see, hence I'm asking. :) |
| |
Mr. SID
Registered: Jan 2003
Posts: 424 |
Why is that problematic? What would you expect the XML output to look like for that release? |
| |
chatGPZ
Registered: Dec 2001
Posts: 11359 |
Quote:There might be reasons against my suggestion
how much time are you willing to spend on a regular basis into making it happen ? :=P |
| |
Feltzkrone
Registered: Oct 2013
Posts: 4 |
@Mr. SID: "&lt;--&gt;" should be "<-->". Apparently here first the < and > and characters have been translated to the < and > entities. What's wrong is that there is a subsequent translation of the & characters to & - it's valid XML but it's wrong data.
@Groepaz: If it's possible to run Java code on the server (or even a Tomcat webserver / servlet container) I'm happy to offer my help by providing SOAP webservice(s) which queries the database and provides results in flawless XML using a well defined interface and message contract. Querying data release by release and playing with the depth levels isn't exactly performant if somebody wants data for all releases. Here a webservice which allows querying multiple releases or a database dump would come in more handy and efficient while being gentle regarding server resources.
And I could also provide tools which create database table snapshots and even provide the delta between two snapshots (although I don't think that it's necessary with a database which takes a tiny fraction of the size a Discogs snapshot is). Both are things which I'm dealing regularly with at work since years. For example a db table made up of customer ID, product ID, sales, proceeds, amount for example is processed at about 200,000 records per second on my workstation computer. Nowadays Java isn't exactly slow anymore. ;)
PHP is not my cup of tea by the way, here are some reasons (not written by me but I share the author's opinion in many cases): http://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/
:P |
| |
Perplex
Registered: Feb 2009
Posts: 255 |
Well, SOAP isn't exactly a shining example of great, modern webservice design either. ;-) |
| |
Feltzkrone
Registered: Oct 2013
Posts: 4 |
But at least it's standardized. :) What would you recommend instead? |
| |
Perplex
Registered: Feb 2009
Posts: 255 |
Some RESTful interface using HTTP GET with response status and headers for metadata and the body reserved for the actual payload serialized as JSON would be nice. |
| |
Merman
Registered: Dec 2002
Posts: 140 |
What about a monthly update by torrent? |
| |
Feltzkrone
Registered: Oct 2013
Posts: 4 |
In my opinion all the info stored in the DB is at least as valuable as the files themselves (thanks to the work done by so many people), having all the files alone would be nice but it's a rather incomplete "backup" then.
SOAP or RESTful: Not much experience with REST but I'm sure this wouldn't be a problem... so why not offer both? :)
I hope that Perff will sooner or later post a reply. Provided he would trust me and give me a backup of the DB (or a datawise stripped-down-version) plus the server architecture would allow for Java then most of the work could be done without taking up too much of his time. Of course the DBMS would have to be accessible using JDBC. |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Considering SOAP vs. REST; both have its merits, but in this particular case I would recommend the latter, as its hypermedia aspect fits the structure of csdb perfectly. Also, end-to-end security and reliable messaging are not really a factor. |
| |
bepp
Registered: Jun 2010
Posts: 265 |
+1 for REST |
| |
JackAsser
Registered: Jun 2002
Posts: 2014 |
Quote: +1 for REST
+1 for REST of course. Kill SOAP. REST+json as data carrier makes it extremely easy to use even from simple shell-scripts and directly compatible with http XHR-requests. |
