| |
Wonder-Boy
Account closed
Registered: Apr 2007
Posts: 3 |
Bulk download?
Hi, is there a way to download a number of releases at the same time? Downloading every release individually can be a bit of a chore. I would like to be able to download for instance "all C64 demos from 2007" in one go. |
|
| |
Jammer
Registered: Nov 2002
Posts: 1335 |
well, csdb server can go down due to frequent refreshing ('edge of disgrace' panic i.e.) thus this option would freeze it totally, i guess. |
| |
iAN CooG
Registered: May 2002
Posts: 3186 |
Quote: Hi, is there a way to download a number of releases at the same time? Downloading every release individually can be a bit of a chore. I would like to be able to download for instance "all C64 demos from 2007" in one go.
There are such things called "batch downloaders": wget, flashget and getright are some of those. An handy plugin for Firefox is Down them all.
Just be gentle to the servers, abuse of these programs lead to server overloads. Allow only one download at a time, and you're done.
|
| |
Wonder-Boy
Account closed
Registered: Apr 2007
Posts: 3 |
Thank you, I will check that out. |
| |
Wonder-Boy
Account closed
Registered: Apr 2007
Posts: 3 |
I tried Down them all, and it seems it only works if you have download links on the page, which you get if you use the simple search function. But if you use the advanced search in order to find e.g. demos from a particular year, you do not get these download links on the results page, so I think these batch downloaders won't work for me. Thanks anyway. |
| |
assiduous
Account closed
Registered: Jun 2007
Posts: 343 |
yes there should be the disk icon  with a download link next to titles in an advanced search result. |
| |
iAN CooG
Registered: May 2002
Posts: 3186 |
strangely enough, I batch-download every day from here and other c64 sites without any problem with both down them all and flashget =)
*download.php* is the key ;) |
| |
Zone
Account closed
Registered: Nov 2002
Posts: 7 |
Come to think of it: It would be cool if some group started to release stuff focusing on certain historical periods of the C64 scene.
For example, imagine a collection of all the releases from the party in Bergen 1990, along with some extra features like pictures, charts and contemporary mag reports. Or let's say, memorabilia from cracker wars of 1990! Pure nostalgia put together to celebrate and remember, and not least to educate and inspire us assholes who missed out on all the fun back then. :)
Anyway, I also do batch downloads from CSDb and I fully acknowledge that site admins are usually not very happy about this! I put in a few seconds of random delay between downloads, sometimes even minutes, to be nicer to the server, but still.
|
| |
iAN CooG
Registered: May 2002
Posts: 3186 |
I have noticed that csdb already has a multiple download prevention check, 2 downloads don't start from the same ip, the 2nd download starts only when the 1st is almost finished. |
| |
Earthshaker
Registered: Sep 2002
Posts: 118 |
Quote: strangely enough, I batch-download every day from here and other c64 sites without any problem with both down them all and flashget =)
*download.php* is the key ;)
How exactly does this work? Everytime i try to do this i get a "you are creating a huge load" warning. I tried DTA and have set it to only do 1 file at a time. I eventually would like to download all the releases here, but without creating problems for the site. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11352 |
set a proper delay between requests, like a couple seconds or so |
| |
AlexC
Registered: Jan 2008
Posts: 298 |
Why not provide periodically a dump of whole release database? This may be later distributed as Torrents or uploaded to AWS servers for example.
Just my 2 cents... |
| |
hedning
Registered: Mar 2009
Posts: 4720 |
CSDb is already available as bulk, with weekly updates. I am not sure the collection is available publically yet, though. I like the idea to have the database mirrored and available for bulk download elsewhere. |
| |
Thierry
Registered: Oct 2009
Posts: 48 |
Quote: CSDb is already available as bulk, with weekly updates. I am not sure the collection is available publically yet, though. I like the idea to have the database mirrored and available for bulk download elsewhere.
+1 !! |
| |
Earthshaker
Registered: Sep 2002
Posts: 118 |
Quote: +1 !!
+1 |
| |
Smasher
Registered: Feb 2003
Posts: 519 |
wtf is this lame "+1" mania? can't we change that in a better proper nerd syntax? like "INX" or "ADC#$01" ? :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11352 |
first sane post in this thread :) |
| |
Moloch
Registered: Jan 2002
Posts: 2924 |
/me clicks the Like button |
| |
T.M.R
Account closed
Registered: Dec 2001
Posts: 749 |
Quoting ZeSmasherwtf is this lame "+1" mania? can't we change that in a better proper nerd syntax? like "INX" or "ADC#$01" ? :)
Isn't +1 just adding a trainer...? |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
I have built a tool that does all this,
available at http://freabesdreamnet.asuscomm.com:1337/assembly or https://facebook.com/assembly64
Will be fun to see if it will cope with the load :-) |
| |
hedning
Registered: Mar 2009
Posts: 4720 |
Thanx Scooby. That was what I was talking about. :) Nice that you came out of the closet! |
| |
Thierry
Registered: Oct 2009
Posts: 48 |
Exelent ! , thanks for this
Its possible to download all releases of 1 scene group ? example : SoftGuru
thanks |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
This is already available in the CSDB_demos item in the assemblytool.All you need to do is rightclick "install" on that particular item, and all groups will be installed on your favourite sd-card/folder with the structure /<A-Z>/<GROUP>/<DEMO>. Important is to NEVER change the name of the folders created OR delete the "db" folder, since this will be used when new content is available |
| |
Thierry
Registered: Oct 2009
Posts: 48 |
It is possible to put a button to pause, is because it happens that cuts, and there have to start all over again, when cut the connection not say what downloading is doing, checking the speed appears some bytes, said that download but not, or if you can put whatt file is downloaded, thank you! |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
I'll see what i can do :-) |
| |
Isildur
Registered: Sep 2006
Posts: 275 |
Is it possible to fetch all screenshots of graphics from CSDB? |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
It was included in earlier versions, but since there were alot of different miscfiles making the distributionsfiles much larger i decided to just keep corefiles like d64, tap, d81 and so on. |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
Thierry, It's now possible to cancel an executing download.
Please update to the latest Assembly64 at http://freabesdreamnet.asuscomm.com:1337/assembly/ |
| |
Thierry
Registered: Oct 2009
Posts: 48 |
Thanks Scooby! , I was able to check, cancel works well for small downloads, but for large downloads hangs: (eg games), there is no possibility to cancel, creates a 3.5GB ZIP file and does nothing more, the zip file is corrupt , download possibility uncompressed ? , Also when hangs starts to consume all memory of VPS, I came to consume 6gb, I will continue doing checks,
Thank you ! |
| |
Low Spirit
Registered: Jan 2010
Posts: 23 |
@thierry:
check the lenght of the path, where u wanna place your downloads.
i noticed the same problem, that assembly notified me, the zip seems to be corrupt.
i examined the log, found the loaded zip and copied this (while assembly tried
to decrunch it) to another place. then, after the errormessage appeared,
i just unzipped it and it went well. so the zip file is 100% fine!!!
maybe scooby can fix this ^^ |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
Hi guys,
So what you say is that the path was too long where you choose your storagefolder, right? I'll try to provoke the same behavior. I tried really long paths and it works just fine. It would be interesting to know the path to your HOME folder. Thx for feedback! Cheers |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
Think i found the bug... Fixed in latest release |
| |
Low Spirit
Registered: Jan 2010
Posts: 23 |
@scooby: yes, it works fine now ^^
btw: did you compile the whole .exe in a new way, that the former needed/delivered .dll's aren't necessary anymore? |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
Great news!
Yep, i switched the linker to a new one! This one is cleaner :-) Happy it works now! |
| |
Isildur
Registered: Sep 2006
Posts: 275 |
Works great. Thank you. |
| |
Thierry
Registered: Oct 2009
Posts: 48 |
Something strange here , in version 1.03 and the new version 1.04 in a real machine and a VM Server 2008r2 stop at 99.9% , the zip file corrupt and im copy the file to other location and same , corrupt file , this happen when download Games , big 3.4gb file , others like 20 years top 20 works ok , cancel ok , but games not working the cancel button
 |
| |
hedning
Registered: Mar 2009
Posts: 4720 |
Works fine here. Never been a problem. Tested on win7, 8 and 10. |
| |
Low Spirit
Registered: Jan 2010
Posts: 23 |
now, that thierry mentioned he had a problem with the games archive, i tried it too. i get the same strange error as thierry. hitting cancel doesn't work either. (Win10 Home)
other archives seem to be ok. i've tried "csdb_demos" and "csdb_demos_year"
btw: deleting an existing archive can be a bit more comfortable if it would be included in the right-click-context-menu. and... a progress bar showing the delete-state would be also nice ^^ |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
Downloading csdb-games was just verified on win10 and it seems to work. My guess is that if there was a problem with the ftp connection to the server, or if it turned really slow it would not possible to hit cancel until that "block" of data was fetched. Probably what happened, please try again!
A dialog for deleting data would probably not be a good idea since it would block the gui in the process, and that's not what you want right? Gave this thought when i designed it, and decided a backgroundjob would be the best, since you don't care about that artifact anyway
Cheers,
Sby |
| |
algorithm
Registered: May 2002
Posts: 705 |
The downloading of all the data works here as well (Windows 10) - there was the issue of the data stopped while fetching (and cancel did not work either) but that was due to my connection issues. Just attempted again (after force closing) and worked fine |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
Great! Found a way around this... Keep looking for updates :-) |
| |
Low Spirit
Registered: Jan 2010
Posts: 23 |
i've no idea if it was the uodate u pushed tonight or your server, but nevertheless:
i retried it and now the whole thing went fine.
i noticed, that the downloadspeed this time was a lot faster, than as i got the error.
my inet connection was both times fine.
thx a lot for your effort ;) |
| |
Slator
Registered: Jan 2002
Posts: 274 |
The tool and the link to it aren't working anymore. Is the site down? |
| |
Cresh
Registered: Jan 2004
Posts: 354 |
FYI: Right now there seems to be an issue with the ddns provider, putting assembly64 out of work!
Assembly64 Turn out my ISP decided to make my IP private. They will re-public it during the day smile emoticon |
| |
Six
Registered: Apr 2002
Posts: 289 |
I have like 25 slots left on my dyndns account if you'd like to use one. |
| |
Scooby
Registered: Dec 2011
Posts: 22 |
Hi,
Assembly64 is now back online at
http://hackerswithstyle.ddns.net:1337/assembly/index.html
Had to change dnsprovider :-)
You need to download new version manually from the site
New content available tonight!
Cheers,
Sby |
| |
hedning
Registered: Mar 2009
Posts: 4720 |
Quote: Hi,
Assembly64 is now back online at
http://hackerswithstyle.ddns.net:1337/assembly/index.html
Had to change dnsprovider :-)
You need to download new version manually from the site
New content available tonight!
Cheers,
Sby
Great news! I love it! |
| |
ѕiηk
Registered: Mar 2014
Posts: 1 |
thnk a lot for your work, that rox! |
| |
dandee
Registered: Jan 2002
Posts: 6 |
It just dawned my that there seems to be complete DB dumps when I watched Mibris Twich channel the other day... however this site/dyndns seems to be long gone. Is there a way to dump CSDB or a torrent to an not so old pack? |
| |
hedning
Registered: Mar 2009
Posts: 4720 |
Quote: It just dawned my that there seems to be complete DB dumps when I watched Mibris Twich channel the other day... however this site/dyndns seems to be long gone. Is there a way to dump CSDB or a torrent to an not so old pack?
Mibri is using Assembly64: https://assembly64.hackerswithstyle.se/assembly/index.html
Use that and you will get whatever you want, plus weekly updates. |
| |
dandee
Registered: Jan 2002
Posts: 6 |
Thanks... that's what I was looking for. The link above was not working anymore.
cheers. |
| |
Mibri
Registered: Feb 2018
Posts: 214 |
Assembly64 rules and Scooby is a saint for running it! |
