| |
Count Zero
Registered: Jan 2003
Posts: 1926 |
Codebase64 mirroring idiots?
I know, I know - Internet is not available everwhere but is that really a reason to crawl codebase64.org with a stupid webcrawler hitting each and every URL it sees?
Frantic and me agreed in adding a plugin which will allow an HTML-ized download for on-the-road-reads and we of course hope that intensified reading leads to more submissions! (Stay tuned for that!)
MEANWHILE we are close to banning _ALL OF POLAND_ via IP rules and deny any access to codebase64.org and other sites hosted on the same server.
HOW SICK ARE YOU generating 60gb of traffic for what appears from the logs as mirroring will result as an unbrowsable data blob.
Oh, cant be wasting our bandwidth with such a silly connection but keeping the shit running for over a week shows some dedication at least. If the initiator is reading this: quit it or deal with the consequences. We are open minded and surely give more than we take - you are abusing it.
(BTW, codebase64 is generating about 8gb per month without stupid mirroring attempts, as a comparison to above give number. We'd like to keep reliability and speed so please report any problems to either Frantic or me) |
|
| |
lft
Registered: Jul 2007
Posts: 369 |
Um, I get that kind of traffic on my site as well. I don't think it's a C64 scener doing it. More like a dumb crawler looking all over the internet for email addresses, unprotected forums, security holes etc. Possibly running from hacked machines in a botnet.
Here's an idea: Add some honeypot links (e.g. black-on-black text saying "Click here to rate-limit my IP"). When they are followed, add rate-limiting rules to your configuration, perhaps with a timeout of a few days. Put "nofollow" on the links to prevent proper search engine crawlers from getting trapped. |
| |
wacek
Registered: Nov 2007
Posts: 513 |
Well,
- I have done it before, but it was June 2011, and let me just check... yes, I turned it off then, so it cannot be me ;)
- please add the download button, I for one would really appreciate it, as I use the offline copy all the time. |
| |
Count Zero
Registered: Jan 2003
Posts: 1926 |
lft: all the traffic is coming from 2-3 ISPs and about 10-15 dynamic IPs in poland and due to the nature of dokuwiki redirecting all requests through its main page there are few possibilities to filter. The main thing I wonder are the different ISPs - could it be a very very limited botnet with just about 1MBit of bandwidth? Nah.
Leads me to apache .htaccess or more drastic blocking methods as neither Frantic nor me are in the mood to adjust IP filters on a daily base.
ALL other proper search engine crawlers won't follow all the links but they can handle and extract the real content just perfectly. This thing tries to obfuscates itself by masquerading as Firefox 38 (all the time it seems - any stupid script kiddie has a list of UA strings...).
Lets just hope the responsible person is reading this, just stops it and we get back to a normal traffic level to allow all real 64 ppl proper services. (I bet there is hardly anyone copy/pasting from codebase as much as I do,though :) ).
*IF* you ppl come accross dokuwiki plugins which may improve site experience, let us know! Just like on the rr.pokefinder.org mediawiki installation we are looking for
anything which may improve the user experience.
We'll be experimenting with a few export plugins in the days to come to see which is best. Likely something allowing local HTML browsing then. Yell now if you prefer some wicked .chm or alike :) |
| |
Conjuror
Registered: Aug 2004
Posts: 168 |
I'd be happy if it was login only past the front page. This is too valuable a resource to be abused.
Its not like we need Google to search the content and its only used by a small community anyway. |
| |
Moloch
Registered: Jan 2002
Posts: 2925 |
Abuse with Codebase64 is nothing new, in the past seven years of hosting it was slammed continually by leechers. I had to block various IPs and websites for abuse of bandwidth or too frequent connections. |
| |
Pex Mahoney Tufvesson
Registered: Sep 2003
Posts: 52 |
I've implemented the same thing as lft above for my production websites. A "blank" link like <a href="info.php"></a> ... something a human could not click on, but a spider/script would trap anytime. And -boom- they're trapped in a ip-block-list, and gone. :)
---
Have a noise night!
http://mahoney.c64.org |
| |
Oswald
Registered: Apr 2002
Posts: 5086 |
" I don't think it's a C64 scener doing it"
maybe its some1 determined to be a coding god, reading 60gb each week :)) |
| |
Bitbreaker
Registered: Oct 2002
Posts: 504 |
Best move all content over to a facebook group *duck* |
| |
Conjuror
Registered: Aug 2004
Posts: 168 |
And there you'll find some interesting coding discussions and mini compos going on atm :p |
| |
Oswald
Registered: Apr 2002
Posts: 5086 |
which group? I'm seriously starving for coder discussions |
| |
Conjuror
Registered: Aug 2004
Posts: 168 |
https://www.facebook.com/groups/RetroAssembler/ |
| |
Oswald
Registered: Apr 2002
Posts: 5086 |
cool, thanks |
| |
Mr. SID
Registered: Jan 2003
Posts: 424 |
Quote: Best move all content over to a facebook group *duck*
No, Google+ ! |
| |
Digger
Registered: Mar 2005
Posts: 427 |
Luckily I live in Sweden now, so no ban for me ;-) |
| |
soci
Registered: Sep 2003
Posts: 479 |
As soon as it's login only and there's no https then it's almost like blocked for me.
If it'd have encryption I might even consider contributing to it (this is not a promise, and as time permits).
As for FB groups, good for my family, but they're not interested in coding, and I'm not interested in FB. Another nice empty set.
Life is damn hard sometimes ;) |
| |
Count Zero
Registered: Jan 2003
Posts: 1926 |
<offtopic>
soci: as much as I favour your paranoia regarding encryption you'd have to find the solution for the regular linux admin to have some sort of self-signed ssl-certificate for several domains on the same IP. Is there something like self-signed SAN certificates maybe? Will this mozilla encrypt campaign have alike?
Until then: we surely would love to see you contributing. Use a unique login/password combination and there shouldn't be too much of a thread to falsify your coding info. Damn that's too high stuff for the NSA even!
soci - dont get me wrong, BUT: As I saw similar complaints regarding encryption from you on the vice ML a while ago: I cannot follow your point of view here. Encrypted submission to about any source (may it be vice, codebase or csdb) does not prevent us from modifying the databases directly and your used encryption does not log your contribution anyhow. :)
Is it really that you wouldn't comment e.g. a blog post if your email is required and there is no https?
</offtopic> |
| |
Burglar
Registered: Dec 2004
Posts: 1088 |
<offtopic answer> almost there: http://www.theregister.co.uk/2015/09/15/lets_encrypt/ </offtopic>
ontopic, if u can do the code, you could write a simple ratelimiter using fail2ban or grep/awk. but takes time and rolling your own can be error prone.
imagine a cronjob that just analyzes the access log and bans ips if a certain threshold is reached. |
| |
Count Zero
Registered: Jan 2003
Posts: 1926 |
Been there already - less error prone is using "Sippenhaft" (sorry, cannot find a proper translation) on all of Poland via geoip. Wouldn't hurt US anyhow :) |
| |
soci
Registered: Sep 2003
Posts: 479 |
Quoting Count Zero<offtopic>
soci: as much as I favour your paranoia regarding encryption you'd have to find the solution for the regular linux admin to have some sort of self-signed ssl-certificate for several domains on the same IP. Is there something like self-signed SAN certificates maybe? Will this mozilla encrypt campaign have alike?
You can have multiple certificates for different domains on a single IP with SNI these days, and it's not difficult to set up (only have experience with apache2). Even self signed is better than nothing.
Quoting Count ZeroUntil then: we surely would love to see you contributing. Use a unique login/password combination and there shouldn't be too much of a thread to falsify your coding info. Damn that's too high stuff for the NSA even!
Unique passwords is the norm of course. It's more about random wifi routers I connect through, not my home connection which I only use for 3 hours a month or so. Same argument as last time, if I suddenly start to spam then sorry ;)
Quoting Count ZeroIs it really that you wouldn't comment e.g. a blog post if your email is required and there is no https?</offtopic>
It's a pain to create a throw away unique e-mail address every time. Especially if the account is dropped after 30 days of inactivity. So I think twice if it's worth to register. |
| |
Bitbreaker
Registered: Oct 2002
Posts: 504 |
Quoting Count Zero<offtopic>
... you'd have to find the solution for the regular linux admin to have some sort of self-signed ssl-certificate for several domains on the same IP. Is there something like self-signed SAN certificates maybe? Will this mozilla encrypt campaign have alike?
Where's the problem having a certificate per VirtualHost directive? Have running that here, just specify a different cert per VirtualHost and you are done? The IP is not the problem, you just need to use the right FQDN per host when creating the certificate. |
| |
JackAsser
Registered: Jun 2002
Posts: 2014 |
It requires SNI-support both on the server and on the browser. It works on major browsers since long though.
Please read: https://en.wikipedia.org/wiki/Server_Name_Indication |
| |
Count Zero
Registered: Jan 2003
Posts: 1926 |
my mod_ssl comments:
# Server Name Indication for SSL named virtual hosts is currently not
# supported by MSIE on Windows XP.
Will setup some https hosts for the paranoid here soonish and let you know then :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11357 |
Quote:
not supported by MSIE on Windows XP.
so? sounds like a plan to me :o) |
| |
Frantic
Registered: Mar 2003
Posts: 1646 |
It is now possible to download the contents of Codebase 64 as a HTML archive that you can browse locally on your own machine, when you have no internet access available or so. The zip file is about 85 megabytes.
http://codebase64.org/doku.php?id=download_the_wiki
Start browsing using the page called "start.html". Let me know if it works OK. I haven't tested it much myself, but merely made use of the "site export" plugin for dokuwiki. |
| |
Count Zero
Registered: Jan 2003
Posts: 1926 |
Also,
https://codebase64.org is available now.
We are looking to add "proper" Mozilla encrypt (or alike) certificates which will not require you to accept the cert soonish.
Yes, more hardening on the available ciphers will be done as well soonish. I am just lazy on testing :)
Basically should not hold back soci from submitting some hardcore drive routines now though! :) |
| |
Mr. SID
Registered: Jan 2003
Posts: 424 |
Quote: It is now possible to download the contents of Codebase 64 as a HTML archive that you can browse locally on your own machine, when you have no internet access available or so. The zip file is about 85 megabytes.
http://codebase64.org/doku.php?id=download_the_wiki
Start browsing using the page called "start.html". Let me know if it works OK. I haven't tested it much myself, but merely made use of the "site export" plugin for dokuwiki.
Seems to work fine. A nice addition to the dev environment on my notebook.
Thanks! |
| |
Vai
Registered: Mar 2002
Posts: 50 |
Quote: Also,
https://codebase64.org is available now.
We are looking to add "proper" Mozilla encrypt (or alike) certificates which will not require you to accept the cert soonish.
Yes, more hardening on the available ciphers will be done as well soonish. I am just lazy on testing :)
Basically should not hold back soci from submitting some hardcore drive routines now though! :)
You can create a free certificate here: https://www.startssl.com. Always better than a self signed one ;) |
| |
soci
Registered: Sep 2003
Posts: 479 |
Quoting Count Zerohttps://codebase64.org is available now.
Many thanks! Works ;)
Quoting Count ZeroBasically should not hold back soci from submitting some hardcore drive routines now though! :)
Which drive code? There are many better loaders than I did before. And there's no interest in real floppies apparently, not even in my own team.
Actually I was thinking about extending the cross development section a bit.
Quoting Count Zerosoci: as much as I favour your paranoia regarding encryption
Possibly it's just a lack of experience. I lived long-long ago in a dormitory with 10Base2 Ethernet and lots of hubs. Was great fun with telnet, http and ftp logins. It thought me a few lessons.
For me it's good enough as it is now. No need to push it further unless you've got interested. |
