| |
Cruzer
Registered: Dec 2001
Posts: 1048 |
Googling for CSDB-entries
Here's something I've noticed. When you search for a c64-demo on Google you usually get the entries from Pouët.net, c64.ch, Demo Dungeon, etc., but not CSDB. If there is a hit from CSDB it's usually some stats page, and not the actual demo.
This is a shame I think, since old sceners who Google for their old productions won't find this great database in the process. My guess is, that it's because of the URL of the prods (noname.c64.org/csdb/release/?id=xxx) - for some reason I guess Google don't like URL's like that, which doesn't have an actual page in it, like "index.php".
So if you want to make CSDB more visible on Google, you might wanna try to alter these URLs. |
|
| |
Moloch
Registered: Jan 2002
Posts: 2928 |
Actually, the reason is because of the robots.txt that this website has configured. This file controls what information a search engine spider will grab from a website.
http://noname.c64.org/robots.txt
|
| |
Cruzer
Registered: Dec 2001
Posts: 1048 |
Ok, "Disallow /csdb/release/" - that explains it.
But what's the idea behind leaving out releases from search engines? |
| |
Perff
Administrator
Posts: 1679 |
Quote: Ok, "Disallow /csdb/release/" - that explains it.
But what's the idea behind leaving out releases from search engines?
The simple answer: To avoid heavy load on the server when crawlers/spiders walk through thousands of pages.
Is there an alternative? Like an "don't leetch as crasy"-option? |
| |
CreaMD
Registered: Dec 2001
Posts: 3057 |
I had to do the same with c64.sk 2 years back. It's content is stored in the MySQL database and once in a time crawlers opened too many connections and crashed the server so I had to disable index all. Meanwhile I'm decided to change the concept of the news displayer so the old months will be published as static pages with titles having static links, etc and then leave google reinxed it completely again. Now I have many hits from google but when users click on the results it displays front page, so they don't find what they were looking for. |
| |
Cruzer
Registered: Dec 2001
Posts: 1048 |
Ok, guess it all makes sense then. Guess you'll just have to get some faster servers. :P |
| |
CreaMD
Registered: Dec 2001
Posts: 3057 |
In my case it would be better to switch to something "dedicated" and not using unstable MySQL database (and ustable outdaated connect code made by my bro) ;-)) |
| |
Perff
Administrator
Posts: 1679 |
I have now removed the lines with release, group, event and scener to allow search engines to store these.
I will keep an eye on the logs the next days to see how much load this will add to the server. |
