Search Daimon blog

Wondering what happens to the data you crawl? Is the project even a live? Why aren’t we release a new index?
At least now you can brows the data online:
http://www.boitho.com/discover/

Google is announcing the idea of a rel=”nofollow” tag on individual links. Forums, blogs, guest books, and others who allow the public to add text and links by themselves, can tell search engines that the links are not necessarily approved by the page and should be treated as untrusted. These links shouldn’t be factored into […]

Thanks to Anders Christensen and The Department of Computer and Information Science at The Norwegian University of Science and Technology (NTNU) for lending us three old computers as well as, hosting space and Internet access. When the servers get operational you can see them crawl as “idi-ntnu” at the user statistics page http://dcsetup.boitho.com/cgi-bin/dc/topCrawlers.cgi.

One of our storage servers have been behaving strange lately. It could behave normal for 2-3 days, and then crash without any warnings. We couldn’t find any thing wrong, and look at it for days. We first believed that it what a bug with the indexing software because it always was the program running when […]