Search Daimon blog

Documents, crm-systems and e-mail servers may be a valuable source for meta-data, such as author, source, project etc. This meta-information can be saved with each document and used for filtering.

We have had support for meta-information in the core search kernel for some time, and you have probably used it to filter on dates and file types. In v2.1 meta-information has been extensively expanded, and almost every document now has some kind of metadata.

The first thing you’ll notice is the more advanced filtering menu on the left of the search interface. The default menu will show document types subdivided into file formats, information type for particular useful meta-data, and date of creation.

On the search results page we now display who emails are to and from. You may click on the email address to filter on it. Clicking on the magnifying glass lists all emails from this email address.

On some documents, selected meta-information will be shown in the snippet, too.

It is also possible to search for specific meta-information. For instance, the query ‘attribute:”author”‘ will return all documents with an author set, and the query ‘book attribute:”author=John Doe”‘ will return all documents of the specified author containing the word ‘book’.

We are happy to announce that Searchdaimon will be a partner for the SearchSummit Norway conference. If you want to learn more about our technology and meet us face to face, drop by our exhibition booth. SearchSummit Norway will also include speakers like Susan Feldman (IDC), Chris Mahoney (Second Life) and John Markus Lervik (FAST). All major companies in the search business in Norway are expected to be there.

The conference is in Oslo, on the 3rd of April. Participation fee is 1290 NOK. However, if you contact us soon, we may be able to give you a pass for free.

Searchdaimon partnered with Jayde Online to deliver the search technology for their web search engine ExactSeek.com. Using our search platform, ExactSeek’s index has grown from 100 million web pages to over 500 million and will increase to 1 billion in the coming months.

News articles (in Norwegian)
http://www.digi.no/505126/norsk-soketeknologi-til-amerikansk-sokegigant
http://www.universitetsavisa.no/ua_lesmer.php?kategori=nyheter&dokid=4798908ae99126.78404924

Full press release:

ExactSeek Partners with Searchdaimon AS on New Search Engine Technology

ExactSeek implements Searchdaimon AS advanced search technology to provide improved search speed, relevancy and index scalability.

ExactSeek.com, a web search engine committed to providing small and medium-sized businesses with affordable search engine advertising alternatives to pay-per-click, today announced an exclusive partnership with Searchdaimon AS, a technology company located in Trondheim, Norway that has developed an advanced search technology capable of high speed indexing and search of huge amounts of unstructured data.

Searchdaimon has long been a participant in the well known search technology environment at the Norwegian University of Science and Technology along with other companies like Google, Yahoo and Fast.

“Searchdaimon’s search technology focuses on scalability through parallelization and advanced text analysis for relevancy,” said Runar Buvik, CTO at Searchdaimon. “Using our search platform, ExactSeek’s index has grown from 100 million web pages to over 500 million and will increase to 1 billion in the coming months.”

ExactSeek’s implementation of the Searchdaimon search platform features rapid crawling and indexing of submitted URLs, incorporation of visitor input through search result voting, a comprehensive news search, a paid inclusion program and keyword ranking tools. Future plans include the release of a distributed, downloadable crawler that can be employed by any PC user to further expand the ExactSeek database and customized search options for webmasters and site owners interested in providing web search to their visitors with their own website look and feel.

“ExactSeek has been focused on improving its search technology for several years. Our partnership with Searchdaimon has enabled us to address the fundamental search issues of speed, relevancy and scalability,” said Mel Strocen, CEO of Jayde Online, Inc., ExactSeek’s parent company. “Most importantly, it provides us with the opportunity to further engage the webmaster community in our growth and to provide a variety of search distribution and advertising options to our partner network which includes the ISEDN (Independent Search Engine & Directory Network).”

ExactSeek’s beta search is available at http://www.exactseek.com

About ExactSeek:
ExactSeek.com is part of the Jayde Online Network which includes GoArticles.com | SiteProNews.com | DropJack.com | Blog-Search.com and FreeWebSubmission.com. Jayde Online Inc. has been internet-focused from its inception in 1998 and primarily involved in the publication of email newsletters and the development of niche and general search engines. ExactSeek was founded in 2002 and has offices in Winnipeg, Canada and San Jose, California.

For more information visit: http://www.exactseek.com

Contact Information:
Barry Green
ExactSeek.com
Vice President, Business Development
http://www.exactseek.com
408-482-5644

We are nearing our first test run of our product, one of the remaining tasks has been to create a administration interface for our users.

This is now close to being complete. Here’s a sneak preview:

Overview:

This the main view, where it’s possible to manage all the collection that have been added.

Managing a collection

The view you get when altering the data for a single collection.

Scan results

This is the results you get after the local network has been scanned for possible collections.

We have hired two new developers. Both are students at NTNU, and will be working part time.

Dagur Johannsson

Dagur is currently on his third year studying Bachelor in informatics at NTNU.

His main focus in computer science is information management and programming, an activity he has done for 5 years. He has developed several applications toward the end user, and thus finds it important to stress user friendliness.

Eirik Nygaard

Mr. Nygaard is currently on his second year studying computer science at NTNU. He has over eight years of programming experience, for most of this time he has been working on open source projects related to operating systems kernels, and low level userland applications and libraries.

He is also a developer for DragonFlyBSD, and the head of Programvareverkstedet one of two student computer clubs at NTNU.

Read more at http://www.searchdaimon.com/people.htm

Daimon is an alternative spelling for daemon, and a captain rank in Star Trek, see http://en.wikipedia.org/wiki/Daimon

We moved into offices in Gløshaugen Innovation Center, in Trondheim to day.

The management has had offices in Oslo Innovation Centre for some time now, but the developers, Runar Buvik, Magnus Galåen and Martin Breda, which lives in Trondheim hasn’t had any place to stay, until now.

For any start up company, getting the first real offices is a large milestone, sow we are very excited.

Below are images of our name on the door, Runar and Magnus working, and the view from the windows.

The crawl has unfortunately been down for a while now.

We managed to run out of space to store new pages. We believed we hade ordered new servers in good time before we needed them, but the ones we wanted were temporally out off stock from our supplier.

When we finally did get them, we hade some software problems that we had to addressed before we where able to use them.

Guess the “just in time” strategy isn’t the best for buying hardware after all.

It do work now. Sow total storage space is now 4.5 TB. See here for live stats:
http://bbh-001.boitho.com/phpSysInfo/

We have reached 200 million crawled pages. Se live status here: http://dcsetup.boitho.com/cgi-bin/dc/topCrawlers.cgi

We are crawling ca 120 million pages a month now, so 300 million isn’t far away.

Now you can get status for your pages. Just submit your url here: http://bbh-001.boitho.com/cgi-bin/UrlToDocID/index.cgi , like this: http://bbh-001.boitho.com/cgi-bin/UrlToDocID/index.cgi?url=http%3A%2F%2Fwww.boitho.com

http://www.boitho.com/beta/

Installed two more storage servers today. Bringing total storage space up to almost 3 TB. The servers are Super Micro 5014C-MT. P4 with Hyper-threading, 512 MB ram and 900 GB disk space.

Photo: Supermicro Inc

See here for an overview over our storage system:
http://bbh-001.boitho.com/phpSysInfo/

Got the papers from Brreg (The Norwegian Register of Business Enterprises) today. Boitho AS have been incorporated.

We have had offices in Oslo Innovation Centre for some time now, and today we where official taken in into there incubator.

http://www.forskningsparken.no/companybase/showcompanypublic.aspx?companyID=8501

The server used to distribute urls for download are having disk problems. We are useing two SCSI disks in raid1 (mirroring), one of this disk have fallen out. It don’t appear to be physical broken, only fallen out of the raid. We will have to rebuild the raid and see if it will start to work again.

This is 140 GB disks sow this will take some time, probably 10-12 hours. The crawling will be down for this period.

Wondering what happens to the data you crawl? Is the project even a live? Why aren’t we release a new index?

At least now you can brows the data online:
http://www.boitho.com/discover/

Google is announcing the idea of a rel=”nofollow” tag on individual links. Forums, blogs, guest books, and others who allow the public to add text and links by themselves, can tell search engines that the links are not necessarily approved by the page and should be treated as untrusted. These links shouldn’t be factored into search engines ranking calculations as a vote from the page they occur on.

This means that adding comments as “Visit my discount pharmaceuticals site” or “Buy cheap Rolexes here” with a link to a spam page, to boost their link population and author text becomes less attractive to spammers.

Boitho, as most search engines, will support this tag.

Related Announcements: Google Msn search Yahoo Searchenginewatch

Thanks to Anders Christensen and The Department of Computer and Information Science at The Norwegian University of Science and Technology (NTNU) for lending us three old computers as well as, hosting space and Internet access. When the servers get operational you can see them crawl as “idi-ntnu” at the user statistics page http://dcsetup.boitho.com/cgi-bin/dc/topCrawlers.cgi.

One of our storage servers have been behaving strange lately. It could behave normal for 2-3 days, and then crash without any warnings. We couldn’t find any thing wrong, and look at it for days. We first believed that it what a bug with the indexing software because it always was the program running when this happened.

It turn out that the cpu fan from time to time would stop, then start again. If we at the same time where using all the disks in the box, by running the indexer, the whole system would get to hot, and the cpu would overheat, crashing the server.