sirdf.com Forum Index sirdf.com
Search & Information Retrieval Development Forum
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

To raid or not to raid

 
Post new topic   Reply to topic    sirdf.com Forum Index -> Running a search engine
View previous topic :: View next topic  
Author Message
runarb
Site Admin


Joined: 29 Oct 2006
Posts: 4

PostPosted: Sat Sep 09, 2006 2:34 am    Post subject: Reply with quote

The Boitho search engine now uses 7 independed servers. All have 4 sata disk, making it a total of 28 disk. The problem is that we are having a lot of disk crashes. Have lost some 6 disks total now.

Every time that happened we have a system that can find out which pages was on that disk, and recrawle them.

The recrawl is time consuming, sow we are thinking about switching to raid5.

I have newer really tested out raid in a high performance system. According to http://www.pcguide.com/ref/hdd/perf/raid/l...leLevel5-c.html “the overhead necessary in dealing with the parity continues to bog down writes”.

How bad is this bog down?

Disk i/o is a big bottleneck to day. To work around this by uses 4 prepossesses in parallel, each indexing data on one disk. Thereby using all 4 disk at ones. If we changes to raid 5 this method will not work.

Have anyone seen any research on this ?


When we become bigger we will use a “redundant array of inexpensive nodes”. Where all data resist on at least 3 independed servers. If one fail we can just add another and copy the data from the two remaining servers. Google uses this approach in the “Google file system”.
_________________
CTO @ Searchdaimon company search.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    sirdf.com Forum Index -> Running a search engine All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group