Develop search engine

I hope this is the right place to ask this querry.
I need to develop a web search engine and arabic search engine.

I need to know the basic working of it. how to develop a crawler, i do not plan to develop sth like msn, or google.. but on a small level but need to understand teh basic concept and technologies behind it.


Can anybody help?

Thanks

Hi,

Well, you know, what you should consider first is if you've got sufficient financial and "brain" resourses for all that. That is pretty expensive to creat and maintain a SE portal (whether you will be operating an independent crawler-based search engine or pay for getting crawler-based search results from another portal). If you are ready for all these things, then in any case you will hire a team of the pros which will technically implement your idea. In that case, believe me, they'll know the basics.
I apologize if my comment is not of help for you. Hope that others will provide you more details on the subject, as I can't boast of too much experience in that particular sphere.

Good luck,

Take a look at my blog, I wrote something on how to write your own search engine with a load of resources and source code also.

I can't post a link here, but it's at msn spaces and caled search science, maybe I can pm you the link if you like.

What size operation are you thinking of?

nice one... you want to build your own SE in arabic to crawl the arabic languages website?...