Other than massive servers and bandwidth, how hard really would it be to make a decent search engine? I’d go old school and have sites self-submit to be crawled. Have a handful of general keywords/categories allowed per domain. Base results on meta tags so the whole page doesn’t even need to be parsed while crawling. I guess it wouldn’t be as advanced as Google, finding strings of text 3 pages into a forum post, but it would be efficient and easier to manage.
Other than massive servers and bandwidth, how hard really would it be to make a decent search engine? I’d go old school and have sites self-submit to be crawled. Have a handful of general keywords/categories allowed per domain. Base results on meta tags so the whole page doesn’t even need to be parsed while crawling. I guess it wouldn’t be as advanced as Google, finding strings of text 3 pages into a forum post, but it would be efficient and easier to manage.