What would be involved in making a site like indeed.com or kayak.com?

I've been working on a recipe site for a while and thought it may be cool to add in results from other sites too like food.com, allrecipes.com etc.

As far as I can tell they don't have their own apis and their metadata is largely lacking. I looked into Google custom search api but I want total control over display.

Any ideas?

The only issue is getting the data, really. If they don't have an api then that pretty much just leaves scraping. Once you have the data it's easy, just use solr or some other search platform.

I don't know what languages you use but I've used a python library called scrapy a couple of times. It seems pretty decent.

Anemone is pretty badass, but it's a ruby library.