Crawl a website with scrapy

 1  Crawl a website with scrapy

scrapy is a crawler framework writtten in Python,  Crawl a website with scrapy described how to use it. 

I follow the guide and wrote a crawler, quite simple and easy to write.

I have written a crawler using BeatifulSoup, urllib2, pyquery, eventlet, MongoDB before,  using Scrapy can simplify our work.

Have fun to use it！

 

2  Is there a better crawler than Scrapy?

Someone asked, "Is there a better crawler than Scrapy?" in quora, and in stackflow  about crawler,

There also an introduction about open-source crawlers in wikipedia,

You can choose one which best suits your case, or you can write one yourself.