Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- import scrapy
- from scrapy.crawler import CrawlerProcess
- class MySpider(scrapy.Spider):
- # Your spider definition
- ...
- process = CrawlerProcess({
- 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
- })
- process.crawl(MySpider)
- process.start() # the script will block here until the crawling is finished
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement