Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- #/usr/bin/env python
- """
- A Crawler implementation in python!
- This program provides the common functionality of a crawler
- To run it just type on a terminal:
- ~$ python run.py
- """
- from multiprocessing import Process
- from config import config
- from crawler import Crawler
- from store import CrawlerStore
- def create_crawler():
- """
- Create a crawler object and start crawling on a single url
- """
- store = CrawlerStore(config["connection"])
- crawler = Crawler(store, config["params"]["log"])
- crawler.start_crawling()
- if __name__ == '__main__':
- create_crawler()
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement