Advertisement
Guest User

Untitled

a guest
Jan 25th, 2015
173
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.62 KB | None | 0 0
  1. #/usr/bin/env python
  2. """
  3.    A Crawler implementation in python!
  4.  
  5.    This program provides the common functionality of a crawler
  6.    To run it just type on a terminal:
  7.  
  8.    ~$ python run.py
  9. """
  10.  
  11. from multiprocessing import Process
  12. from config import config
  13. from crawler import Crawler
  14. from store import CrawlerStore
  15.  
  16.  
  17. def create_crawler():
  18.     """
  19.        Create a crawler object and start crawling on a single url
  20.    """
  21.     store = CrawlerStore(config["connection"])
  22.     crawler = Crawler(store, config["params"]["log"])
  23.     crawler.start_crawling()
  24.  
  25.  
  26. if __name__ == '__main__':
  27.  
  28.     create_crawler()
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement