Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- # my_sls_scraper/settings.py
- # ...
- HTTPCACHE_ENABLED = True
- HTTPCACHE_EXPIRATION_SECS = 60 * 60 * 24 * 7
- HTTPCACHE_DIR = 'httpcache'
- CLOSESPIDER_PAGECOUNT = 32
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement