Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- mona@pascal:~/computer_vision/instagram/insta$ scrapy crawl instagramspider
- 2017-03-03 20:05:08-0600 [scrapy] INFO: Scrapy 0.14.4 started (bot: insta)
- 2017-03-03 20:05:08-0600 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState
- /usr/lib/python2.7/dist-packages/scrapy/__init__.pyc
- 2017-03-03 20:05:09-0600 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
- 2017-03-03 20:05:09-0600 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
- 2017-03-03 20:05:09-0600 [scrapy] DEBUG: Enabled item pipelines:
- 2017-03-03 20:05:09-0600 [instagramspider] INFO: Spider opened
- 2017-03-03 20:05:09-0600 [instagramspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
- 2017-03-03 20:05:09-0600 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6024
- 2017-03-03 20:05:09-0600 [scrapy] DEBUG: Web service listening on 0.0.0.0:6081
- 2017-03-03 20:05:09-0600 [instagramspider] DEBUG: Crawled (200) <GET https://www.instagram.com/mona_of_green_gables/?__a=1> (referer: None)
- https://www.instagram.com/p/BRJcqBrFjYx/?__a=1
- {u'media': {u'caption': u"Handling winterland one winter jacket at a time...missing my Old Navy comfty jacket which I broke a few weeks ago by hitting a chair in deep learning class! Oh you deep learnig! You're killing me hahahah \nWhen is this winter thing over? smh smh",
- u'caption_is_edited': False,
- u'code': u'BRJcqBrFjYx',
- u'comments': {u'count': 2,
- u'nodes': [{u'created_at': 1488483858,
- u'id': u'17874332911057376',
- u'text': u'\u0627\u0648\u0646\u062c\u0627 \u0647\u0646\u0648\u0632 \u06af\u0631\u0645 \u0646\u0634\u062f\u0647\u061f \u0627\u06cc\u0646\u062c\u0627 \u0627\u06cc\u0646 \u0647\u0641\u062a\u0647 \u06cc\u0647\u0648 \u0628\u0647\u0627\u0631\u06cc \u0634\u062f!',
- u'user': {u'id': u'30887768',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/15259093_1754962318157984_5418431664328015872_a.jpg',
- u'username': u'shahrzad87'}},
- {u'created_at': 1488483884,
- u'id': u'17860945474091787',
- u'text': u'\u0646\u0647 \u0627\u0632 \u0627\u0648\u0644 \u0645\u0627\u0631\u0686 \u0647\u0645\u06cc\u0646\u062c\u0648\u0631 \u062f\u0627\u0631\u0647 \u0628\u0631\u0641 \u0645\u06cc\u0627\u062f \u0644\u0627\u0645\u0635\u0628 @shahrzad87',
- u'user': {u'id': u'53112486',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/11909374_962415803801353_153711560_a.jpg',
- u'username': u'mona_of_green_gables'}}],
- u'page_info': {u'end_cursor': None,
- u'has_next_page': False,
- u'has_previous_page': False,
- u'start_cursor': None}},
- u'comments_disabled': False,
- u'date': 1488483211,
- u'dimensions': {u'height': 1350, u'width': 1080},
- u'display_src': u'https://scontent.cdninstagram.com/t51.2885-15/e35/16906893_399221317103184_7977034770221629440_n.jpg',
- u'id': u'1461825587375388209',
- u'is_ad': False,
- u'is_video': False,
- u'likes': {u'count': 76,
- u'nodes': [{u'user': {u'id': u'1690824627',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/17076734_1456139381063157_5493655894803611648_a.jpg',
- u'username': u't.tvsl'}},
- {u'user': {u'id': u'2266986245',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/12142608_164741927208409_1388296493_a.jpg',
- u'username': u'abb.yazdani'}},
- {u'user': {u'id': u'1087767088',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/16906800_674341959434226_8512775500232392704_a.jpg',
- u'username': u'sara_katebifar'}},
- {u'user': {u'id': u'1700039591',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/1538354_715449765234462_502906170_a.jpg',
- u'username': u'jayasruthivr'}},
- {u'user': {u'id': u'1290345729',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/14566776_1294751307230659_1321790208412221440_a.jpg',
- u'username': u'fatemeh.ezazi'}},
- {u'user': {u'id': u'297955830',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/15275487_1622703004697619_5693059939182837760_a.jpg',
- u'username': u'arez0oo'}},
- {u'user': {u'id': u'726587651',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/16110422_1415562345128973_717139223513137152_n.jpg',
- u'username': u'baharehhoj'}},
- {u'user': {u'id': u'16087947',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/15623787_1740888302894826_992606159451979776_a.jpg',
- u'username': u'rminanikoo'}},
- {u'user': {u'id': u'43975385',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/12950453_230370707323357_2074026184_a.jpg',
- u'username': u'mostal'}},
- {u'user': {u'id': u'569801849',
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/s150x150/15403392_1239348219492323_2643269847739269120_a.jpg',
- u'username': u'haleh.ramian'}}],
- u'viewer_has_liked': False},
- u'location': {u'has_public_page': True,
- u'id': u'267480400',
- u'name': u'UW-Madison Dept. of Computer Sciences',
- u'slug': u'uw-madison-dept-of-computer-sciences'},
- u'owner': {u'blocked_by_viewer': False,
- u'followed_by_viewer': False,
- u'full_name': u'Mona Jalal',
- u'has_blocked_viewer': False,
- u'id': u'53112486',
- u'is_private': False,
- u'is_unpublished': False,
- u'profile_pic_url': u'https://scontent.cdninstagram.com/t51.2885-19/11909374_962415803801353_153711560_a.jpg',
- u'requested_by_viewer': False,
- u'username': u'mona_of_green_gables'},
- u'related_media': {u'nodes': []},
- u'usertags': {u'nodes': []}}}
- *********************************
- 2017-03-03 20:05:09-0600 [instagramspider] INFO: Closing spider (finished)
- 2017-03-03 20:05:09-0600 [instagramspider] INFO: Dumping spider stats:
- {'downloader/request_bytes': 225,
- 'downloader/request_count': 1,
- 'downloader/request_method_count/GET': 1,
- 'downloader/response_bytes': 3445,
- 'downloader/response_count': 1,
- 'downloader/response_status_count/200': 1,
- 'finish_reason': 'finished',
- 'finish_time': datetime.datetime(2017, 3, 4, 2, 5, 9, 791302),
- 'scheduler/memory_enqueued': 1,
- 'start_time': datetime.datetime(2017, 3, 4, 2, 5, 9, 324161)}
- 2017-03-03 20:05:09-0600 [instagramspider] INFO: Spider closed (finished)
- 2017-03-03 20:05:09-0600 [scrapy] INFO: Dumping global stats:
- {'memusage/max': 127696896, 'memusage/startup': 127696896}
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement