daily pastebin goal
30%
SHARE
TWEET

Untitled

a guest Sep 13th, 2017 50 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1.     def parse(self, response):
  2.  
  3.         data = response.body
  4.         data = json.loads(data)
  5.  
  6.         exception_result = ''
  7.         try:
  8.             social_links_data = data['screennames']['data']
  9.             exception_result = social_links_data
  10.         except KeyError:
  11.             pass
  12.  
  13.         # get social links if they are in dict
  14.         social_links = get_social_links(exception_result)
  15.  
  16.         if response.status == 200:
  17.  
  18.             item = CompanyCrawlerItem()
  19.  
  20.             item['title'] = data.get('name', 'No information')
  21.             item['url'] = data.get('link', 'No information')
  22.             item['logo_url'] = data['picture']['data'].get('url', 'No information')
  23.             item['location'] = data.get('single_line_address', 'No information')
  24.             item['website'] = data.get('website', 'No information')
  25.             item['twitter'] = social_links.get('Twitter', 'No information')
  26.  
  27.             # надо отправить по item = FacebookCompanyCrawlerItem()
  28.             item['phone'] = data.get('phone', 'No information')
  29.             item['message'] = data.get('link', 'No information')
  30.             item['founded_year'] = data['start_info'].get('date', 'No information')
  31.             item['youtube'] = social_links.get('YouTube', 'No information')
  32.             item['instagram'] = social_links.get('Instagram', 'No information')
  33.             item['snapchat'] = social_links.get('Snapchat', 'No information')
  34.  
  35.         yield item
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top