Advertisement
katkid

scrapy issue js source

Jun 12th, 2018
114
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.51 KB | None | 0 0
  1. ```
  2. save content from https://pastebin.com/cu4dHdtG to js_source.html. Then run
  3. scrapy shell ./js_source.html
  4. Code issued at '5.5&quot;</p>', when re.compile run, it did converted to '5.5"</p>'
  5. ```
  6. import re
  7. import json
  8. pattern = re.compile(r"\"fields\": {.*?}}\W+},\W+\"", re.MULTILINE | re.DOTALL)
  9. json_source = response.xpath("//script[contains(., 'app.run({')]/text()").re(pattern)[0].replace(',\n        "', '').replace('}\n        }', '}').replace('"fields": ', '')
  10.  
  11. json_data = json.loads(json_source)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement