Advertisement
Dundre32

Untitled

Apr 25th, 2020
111
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.29 KB | None | 0 0
  1. val path = "reviews_devset.json"
  2. val rdd = sc.textFile(path)
  3.  
  4. val rdMapped = rdd.map{ row =>
  5. val json_row = parse(row)
  6.  
  7. (compact(json_row \ "category"), compact(json_row \ "reviewText").toLowerCase())
  8. }.collect()
  9.  
  10. val counts = rdMapped.flatMap(x => x._2.split("\\W")).map(x => (x, 1))
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement