Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- val path = "reviews_devset.json"
- val rdd = sc.textFile(path)
- val rdMapped = rdd.map{ row =>
- val json_row = parse(row)
- (compact(json_row \ "category"), compact(json_row \ "reviewText").toLowerCase())
- }.collect()
- val counts = rdMapped.flatMap(x => x._2.split("\\W")).map(x => (x, 1))
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement