Guest User

Untitled

a guest
Mar 23rd, 2018
77
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.63 KB | None | 0 0
  1. val r = sc.textFile("data96/data.csv")
  2. val rm = r.map(x=>(x.split(",")(0), x.split(",")(1)))
  3.  
  4. val r_grp = rm.groupByKey
  5.  
  6. r_grp.foreach(println(_))
  7. (1,CompactBuffer(Lokesh, Pavan, Tejas, Kumar, Venkat))
  8. (2,CompactBuffer(Bhupesh, Amit, Ratan, Dinesh, Sheela))
  9.  
  10. val swapped = rm.map(item=>item.swap)
  11.  
  12. val com= rm.combineByKey(List(_), (x:List[String], y:String) =>y::x,(x:List[String], y:List[String])=>x:::y) //this is terrible to me, I don't want to do this way
  13.  
  14. com.repartition(1).saveAsTextFile("data96/s43")
  15.  
  16. hdfs dfs -cat data96/s43/*
  17. (2,List(Dinesh, Ratan, Amit, Bhupesh, Sheela))
  18. (1,List(Lokesh, Venkat, Kumar, Tejas, Pavan))
Add Comment
Please, Sign In to add comment