Guest User

Untitled

a guest
Jan 22nd, 2018
91
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.10 KB | None | 0 0
  1. scala> val inp=sc.textFile("Telephone.txt")
  2. inp: org.apache.spark.rdd.RDD[String] = Telephone.txt MapPartitionsRDD[1] at textFile at <console>:35
  3.  
  4. scala> inp.first()
  5. res0: String = 2014-03-15:10:10:20,Sorrento,8cc3b47e-bd01-4482-b500-28f2342679af,33.6894754264,-117.543308253
  6.  
  7. scala> case class Telephone(dt:String,ct:String,s:String,lat:Double,lon:Double)
  8. defined class Telephone
  9.  
  10. scala> val inp_split=inp.map(x=>x.split(","))
  11. inp_split: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[2] at map at <console>:37
  12.  
  13. scala> val telrdd=inp_split.map(x=>Telephone(x(0),x(1),x(2),x(3).toDouble,x(4).toDouble))
  14. telrdd: org.apache.spark.rdd.RDD[Telephone] = MapPartitionsRDD[3] at map at <console>:41
  15.  
  16. scala> val telDF=telrdd.toDF()
  17. telDF: org.apache.spark.sql.DataFrame = [dt: string, ct: string, s: string, lat: double, lon: double]
  18.  
  19. scala> teldf.count()
  20.  
  21. [Stage 31:=============================> (1 + 1) / 2]18/01/22 20:16:19 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 31.0 (TID 5
  22. 3, cloudera-slavenode2.cloudlab.com, executor 16): java.lang.ArrayIndexOutOfBoundsException: 1
Add Comment
Please, Sign In to add comment