Guest User

Untitled

a guest
Feb 19th, 2018
67
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.44 KB | None | 0 0
  1. /**
  2. * myDataset: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [key: string, absMal: array<struct<absDjt:string,absDr:string>>]
  3. */
  4.  
  5. // one array element => a row (explode)
  6.  
  7. val explodeDataset = myDataset.select(col("key"), explode(col("absMal")).as("abs")).filter("absMal is not null")
  8.  
  9. // one struct element => one column
  10.  
  11. val finalDataset = explodeDataset.select(col("key"), col("abs.absDr").as("absDr"), col("abs.absDjt").as("absDjt"))
Add Comment
Please, Sign In to add comment