Guest User

Untitled

a guest
Nov 19th, 2018
107
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.88 KB | None | 0 0
  1. Test data:
  2. ```json
  3. {"maker":"Apple", "products":[{"name": "iPhobe", "price": 100000}, {"name": "iPad", "price": 120000}]}
  4. {"maker":"ASUS", "products":[{"name": "Zenfone", "price": 20000}]}
  5. ```
  6.  
  7. DataFrame:
  8. ```
  9. scala> val df = ds.select(ds("maker"),explode(ds("products")).as("p"))
  10. scala> df.select("maker", "p.name", "p.price").show
  11. +-----+-------+------+
  12. |maker| name| price|
  13. +-----+-------+------+
  14. |Apple| iPhobe|100000|
  15. |Apple| iPad|120000|
  16. | ASUS|Zenfone| 20000|
  17. +-----+-------+------+
  18. ```
  19.  
  20. SQL:
  21. ```
  22. scala> val ds = spark.read.json("/Users/takezoe/test.json")
  23. scala> ds.createOrReplaceTempView("test")
  24. scala> val df = spark.sql("SELECT maker, p.name, p.price FROM (SELECT maker, explode(products) as p FROM test)")
  25. scala> df.show
  26. +-----+-------+------+
  27. |maker| name| price|
  28. +-----+-------+------+
  29. |Apple| iPhobe|100000|
  30. |Apple| iPad|120000|
  31. | ASUS|Zenfone| 20000|
  32. +-----+-------+------+
  33. ```
Add Comment
Please, Sign In to add comment