Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Test data:
- ```json
- {"maker":"Apple", "products":[{"name": "iPhobe", "price": 100000}, {"name": "iPad", "price": 120000}]}
- {"maker":"ASUS", "products":[{"name": "Zenfone", "price": 20000}]}
- ```
- DataFrame:
- ```
- scala> val df = ds.select(ds("maker"),explode(ds("products")).as("p"))
- scala> df.select("maker", "p.name", "p.price").show
- +-----+-------+------+
- |maker| name| price|
- +-----+-------+------+
- |Apple| iPhobe|100000|
- |Apple| iPad|120000|
- | ASUS|Zenfone| 20000|
- +-----+-------+------+
- ```
- SQL:
- ```
- scala> val ds = spark.read.json("/Users/takezoe/test.json")
- scala> ds.createOrReplaceTempView("test")
- scala> val df = spark.sql("SELECT maker, p.name, p.price FROM (SELECT maker, explode(products) as p FROM test)")
- scala> df.show
- +-----+-------+------+
- |maker| name| price|
- +-----+-------+------+
- |Apple| iPhobe|100000|
- |Apple| iPad|120000|
- | ASUS|Zenfone| 20000|
- +-----+-------+------+
- ```
Add Comment
Please, Sign In to add comment