Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- >>> df.show()
- +---+-------+-----+-------+
- | id| ranges|score| uom|
- +---+-------+-----+-------+
- | 1| low| 20|percent|
- | 1|verylow| 10|percent|
- | 1| high| 70| bytes|
- | 1| medium| 40|percent|
- | 1| high| 60|percent|
- | 1|verylow| 10|percent|
- | 1| high| 70|percent|
- +---+-------+-----+-------+
- results = spark.sql('select percentile_approx(score,0.95) as score, first(ranges) from subset GROUP BY id')
- >>> results.show()
- +-----+--------------------+
- |score|first(ranges, false)|
- +-----+--------------------+
- | 70| low|
- +-----+--------------------+
- > pyspark.sql.utils.AnalysisException: u"expression 'subset.`ranges`' is
- > neither present in the group by, nor is it an aggregate function. Add
- > to group by or wrap in first() (or first_value) if you don't care
- > which value you get.;;nAggregate [id#0L],
- > [percentile_approx(score#2L, cast(0.95 as double), 10000, 0, 0) AS
- > score#353L, ranges#1]n+- SubqueryAlias subsetn +- LogicalRDD
- > [id#0L, ranges#1, score#2L, uom#3], falsen
- >>> map = spark.sql('select ranges, score from df')
- >>> results = spark.sql('select percentile_approx(score,0.95) as score from subset GROUP BY id')
- >>> final_result = spark.sql('select r.score, m.ranges from results as r join map as m on r.score = m.score')
Add Comment
Please, Sign In to add comment