API tools faq
paste
Guest User

SparkPlan

a guest
Apr 19th, 2018
165
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 7.50 KB | None | 0 0
raw download clone embed print report
  1.  
  2. == Parsed Logical Plan ==
  3. 'Aggregate ['source, 'first_leg, 'second_leg, 'dest], ['one.sourcehub AS source#449, 'one.mappedhub AS first_leg#450, 'two.mappedhub AS second_leg#451, 'one.destinationhub AS dest#452]
  4. +- 'Join Inner, ((('two.sourcehub = 'one.mappedhub) && ('three.sourcehub = 'two.mappedhub)) && (('one.destinationhub = 'two.destinationhub) && ('two.destinationhub = 'three.destinationhub)))
  5. :- 'Join Inner
  6. : :- 'SubqueryAlias one
  7. : : +- 'Project [*]
  8. : : +- 'UnresolvedRelation `network_design`
  9. : +- 'SubqueryAlias two
  10. : +- 'Project [*]
  11. : +- 'UnresolvedRelation `network_design`
  12. +- 'SubqueryAlias three
  13. +- 'Project [*]
  14. +- 'UnresolvedRelation `network_design`
  15.  
  16.  
  17. == Analyzed Logical Plan ==
  18. source: string, first_leg: string, second_leg: string, dest: string
  19. Aggregate [sourcehub#407, mappedhub#409, mappedhub#467, destinationhub#408], [sourcehub#407 AS source#449, mappedhub#409 AS first_leg#450, mappedhub#467 AS second_leg#451, destinationhub#408 AS dest#452]
  20. +- Join Inner, (((sourcehub#465 = mappedhub#409) && (sourcehub#468 = mappedhub#467)) && ((destinationhub#408 = destinationhub#466) && (destinationhub#466 = destinationhub#469)))
  21. :- Join Inner
  22. : :- SubqueryAlias one
  23. : : +- Project [sourcehub#407, destinationhub#408, mappedhub#409]
  24. : : +- SubqueryAlias network_design
  25. : : +- Project [sourcehub#407, destinationhub#408, mappedhub#409]
  26. : : +- SubqueryAlias network_file
  27. : : +- HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#407, destinationhub#408, mappedhub#409]
  28. : +- SubqueryAlias two
  29. : +- Project [sourcehub#465, destinationhub#466, mappedhub#467]
  30. : +- SubqueryAlias network_design
  31. : +- Project [sourcehub#465, destinationhub#466, mappedhub#467]
  32. : +- SubqueryAlias network_file
  33. : +- HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#465, destinationhub#466, mappedhub#467]
  34. +- SubqueryAlias three
  35. +- Project [sourcehub#468, destinationhub#469, mappedhub#470]
  36. +- SubqueryAlias network_design
  37. +- Project [sourcehub#468, destinationhub#469, mappedhub#470]
  38. +- SubqueryAlias network_file
  39. +- HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#468, destinationhub#469, mappedhub#470]
  40.  
  41. == Optimized Logical Plan ==
  42. Aggregate [sourcehub#407, mappedhub#409, mappedhub#467, destinationhub#408], [sourcehub#407 AS source#449, mappedhub#409 AS first_leg#450, mappedhub#467 AS second_leg#451, destinationhub#408 AS dest#452]
  43. +- Project [sourcehub#407, destinationhub#408, mappedhub#409, mappedhub#467]
  44. +- Join Inner, ((sourcehub#468 = mappedhub#467) && (destinationhub#466 = destinationhub#469))
  45. :- Project [sourcehub#407, destinationhub#408, mappedhub#409, destinationhub#466, mappedhub#467]
  46. : +- Join Inner, ((sourcehub#465 = mappedhub#409) && (destinationhub#408 = destinationhub#466))
  47. : :- Filter (isnotnull(mappedhub#409) && isnotnull(destinationhub#408))
  48. : : +- InMemoryRelation [sourcehub#407, destinationhub#408, mappedhub#409], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
  49. : : +- HiveTableScan [sourcehub#0, destinationhub#1, mappedhub#2], HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#0, destinationhub#1, mappedhub#2]
  50. : +- Filter ((isnotnull(sourcehub#465) && isnotnull(destinationhub#466)) && isnotnull(mappedhub#467))
  51. : +- InMemoryRelation [sourcehub#465, destinationhub#466, mappedhub#467], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
  52. : +- HiveTableScan [sourcehub#0, destinationhub#1, mappedhub#2], HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#0, destinationhub#1, mappedhub#2]
  53. +- Project [sourcehub#468, destinationhub#469]
  54. +- Filter (isnotnull(destinationhub#469) && isnotnull(sourcehub#468))
  55. +- InMemoryRelation [sourcehub#468, destinationhub#469, mappedhub#470], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
  56. +- HiveTableScan [sourcehub#0, destinationhub#1, mappedhub#2], HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#0, destinationhub#1, mappedhub#2]
  57. == Physical Plan ==
  58. *HashAggregate(keys=[sourcehub#407, mappedhub#409, mappedhub#467, destinationhub#408], functions=[], output=[source#449, first_leg#450, second_leg#451, dest#452])
  59. +- Exchange hashpartitioning(sourcehub#407, mappedhub#409, mappedhub#467, destinationhub#408, 200)
  60. +- *HashAggregate(keys=[sourcehub#407, mappedhub#409, mappedhub#467, destinationhub#408], functions=[], output=[sourcehub#407, mappedhub#409, mappedhub#467, destinationhub#408])
  61. +- *Project [sourcehub#407, destinationhub#408, mappedhub#409, mappedhub#467]
  62. +- *BroadcastHashJoin [mappedhub#467, destinationhub#466], [sourcehub#468, destinationhub#469], Inner, BuildRight
  63. :- *Project [sourcehub#407, destinationhub#408, mappedhub#409, destinationhub#466, mappedhub#467]
  64. : +- *BroadcastHashJoin [mappedhub#409, destinationhub#408], [sourcehub#465, destinationhub#466], Inner, BuildRight
  65. : :- *Filter (isnotnull(mappedhub#409) && isnotnull(destinationhub#408))
  66. : : +- InMemoryTableScan [sourcehub#407, destinationhub#408, mappedhub#409], [isnotnull(mappedhub#409), isnotnull(destinationhub#408)]
  67. : : +- InMemoryRelation [sourcehub#407, destinationhub#408, mappedhub#409], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
  68. : : +- HiveTableScan [sourcehub#0, destinationhub#1, mappedhub#2], HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#0, destinationhub#1, mappedhub#2]
  69. : +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]))
  70. : +- *Filter ((isnotnull(sourcehub#465) && isnotnull(destinationhub#466)) && isnotnull(mappedhub#467))
  71. : +- InMemoryTableScan [sourcehub#465, destinationhub#466, mappedhub#467], [isnotnull(sourcehub#465), isnotnull(destinationhub#466), isnotnull(mappedhub#467)]
  72. : +- InMemoryRelation [sourcehub#465, destinationhub#466, mappedhub#467], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
  73. : +- HiveTableScan [sourcehub#0, destinationhub#1, mappedhub#2], HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#0, destinationhub#1, mappedhub#2]
  74. +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]))
  75. +- *Filter (isnotnull(destinationhub#469) && isnotnull(sourcehub#468))
  76. +- InMemoryTableScan [sourcehub#468, destinationhub#469], [isnotnull(destinationhub#469), isnotnull(sourcehub#468)]
  77. +- InMemoryRelation [sourcehub#468, destinationhub#469, mappedhub#470], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
  78. +- HiveTableScan [sourcehub#0, destinationhub#1, mappedhub#2], HiveTableRelation `test`.`network_file`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sourcehub#0, destinationhub#1, mappedhub#2]
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!