SHARE
TWEET

profile_5nodes

h_fabian Jun 18th, 2014 7 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. Query (id=754caf76397b19d3:1be6cb02414de49a)
  2.   - AsyncTotalTime: 0
  3.   - TotalTime: 0
  4.   Summary
  5.     Session ID: b345703f00825ab0:60309011dc1cd7a3
  6.     Session Type: BEESWAX
  7.     Start Time: 2014-06-18 10:22:52.733483000
  8.     End Time: 2014-06-18 10:23:00.363519000
  9.     Query Type: DML
  10.     Query State: FINISHED
  11.     Query Status: OK
  12.     Impala Version: impalad version 1.3.1-cdh4 RELEASE (build 907481bf45b248a7bb3bb077d54831a71f484e5f)
  13.     User: hadoop
  14.     Network Address: ::ffff:x.y.z..a:port
  15.     Default Db: test_parquet_part
  16.     Sql Statement: insert INTO transactions PARTITION(year=2000) SELECT id, customer_id, book_id, quantity, transaction_date from test_external_part.transactions WHERE year=2000
  17.     Plan:
  18. ----------------
  19. Estimated Per-Host Requirements: Memory=343.32MB VCores=1
  20.  
  21. F00:PLAN FRAGMENT [PARTITION=RANDOM]
  22.   WRITE TO HDFS [test_parquet_part.transactions, OVERWRITE=false, PARTITION-KEYS=(2000)]
  23.   |  partitions=1
  24.   |  hosts=5 per-host-mem=167.32MB
  25.   |
  26.   00:SCAN HDFS [test_external_part.transactions, PARTITION=RANDOM]
  27.      partitions=1/15 size=851.69MB
  28.      table stats: 263673113 rows total
  29.      column stats: all
  30.      hosts=5 per-host-mem=176.00MB
  31.      tuple-ids=0 row-size=48B cardinality=18275489
  32. ----------------
  33.     Estimated Per-Host Mem: 359994032
  34.     Estimated Per-Host VCores: 1
  35.     Request Pool: default-pool
  36.     - AsyncTotalTime: 0
  37.     - TotalTime: 0
  38.   ImpalaServer
  39.     - AsyncTotalTime: 0
  40.     - ClientFetchWaitTimer: 14145809
  41.     - MetastoreUpdateTimer: 2053720636
  42.     - RowMaterializationTimer: 0
  43.     - TotalTime: 0
  44.   Execution Profile 754caf76397b19d3:1be6cb02414de49a
  45.     Insert Stats:
  46. Partition: year=2000/
  47.   BytesWritten: 421.17 MB
  48.   Per Column Sizes:
  49.     book_id: 94.33 MB
  50.     customer_id: 93.25 MB
  51.     id: 79.62 MB
  52.     quantity: 13.12 MB
  53.     transaction_date: 140.84 MB
  54.     Per Node Peak Memory Usage: DataNode1.domain.name:port(175.74 MB) DataNode2.domain.name:port(183.80 MB) DataNode5.domain.name:port(167.97 MB) DataNode3.domain.name:port(100.88 MB) DataNode4.domain.name:port(100.88 MB)
  55.     - AsyncTotalTime: 0
  56.     - FinalizationTimer: 153692181
  57.       - FileDeletionTimer: 58978091
  58.       - FileMoveTimer: 58116648
  59.       - Overwrite/PartitionCreationTimer: 26622048
  60.     - TotalTime: 5483750544
  61.     Averaged Fragment 0
  62.       split sizes:  min: 128.00 MB, max: 256.00 MB, avg: 170.34 MB, stddev: 46.77 MB
  63.       completion times: min:2s313ms  max:5s237ms  mean: 3s900ms  stddev:981.932ms
  64.       execution rates: min:33.59 MB/sec  max:55.32 MB/sec  mean:44.45 MB/sec  stddev:7.79 MB/sec
  65.       num instances: 5
  66.       - AverageThreadTokens: 1.8065079365079364
  67.       - PeakMemoryUsage: 152941712
  68.       - PrepareTime: 50083
  69.       - RowsProduced: 3655097
  70.       - TotalCpuTime: 4983494295
  71.       - TotalNetworkReceiveTime: 0
  72.       - TotalNetworkSendTime: 0
  73.       - TotalStorageWaitTime: 2502582780
  74.       - TotalTime: 3894164210
  75.       CodeGen
  76.         - CodegenTime: 181
  77.         - CompileTime: 44464303
  78.         - LoadTime: 11291071
  79.         - ModuleFileSize: 108040
  80.         - TotalTime: 56296341
  81.       HdfsTableSink
  82.         - BytesWritten: 88325167
  83.         - EncodeTimer: 845860806
  84.         - FinalizePartitionFileTimer: 1530076070
  85.         - HdfsWriteTimer: 1409855885
  86.         - PeakMemoryUsage: 127548150
  87.         - RowsInserted: 3655097
  88.         - TmpFileCreateTimer: 27963774
  89.         - TotalTime: 2405319399
  90.       HDFS_SCAN_NODE (id=0)
  91.         - AverageHdfsReadThreadConcurrency: 1.0
  92.         - AverageScannerThreadConcurrency: 1.4333333333333333
  93.         - BytesRead: 178614480
  94.         - BytesReadDataNodeCache: 0
  95.         - BytesReadLocal: 178613865
  96.         - BytesReadShortCircuit: 178613865
  97.         - NumDisksAccessed: 1
  98.         - NumScannerThreadsStarted: 1
  99.         - PeakMemoryUsage: 30605312
  100.         - PerReadThreadRawHdfsThroughput: 82249536
  101.         - RowsRead: 3655097
  102.         - RowsReturned: 3655097
  103.         - RowsReturnedRate: 2712426
  104.         - ScanRangesComplete: 1
  105.         - ScannerThreadsInvoluntaryContextSwitches: 124
  106.         - ScannerThreadsTotalWallClockTime: 3532027956
  107.           - DelimiterParseTime: 162742257
  108.           - MaterializeTupleTime(*): 800519521
  109.           - ScannerThreadsSysTime: 4799000
  110.           - ScannerThreadsUserTime: 965652400
  111.         - ScannerThreadsVoluntaryContextSwitches: 90
  112.         - TotalRawHdfsReadTime(*): 2258346009
  113.         - TotalReadThroughput: 64831773
  114.         - TotalTime: 1486192361
  115.     Fragment 0
  116.       - AsyncTotalTime: 0
  117.       - TotalTime: 0
  118.       Instance 754caf76397b19d3:1be6cb02414de49c (host=DataNode2.domain.name:port)
  119.         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/256.00 MB
  120.         - AsyncTotalTime: 0
  121.         - AverageThreadTokens: 2.2
  122.         - PeakMemoryUsage: 192730256
  123.         - PrepareTime: 46235
  124.         - RowsProduced: 5498758
  125.         - TotalCpuTime: 6813475047
  126.         - TotalNetworkReceiveTime: 0
  127.         - TotalNetworkSendTime: 0
  128.         - TotalStorageWaitTime: 4599419050
  129.         - TotalTime: 5220078172
  130.         CodeGen
  131.           - AsyncTotalTime: 0
  132.           - CodegenTime: 159
  133.           - CompileTime: 43891505
  134.           - LoadTime: 7183744
  135.           - ModuleFileSize: 108040
  136.           - TotalTime: 51524321
  137.         HdfsTableSink
  138.           - AsyncTotalTime: 0
  139.           - BytesWritten: 132607405
  140.           - EncodeTimer: 1261734977
  141.           - FinalizePartitionFileTimer: 2110027568
  142.           - HdfsWriteTimer: 2024262885
  143.           - PeakMemoryUsage: 158946448
  144.           - RowsInserted: 5498758
  145.           - TmpFileCreateTimer: 34911840
  146.           - TotalTime: 3408901942
  147.         HDFS_SCAN_NODE (id=0)
  148.           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/256.00 MB
  149.           Hdfs Read Thread Concurrency Bucket: 0:0% 1:100% 2:0%
  150.           File Formats: TEXT/NONE:2
  151.           ExecOption: Codegen enabled: 0 out of 2
  152.           - AsyncTotalTime: 0
  153.           - AverageHdfsReadThreadConcurrency: 1.0
  154.           - AverageScannerThreadConcurrency: 2.0
  155.           - BytesRead: 268437504
  156.           - BytesReadDataNodeCache: 0
  157.           - BytesReadLocal: 268436480
  158.           - BytesReadShortCircuit: 268436480
  159.           - NumDisksAccessed: 1
  160.           - NumScannerThreadsStarted: 2
  161.           - PeakMemoryUsage: 33775616
  162.           - PerReadThreadRawHdfsThroughput: 88101698
  163.           - RowsRead: 5498758
  164.           - RowsReturned: 5498758
  165.           - RowsReturnedRate: 3042828
  166.           - ScanRangesComplete: 2
  167.           - ScannerThreadsInvoluntaryContextSwitches: 183
  168.           - ScannerThreadsTotalWallClockTime: 6136132122
  169.             - DelimiterParseTime: 248314944
  170.             - MaterializeTupleTime(*): 1207349698
  171.             - ScannerThreadsSysTime: 19996000
  172.             - ScannerThreadsUserTime: 1446779000
  173.           - ScannerThreadsVoluntaryContextSwitches: 102
  174.           - TotalRawHdfsReadTime(*): 3046904990
  175.           - TotalReadThroughput: 78293674
  176.           - TotalTime: 1807120313
  177.       Instance 754caf76397b19d3:1be6cb02414de49b (host=DataNode1.domain.name:port)
  178.         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/172.23 MB
  179.         - AsyncTotalTime: 0
  180.         - AverageThreadTokens: 2.111111111111111
  181.         - PeakMemoryUsage: 184280208
  182.         - PrepareTime: 43879
  183.         - RowsProduced: 3690077
  184.         - TotalCpuTime: 5659865638
  185.         - TotalNetworkReceiveTime: 0
  186.         - TotalNetworkSendTime: 0
  187.         - TotalStorageWaitTime: 3633646344
  188.         - TotalTime: 4541521457
  189.         CodeGen
  190.           - AsyncTotalTime: 0
  191.           - CodegenTime: 154
  192.           - CompileTime: 43973489
  193.           - LoadTime: 26040206
  194.           - ModuleFileSize: 108040
  195.           - TotalTime: 70467145
  196.         HdfsTableSink
  197.           - AsyncTotalTime: 0
  198.           - BytesWritten: 89164406
  199.           - EncodeTimer: 861715092
  200.           - FinalizePartitionFileTimer: 1629742080
  201.           - HdfsWriteTimer: 1504999606
  202.           - PeakMemoryUsage: 158946448
  203.           - RowsInserted: 3690077
  204.           - TmpFileCreateTimer: 17410834
  205.           - TotalTime: 2510351544
  206.         HDFS_SCAN_NODE (id=0)
  207.           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/172.23 MB
  208.           Hdfs Read Thread Concurrency Bucket: 0:0% 1:100% 2:0%
  209.           File Formats: TEXT/NONE:2
  210.           ExecOption: Codegen enabled: 0 out of 2
  211.           - AsyncTotalTime: 0
  212.           - AverageHdfsReadThreadConcurrency: 1.0
  213.           - AverageScannerThreadConcurrency: 1.6666666666666667
  214.           - BytesRead: 180598464
  215.           - BytesReadDataNodeCache: 0
  216.           - BytesReadLocal: 180598464
  217.           - BytesReadShortCircuit: 180598464
  218.           - NumDisksAccessed: 1
  219.           - NumScannerThreadsStarted: 2
  220.           - PeakMemoryUsage: 33775616
  221.           - PerReadThreadRawHdfsThroughput: 63868992
  222.           - RowsRead: 3690077
  223.           - RowsReturned: 3690077
  224.           - RowsReturnedRate: 1819183
  225.           - ScanRangesComplete: 2
  226.           - ScannerThreadsInvoluntaryContextSwitches: 142
  227.           - ScannerThreadsTotalWallClockTime: 4677818661
  228.             - DelimiterParseTime: 167520671
  229.             - MaterializeTupleTime(*): 821017082
  230.             - ScannerThreadsSysTime: 3999000
  231.             - ScannerThreadsUserTime: 991848000
  232.           - ScannerThreadsVoluntaryContextSwitches: 92
  233.           - TotalRawHdfsReadTime(*): 2827639154
  234.           - TotalReadThroughput: 57402944
  235.           - TotalTime: 2028424558
  236.       Instance 754caf76397b19d3:1be6cb02414de49e (host=DataNode3.domain.name:port)
  237.         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/128.00 MB
  238.         - AsyncTotalTime: 0
  239.         - AverageThreadTokens: 1.5714285714285714
  240.         - PeakMemoryUsage: 105784464
  241.         - PrepareTime: 41560
  242.         - RowsProduced: 2749191
  243.         - TotalCpuTime: 4639203067
  244.         - TotalNetworkReceiveTime: 0
  245.         - TotalNetworkSendTime: 0
  246.         - TotalStorageWaitTime: 1658910414
  247.         - TotalTime: 3806581977
  248.         CodeGen
  249.           - AsyncTotalTime: 0
  250.           - CodegenTime: 154
  251.           - CompileTime: 43651796
  252.           - LoadTime: 7805914
  253.           - ModuleFileSize: 108040
  254.           - TotalTime: 51982862
  255.         HdfsTableSink
  256.           - AsyncTotalTime: 0
  257.           - BytesWritten: 66567863
  258.           - EncodeTimer: 658828231
  259.           - FinalizePartitionFileTimer: 1371725596
  260.           - HdfsWriteTimer: 1264036070
  261.           - PeakMemoryUsage: 80450704
  262.           - RowsInserted: 2749191
  263.           - TmpFileCreateTimer: 18521554
  264.           - TotalTime: 2050169869
  265.         HDFS_SCAN_NODE (id=0)
  266.           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/128.00 MB
  267.           Hdfs Read Thread Concurrency Bucket: 0:0% 1:100% 2:0%
  268.           File Formats: TEXT/NONE:1
  269.           ExecOption: Codegen enabled: 0 out of 1
  270.           - AsyncTotalTime: 0
  271.           - AverageHdfsReadThreadConcurrency: 1.0
  272.           - AverageScannerThreadConcurrency: 1.0
  273.           - BytesRead: 134218752
  274.           - BytesReadDataNodeCache: 0
  275.           - BytesReadLocal: 134217728
  276.           - BytesReadShortCircuit: 134217728
  277.           - NumDisksAccessed: 1
  278.           - NumScannerThreadsStarted: 1
  279.           - PeakMemoryUsage: 25325568
  280.           - PerReadThreadRawHdfsThroughput: 58113575
  281.           - RowsRead: 2749191
  282.           - RowsReturned: 2749191
  283.           - RowsReturnedRate: 1566945
  284.           - ScanRangesComplete: 1
  285.           - ScannerThreadsInvoluntaryContextSwitches: 94
  286.           - ScannerThreadsTotalWallClockTime: 2436686085
  287.             - DelimiterParseTime: 119908593
  288.             - MaterializeTupleTime(*): 605411989
  289.             - ScannerThreadsSysTime: 0
  290.             - ScannerThreadsUserTime: 729889000
  291.           - ScannerThreadsVoluntaryContextSwitches: 84
  292.           - TotalRawHdfsReadTime(*): 2309593755
  293.           - TotalReadThroughput: 54525952
  294.           - TotalTime: 1754490726
  295.       Instance 754caf76397b19d3:1be6cb02414de49d (host=DataNode5.domain.name:port)
  296.         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/167.46 MB
  297.         - AsyncTotalTime: 0
  298.         - AverageThreadTokens: 1.75
  299.         - PeakMemoryUsage: 176129168
  300.         - PrepareTime: 45069
  301.         - RowsProduced: 3588681
  302.         - TotalCpuTime: 4648706631
  303.         - TotalNetworkReceiveTime: 0
  304.         - TotalNetworkSendTime: 0
  305.         - TotalStorageWaitTime: 2065003295
  306.         - TotalTime: 3591876918
  307.         CodeGen
  308.           - AsyncTotalTime: 0
  309.           - CodegenTime: 138
  310.           - CompileTime: 44129340
  311.           - LoadTime: 7507659
  312.           - ModuleFileSize: 108040
  313.           - TotalTime: 52069184
  314.         HdfsTableSink
  315.           - AsyncTotalTime: 0
  316.           - BytesWritten: 86731573
  317.           - EncodeTimer: 820217644
  318.           - FinalizePartitionFileTimer: 1569556560
  319.           - HdfsWriteTimer: 1396340681
  320.           - PeakMemoryUsage: 158946448
  321.           - RowsInserted: 3588681
  322.           - TmpFileCreateTimer: 37370751
  323.           - TotalTime: 2428401127
  324.         HDFS_SCAN_NODE (id=0)
  325.           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/167.46 MB
  326.           Hdfs Read Thread Concurrency Bucket: 0:0% 1:100% 2:0%
  327.           File Formats: TEXT/NONE:2
  328.           ExecOption: Codegen enabled: 0 out of 2
  329.           - AsyncTotalTime: 0
  330.           - AverageHdfsReadThreadConcurrency: 1.0
  331.           - AverageScannerThreadConcurrency: 1.5
  332.           - BytesRead: 175598929
  333.           - BytesReadDataNodeCache: 0
  334.           - BytesReadLocal: 175598929
  335.           - BytesReadShortCircuit: 175598929
  336.           - NumDisksAccessed: 1
  337.           - NumScannerThreadsStarted: 2
  338.           - PeakMemoryUsage: 33775616
  339.           - PerReadThreadRawHdfsThroughput: 95693578
  340.           - RowsRead: 3588681
  341.           - RowsReturned: 3588681
  342.           - RowsReturnedRate: 3091524
  343.           - ScanRangesComplete: 2
  344.           - ScannerThreadsInvoluntaryContextSwitches: 111
  345.           - ScannerThreadsTotalWallClockTime: 3066334040
  346.             - DelimiterParseTime: 159015457
  347.             - MaterializeTupleTime(*): 769671311
  348.             - ScannerThreadsSysTime: 0
  349.             - ScannerThreadsUserTime: 935857000
  350.           - ScannerThreadsVoluntaryContextSwitches: 91
  351.           - TotalRawHdfsReadTime(*): 1835012674
  352.           - TotalReadThroughput: 66827432
  353.           - TotalTime: 1160812557
  354.       Instance 754caf76397b19d3:1be6cb02414de49f (host=DataNode4.domain.name:port)
  355.         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/128.00 MB
  356.         - AsyncTotalTime: 0
  357.         - AverageThreadTokens: 1.4
  358.         - PeakMemoryUsage: 105784464
  359.         - PrepareTime: 73674
  360.         - RowsProduced: 2748782
  361.         - TotalCpuTime: 3156221095
  362.         - TotalNetworkReceiveTime: 0
  363.         - TotalNetworkSendTime: 0
  364.         - TotalStorageWaitTime: 555934798
  365.         - TotalTime: 2310762526
  366.         CodeGen
  367.           - AsyncTotalTime: 0
  368.           - CodegenTime: 300
  369.           - CompileTime: 46675385
  370.           - LoadTime: 7917832
  371.           - ModuleFileSize: 108040
  372.           - TotalTime: 55438197
  373.         HdfsTableSink
  374.           - AsyncTotalTime: 0
  375.           - BytesWritten: 66554592
  376.           - EncodeTimer: 626808088
  377.           - FinalizePartitionFileTimer: 969328547
  378.           - HdfsWriteTimer: 859640183
  379.           - PeakMemoryUsage: 80450704
  380.           - RowsInserted: 2748782
  381.           - TmpFileCreateTimer: 31603894
  382.           - TotalTime: 1628772516
  383.         HDFS_SCAN_NODE (id=0)
  384.           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/128.00 MB
  385.           Hdfs Read Thread Concurrency Bucket: 0:0% 1:100% 2:0%
  386.           File Formats: TEXT/NONE:1
  387.           ExecOption: Codegen enabled: 0 out of 1
  388.           - AsyncTotalTime: 0
  389.           - AverageHdfsReadThreadConcurrency: 1.0
  390.           - AverageScannerThreadConcurrency: 1.0
  391.           - BytesRead: 134218752
  392.           - BytesReadDataNodeCache: 0
  393.           - BytesReadLocal: 134217728
  394.           - BytesReadShortCircuit: 134217728
  395.           - NumDisksAccessed: 1
  396.           - NumScannerThreadsStarted: 1
  397.           - PeakMemoryUsage: 26374144
  398.           - PerReadThreadRawHdfsThroughput: 105469838
  399.           - RowsRead: 2748782
  400.           - RowsReturned: 2748782
  401.           - RowsReturnedRate: 4041650
  402.           - ScanRangesComplete: 1
  403.           - ScannerThreadsInvoluntaryContextSwitches: 92
  404.           - ScannerThreadsTotalWallClockTime: 1343168874
  405.             - DelimiterParseTime: 118951624
  406.             - MaterializeTupleTime(*): 599147529
  407.             - ScannerThreadsSysTime: 0
  408.             - ScannerThreadsUserTime: 723889000
  409.           - ScannerThreadsVoluntaryContextSwitches: 81
  410.           - TotalRawHdfsReadTime(*): 1272579474
  411.           - TotalReadThroughput: 67108864
  412.           - TotalTime: 680113652
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top