API tools faq
paste
Advertisement
Guest User

TPCH Data Loading

a guest
Aug 1st, 2018
66
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Julia 12.57 KB | None | 0 0
raw download clone embed print report
  1. #########################
  2. Before IMPROVEMENT
  3. #########################
  4.  
  5. julia> @time init_db(BenchmarkConfig("0.01"))
  6. Schema  0.292211 seconds (5.04 k allocations: 179.828 KiB)
  7. Importing TPC-H SF 0.01
  8. Importing NATION ... 10.083640 seconds (4.00 M allocations: 188.681 MiB, 1.38% gc time)
  9. Importing REGION ...  0.266748 seconds (53.76 k allocations: 3.092 MiB)
  10. Importing PART ...  5.096442 seconds (1.87 M allocations: 116.497 MiB, 1.22% gc time)
  11. Importing SUPPLIER ...  4.726727 seconds (1.94 M allocations: 87.552 MiB, 1.06% gc time)
  12. Importing PARTSUPP ...  8.374566 seconds (3.72 M allocations: 305.342 MiB, 1.52% gc time)
  13. Importing CUSTOMER ...  3.242603 seconds (1.45 M allocations: 81.287 MiB, 1.27% gc time)
  14. Importing ORDERS ...  7.313970 seconds (2.74 M allocations: 388.154 MiB, 1.52% gc time)
  15. Importing LINEITEM ... 32.649077 seconds (12.09 M allocations: 2.955 GiB, 2.58% gc time)
  16. Commit ...   3.637549 seconds (295.92 k allocations: 15.504 MiB, 0.34% gc time)
  17.  75.694978 seconds (28.16 M allocations: 4.113 GiB, 1.83% gc time)
  18. true
  19.  
  20. julia> @time init_db(BenchmarkConfig("0.01"))
  21. Schema  0.360325 seconds (5.04 k allocations: 179.828 KiB)
  22. Importing TPC-H SF 0.01
  23. Importing NATION ...  0.006116 seconds (1.48 k allocations: 75.891 KiB)
  24. Importing REGION ...  0.002276 seconds (701 allocations: 36.500 KiB)
  25. Importing PART ...  0.489276 seconds (187.21 k allocations: 34.102 MiB, 2.23% gc time)
  26. Importing SUPPLIER ...  0.016822 seconds (7.70 k allocations: 547.383 KiB)
  27. Importing PARTSUPP ...  1.163101 seconds (417.28 k allocations: 167.195 MiB, 4.97% gc time)
  28. Importing CUSTOMER ...  0.329717 seconds (125.38 k allocations: 25.144 MiB, 2.47% gc time)
  29. Importing ORDERS ...  3.988790 seconds (1.38 M allocations: 325.093 MiB, 2.22% gc time)
  30. Importing LINEITEM ... 28.456206 seconds (10.62 M allocations: 2.885 GiB, 3.11% gc time)
  31. Commit ...   2.557443 seconds (2.67 k allocations: 98.938 KiB)
  32.  37.409427 seconds (12.75 M allocations: 3.425 GiB, 2.81% gc time)
  33. true
  34.  
  35. [~/.pager 17:21:06]$ du -hs
  36. 600M
  37.  
  38. julia> @time init_db(BenchmarkConfig("0.01"))
  39. Schema  0.283145 seconds (5.04 k allocations: 179.828 KiB)
  40. Importing TPC-H SF 0.01
  41. Importing NATION ...  0.003075 seconds (1.48 k allocations: 75.891 KiB)
  42. Importing REGION ...  0.008416 seconds (701 allocations: 36.500 KiB)
  43. Importing PART ...  0.426589 seconds (187.21 k allocations: 34.102 MiB, 1.52% gc time)
  44. Importing SUPPLIER ...  0.015983 seconds (7.70 k allocations: 547.383 KiB)
  45. Importing PARTSUPP ...  1.097394 seconds (417.28 k allocations: 167.195 MiB, 3.17% gc time)
  46. Importing CUSTOMER ...  0.303164 seconds (125.38 k allocations: 25.144 MiB, 1.81% gc time)
  47. Importing ORDERS ...  3.695322 seconds (1.38 M allocations: 325.093 MiB, 1.94% gc time)
  48. Importing LINEITEM ... 36.296721 seconds (10.62 M allocations: 2.885 GiB, 3.30% gc time)
  49. Commit ...   3.293454 seconds (2.67 k allocations: 98.938 KiB)
  50.  45.432428 seconds (12.75 M allocations: 3.429 GiB, 2.90% gc time)
  51. true
  52.     .
  53. [~/.pager 17:21:19]$ du -hs
  54. 900M
  55.  
  56. julia> @time init_db(BenchmarkConfig("0.1"))
  57. Schema  0.290044 seconds (5.04 k allocations: 179.828 KiB)
  58. Importing TPC-H SF 0.1
  59. Importing NATION ...  0.004784 seconds (1.48 k allocations: 75.891 KiB)
  60. Importing REGION ...  0.001729 seconds (701 allocations: 36.500 KiB)
  61. Importing PART ...  5.294922 seconds (1.91 M allocations: 570.776 MiB, 2.48% gc time)
  62. Importing SUPPLIER ...  0.181775 seconds (73.85 k allocations: 12.953 MiB)
  63. Importing PARTSUPP ... 27.931067 seconds (4.70 M allocations: 4.104 GiB, 5.20% gc time)
  64. Importing CUSTOMER ...  5.529738 seconds (1.28 M allocations: 438.088 MiB, 3.19% gc time)
  65. Importing ORDERS ...102.302952 seconds (17.54 M allocations: 8.502 GiB, 3.04% gc time)
  66. Importing LINEITEM ...780.073686 seconds (142.25 M allocations: 75.013 GiB, 3.08% gc time)
  67. Commit ...  23.341912 seconds (4.47 k allocations: 127.047 KiB)
  68. 944.957580 seconds (167.76 M allocations: 88.617 GiB, 3.06% gc time)
  69. true
  70.  
  71. [~/.pager 17:22:23]$ du -hs
  72. 3.4G
  73.  
  74. [~/.pager 17:46:04]$ cd pager_client/
  75. [~/.pager/pager_client 17:46:09]$ ls -1 | wc -l
  76.     1083
  77.  
  78.  
  79. #########################
  80. AFTER IMPROVEMENT
  81. #########################
  82.  
  83. [~ 18:15:21]$ julia
  84.                _
  85.    _       _ _(_)_     |  A fresh approach to technical computing
  86.   (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
  87.    _ _   _| |_  __ _   |  Type "?help" for help.
  88.   | | | | | | |/ _` |  |
  89.   | | |_| | | | (_| |  |  Version 0.6.3 (2018-05-28 20:20 UTC)
  90.  _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
  91. |__/                   |  x86_64-apple-darwin14.5.0
  92.  
  93. shell> cd /Users/dashti/Dropbox/workspaces/RelationalAI/Delve
  94. /Users/dashti/Dropbox/workspaces/RelationalAI/Delve
  95.  
  96. shell> du -sh ~/.pager
  97.   0B    /Users/dashti/.pager
  98.  
  99. julia> using Revise; using TPCH; using Delve.Database; using Delve.REPL; using Delve.Compiler; Delve.init_delve(); @time init_db(BenchmarkConfig("0.01"))
  100. Schema  4.675610 seconds (1.95 M allocations: 99.088 MiB, 1.11% gc time)
  101. Importing TPC-H SF 0.01
  102. Importing NATION ...
  103.     reading CSV file:   6.682030 seconds (1.48 M allocations: 81.335 MiB, 3.43% gc time)
  104.     loading into BeTree:   6.622713 seconds (2.71 M allocations: 116.361 MiB, 0.90% gc time)
  105. Importing REGION ...
  106.     reading CSV file:   0.279829 seconds (53.66 k allocations: 2.903 MiB)
  107.     loading into BeTree:   0.000208 seconds (158 allocations: 454.930 KiB)
  108. Importing PART ...
  109.     reading CSV file:   1.570982 seconds (547.72 k allocations: 28.679 MiB, 0.78% gc time)
  110.     loading into BeTree:   4.072815 seconds (1.96 M allocations: 86.880 MiB, 1.20% gc time)
  111. Importing SUPPLIER ...
  112.     reading CSV file:   1.599461 seconds (342.81 k allocations: 18.966 MiB, 0.75% gc time)
  113.     loading into BeTree:   4.283792 seconds (1.84 M allocations: 80.378 MiB, 1.09% gc time)
  114. Importing PARTSUPP ...
  115.     reading CSV file:   2.580306 seconds (809.41 k allocations: 49.700 MiB, 0.93% gc time)
  116.     loading into BeTree:   6.945955 seconds (5.09 M allocations: 211.679 MiB, 1.66% gc time)
  117. Importing CUSTOMER ...
  118.     reading CSV file:   1.355911 seconds (370.04 k allocations: 20.750 MiB, 0.80% gc time)
  119.     loading into BeTree:   2.236550 seconds (1.44 M allocations: 55.460 MiB, 1.13% gc time)
  120. Importing ORDERS ...
  121.     reading CSV file:   1.769108 seconds (1.44 M allocations: 65.710 MiB, 1.90% gc time)
  122.     loading into BeTree:   3.754897 seconds (4.72 M allocations: 158.692 MiB, 2.12% gc time)
  123. Importing LINEITEM ...
  124.     reading CSV file:   5.119087 seconds (9.74 M allocations: 386.317 MiB, 6.65% gc time)
  125.     loading into BeTree:   6.779528 seconds (59.11 M allocations: 1.890 GiB, 9.21% gc time)
  126. Commit ...   1.018742 seconds (297.58 k allocations: 15.614 MiB, 1.16% gc time)
  127.  77.995584 seconds (106.11 M allocations: 3.881 GiB, 2.61% gc time)
  128. true
  129.  
  130. shell> du -sh ~/.pager
  131.  39M    /Users/dashti/.pager
  132.  
  133. shell> find ~/.pager -type f | wc -l
  134.      659
  135.  
  136. julia> using Revise; using TPCH; using Delve.Database; using Delve.REPL; using Delve.Compiler; Delve.init_delve(); @time init_db(BenchmarkConfig("0.01"))
  137. Schema  0.006363 seconds (5.04 k allocations: 179.828 KiB)
  138. Importing TPC-H SF 0.01
  139. Importing NATION ...
  140.     reading CSV file:   0.002006 seconds (1.33 k allocations: 87.984 KiB)
  141.     loading into BeTree:   0.000314 seconds (1.00 k allocations: 524.867 KiB)
  142. Importing REGION ...
  143.     reading CSV file:   0.001251 seconds (582 allocations: 34.922 KiB)
  144.     loading into BeTree:   0.000177 seconds (158 allocations: 454.930 KiB)
  145. Importing PART ...
  146.     reading CSV file:   0.119323 seconds (168.87 k allocations: 7.975 MiB)
  147.     loading into BeTree:   0.055351 seconds (392.87 k allocations: 11.955 MiB, 65.81% gc time)
  148. Importing SUPPLIER ...
  149.     reading CSV file:   0.006618 seconds (7.21 k allocations: 408.313 KiB)
  150.     loading into BeTree:   0.000965 seconds (9.68 k allocations: 932.523 KiB)
  151. Importing PARTSUPP ...
  152.     reading CSV file:   0.272466 seconds (381.17 k allocations: 24.167 MiB, 3.91% gc time)
  153.     loading into BeTree:   0.106114 seconds (1.83 M allocations: 79.665 MiB, 27.84% gc time)
  154. Importing CUSTOMER ...
  155.     reading CSV file:   0.078478 seconds (113.31 k allocations: 6.377 MiB)
  156.     loading into BeTree:   0.012913 seconds (261.14 k allocations: 8.964 MiB)
  157. Importing ORDERS ...
  158.     reading CSV file:   0.723830 seconds (1.22 M allocations: 53.603 MiB, 3.33% gc time)
  159.     loading into BeTree:   0.187262 seconds (3.34 M allocations: 94.574 MiB, 17.69% gc time)
  160. Importing LINEITEM ...
  161.     reading CSV file:   4.322968 seconds (9.55 M allocations: 376.080 MiB, 8.83% gc time)
  162.     loading into BeTree:   1.732302 seconds (57.44 M allocations: 1.811 GiB, 30.43% gc time)
  163. Commit ...   0.548349 seconds (4.29 k allocations: 124.156 KiB)
  164.   8.186243 seconds (74.73 M allocations: 2.461 GiB, 12.74% gc time)
  165. true
  166.  
  167. shell> du -sh ~/.pager
  168.  78M    /Users/dashti/.pager
  169.  
  170. shell> find ~/.pager -type f | wc -l
  171.     1316
  172.  
  173. julia> using Revise; using TPCH; using Delve.Database; using Delve.REPL; using Delve.Compiler; Delve.init_delve(); @time init_db(BenchmarkConfig("0.1"))
  174. Schema  0.007731 seconds (5.04 k allocations: 179.828 KiB)
  175. Importing TPC-H SF 0.1
  176. Importing NATION ...
  177.     reading CSV file:   0.002288 seconds (1.33 k allocations: 87.984 KiB)
  178.     loading into BeTree:   0.000419 seconds (1.00 k allocations: 524.867 KiB)
  179. Importing REGION ...
  180.     reading CSV file:   0.001471 seconds (582 allocations: 34.922 KiB)
  181.     loading into BeTree:   0.000172 seconds (158 allocations: 454.930 KiB)
  182. Importing PART ...
  183.     reading CSV file:   1.101828 seconds (1.70 M allocations: 84.706 MiB, 5.35% gc time)
  184.     loading into BeTree:   0.440527 seconds (5.04 M allocations: 153.391 MiB, 9.21% gc time)
  185. Importing SUPPLIER ...
  186.     reading CSV file:   0.053924 seconds (67.22 k allocations: 3.559 MiB)
  187.     loading into BeTree:   0.006777 seconds (138.82 k allocations: 5.260 MiB)
  188. Importing PARTSUPP ...
  189.     reading CSV file:   3.030867 seconds (3.87 M allocations: 210.616 MiB, 10.69% gc time)
  190.     loading into BeTree:   0.953516 seconds (21.95 M allocations: 953.162 MiB, 26.67% gc time)
  191. Importing CUSTOMER ...
  192.     reading CSV file:   0.837614 seconds (1.14 M allocations: 59.876 MiB, 1.57% gc time)
  193.     loading into BeTree:   0.269356 seconds (3.35 M allocations: 114.876 MiB, 16.02% gc time)
  194. Importing ORDERS ...
  195.     reading CSV file:   7.225758 seconds (12.29 M allocations: 479.478 MiB, 4.88% gc time)
  196.     loading into BeTree:   2.001342 seconds (40.63 M allocations: 1.115 GiB, 22.83% gc time)
  197. Importing LINEITEM ...
  198.     reading CSV file:  41.078102 seconds (95.77 M allocations: 3.292 GiB, 4.82% gc time)
  199.     loading into BeTree:  20.705375 seconds (682.08 M allocations: 21.470 GiB, 31.27% gc time)
  200. Commit ...   4.961543 seconds (18.84 k allocations: 351.578 KiB)
  201.  82.692114 seconds (868.06 M allocations: 27.896 GiB, 12.09% gc time)
  202. true
  203.  
  204. shell> du -sh ~/.pager
  205. 425M    /Users/dashti/.pager
  206.  
  207. shell> find ~/.pager -type f | wc -l
  208.     6826
  209.  
  210. julia> using Revise; using TPCH; using Delve.Database; using Delve.REPL; using Delve.Compiler; Delve.init_delve(); @time init_db(BenchmarkConfig("1"))
  211. Schema  0.008204 seconds (5.04 k allocations: 179.828 KiB)
  212. Importing TPC-H SF 1
  213. Importing NATION ...
  214.     reading CSV file:   0.002321 seconds (1.33 k allocations: 87.969 KiB)
  215.     loading into BeTree:   0.000407 seconds (1.00 k allocations: 524.867 KiB)
  216. Importing REGION ...
  217.     reading CSV file:   0.001487 seconds (582 allocations: 34.906 KiB)
  218.     loading into BeTree:   0.000246 seconds (158 allocations: 454.930 KiB)
  219. Importing PART ...
  220.     reading CSV file:  11.076770 seconds (17.00 M allocations: 695.186 MiB, 6.86% gc time)
  221.     loading into BeTree:   3.177314 seconds (64.38 M allocations: 1.895 GiB, 14.21% gc time)
  222. Importing SUPPLIER ...
  223.     reading CSV file:   0.485690 seconds (679.25 k allocations: 38.948 MiB, 3.55% gc time)
  224.     loading into BeTree:   0.142606 seconds (1.82 M allocations: 61.234 MiB, 13.83% gc time)
  225. Importing PARTSUPP ...
  226.     reading CSV file:  28.905071 seconds (39.12 M allocations: 1.949 GiB, 4.82% gc time)
  227.     loading into BeTree:  11.638870 seconds (268.97 M allocations: 11.380 GiB, 27.57% gc time)
  228. Importing CUSTOMER ...
  229.     reading CSV file:   8.819493 seconds (11.40 M allocations: 521.347 MiB, 4.51% gc time)
  230.     loading into BeTree:   2.159210 seconds (40.63 M allocations: 1.349 GiB, 15.84% gc time)
  231. Importing ORDERS ...
  232.     reading CSV file:  72.312415 seconds (122.99 M allocations: 4.501 GiB, 3.28% gc time)
  233.     loading into BeTree:  23.088671 seconds (501.24 M allocations: 13.711 GiB, 15.58% gc time)
  234. Importing LINEITEM ...
  235.     reading CSV file: 416.117654 seconds (959.87 M allocations: 32.559 GiB, 3.45% gc time)
  236.     loading into BeTree: 262.400409 seconds (8.24 G allocations: 259.087 GiB, 29.17% gc time)
  237. Commit ...  77.959596 seconds (163.67 k allocations: 2.553 MiB)
  238. 918.310715 seconds (10.27 G allocations: 327.725 GiB, 11.27% gc time)
  239. true
  240.  
  241. shell> du -sh ~/.pager
  242. 3.8G    /Users/dashti/.pager
  243.  
  244. shell> find ~/.pager -type f | wc -l
  245.    60611
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!