Advertisement
Guest User

Untitled

a guest
Oct 26th, 2016
65
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.79 KB | None | 0 0
  1. 4 workers, 15 cores each, another shell with 40 cores running at the same time.
  2. 2tags, 100k securities, 2.5k pvs
  3.  
  4. I: input - the size of data input to all the tasks
  5. O: ouput - the size of data output from all tasks
  6. SR: shuffle read - the size of data read from other partitions
  7. SW: shuffle write - the size of data written to other partitions
  8.  
  9.  
  10. baseline: read from data table and write back to another table
  11. 1s 20/20 I: 1932.5KB O: 1932.5KB
  12. spark smart enough to push down the select into cassandra, so not reading all the pvs
  13.  
  14.  
  15. basic VaR
  16. 3 s 20/20 I: 1909.2MB SW: 2.3MB
  17.  
  18.  
  19. conditional VaR
  20. 3s 20/20 I: 1909.2MB SW: 2.3MB
  21.  
  22.  
  23. marginal VaR
  24. read, rollup and collect:
  25. 5s T: 2/2 S: 80/80 :
  26. 4s 20/20 I: 1909.2 MB SW: 48.1 MB
  27. 0.4s 60/60 SR: 48.1 MB
  28. marginal:
  29. 5s T: 2/2 S: 80/80 :
  30. 2s T: 20/20 I: 1909.2 MB SW: 2018.8 MB
  31. 3s T: 60/60 O: 2.3 MB SR: 2018.8 MB
  32.  
  33. incremental VaR
  34. read, rollup and collect:
  35. 5s T: 2/2 S: 80/80 :
  36. 4s 20/20 I: 1909.2 MB SW: 48.1 MB
  37. 0.4s 60/60 SR: 48.1 MB
  38. incremental:
  39. 5s T: 2/2 S: 80/80 :
  40. 2s 20/20 I: 1909.2 MB SW: 2018.8 MB
  41. 2s 60/60 O: 2.3 MB SR: 2018.3 MB
  42.  
  43. all VaR:
  44. read, rollup and collect:
  45. 5 s 2/2 80/80
  46. VaR calcs:
  47. 8 s 6/6 240/240
  48. 1 - persist:
  49. 4s 20/20 I: 2.8GB SW: 2.0GB
  50. 2 - save (bvar):
  51. 2s 60/60 SR: 2.0GB SW: 1549.7KB
  52. 3 - save (cvar):
  53. 2s 60/60 I: 2.8GB SW: 1105.2KB
  54. 4 - save (ivar):
  55. 2s 60/60 I: 2.8GB SW: 1874.8KB
  56. 5 - save (mvar):
  57. 2s 60/60 I: 2.8GB SW: 1872.0KB
  58. 6 - runJob (join and save):
  59. 1s 60/60 O: 3.4MB SR: 6.3MB
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement