Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- 4 workers, 15 cores each, another shell with 40 cores running at the same time.
- 2tags, 100k securities, 2.5k pvs
- I: input - the size of data input to all the tasks
- O: ouput - the size of data output from all tasks
- SR: shuffle read - the size of data read from other partitions
- SW: shuffle write - the size of data written to other partitions
- baseline: read from data table and write back to another table
- 1s 20/20 I: 1932.5KB O: 1932.5KB
- spark smart enough to push down the select into cassandra, so not reading all the pvs
- basic VaR
- 3 s 20/20 I: 1909.2MB SW: 2.3MB
- conditional VaR
- 3s 20/20 I: 1909.2MB SW: 2.3MB
- marginal VaR
- read, rollup and collect:
- 5s T: 2/2 S: 80/80 :
- 4s 20/20 I: 1909.2 MB SW: 48.1 MB
- 0.4s 60/60 SR: 48.1 MB
- marginal:
- 5s T: 2/2 S: 80/80 :
- 2s T: 20/20 I: 1909.2 MB SW: 2018.8 MB
- 3s T: 60/60 O: 2.3 MB SR: 2018.8 MB
- incremental VaR
- read, rollup and collect:
- 5s T: 2/2 S: 80/80 :
- 4s 20/20 I: 1909.2 MB SW: 48.1 MB
- 0.4s 60/60 SR: 48.1 MB
- incremental:
- 5s T: 2/2 S: 80/80 :
- 2s 20/20 I: 1909.2 MB SW: 2018.8 MB
- 2s 60/60 O: 2.3 MB SR: 2018.3 MB
- all VaR:
- read, rollup and collect:
- 5 s 2/2 80/80
- VaR calcs:
- 8 s 6/6 240/240
- 1 - persist:
- 4s 20/20 I: 2.8GB SW: 2.0GB
- 2 - save (bvar):
- 2s 60/60 SR: 2.0GB SW: 1549.7KB
- 3 - save (cvar):
- 2s 60/60 I: 2.8GB SW: 1105.2KB
- 4 - save (ivar):
- 2s 60/60 I: 2.8GB SW: 1874.8KB
- 5 - save (mvar):
- 2s 60/60 I: 2.8GB SW: 1872.0KB
- 6 - runJob (join and save):
- 1s 60/60 O: 3.4MB SR: 6.3MB
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement