Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- 4 instances each have 4 processors.
- Set refresh interval to -1 and replications to '0' and other basic
- configurations required for better writing.
- 2 Core instances
- - 8 vCPU, 16 GiB memory, EBS only storage
- - EBS Storage:1000 GiB
- 1 Master node
- - 1 vCPU, 3.8 GiB memory, 410 SSD GB storage
- executor-memory - 8g
- spark.executor.instances=2
- spark.executor.cores=4
- es.batch.size.bytes - 6MB
- es.batch.size.entries - 10000
- es.batch.write.refresh - false
- -1116 bytes result sent to driver
- JavaRDD<String> javaRDD = jsc.textFile("<S3 Path>");
- JavaEsSpark.saveJsonToEs(javaRDD,"<Index name>");
Add Comment
Please, Sign In to add comment