Guest User

Untitled

a guest
Oct 18th, 2017
72
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.61 KB | None | 0 0
  1. 4 instances each have 4 processors.
  2. Set refresh interval to -1 and replications to '0' and other basic
  3. configurations required for better writing.
  4.  
  5. 2 Core instances
  6. - 8 vCPU, 16 GiB memory, EBS only storage
  7. - EBS Storage:1000 GiB
  8.  
  9. 1 Master node
  10. - 1 vCPU, 3.8 GiB memory, 410 SSD GB storage
  11.  
  12. executor-memory - 8g
  13. spark.executor.instances=2
  14. spark.executor.cores=4
  15.  
  16. es.batch.size.bytes - 6MB
  17. es.batch.size.entries - 10000
  18. es.batch.write.refresh - false
  19.  
  20. -1116 bytes result sent to driver
  21.  
  22. JavaRDD<String> javaRDD = jsc.textFile("<S3 Path>");
  23. JavaEsSpark.saveJsonToEs(javaRDD,"<Index name>");
Add Comment
Please, Sign In to add comment