Spark Shuffle Tuning Stuff

https://issues.apache.org/jira/browse/SPARK-6235
https://issues.apache.org/jira/browse/SPARK-19908
https://issues.apache.org/jira/browse/SPARK-13510
https://www.slideshare.net/ilganeli/how-to-actually-tune-your-spark-jobs-so-they-work


•	Can you confirm this was compiled with the correct versions of all jars that are only HWX issued jars?
•	Whats the AccuRev Location of this code? Please Add gray and myself CDSID

•	Increase your Driver memory to 4g please
•	Add this to your spark.executor.extraJavaOptions: ‘-Dio.netty.noUnsafe=true’  and ‘-XX:MaxDirectMemorySize=512m’ and ‘-XX:+PrintGCDetails -XX:+PrintGCTimeStamps’
o	This one is important to get a run done with so we can get the real error if its netty related…
•	Is all input data in ORC?
•	Are you using the KyroSerilizer ? if so please Set spark.kryoserializer.buffer.max=512m
•	What volumes was this tested on in the past that it worked?
•	Set to – 16gb executor memory, and executors to 200.
•	How many shuffle partitions were used in this one?


spark.shuffle.memoryFraction	0.2
spark.storage.memoryFraction	0.6


2017-07-26 02:49:35,171 ERROR [shuffle-client-0] shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_4_156_193, and will not retry (0 retries)
java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:645)
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:228)
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:212)
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:132)
	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271)
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146)
	at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107)
	at io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)