Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Deeplearning4j OOM Exception Encountered for ComputationGraph
- Timestamp: 2018-12-19 11:44:25.529
- Thread ID 1
- Thread Name main
- Stack Trace:
- java.lang.OutOfMemoryError: Cannot allocate new LongPointer(9): totalBytes = 1040M, physicalBytes = 29938M
- at org.bytedeco.javacpp.LongPointer.<init>(LongPointer.java:76)
- at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1815)
- at org.nd4j.linalg.convolution.Convolution.im2col(Convolution.java:211)
- at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.preOutput(ConvolutionLayer.java:367)
- at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.activate(ConvolutionLayer.java:411)
- at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:110)
- at org.deeplearning4j.nn.graph.ComputationGraph.ffToLayerActivationsInWS(ComputationGraph.java:2080)
- at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1360)
- at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1329)
- at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:160)
- at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:63)
- at org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)
- at org.deeplearning4j.nn.graph.ComputationGraph.fitHelper(ComputationGraph.java:1149)
- at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1098)
- at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1085)
- at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:965)
- at com.pcng.atrain.common.dl4j.TransferLearningDetector.main(TransferLearningDetector.java:145)
- Caused by: java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes (29938M) > maxPhysicalBytes (29696M)
- at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:584)
- at org.bytedeco.javacpp.Pointer.init(Pointer.java:124)
- at org.bytedeco.javacpp.LongPointer.allocateArray(Native Method)
- at org.bytedeco.javacpp.LongPointer.<init>(LongPointer.java:68)
- ... 16 more
- ========== Memory Information ==========
- ----- Version Information -----
- Deeplearning4j Version 1.0.0-beta3
- Deeplearning4j CUDA <not present>
- ----- System Information -----
- Operating System GNU/Linux CentOS Linux 7
- CPU AMD A8-7650K Radeon R7, 10 Compute Cores 4C+6G
- CPU Cores - Physical 2
- CPU Cores - Logical 4
- Total System Memory 30.32 GB (32552964096)
- ----- ND4J Environment Information -----
- Data Type FLOAT
- backend CPU
- blas.vendor MKL
- os Linux
- ----- Memory Configuration -----
- JVM Memory: XMX 910.50 MB (954728448)
- JVM Memory: current 601 MB (630194176)
- JavaCPP Memory: Max Bytes 29 GB (31138512896)
- JavaCPP Memory: Max Physical 29 GB (31138512896)
- JavaCPP Memory: Current Bytes 1.02 GB (1091501677)
- JavaCPP Memory: Current Physical 29.29 GB (31449874432)
- Periodic GC Enabled false
- ----- Workspace Information -----
- Workspaces: # for current thread 4
- Current thread workspaces:
- Name State Size # Cycles
- WS_LAYER_WORKING_MEM CLOSED 4.38 GB (4700160000) 153
- WS_ALL_LAYERS_ACT CLOSED 24.44 GB (26241184048) 4
- WS_LAYER_ACT_0 CLOSED 0 B 1
- WS_LAYER_ACT_1 CLOSED 0 B 1
- Workspaces total size 28.82 GB (30941344048)
- ----- Network Information -----
- Network # Parameters 51065561
- Parameter Memory 194.80 MB (204262244)
- Parameter Gradients Memory 194.80 MB (204262244)
- Updater Number of Elements 1035250
- Updater Memory 3.95 MB (4141000)
- Updater Classes:
- org.nd4j.linalg.learning.AdamUpdater
- org.nd4j.linalg.learning.NoOpUpdater
- Params + Gradient + Updater Memory 198.75 MB (208403244)
- Iteration Count 1
- Epoch Count 0
- Backprop Type Standard
- Workspace Mode: Training ENABLED
- Workspace Mode: Inference ENABLED
- Number of Layers 74
- Layer Counts
- ConvolutionLayer 1
- FrozenLayer 72
- Yolo2OutputLayer 1
- Layer Parameter Breakdown
- Idx Name Layer Type Layer # Parameters Layer Parameter Memory
- 1 conv2d_1 FrozenLayer 864 3.38 KB (3456)
- 2 batch_normalization_1 FrozenLayer 128 512 B
- 3 leaky_re_lu_1 FrozenLayer 0 0 B
- 4 max_pooling2d_1 FrozenLayer 0 0 B
- 5 conv2d_2 FrozenLayer 18432 72 KB (73728)
- 6 batch_normalization_2 FrozenLayer 256 1 KB (1024)
- 7 leaky_re_lu_2 FrozenLayer 0 0 B
- 8 max_pooling2d_2 FrozenLayer 0 0 B
- 9 conv2d_3 FrozenLayer 73728 288 KB (294912)
- 10 batch_normalization_3 FrozenLayer 512 2 KB (2048)
- 11 leaky_re_lu_3 FrozenLayer 0 0 B
- 12 conv2d_4 FrozenLayer 8192 32 KB (32768)
- 13 batch_normalization_4 FrozenLayer 256 1 KB (1024)
- 14 leaky_re_lu_4 FrozenLayer 0 0 B
- 15 conv2d_5 FrozenLayer 73728 288 KB (294912)
- 16 batch_normalization_5 FrozenLayer 512 2 KB (2048)
- 17 leaky_re_lu_5 FrozenLayer 0 0 B
- 18 max_pooling2d_3 FrozenLayer 0 0 B
- 19 conv2d_6 FrozenLayer 294912 1.13 MB (1179648)
- 20 batch_normalization_6 FrozenLayer 1024 4 KB (4096)
- 21 leaky_re_lu_6 FrozenLayer 0 0 B
- 22 conv2d_7 FrozenLayer 32768 128 KB (131072)
- 23 batch_normalization_7 FrozenLayer 512 2 KB (2048)
- 24 leaky_re_lu_7 FrozenLayer 0 0 B
- 25 conv2d_8 FrozenLayer 294912 1.13 MB (1179648)
- 26 batch_normalization_8 FrozenLayer 1024 4 KB (4096)
- 27 leaky_re_lu_8 FrozenLayer 0 0 B
- 28 max_pooling2d_4 FrozenLayer 0 0 B
- 29 conv2d_9 FrozenLayer 1179648 4.50 MB (4718592)
- 30 batch_normalization_9 FrozenLayer 2048 8 KB (8192)
- 31 leaky_re_lu_9 FrozenLayer 0 0 B
- 32 conv2d_10 FrozenLayer 131072 512 KB (524288)
- 33 batch_normalization_10 FrozenLayer 1024 4 KB (4096)
- 34 leaky_re_lu_10 FrozenLayer 0 0 B
- 35 conv2d_11 FrozenLayer 1179648 4.50 MB (4718592)
- 36 batch_normalization_11 FrozenLayer 2048 8 KB (8192)
- 37 leaky_re_lu_11 FrozenLayer 0 0 B
- 38 conv2d_12 FrozenLayer 131072 512 KB (524288)
- 39 batch_normalization_12 FrozenLayer 1024 4 KB (4096)
- 40 leaky_re_lu_12 FrozenLayer 0 0 B
- 41 conv2d_13 FrozenLayer 1179648 4.50 MB (4718592)
- 42 batch_normalization_13 FrozenLayer 2048 8 KB (8192)
- 43 leaky_re_lu_13 FrozenLayer 0 0 B
- 44 max_pooling2d_5 FrozenLayer 0 0 B
- 45 conv2d_14 FrozenLayer 4718592 18 MB (18874368)
- 46 batch_normalization_14 FrozenLayer 4096 16 KB (16384)
- 47 leaky_re_lu_14 FrozenLayer 0 0 B
- 48 conv2d_15 FrozenLayer 524288 2 MB (2097152)
- 49 batch_normalization_15 FrozenLayer 2048 8 KB (8192)
- 50 leaky_re_lu_15 FrozenLayer 0 0 B
- 51 conv2d_16 FrozenLayer 4718592 18 MB (18874368)
- 52 batch_normalization_16 FrozenLayer 4096 16 KB (16384)
- 53 leaky_re_lu_16 FrozenLayer 0 0 B
- 54 conv2d_17 FrozenLayer 524288 2 MB (2097152)
- 55 batch_normalization_17 FrozenLayer 2048 8 KB (8192)
- 56 leaky_re_lu_17 FrozenLayer 0 0 B
- 57 conv2d_18 FrozenLayer 4718592 18 MB (18874368)
- 58 batch_normalization_18 FrozenLayer 4096 16 KB (16384)
- 59 leaky_re_lu_18 FrozenLayer 0 0 B
- 60 conv2d_19 FrozenLayer 9437184 36 MB (37748736)
- 61 batch_normalization_19 FrozenLayer 4096 16 KB (16384)
- 62 conv2d_21 FrozenLayer 32768 128 KB (131072)
- 63 leaky_re_lu_19 FrozenLayer 0 0 B
- 64 batch_normalization_21 FrozenLayer 256 1 KB (1024)
- 65 conv2d_20 FrozenLayer 9437184 36 MB (37748736)
- 66 leaky_re_lu_21 FrozenLayer 0 0 B
- 67 batch_normalization_20 FrozenLayer 4096 16 KB (16384)
- 68 space_to_depth_x2 FrozenLayer 0 0 B
- 69 leaky_re_lu_20 FrozenLayer 0 0 B
- 71 conv2d_22 FrozenLayer 11796480 45 MB (47185920)
- 72 batch_normalization_22 FrozenLayer 4096 16 KB (16384)
- 73 leaky_re_lu_22 FrozenLayer 0 0 B
- 74 convolution2d_23 ConvolutionLayer 517625 1.97 MB (2070500)
- 75 outputs Yolo2OutputLayer 0 0 B
- ----- Layer Helpers - Memory Use -----
- Total Helper Count 0
- Helper Count w/ Memory 0
- Total Helper Persistent Memory Use 0 B
- ----- Network Activations: Inferred Activation Shapes -----
- Current Minibatch Size 25
- Current Input Shape (Input 0) [25, 3, 800, 800]
- Idx Name Layer Type Activations Type Activations Shape # Elements Memory
- 0 input_1 InputVertex InputTypeConvolutional(h=800,w=800,c=3) [25, 3, 800, 800] 48000000 183.11 MB (192000000)
- 1 conv2d_1 FrozenLayer InputTypeConvolutional(h=800,w=800,c=32) [25, 32, 800, 800] 512000000 1.91 GB (2048000000)
- 2 batch_normalization_1 FrozenLayer InputTypeConvolutional(h=800,w=800,c=32) [25, 32, 800, 800] 512000000 1.91 GB (2048000000)
- 3 leaky_re_lu_1 FrozenLayer InputTypeConvolutional(h=800,w=800,c=32) [25, 32, 800, 800] 512000000 1.91 GB (2048000000)
- 4 max_pooling2d_1 FrozenLayer InputTypeConvolutional(h=400,w=400,c=32) [25, 32, 400, 400] 128000000 488.28 MB (512000000)
- 5 conv2d_2 FrozenLayer InputTypeConvolutional(h=400,w=400,c=64) [25, 64, 400, 400] 256000000 976.56 MB (1024000000)
- 6 batch_normalization_2 FrozenLayer InputTypeConvolutional(h=400,w=400,c=64) [25, 64, 400, 400] 256000000 976.56 MB (1024000000)
- 7 leaky_re_lu_2 FrozenLayer InputTypeConvolutional(h=400,w=400,c=64) [25, 64, 400, 400] 256000000 976.56 MB (1024000000)
- 8 max_pooling2d_2 FrozenLayer InputTypeConvolutional(h=200,w=200,c=64) [25, 64, 200, 200] 64000000 244.14 MB (256000000)
- 9 conv2d_3 FrozenLayer InputTypeConvolutional(h=200,w=200,c=128) [25, 128, 200, 200] 128000000 488.28 MB (512000000)
- 10 batch_normalization_3 FrozenLayer InputTypeConvolutional(h=200,w=200,c=128) [25, 128, 200, 200] 128000000 488.28 MB (512000000)
- 11 leaky_re_lu_3 FrozenLayer InputTypeConvolutional(h=200,w=200,c=128) [25, 128, 200, 200] 128000000 488.28 MB (512000000)
- 12 conv2d_4 FrozenLayer InputTypeConvolutional(h=200,w=200,c=64) [25, 64, 200, 200] 64000000 244.14 MB (256000000)
- 13 batch_normalization_4 FrozenLayer InputTypeConvolutional(h=200,w=200,c=64) [25, 64, 200, 200] 64000000 244.14 MB (256000000)
- 14 leaky_re_lu_4 FrozenLayer InputTypeConvolutional(h=200,w=200,c=64) [25, 64, 200, 200] 64000000 244.14 MB (256000000)
- 15 conv2d_5 FrozenLayer InputTypeConvolutional(h=200,w=200,c=128) [25, 128, 200, 200] 128000000 488.28 MB (512000000)
- 16 batch_normalization_5 FrozenLayer InputTypeConvolutional(h=200,w=200,c=128) [25, 128, 200, 200] 128000000 488.28 MB (512000000)
- 17 leaky_re_lu_5 FrozenLayer InputTypeConvolutional(h=200,w=200,c=128) [25, 128, 200, 200] 128000000 488.28 MB (512000000)
- 18 max_pooling2d_3 FrozenLayer InputTypeConvolutional(h=100,w=100,c=128) [25, 128, 100, 100] 32000000 122.07 MB (128000000)
- 19 conv2d_6 FrozenLayer InputTypeConvolutional(h=100,w=100,c=256) [25, 256, 100, 100] 64000000 244.14 MB (256000000)
- 20 batch_normalization_6 FrozenLayer InputTypeConvolutional(h=100,w=100,c=256) [25, 256, 100, 100] 64000000 244.14 MB (256000000)
- 21 leaky_re_lu_6 FrozenLayer InputTypeConvolutional(h=100,w=100,c=256) [25, 256, 100, 100] 64000000 244.14 MB (256000000)
- 22 conv2d_7 FrozenLayer InputTypeConvolutional(h=100,w=100,c=128) [25, 128, 100, 100] 32000000 122.07 MB (128000000)
- 23 batch_normalization_7 FrozenLayer InputTypeConvolutional(h=100,w=100,c=128) [25, 128, 100, 100] 32000000 122.07 MB (128000000)
- 24 leaky_re_lu_7 FrozenLayer InputTypeConvolutional(h=100,w=100,c=128) [25, 128, 100, 100] 32000000 122.07 MB (128000000)
- 25 conv2d_8 FrozenLayer InputTypeConvolutional(h=100,w=100,c=256) [25, 256, 100, 100] 64000000 244.14 MB (256000000)
- 26 batch_normalization_8 FrozenLayer InputTypeConvolutional(h=100,w=100,c=256) [25, 256, 100, 100] 64000000 244.14 MB (256000000)
- 27 leaky_re_lu_8 FrozenLayer InputTypeConvolutional(h=100,w=100,c=256) [25, 256, 100, 100] 64000000 244.14 MB (256000000)
- 28 max_pooling2d_4 FrozenLayer InputTypeConvolutional(h=50,w=50,c=256) [25, 256, 50, 50] 16000000 61.04 MB (64000000)
- 29 conv2d_9 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 30 batch_normalization_9 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 31 leaky_re_lu_9 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 32 conv2d_10 FrozenLayer InputTypeConvolutional(h=50,w=50,c=256) [25, 256, 50, 50] 16000000 61.04 MB (64000000)
- 33 batch_normalization_10 FrozenLayer InputTypeConvolutional(h=50,w=50,c=256) [25, 256, 50, 50] 16000000 61.04 MB (64000000)
- 34 leaky_re_lu_10 FrozenLayer InputTypeConvolutional(h=50,w=50,c=256) [25, 256, 50, 50] 16000000 61.04 MB (64000000)
- 35 conv2d_11 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 36 batch_normalization_11 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 37 leaky_re_lu_11 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 38 conv2d_12 FrozenLayer InputTypeConvolutional(h=50,w=50,c=256) [25, 256, 50, 50] 16000000 61.04 MB (64000000)
- 39 batch_normalization_12 FrozenLayer InputTypeConvolutional(h=50,w=50,c=256) [25, 256, 50, 50] 16000000 61.04 MB (64000000)
- 40 leaky_re_lu_12 FrozenLayer InputTypeConvolutional(h=50,w=50,c=256) [25, 256, 50, 50] 16000000 61.04 MB (64000000)
- 41 conv2d_13 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 42 batch_normalization_13 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 43 leaky_re_lu_13 FrozenLayer InputTypeConvolutional(h=50,w=50,c=512) [25, 512, 50, 50] 32000000 122.07 MB (128000000)
- 44 max_pooling2d_5 FrozenLayer InputTypeConvolutional(h=25,w=25,c=512) [25, 512, 25, 25] 8000000 30.52 MB (32000000)
- 45 conv2d_14 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 46 batch_normalization_14 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 47 leaky_re_lu_14 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 48 conv2d_15 FrozenLayer InputTypeConvolutional(h=25,w=25,c=512) [25, 512, 25, 25] 8000000 30.52 MB (32000000)
- 49 batch_normalization_15 FrozenLayer InputTypeConvolutional(h=25,w=25,c=512) [25, 512, 25, 25] 8000000 30.52 MB (32000000)
- 50 leaky_re_lu_15 FrozenLayer InputTypeConvolutional(h=25,w=25,c=512) [25, 512, 25, 25] 8000000 30.52 MB (32000000)
- 51 conv2d_16 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 52 batch_normalization_16 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 53 leaky_re_lu_16 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 54 conv2d_17 FrozenLayer InputTypeConvolutional(h=25,w=25,c=512) [25, 512, 25, 25] 8000000 30.52 MB (32000000)
- 55 batch_normalization_17 FrozenLayer InputTypeConvolutional(h=25,w=25,c=512) [25, 512, 25, 25] 8000000 30.52 MB (32000000)
- 56 leaky_re_lu_17 FrozenLayer InputTypeConvolutional(h=25,w=25,c=512) [25, 512, 25, 25] 8000000 30.52 MB (32000000)
- 57 conv2d_18 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 58 batch_normalization_18 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 59 leaky_re_lu_18 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 60 conv2d_19 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 61 batch_normalization_19 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 62 conv2d_21 FrozenLayer InputTypeConvolutional(h=50,w=50,c=64) [25, 64, 50, 50] 4000000 15.26 MB (16000000)
- 63 leaky_re_lu_19 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 64 batch_normalization_21 FrozenLayer InputTypeConvolutional(h=50,w=50,c=64) [25, 64, 50, 50] 4000000 15.26 MB (16000000)
- 65 conv2d_20 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 66 leaky_re_lu_21 FrozenLayer InputTypeConvolutional(h=50,w=50,c=64) [25, 64, 50, 50] 4000000 15.26 MB (16000000)
- 67 batch_normalization_20 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 68 space_to_depth_x2 FrozenLayer InputTypeConvolutional(h=25,w=25,c=256) [25, 256, 25, 25] 4000000 15.26 MB (16000000)
- 69 leaky_re_lu_20 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 70 concatenate_1 MergeVertex InputTypeConvolutional(h=25,w=25,c=1280) [25, 1280, 25, 25] 20000000 76.29 MB (80000000)
- 71 conv2d_22 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 72 batch_normalization_22 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 73 leaky_re_lu_22 FrozenLayer InputTypeConvolutional(h=25,w=25,c=1024) [25, 1024, 25, 25] 16000000 61.04 MB (64000000)
- 74 convolution2d_23 ConvolutionLayer InputTypeConvolutional(h=25,w=25,c=505) [25, 505, 25, 25] 7890625 30.10 MB (31562500)
- 75 outputs Yolo2OutputLayer InputTypeConvolutional(h=25,w=25,c=505) [25, 505, 25, 25] 7890625 30.10 MB (31562500)
- Total Activations Memory 17.93 GB (19247125000)
- Total Activation Gradient Memory 17.90 GB (19215562500)
- ----- Network Training Listeners -----
- Number of Listeners 1
- Listener 0 org.deeplearning4j.ui.stats.StatsListener@39ffda4a
Add Comment
Please, Sign In to add comment