API tools faq
paste
Guest User

Untitled

a guest
Nov 27th, 2018
143
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 9.32 KB | None | 0 0
raw download clone embed print report
  1.  
  2. Logged in as: dr.who
  3. Application
  4. About
  5. Jobs
  6. Tools
  7. Log Type: syslog
  8.  
  9. Log Upload Time: Sun Nov 25 08:43:13 +0000 2018
  10.  
  11. Log Length: 9306
  12.  
  13. 2018-11-25 08:29:13,707 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
  14. 2018-11-25 08:29:13,771 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
  15. 2018-11-25 08:29:13,771 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
  16. 2018-11-25 08:29:13,778 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
  17. 2018-11-25 08:29:13,779 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1524597382992_21544, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@31f9b85e)
  18. 2018-11-25 08:29:13,808 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: YARN_AM_RM_TOKEN, Service: 10.19.65.12:8030,10.19.65.13:8030, Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@2cdd0d4b)
  19. 2018-11-25 08:29:13,848 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: RM_DELEGATION_TOKEN, Service: 10.19.65.12:8032,10.19.65.13:8032, Ident: (RM_DELEGATION_TOKEN owner=dnet.beta, renewer=oozie mr token, realUser=oozie, issueDate=1543133170537, maxDate=1543737970537, sequenceNumber=288386, masterKeyId=1084)
  20. 2018-11-25 08:29:13,881 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
  21. 2018-11-25 08:29:14,056 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data/1/yarn/nm/usercache/dnet.beta/appcache/application_1524597382992_21544,/data/2/yarn/nm/usercache/dnet.beta/appcache/application_1524597382992_21544,/data/3/yarn/nm/usercache/dnet.beta/appcache/application_1524597382992_21544,/data/4/yarn/nm/usercache/dnet.beta/appcache/application_1524597382992_21544
  22. 2018-11-25 08:29:14,282 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
  23. 2018-11-25 08:29:14,779 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
  24. 2018-11-25 08:29:14,789 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
  25. 2018-11-25 08:29:14,983 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Number of splits :1
  26. Total Length = 134217728
  27. Input split[0]:
  28. Length = 134217728
  29. Locations:
  30.  
  31. -----------------------
  32.  
  33. 2018-11-25 08:29:14,999 INFO [main] org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: Current split being processed hdfs://nameservice1/tmp/temp2000951168/tmp1857898809/part-r-00001:0+134217728
  34. 2018-11-25 08:29:15,107 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 67108860(268435440)
  35. 2018-11-25 08:29:15,107 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 256
  36. 2018-11-25 08:29:15,107 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 214748368
  37. 2018-11-25 08:29:15,107 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 268435456
  38. 2018-11-25 08:29:15,107 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 67108860; length = 16777216
  39. 2018-11-25 08:29:15,115 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  40. 2018-11-25 08:29:15,155 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
  41. 2018-11-25 08:29:15,184 INFO [main] org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 1
  42. 2018-11-25 08:29:15,184 INFO [main] org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil: Total input paths to process : 1
  43. 2018-11-25 08:29:15,197 INFO [main] org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
  44. 2018-11-25 08:29:15,219 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.deflate]
  45. 2018-11-25 08:29:15,296 INFO [main] org.apache.pig.data.SchemaTupleBackend: Key [pig.schematuple] was not set... will not generate code.
  46. 2018-11-25 08:29:15,342 INFO [main] org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map: Aliases being processed per job phase (AliasName[line,offset]): M: wc_ranked[128,12] C: R:
  47. 2018-11-25 08:30:19,802 INFO [Service Thread] org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call- Usage threshold init = 1402470400(1369600K) used = 6087127680(5944460K) committed = 7158628352(6990848K) max = 7158628352(6990848K)
  48. 2018-11-25 08:31:14,239 INFO [Service Thread] org.apache.pig.impl.util.SpillableMemoryManager: Spilled an estimate of 17575035332 bytes from 1 objects. init = 1402470400(1369600K) used = 6087127680(5944460K) committed = 7158628352(6990848K) max = 7158628352(6990848K)
  49. 2018-11-25 08:31:44,958 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 67108860(268435440)
  50. 2018-11-25 08:31:44,958 INFO [main] org.apache.hadoop.mapred.MapTask: Record too large for in-memory buffer: 268435471 bytes
  51. 2018-11-25 08:31:44,966 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.snappy]
  52. 2018-11-25 08:32:36,898 INFO [Service Thread] org.apache.pig.impl.util.SpillableMemoryManager: Spilled an estimate of 7768757572 bytes from 1 objects. init = 1402470400(1369600K) used = 6746811088(6588682K) committed = 7158628352(6990848K) max = 7158628352(6990848K)
  53. 2018-11-25 08:32:41,378 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output
  54. 2018-11-25 08:32:41,380 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.snappy]
  55. 2018-11-25 08:32:41,424 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError
  56. at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
  57. at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
  58. at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
  59. at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
  60. at java.io.DataOutputStream.write(DataOutputStream.java:107)
  61. at java.io.DataOutputStream.writeUTF(DataOutputStream.java:401)
  62. at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
  63. at org.apache.pig.data.utils.SedesHelper.writeChararray(SedesHelper.java:66)
  64. at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:580)
  65. at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:462)
  66. at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
  67. at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:650)
  68. at org.apache.pig.data.BinInterSedes.writeBag(BinInterSedes.java:641)
  69. at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:474)
  70. at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:462)
  71. at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
  72. at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:650)
  73. at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:470)
  74. at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:40)
  75. at org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:126)
  76. at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98)
  77. at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82)
  78. at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:206)
  79. at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.spillSingleRecord(MapTask.java:1700)
  80. at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1184)
  81. at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
  82. at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
  83. at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
  84. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:122)
  85. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284)
  86. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
  87. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
  88. at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
  89. at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
  90. at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
  91. at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
  92. at java.security.AccessController.doPrivileged(Native Method)
  93. at javax.security.auth.Subject.doAs(Subject.java:422)
  94. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
  95. at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
  96.  
  97. 2018-11-25 08:32:41,527 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
  98. 2018-11-25 08:32:41,528 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
  99. 2018-11-25 08:32:41,528 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!