API tools faq
paste
Advertisement
Guest User

Datanode log

a guest
Dec 4th, 2012
56
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 39.28 KB | None | 0 0
raw download clone embed print report
  1. 2012-12-04 10:05:30,509 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  2. 2012-12-04 10:05:30,510 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201212040948_0005_m_000001_1 task's state:UNASSIGNED
  3. 2012-12-04 10:05:30,510 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201212040948_0005_m_000001_1 which needs 1 slots
  4. 2012-12-04 10:05:30,510 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201212040948_0005_m_000001_1 which needs 1 slots
  5. 2012-12-04 10:05:32,480 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating exeef84dfe5ac2345a288fcba003ba59882 in /data/2/mapred/local/taskTracker/distcache/1299361856892079245_638392187_1700177199/namenode/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276-work-7929800928919185954 with rwxr-xr-x
  6. 2012-12-04 10:05:32,562 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Cached hdfs://namenode:8020/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276/exeef84dfe5ac2345a288fcba003ba59882 as /data/2/mapred/local/taskTracker/distcache/1299361856892079245_638392187_1700177199/namenode/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276/exeef84dfe5ac2345a288fcba003ba59882
  7. 2012-12-04 10:05:32,580 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Cached hdfs://namenode:8020/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276/exeef84dfe5ac2345a288fcba003ba59882 as /data/2/mapred/local/taskTracker/distcache/1299361856892079245_638392187_1700177199/namenode/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276/exeef84dfe5ac2345a288fcba003ba59882
  8. 2012-12-04 10:05:32,590 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating lower.py in /data/1/mapred/local/taskTracker/distcache/4563967717847113416_-1407662699_1700177804/namenode/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276-work--5628624305436195821 with rwxr-xr-x
  9. 2012-12-04 10:05:32,885 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Cached hdfs://namenode:8020/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276/lower.py#lower.py as /data/1/mapred/local/taskTracker/distcache/4563967717847113416_-1407662699_1700177804/namenode/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276/lower.py
  10. 2012-12-04 10:05:32,886 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Cached hdfs://namenode:8020/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276/lower.py#lower.py as /data/1/mapred/local/taskTracker/distcache/4563967717847113416_-1407662699_1700177804/namenode/user/vagrant/examples/pydoop_script_95f1687ec2594290855d508029d83276/lower.py
  11. 2012-12-04 10:05:32,906 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user vagrant on this TT.
  12. 2012-12-04 10:05:32,942 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating pydoop_2_0_0_cdh_4_1_2.jar in /data/2/mapred/local/taskTracker/vagrant/distcache/-8478138521094200178_-1346713089_1700184960/namenode/tmp/hadoop-mapred/mapred/staging/vagrant/.staging/job_201212040948_0005/libjars-work-1083017459797455713 with rwxr-xr-x
  13. 2012-12-04 10:05:32,964 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Cached hdfs://namenode/tmp/hadoop-mapred/mapred/staging/vagrant/.staging/job_201212040948_0005/libjars/pydoop_2_0_0_cdh_4_1_2.jar as /data/2/mapred/local/taskTracker/vagrant/distcache/-8478138521094200178_-1346713089_1700184960/namenode/tmp/hadoop-mapred/mapred/staging/vagrant/.staging/job_201212040948_0005/libjars/pydoop_2_0_0_cdh_4_1_2.jar
  14. 2012-12-04 10:05:33,138 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201212040948_0005_m_-1576027144
  15. 2012-12-04 10:05:33,142 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201212040948_0005_m_-1576027144 spawned.
  16. 2012-12-04 10:05:33,146 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /data/1/mapred/local/ttprivate/taskTracker/vagrant/jobcache/job_201212040948_0005/attempt_201212040948_0005_m_000001_1/taskjvm.sh
  17. 2012-12-04 10:05:34,870 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201212040948_0005_m_-1576027144 given task: attempt_201212040948_0005_m_000001_1
  18. 2012-12-04 10:05:43,337 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212040948_0005_m_000001_1 0.0% hdfs://namenode/user/vagrant/examples/lower/lower.txt:31+31
  19. 2012-12-04 10:15:43,518 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212040948_0005_m_000001_1: Task attempt_201212040948_0005_m_000001_1 failed to report status for 600 seconds. Killing!
  20. 2012-12-04 10:15:43,530 INFO org.apache.hadoop.mapred.TaskTracker: Process Thread Dump: lost task
  21. 27 active threads
  22. Thread 135 (process reaper):
  23. State: RUNNABLE
  24. Blocked count: 0
  25. Waited count: 0
  26. Stack:
  27. java.lang.UNIXProcess.waitForProcessExit(Native Method)
  28. java.lang.UNIXProcess.access$900(UNIXProcess.java:36)
  29. java.lang.UNIXProcess$1$1.run(UNIXProcess.java:148)
  30. Thread 134 (JVM Runner jvm_201212040948_0005_m_-1576027144 spawned.):
  31. State: WAITING
  32. Blocked count: 1
  33. Waited count: 2
  34. Waiting on java.lang.UNIXProcess@7df36af6
  35. Stack:
  36. java.lang.Object.wait(Native Method)
  37. java.lang.Object.wait(Object.java:502)
  38. java.lang.UNIXProcess.waitFor(UNIXProcess.java:181)
  39. org.apache.hadoop.util.Shell.runCommand(Shell.java:250)
  40. org.apache.hadoop.util.Shell.run(Shell.java:188)
  41. org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
  42. org.apache.hadoop.mapred.DefaultTaskController.launchTask(DefaultTaskController.java:146)
  43. org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:497)
  44. org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:471)
  45. Thread 132 (Thread-81):
  46. State: WAITING
  47. Blocked count: 2
  48. Waited count: 3
  49. Waiting on java.lang.Object@1298a7eb
  50. Stack:
  51. java.lang.Object.wait(Native Method)
  52. java.lang.Object.wait(Object.java:502)
  53. org.apache.hadoop.mapred.TaskRunner.launchJvmAndWait(TaskRunner.java:275)
  54. org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:230)
  55. Thread 127 (Readahead Thread #0):
  56. State: WAITING
  57. Blocked count: 0
  58. Waited count: 1
  59. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a58701
  60. Stack:
  61. sun.misc.Unsafe.park(Native Method)
  62. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  63. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  64. java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:345)
  65. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
  66. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
  67. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  68. java.lang.Thread.run(Thread.java:679)
  69. Thread 53 (IPC Client (162291475) connection to namenode/10.10.10.2:8021 from mapred):
  70. State: TIMED_WAITING
  71. Blocked count: 11576
  72. Waited count: 11384
  73. Stack:
  74. java.lang.Object.wait(Native Method)
  75. org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:769)
  76. org.apache.hadoop.ipc.Client$Connection.run(Client.java:812)
  77. Thread 9 (taskCleanup):
  78. State: WAITING
  79. Blocked count: 7
  80. Waited count: 13
  81. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153b2cb
  82. Stack:
  83. sun.misc.Unsafe.park(Native Method)
  84. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  85. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  86. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  87. org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:568)
  88. java.lang.Thread.run(Thread.java:679)
  89. Thread 20 (Thread-6):
  90. State: WAITING
  91. Blocked count: 2
  92. Waited count: 9
  93. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c2bfdff
  94. Stack:
  95. sun.misc.Unsafe.park(Native Method)
  96. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  97. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  98. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  99. org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.monitor(UserLogManager.java:131)
  100. org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager$1.run(UserLogManager.java:66)
  101. Thread 21 (Thread-7):
  102. State: TIMED_WAITING
  103. Blocked count: 0
  104. Waited count: 1
  105. Stack:
  106. java.lang.Thread.sleep(Native Method)
  107. org.apache.hadoop.mapred.UserLogCleaner.run(UserLogCleaner.java:93)
  108. Thread 51 (Timer-0):
  109. State: TIMED_WAITING
  110. Blocked count: 0
  111. Waited count: 53
  112. Stack:
  113. java.lang.Object.wait(Native Method)
  114. java.util.TimerThread.mainLoop(Timer.java:531)
  115. java.util.TimerThread.run(Timer.java:484)
  116. Thread 50 (1958303722@qtp-108966758-0 - Acceptor0 SelectChannelConnector@0.0.0.0:50060):
  117. State: RUNNABLE
  118. Blocked count: 4
  119. Waited count: 1
  120. Stack:
  121. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  122. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  123. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
  124. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  125. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  126. org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:511)
  127. org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:193)
  128. org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
  129. org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
  130. org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  131. Thread 49 (Monitor for Jetty bugs):
  132. State: TIMED_WAITING
  133. Blocked count: 1
  134. Waited count: 107
  135. Stack:
  136. java.lang.Thread.sleep(Native Method)
  137. org.apache.hadoop.mapred.JettyBugMonitor.monitorThreads(JettyBugMonitor.java:142)
  138. org.apache.hadoop.mapred.JettyBugMonitor.doRun(JettyBugMonitor.java:121)
  139. org.apache.hadoop.mapred.JettyBugMonitor.run(JettyBugMonitor.java:102)
  140. Thread 48 (TaskLauncher for REDUCE tasks):
  141. State: WAITING
  142. Blocked count: 0
  143. Waited count: 1
  144. Waiting on java.util.LinkedList@63779885
  145. Stack:
  146. java.lang.Object.wait(Native Method)
  147. java.lang.Object.wait(Object.java:502)
  148. org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2631)
  149. Thread 47 (TaskLauncher for MAP tasks):
  150. State: WAITING
  151. Blocked count: 64
  152. Waited count: 65
  153. Waiting on java.util.LinkedList@18682406
  154. Stack:
  155. java.lang.Object.wait(Native Method)
  156. java.lang.Object.wait(Object.java:502)
  157. org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2631)
  158. Thread 40 (Map-events fetcher for all reduce tasks on tracker_datanode1:localhost/127.0.0.1:35316):
  159. State: WAITING
  160. Blocked count: 6
  161. Waited count: 7
  162. Waiting on java.util.TreeMap@126d2380
  163. Stack:
  164. java.lang.Object.wait(Native Method)
  165. java.lang.Object.wait(Object.java:502)
  166. org.apache.hadoop.mapred.TaskTracker$MapEventsFetcherThread.run(TaskTracker.java:1157)
  167. Thread 39 (IPC Server handler 3 on 35316):
  168. State: WAITING
  169. Blocked count: 0
  170. Waited count: 59
  171. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4235e6e3
  172. Stack:
  173. sun.misc.Unsafe.park(Native Method)
  174. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  175. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  176. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  177. org.apache.hadoop.ipc.Server$Handler.run(Server.java:1670)
  178. Thread 38 (IPC Server handler 2 on 35316):
  179. State: WAITING
  180. Blocked count: 0
  181. Waited count: 59
  182. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4235e6e3
  183. Stack:
  184. sun.misc.Unsafe.park(Native Method)
  185. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  186. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  187. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  188. org.apache.hadoop.ipc.Server$Handler.run(Server.java:1670)
  189. Thread 37 (IPC Server handler 1 on 35316):
  190. State: WAITING
  191. Blocked count: 0
  192. Waited count: 60
  193. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4235e6e3
  194. Stack:
  195. sun.misc.Unsafe.park(Native Method)
  196. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  197. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  198. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  199. org.apache.hadoop.ipc.Server$Handler.run(Server.java:1670)
  200. Thread 36 (IPC Server handler 0 on 35316):
  201. State: WAITING
  202. Blocked count: 0
  203. Waited count: 60
  204. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4235e6e3
  205. Stack:
  206. sun.misc.Unsafe.park(Native Method)
  207. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  208. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  209. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  210. org.apache.hadoop.ipc.Server$Handler.run(Server.java:1670)
  211. Thread 33 (IPC Server listener on 35316):
  212. State: RUNNABLE
  213. Blocked count: 6
  214. Waited count: 0
  215. Stack:
  216. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  217. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  218. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
  219. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  220. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  221. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
  222. org.apache.hadoop.ipc.Server$Listener.run(Server.java:612)
  223. Thread 35 (IPC Server Responder):
  224. State: RUNNABLE
  225. Blocked count: 0
  226. Waited count: 0
  227. Stack:
  228. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  229. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  230. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
  231. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  232. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  233. org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:796)
  234. org.apache.hadoop.ipc.Server$Responder.run(Server.java:779)
  235. Thread 34 (Socket Reader #1 for port 35316):
  236. State: RUNNABLE
  237. Blocked count: 6
  238. Waited count: 6
  239. Stack:
  240. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  241. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  242. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
  243. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  244. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  245. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
  246. org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:499)
  247. org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:485)
  248. Thread 22 (Directory/File cleanup thread):
  249. State: WAITING
  250. Blocked count: 2
  251. Waited count: 16
  252. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c5aebd9
  253. Stack:
  254. sun.misc.Unsafe.park(Native Method)
  255. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  256. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  257. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  258. org.apache.hadoop.mapred.CleanupQueue$PathCleanupThread.run(CleanupQueue.java:130)
  259. Thread 11 (HftpFileSystem-DelegationTokenRenewer):
  260. State: WAITING
  261. Blocked count: 0
  262. Waited count: 1
  263. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62803d5
  264. Stack:
  265. sun.misc.Unsafe.park(Native Method)
  266. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  267. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  268. java.util.concurrent.DelayQueue.take(DelayQueue.java:189)
  269. org.apache.hadoop.fs.DelegationTokenRenewer.run(DelegationTokenRenewer.java:152)
  270. Thread 4 (Signal Dispatcher):
  271. State: RUNNABLE
  272. Blocked count: 0
  273. Waited count: 0
  274. Stack:
  275. Thread 3 (Finalizer):
  276. State: WAITING
  277. Blocked count: 729
  278. Waited count: 727
  279. Waiting on java.lang.ref.ReferenceQueue$Lock@3c6833f2
  280. Stack:
  281. java.lang.Object.wait(Native Method)
  282. java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
  283. java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
  284. java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
  285. Thread 2 (Reference Handler):
  286. State: WAITING
  287. Blocked count: 729
  288. Waited count: 730
  289. Waiting on java.lang.ref.Reference$Lock@e1bdb58
  290. Stack:
  291. java.lang.Object.wait(Native Method)
  292. java.lang.Object.wait(Object.java:502)
  293. java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
  294. Thread 1 (main):
  295. State: RUNNABLE
  296. Blocked count: 11661
  297. Waited count: 22782
  298. Stack:
  299. sun.management.ThreadImpl.getThreadInfo0(Native Method)
  300. sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:165)
  301. sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:141)
  302. org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:162)
  303. org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:216)
  304. org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(TaskTracker.java:2335)
  305. org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:2002)
  306. org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2808)
  307. org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4033)
  308.  
  309. 2012-12-04 10:15:43,530 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201212040948_0005_m_000001_1
  310. 2012-12-04 10:15:43,554 INFO org.apache.hadoop.util.ProcessTree: Killing process group5632 with signal TERM. Exit code 0
  311. 2012-12-04 10:15:43,555 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
  312. 2012-12-04 10:15:43,558 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201212040948_0005_m_000001_1 not found in cache
  313. 2012-12-04 10:15:43,825 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  314. 2012-12-04 10:15:43,826 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201212040948_0005_m_000001_1 task's state:FAILED_UNCLEAN
  315. 2012-12-04 10:15:43,826 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201212040948_0005_m_000001_1 which needs 1 slots
  316. 2012-12-04 10:15:43,826 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201212040948_0005_m_000001_1 which needs 1 slots
  317. 2012-12-04 10:15:43,859 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201212040948_0005_m_-1075492596
  318. 2012-12-04 10:15:43,860 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201212040948_0005_m_-1075492596 spawned.
  319. 2012-12-04 10:15:43,864 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /data/2/mapred/local/ttprivate/taskTracker/vagrant/jobcache/job_201212040948_0005/attempt_201212040948_0005_m_000001_1.cleanup/taskjvm.sh
  320. 2012-12-04 10:15:44,042 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 143
  321. 2012-12-04 10:15:44,043 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201212040948_0005_m_-1576027144 exited with exit code 143. Number of tasks it ran: 0
  322. 2012-12-04 10:15:45,446 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201212040948_0005_m_-1075492596 given task: attempt_201212040948_0005_m_000001_1
  323. 2012-12-04 10:15:48,272 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212040948_0005_m_000001_1 0.0%
  324. 2012-12-04 10:15:48,476 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212040948_0005_m_000001_1 0.0% cleanup
  325. 2012-12-04 10:15:48,486 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201212040948_0005_m_000001_1 is done.
  326. 2012-12-04 10:15:48,486 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201212040948_0005_m_000001_1 was -1
  327. 2012-12-04 10:15:48,488 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
  328. 2012-12-04 10:15:48,781 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201212040948_0005_m_-1075492596 exited with exit code 0. Number of tasks it ran: 1
  329. 2012-12-04 10:15:51,213 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  330. 2012-12-04 10:15:51,216 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201212040948_0005_m_000000_2 task's state:UNASSIGNED
  331. 2012-12-04 10:15:51,217 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201212040948_0005_m_000000_2 which needs 1 slots
  332. 2012-12-04 10:15:51,217 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201212040948_0005_m_000000_2 which needs 1 slots
  333. 2012-12-04 10:15:51,281 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201212040948_0005_m_-1146419237
  334. 2012-12-04 10:15:51,282 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201212040948_0005_m_-1146419237 spawned.
  335. 2012-12-04 10:15:51,287 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /data/2/mapred/local/ttprivate/taskTracker/vagrant/jobcache/job_201212040948_0005/attempt_201212040948_0005_m_000000_2/taskjvm.sh
  336. 2012-12-04 10:15:53,192 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201212040948_0005_m_-1146419237 given task: attempt_201212040948_0005_m_000000_2
  337. 2012-12-04 10:16:01,504 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212040948_0005_m_000000_2 0.0% hdfs://namenode/user/vagrant/examples/lower/lower.txt:0+31
  338. 2012-12-04 10:26:01,677 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212040948_0005_m_000000_2: Task attempt_201212040948_0005_m_000000_2 failed to report status for 600 seconds. Killing!
  339. 2012-12-04 10:26:01,687 INFO org.apache.hadoop.mapred.TaskTracker: Process Thread Dump: lost task
  340. 27 active threads
  341. Thread 160 (process reaper):
  342. State: RUNNABLE
  343. Blocked count: 0
  344. Waited count: 0
  345. Stack:
  346. java.lang.UNIXProcess.waitForProcessExit(Native Method)
  347. java.lang.UNIXProcess.access$900(UNIXProcess.java:36)
  348. java.lang.UNIXProcess$1$1.run(UNIXProcess.java:148)
  349. Thread 159 (JVM Runner jvm_201212040948_0005_m_-1146419237 spawned.):
  350. State: WAITING
  351. Blocked count: 1
  352. Waited count: 2
  353. Waiting on java.lang.UNIXProcess@19129103
  354. Stack:
  355. java.lang.Object.wait(Native Method)
  356. java.lang.Object.wait(Object.java:502)
  357. java.lang.UNIXProcess.waitFor(UNIXProcess.java:181)
  358. org.apache.hadoop.util.Shell.runCommand(Shell.java:250)
  359. org.apache.hadoop.util.Shell.run(Shell.java:188)
  360. org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
  361. org.apache.hadoop.mapred.DefaultTaskController.launchTask(DefaultTaskController.java:146)
  362. org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:497)
  363. org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:471)
  364. Thread 157 (Thread-96):
  365. State: WAITING
  366. Blocked count: 2
  367. Waited count: 3
  368. Waiting on java.lang.Object@3b237a89
  369. Stack:
  370. java.lang.Object.wait(Native Method)
  371. java.lang.Object.wait(Object.java:502)
  372. org.apache.hadoop.mapred.TaskRunner.launchJvmAndWait(TaskRunner.java:275)
  373. org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:230)
  374. Thread 127 (Readahead Thread #0):
  375. State: WAITING
  376. Blocked count: 0
  377. Waited count: 1
  378. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a58701
  379. Stack:
  380. sun.misc.Unsafe.park(Native Method)
  381. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  382. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  383. java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:345)
  384. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
  385. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
  386. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  387. java.lang.Thread.run(Thread.java:679)
  388. Thread 53 (IPC Client (162291475) connection to namenode/10.10.10.2:8021 from mapred):
  389. State: TIMED_WAITING
  390. Blocked count: 13690
  391. Waited count: 13402
  392. Stack:
  393. java.lang.Object.wait(Native Method)
  394. org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:769)
  395. org.apache.hadoop.ipc.Client$Connection.run(Client.java:812)
  396. Thread 9 (taskCleanup):
  397. State: WAITING
  398. Blocked count: 7
  399. Waited count: 13
  400. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@153b2cb
  401. Stack:
  402. sun.misc.Unsafe.park(Native Method)
  403. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  404. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  405. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  406. org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:568)
  407. java.lang.Thread.run(Thread.java:679)
  408. Thread 20 (Thread-6):
  409. State: WAITING
  410. Blocked count: 2
  411. Waited count: 11
  412. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c2bfdff
  413. Stack:
  414. sun.misc.Unsafe.park(Native Method)
  415. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  416. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  417. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  418. org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.monitor(UserLogManager.java:131)
  419. org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager$1.run(UserLogManager.java:66)
  420. Thread 21 (Thread-7):
  421. State: TIMED_WAITING
  422. Blocked count: 0
  423. Waited count: 1
  424. Stack:
  425. java.lang.Thread.sleep(Native Method)
  426. org.apache.hadoop.mapred.UserLogCleaner.run(UserLogCleaner.java:93)
  427. Thread 51 (Timer-0):
  428. State: TIMED_WAITING
  429. Blocked count: 0
  430. Waited count: 74
  431. Stack:
  432. java.lang.Object.wait(Native Method)
  433. java.util.TimerThread.mainLoop(Timer.java:531)
  434. java.util.TimerThread.run(Timer.java:484)
  435. Thread 50 (1958303722@qtp-108966758-0 - Acceptor0 SelectChannelConnector@0.0.0.0:50060):
  436. State: RUNNABLE
  437. Blocked count: 4
  438. Waited count: 1
  439. Stack:
  440. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  441. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  442. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
  443. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  444. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  445. org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:511)
  446. org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:193)
  447. org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
  448. org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
  449. org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  450. Thread 49 (Monitor for Jetty bugs):
  451. State: TIMED_WAITING
  452. Blocked count: 1
  453. Waited count: 148
  454. Stack:
  455. java.lang.Thread.sleep(Native Method)
  456. org.apache.hadoop.mapred.JettyBugMonitor.monitorThreads(JettyBugMonitor.java:142)
  457. org.apache.hadoop.mapred.JettyBugMonitor.doRun(JettyBugMonitor.java:121)
  458. org.apache.hadoop.mapred.JettyBugMonitor.run(JettyBugMonitor.java:102)
  459. Thread 48 (TaskLauncher for REDUCE tasks):
  460. State: WAITING
  461. Blocked count: 0
  462. Waited count: 1
  463. Waiting on java.util.LinkedList@63779885
  464. Stack:
  465. java.lang.Object.wait(Native Method)
  466. java.lang.Object.wait(Object.java:502)
  467. org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2631)
  468. Thread 47 (TaskLauncher for MAP tasks):
  469. State: WAITING
  470. Blocked count: 66
  471. Waited count: 67
  472. Waiting on java.util.LinkedList@18682406
  473. Stack:
  474. java.lang.Object.wait(Native Method)
  475. java.lang.Object.wait(Object.java:502)
  476. org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2631)
  477. Thread 40 (Map-events fetcher for all reduce tasks on tracker_datanode1:localhost/127.0.0.1:35316):
  478. State: WAITING
  479. Blocked count: 8
  480. Waited count: 9
  481. Waiting on java.util.TreeMap@126d2380
  482. Stack:
  483. java.lang.Object.wait(Native Method)
  484. java.lang.Object.wait(Object.java:502)
  485. org.apache.hadoop.mapred.TaskTracker$MapEventsFetcherThread.run(TaskTracker.java:1157)
  486. Thread 39 (IPC Server handler 3 on 35316):
  487. State: WAITING
  488. Blocked count: 0
  489. Waited count: 111
  490. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4235e6e3
  491. Stack:
  492. sun.misc.Unsafe.park(Native Method)
  493. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  494. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  495. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  496. org.apache.hadoop.ipc.Server$Handler.run(Server.java:1670)
  497. Thread 38 (IPC Server handler 2 on 35316):
  498. State: WAITING
  499. Blocked count: 0
  500. Waited count: 111
  501. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4235e6e3
  502. Stack:
  503. sun.misc.Unsafe.park(Native Method)
  504. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  505. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  506. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  507. org.apache.hadoop.ipc.Server$Handler.run(Server.java:1670)
  508. Thread 37 (IPC Server handler 1 on 35316):
  509. State: WAITING
  510. Blocked count: 0
  511. Waited count: 111
  512. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4235e6e3
  513. Stack:
  514. sun.misc.Unsafe.park(Native Method)
  515. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  516. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  517. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  518. org.apache.hadoop.ipc.Server$Handler.run(Server.java:1670)
  519. Thread 36 (IPC Server handler 0 on 35316):
  520. State: WAITING
  521. Blocked count: 0
  522. Waited count: 111
  523. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4235e6e3
  524. Stack:
  525. sun.misc.Unsafe.park(Native Method)
  526. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  527. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  528. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  529. org.apache.hadoop.ipc.Server$Handler.run(Server.java:1670)
  530. Thread 33 (IPC Server listener on 35316):
  531. State: RUNNABLE
  532. Blocked count: 8
  533. Waited count: 0
  534. Stack:
  535. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  536. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  537. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
  538. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  539. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  540. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
  541. org.apache.hadoop.ipc.Server$Listener.run(Server.java:612)
  542. Thread 35 (IPC Server Responder):
  543. State: RUNNABLE
  544. Blocked count: 0
  545. Waited count: 0
  546. Stack:
  547. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  548. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  549. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
  550. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  551. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  552. org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:796)
  553. org.apache.hadoop.ipc.Server$Responder.run(Server.java:779)
  554. Thread 34 (Socket Reader #1 for port 35316):
  555. State: RUNNABLE
  556. Blocked count: 8
  557. Waited count: 8
  558. Stack:
  559. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  560. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  561. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
  562. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  563. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  564. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
  565. org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:499)
  566. org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:485)
  567. Thread 22 (Directory/File cleanup thread):
  568. State: WAITING
  569. Blocked count: 2
  570. Waited count: 21
  571. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c5aebd9
  572. Stack:
  573. sun.misc.Unsafe.park(Native Method)
  574. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  575. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  576. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
  577. org.apache.hadoop.mapred.CleanupQueue$PathCleanupThread.run(CleanupQueue.java:130)
  578. Thread 11 (HftpFileSystem-DelegationTokenRenewer):
  579. State: WAITING
  580. Blocked count: 0
  581. Waited count: 1
  582. Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62803d5
  583. Stack:
  584. sun.misc.Unsafe.park(Native Method)
  585. java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  586. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  587. java.util.concurrent.DelayQueue.take(DelayQueue.java:189)
  588. org.apache.hadoop.fs.DelegationTokenRenewer.run(DelegationTokenRenewer.java:152)
  589. Thread 4 (Signal Dispatcher):
  590. State: RUNNABLE
  591. Blocked count: 0
  592. Waited count: 0
  593. Stack:
  594. Thread 3 (Finalizer):
  595. State: WAITING
  596. Blocked count: 847
  597. Waited count: 845
  598. Waiting on java.lang.ref.ReferenceQueue$Lock@3c6833f2
  599. Stack:
  600. java.lang.Object.wait(Native Method)
  601. java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
  602. java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
  603. java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
  604. Thread 2 (Reference Handler):
  605. State: WAITING
  606. Blocked count: 848
  607. Waited count: 848
  608. Waiting on java.lang.ref.Reference$Lock@e1bdb58
  609. Stack:
  610. java.lang.Object.wait(Native Method)
  611. java.lang.Object.wait(Object.java:502)
  612. java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
  613. Thread 1 (main):
  614. java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
  615. java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
  616. java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
  617. Thread 2 (Reference Handler):
  618. State: WAITING
  619. Blocked count: 848
  620. Waited count: 848
  621. Waiting on java.lang.ref.Reference$Lock@e1bdb58
  622. Stack:
  623. java.lang.Object.wait(Native Method)
  624. java.lang.Object.wait(Object.java:502)
  625. java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
  626. Thread 1 (main):
  627. State: RUNNABLE
  628. Blocked count: 13817
  629. Waited count: 26823
  630. Stack:
  631. sun.management.ThreadImpl.getThreadInfo0(Native Method)
  632. sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:165)
  633. sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:141)
  634. org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:162)
  635. org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:216)
  636. org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(TaskTracker.java:2335)
  637. org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:2002)
  638. org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2808)
  639. org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4033)
  640.  
  641. 2012-12-04 10:26:01,687 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201212040948_0005_m_000000_2
  642. 2012-12-04 10:26:01,712 INFO org.apache.hadoop.util.ProcessTree: Killing process group5759 with signal TERM. Exit code 0
  643. 2012-12-04 10:26:01,713 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
  644. 2012-12-04 10:26:01,713 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201212040948_0005_m_000000_2 not found in cache
  645. 2012-12-04 10:26:02,082 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  646. 2012-12-04 10:26:02,085 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201212040948_0005_m_000000_2 task's state:FAILED_UNCLEAN
  647. 2012-12-04 10:26:02,085 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201212040948_0005_m_000000_2 which needs 1 slots
  648. 2012-12-04 10:26:02,085 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201212040948_0005_m_000000_2 which needs 1 slots
  649. 2012-12-04 10:26:02,126 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201212040948_0005_m_2110674336
  650. 2012-12-04 10:26:02,127 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201212040948_0005_m_2110674336 spawned.
  651. 2012-12-04 10:26:02,135 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /data/2/mapred/local/ttprivate/taskTracker/vagrant/jobcache/job_201212040948_0005/attempt_201212040948_0005_m_000000_2.cleanup/taskjvm.sh
  652. 2012-12-04 10:26:02,300 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 143
  653. 2012-12-04 10:26:02,303 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201212040948_0005_m_-1146419237 exited with exit code 143. Number of tasks it ran: 0
  654. 2012-12-04 10:26:03,495 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201212040948_0005_m_2110674336 given task: attempt_201212040948_0005_m_000000_2
  655. 2012-12-04 10:26:05,172 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212040948_0005_m_000000_2 0.0%
  656. 2012-12-04 10:26:05,489 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212040948_0005_m_000000_2 0.0% cleanup
  657. 2012-12-04 10:26:05,491 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201212040948_0005_m_000000_2 is done.
  658. 2012-12-04 10:26:05,491 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201212040948_0005_m_000000_2 was -1
  659. 2012-12-04 10:26:05,491 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
  660. 2012-12-04 10:26:05,657 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201212040948_0005_m_2110674336 exited with exit code 0. Number of tasks it ran: 1
  661. 2012-12-04 10:36:29,513 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201212040948_0005
  662. 2012-12-04 10:36:29,515 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201212040948_0005 for user-log deletion with retainTimeStamp:1354703789514
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!