Tasktracker log:
2011-05-21 14:51:59,116 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = ap100/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.203.0
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
************************************************************/
2011-05-21 14:51:59,759 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2011-05-21 14:51:59,934 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2011-05-21 14:51:59,944 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2011-05-21 14:51:59,945 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
2011-05-21 14:52:01,231 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2011-05-21 14:52:01,239 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2011-05-21 14:52:06,654 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2011-05-21 14:52:06,843 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2011-05-21 14:52:06,916 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2011-05-21 14:52:06,924 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hadoop
2011-05-21 14:52:07,084 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2011-05-21 14:52:07,085 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
2011-05-21 14:52:07,139 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2011-05-21 14:52:07,143 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort49008 registered.
2011-05-21 14:52:07,143 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort49008 registered.
2011-05-21 14:52:07,147 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2011-05-21 14:52:07,149 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 49008: starting
2011-05-21 14:52:07,150 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 49008: starting
2011-05-21 14:52:07,150 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 49008: starting
2011-05-21 14:52:07,150 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 49008: starting
2011-05-21 14:52:07,154 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 49008: starting
2011-05-21 14:52:07,154 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost.localdomain/127.0.0.1:49008
2011-05-21 14:52:07,154 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:49008
2011-05-21 14:52:56,586 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:49008
2011-05-21 14:52:56,588 INFO org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@f9c40
2011-05-21 14:52:56,622 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2011-05-21 14:52:56,623 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2011-05-21 14:52:56,647 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
2011-05-21 14:52:56,675 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
2011-05-21 14:52:56,678 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
2011-05-21 14:52:56,679 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
2011-05-21 14:52:56,679 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
2011-05-21 14:52:56,679 INFO org.mortbay.log: jetty-6.1.26
2011-05-21 14:52:57,082 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
2011-05-21 14:52:57,082 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
2011-05-21 14:52:57,091 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105191244_0002 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,092 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105201436_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,092 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105201430_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,092 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105171858_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,092 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105191520_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,092 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105201316_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,092 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105191739_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,092 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105201343_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,092 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105191425_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,093 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105191441_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,093 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105191244_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,093 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105201500_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,093 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105191907_0001 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:52:57,093 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105191244_0003 for user-log deletion with retainTimeStamp:1306072376635
2011-05-21 14:53:48,280 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.io.IOException: Call to master/192.168.0.2:54311 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
at org.apache.hadoop.ipc.Client.call(Client.java:1033)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
at org.apache.hadoop.mapred.$Proxy5.heartbeat(Unknown Source)
at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1667)
at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1473)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2233)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3432)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
2011-05-21 14:53:48,280 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '15
2011-05-21 14:53:49,282 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 0 time(s).
2011-05-21 14:53:50,283 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 1 time(s).
2011-05-21 14:53:51,284 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 2 time(s).
2011-05-21 14:53:52,285 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 3 time(s).
2011-05-21 14:53:53,286 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 4 time(s).
2011-05-21 14:53:54,286 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 5 time(s).
2011-05-21 14:53:55,287 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 6 time(s).
2011-05-21 14:53:56,288 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 7 time(s).
2011-05-21 14:53:57,289 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 8 time(s).
2011-05-21 14:53:58,290 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 9 time(s).
2011-05-21 14:53:58,291 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.net.ConnectException: Call to master/192.168.0.2:54311 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1057)
at org.apache.hadoop.ipc.Client.call(Client.java:1033)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
at org.apache.hadoop.mapred.$Proxy5.heartbeat(Unknown Source)
at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1667)
at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1473)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2233)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3432)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:406)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:414)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:527)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:187)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164)
at org.apache.hadoop.ipc.Client.call(Client.java:1010)
... 6 more
2011-05-21 14:53:58,291 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '15
2011-05-21 14:53:59,293 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 0 time(s).
2011-05-21 14:54:00,294 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 1 time(s).
2011-05-21 14:54:01,294 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 2 time(s).
2011-05-21 14:54:02,295 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 3 time(s).
2011-05-21 14:54:03,296 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 4 time(s).
2011-05-21 14:54:04,297 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 5 time(s).
2011-05-21 14:54:05,298 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 6 time(s).
2011-05-21 14:54:06,299 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 7 time(s).
2011-05-21 14:54:07,300 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 8 time(s).
2011-05-21 14:54:08,301 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 9 time(s).
2011-05-21 14:54:08,302 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.net.ConnectException: Call to master/192.168.0.2:54311 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1057)
at org.apache.hadoop.ipc.Client.call(Client.java:1033)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
at org.apache.hadoop.mapred.$Proxy5.heartbeat(Unknown Source)
at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1667)
at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1473)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2233)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3432)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:406)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:414)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:527)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:187)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164)
at org.apache.hadoop.ipc.Client.call(Client.java:1010)
... 6 more
2011-05-21 14:54:08,302 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '15
2011-05-21 14:54:09,303 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.2:54311. Already tried 0 time(s).
2011-05-21 14:54:14,625 INFO org.apache.hadoop.mapred.TaskTracker: Recieved RenitTrackerAction from JobTracker
2011-05-21 14:54:14,652 INFO org.apache.hadoop.mapred.TaskTracker: Shutting down: Map-events fetcher for all reduce tasks on tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:49008
2011-05-21 14:54:14,653 INFO org.apache.hadoop.ipc.Server: Stopping server on 49008
2011-05-21 14:54:14,654 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 49008: exiting
2011-05-21 14:54:14,654 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 49008: exiting
2011-05-21 14:54:14,654 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 49008: exiting
2011-05-21 14:54:14,655 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 49008
2011-05-21 14:54:14,655 INFO org.apache.hadoop.ipc.metrics.RpcInstrumentation: shut down
2011-05-21 14:54:14,655 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 49008: exiting
2011-05-21 14:54:14,656 WARN org.apache.hadoop.mapred.TaskTracker: Reinitializing local state
2011-05-21 14:54:14,656 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2011-05-21 14:54:14,656 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hadoop
2011-05-21 14:54:14,660 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name jvm already exists!
2011-05-21 14:54:14,661 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name TaskTrackerMetrics already exists!
2011-05-21 14:54:14,662 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2011-05-21 14:54:14,662 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort38271 registered.
2011-05-21 14:54:14,663 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort38271 registered.
2011-05-21 14:54:14,663 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2011-05-21 14:54:14,665 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost.localdomain/127.0.0.1:38271
2011-05-21 14:54:14,665 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:38271
2011-05-21 14:54:14,674 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 38271: starting
2011-05-21 14:54:14,675 INFO org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@1eb5666
2011-05-21 14:54:14,675 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 38271: starting
2011-05-21 14:54:14,675 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 38271: starting
2011-05-21 14:54:14,675 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 38271: starting
2011-05-21 14:54:14,675 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 38271: starting
2011-05-21 14:54:14,678 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:38271
2011-05-21 14:54:14,677 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2011-05-21 14:54:14,679 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
2011-05-21 14:55:35,864 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000008_0 task's state:UNASSIGNED
2011-05-21 14:55:35,868 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000008_0 which needs 1 slots
2011-05-21 14:55:35,868 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201105211454_0001_m_000008_0 which needs 1 slots
2011-05-21 14:55:36,226 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user hadoop on this TT.
2011-05-21 14:55:36,477 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_-1435574959
2011-05-21 14:55:36,477 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_-1435574959 spawned.
2011-05-21 14:55:36,481 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000008_0/taskjvm.sh
2011-05-21 14:55:37,367 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_-1435574959 given task: attempt_201105211454_0001_m_000008_0
2011-05-21 14:55:41,066 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000008_0 0.0% setup
2011-05-21 14:55:41,068 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000008_0 is done.
2011-05-21 14:55:41,069 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000008_0 was -1
2011-05-21 14:55:41,070 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2011-05-21 14:55:41,155 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_-1435574959 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:55:41,188 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2011-05-21 14:55:41,233 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2011-05-21 14:55:41,234 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName hadoop for UID 1001 from the native implementation
2011-05-21 14:55:41,891 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000000_0 task's state:UNASSIGNED
2011-05-21 14:55:41,891 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000001_0 task's state:UNASSIGNED
2011-05-21 14:55:41,891 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000000_0 which needs 1 slots
2011-05-21 14:55:41,891 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201105211454_0001_m_000000_0 which needs 1 slots
2011-05-21 14:55:41,891 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201105211454_0001_m_000008_0
2011-05-21 14:55:41,892 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201105211454_0001_m_000008_0
2011-05-21 14:55:41,892 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201105211454_0001_m_000008_0 not found in cache
2011-05-21 14:55:41,898 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000001_0 which needs 1 slots
2011-05-21 14:55:41,898 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201105211454_0001_m_000001_0 which needs 1 slots
2011-05-21 14:55:41,903 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_147283024
2011-05-21 14:55:41,904 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_147283024 spawned.
2011-05-21 14:55:41,905 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_1732511660
2011-05-21 14:55:41,906 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_1732511660 spawned.
2011-05-21 14:55:41,907 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000000_0/taskjvm.sh
2011-05-21 14:55:41,925 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000001_0/taskjvm.sh
2011-05-21 14:55:43,470 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_147283024 given task: attempt_201105211454_0001_m_000000_0
2011-05-21 14:55:43,492 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_1732511660 given task: attempt_201105211454_0001_m_000001_0
2011-05-21 14:55:50,619 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000001_0 1.0%
2011-05-21 14:55:50,621 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000000_0 1.0%
2011-05-21 14:55:50,626 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000001_0 1.0%
2011-05-21 14:55:50,633 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000000_0 1.0%
2011-05-21 14:55:50,635 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000001_0 is done.
2011-05-21 14:55:50,635 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000001_0 was 740452
2011-05-21 14:55:50,636 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1
2011-05-21 14:55:50,639 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000000_0 is done.
2011-05-21 14:55:50,639 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000000_0 was 807006
2011-05-21 14:55:50,639 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2011-05-21 14:55:50,756 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_147283024 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:55:50,841 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_1732511660 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:55:50,955 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000004_0 task's state:UNASSIGNED
2011-05-21 14:55:50,957 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000005_0 task's state:UNASSIGNED
2011-05-21 14:55:50,957 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_r_000000_0 task's state:UNASSIGNED
2011-05-21 14:55:50,957 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_r_000000_0 which needs 1 slots
2011-05-21 14:55:50,958 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201105211454_0001_r_000000_0 which needs 1 slots
2011-05-21 14:55:50,958 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000004_0 which needs 1 slots
2011-05-21 14:55:50,958 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201105211454_0001_m_000004_0 which needs 1 slots
2011-05-21 14:55:50,968 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000005_0 which needs 1 slots
2011-05-21 14:55:50,968 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201105211454_0001_m_000005_0 which needs 1 slots
2011-05-21 14:55:50,973 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_r_-1435574959
2011-05-21 14:55:50,975 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_534546110
2011-05-21 14:55:50,976 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_r_-1435574959 spawned.
2011-05-21 14:55:50,976 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_534546110 spawned.
2011-05-21 14:55:50,983 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_-887848243
2011-05-21 14:55:50,983 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_-887848243 spawned.
2011-05-21 14:55:50,988 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000004_0/taskjvm.sh
2011-05-21 14:55:50,991 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000005_0/taskjvm.sh
2011-05-21 14:55:50,996 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_r_000000_0/taskjvm.sh
2011-05-21 14:55:52,929 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_r_-1435574959 given task: attempt_201105211454_0001_r_000000_0
2011-05-21 14:55:53,611 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_-887848243 given task: attempt_201105211454_0001_m_000005_0
2011-05-21 14:55:53,656 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_534546110 given task: attempt_201105211454_0001_m_000004_0
2011-05-21 14:55:58,027 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000004_0 1.0%
2011-05-21 14:55:58,029 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000004_0 is done.
2011-05-21 14:55:58,029 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000004_0 was 217654
2011-05-21 14:55:58,029 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1
2011-05-21 14:55:58,065 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000005_0 1.0%
2011-05-21 14:55:58,068 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000005_0 is done.
2011-05-21 14:55:58,068 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000005_0 was 252620
2011-05-21 14:55:58,069 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2011-05-21 14:55:58,198 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_-887848243 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:55:58,261 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_534546110 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:56:00,416 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.0% reduce > copy >
2011-05-21 14:56:01,294 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50184, bytes: 807006, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000000_0, duration: 71903989
2011-05-21 14:56:01,339 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50184, bytes: 740452, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000001_0, duration: 28054855
2011-05-21 14:56:01,373 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000003_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:01,374 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000003_0. Ignored.
2011-05-21 14:56:01,403 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50185, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000003_0, duration: 30652922
2011-05-21 14:56:01,403 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:06,416 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000003_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:06,417 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000003_0. Ignored.
2011-05-21 14:56:06,422 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:06,425 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50186, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000003_0, duration: 9686895
2011-05-21 14:56:06,426 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:11,449 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000003_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:11,450 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000003_0. Ignored.
2011-05-21 14:56:11,454 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50187, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000003_0, duration: 5148548
2011-05-21 14:56:11,454 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:12,428 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:14,998 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000003_1 task's state:UNASSIGNED
2011-05-21 14:56:14,998 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000003_1 which needs 1 slots
2011-05-21 14:56:14,998 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201105211454_0001_m_000003_1 which needs 1 slots
2011-05-21 14:56:15,001 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_-1887686110
2011-05-21 14:56:15,001 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_-1887686110 spawned.
2011-05-21 14:56:15,004 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_1/taskjvm.sh
2011-05-21 14:56:15,433 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:15,862 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_-1887686110 given task: attempt_201105211454_0001_m_000003_1
2011-05-21 14:56:16,502 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000003_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:16,503 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000003_0. Ignored.
2011-05-21 14:56:16,506 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50191, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000003_0, duration: 3102166
2011-05-21 14:56:16,506 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000003_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:19,521 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000003_1 1.0%
2011-05-21 14:56:19,523 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000003_1 is done.
2011-05-21 14:56:19,523 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000003_1 was 267013
2011-05-21 14:56:19,524 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2011-05-21 14:56:19,618 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_-1887686110 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:56:21,437 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:21,518 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000002_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:21,518 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000002_0. Ignored.
2011-05-21 14:56:21,528 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50194, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000002_0, duration: 10930024
2011-05-21 14:56:21,528 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:26,536 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000002_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:26,537 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000002_0. Ignored.
2011-05-21 14:56:26,539 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50195, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000002_0, duration: 3771412
2011-05-21 14:56:26,539 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:27,442 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:30,445 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:31,549 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000002_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:31,549 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000002_0. Ignored.
2011-05-21 14:56:31,552 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50196, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000002_0, duration: 4560347
2011-05-21 14:56:31,553 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:36,451 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:36,560 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000002_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:36,560 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000002_0. Ignored.
2011-05-21 14:56:36,562 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50197, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000002_0, duration: 3483028
2011-05-21 14:56:36,563 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:39,031 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000002_1 task's state:UNASSIGNED
2011-05-21 14:56:39,031 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000002_1 which needs 1 slots
2011-05-21 14:56:39,031 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201105211454_0001_m_000002_1 which needs 1 slots
2011-05-21 14:56:39,034 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_2057831501
2011-05-21 14:56:39,035 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_2057831501 spawned.
2011-05-21 14:56:39,037 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000002_1/taskjvm.sh
2011-05-21 14:56:39,897 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_2057831501 given task: attempt_201105211454_0001_m_000002_1
2011-05-21 14:56:41,574 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000006_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:41,575 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000006_0. Ignored.
2011-05-21 14:56:41,577 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50203, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000006_0, duration: 3580230
2011-05-21 14:56:41,578 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:42,457 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:43,506 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000002_1 1.0%
2011-05-21 14:56:43,508 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000002_1 is done.
2011-05-21 14:56:43,508 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000002_1 was 466814
2011-05-21 14:56:43,508 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2011-05-21 14:56:43,601 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_2057831501 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:56:45,461 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:46,583 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000006_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:46,584 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000006_0. Ignored.
2011-05-21 14:56:46,587 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50204, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000006_0, duration: 4018860
2011-05-21 14:56:46,587 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:51,465 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:56:51,594 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000006_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:51,594 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000006_0. Ignored.
2011-05-21 14:56:51,600 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50205, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000006_0, duration: 6509296
2011-05-21 14:56:51,600 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:56,615 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000006_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:56:56,616 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000006_0. Ignored.
2011-05-21 14:56:56,618 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50206, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000006_0, duration: 3181551
2011-05-21 14:56:56,618 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
2011-05-21 14:56:57,471 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:57:00,061 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000006_1 task's state:UNASSIGNED
2011-05-21 14:57:00,061 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000006_1 which needs 1 slots
2011-05-21 14:57:00,062 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201105211454_0001_m_000006_1 which needs 1 slots
2011-05-21 14:57:00,066 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_-1200608404
2011-05-21 14:57:00,066 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_-1200608404 spawned.
2011-05-21 14:57:00,069 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_1/taskjvm.sh
2011-05-21 14:57:00,474 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:57:00,930 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_-1200608404 given task: attempt_201105211454_0001_m_000006_1
2011-05-21 14:57:01,627 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201105211454_0001_m_000006_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3543)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2011-05-21 14:57:01,635 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201105211454_0001_m_000006_0. Ignored.
2011-05-21 14:57:01,642 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50211, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000006_0, duration: 15388871
2011-05-21 14:57:01,643 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000006_0/output/file.out.index in any of the configured local directories
2011-05-21 14:57:04,542 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000006_1 1.0%
2011-05-21 14:57:04,544 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000006_1 is done.
2011-05-21 14:57:04,545 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000006_1 was 163388
2011-05-21 14:57:04,545 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2011-05-21 14:57:04,638 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_-1200608404 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:57:06,479 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:57:06,660 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50213, bytes: 217654, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000004_0, duration: 12219858
2011-05-21 14:57:06,678 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50213, bytes: 252620, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000005_0, duration: 11314179
2011-05-21 14:57:06,723 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50213, bytes: 267013, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000003_1, duration: 18333850
2011-05-21 14:57:06,754 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50214, bytes: 466814, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000002_1, duration: 18786111
2011-05-21 14:57:11,769 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50215, bytes: 163388, op: MAPRED_SHUFFLE, cliID: attempt_201105211454_0001_m_000006_1, duration: 7220652
2011-05-21 14:57:11,777 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:57:12,325 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.095238104% reduce > copy (2 of 7 at 0.21 MB/s) >
2011-05-21 14:57:12,491 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.6685357% reduce > reduce
2011-05-21 14:57:13,366 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_r_000000_0 is in commit-pending, task state:COMMIT_PENDING
2011-05-21 14:57:13,367 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 0.6685357% reduce > reduce
2011-05-21 14:57:15,086 INFO org.apache.hadoop.mapred.TaskTracker: Received commit task action for attempt_201105211454_0001_r_000000_0
2011-05-21 14:57:15,494 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 1.0% reduce > reduce
2011-05-21 14:57:15,497 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_r_000000_0 1.0% reduce > reduce
2011-05-21 14:57:15,498 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_r_000000_0 is done.
2011-05-21 14:57:15,499 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_r_000000_0 was -1
2011-05-21 14:57:15,499 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2011-05-21 14:57:15,564 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_r_-1435574959 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:57:18,102 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201105211454_0001_m_000007_0 task's state:UNASSIGNED
2011-05-21 14:57:18,102 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201105211454_0001_r_000000_0
2011-05-21 14:57:18,104 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201105211454_0001_r_000000_0
2011-05-21 14:57:18,102 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201105211454_0001_m_000007_0 which needs 1 slots
2011-05-21 14:57:18,104 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201105211454_0001_m_000007_0 which needs 1 slots
2011-05-21 14:57:18,108 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201105211454_0001_m_1594890384
2011-05-21 14:57:18,109 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201105211454_0001_m_1594890384 spawned.
2011-05-21 14:57:18,111 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /usr/local/hadoop/temp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105211454_0001/attempt_201105211454_0001_m_000007_0/taskjvm.sh
2011-05-21 14:57:18,967 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201105211454_0001_m_1594890384 given task: attempt_201105211454_0001_m_000007_0
2011-05-21 14:57:19,588 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000007_0 0.0%
2011-05-21 14:57:22,578 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201105211454_0001_m_000007_0 0.0% cleanup
2011-05-21 14:57:22,580 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201105211454_0001_m_000007_0 is done.
2011-05-21 14:57:22,580 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201105211454_0001_m_000007_0 was -1
2011-05-21 14:57:22,581 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2011-05-21 14:57:22,653 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201105211454_0001_m_1594890384 exited with exit code 0. Number of tasks it ran: 1
2011-05-21 14:57:24,205 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201105211454_0001
2011-05-21 14:57:24,205 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201105211454_0001_m_000007_0 not found in cache
2011-05-21 14:57:24,207 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201105211454_0001 for user-log deletion with retainTimeStamp:1306072644207