DataNode log: 2011-05-21 14:51:57,501 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting DataNode STARTUP_MSG: host = ap100/127.0.0.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.203.0 STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 ************************************************************/ 2011-05-21 14:51:59,297 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2011-05-21 14:51:59,383 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2011-05-21 14:51:59,438 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2011-05-21 14:51:59,438 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2011-05-21 14:52:00,003 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2011-05-21 14:52:00,012 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2011-05-21 14:52:06,654 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean 2011-05-21 14:52:06,690 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010 2011-05-21 14:52:06,696 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s 2011-05-21 14:52:11,825 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-05-21 14:52:12,030 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2011-05-21 14:52:12,056 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 2011-05-21 14:52:12,057 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 2011-05-21 14:52:12,057 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075 2011-05-21 14:52:12,057 INFO org.mortbay.log: jetty-6.1.26 2011-05-21 14:52:13,025 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 2011-05-21 14:52:13,034 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered. 2011-05-21 14:52:13,036 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source DataNode registered. 2011-05-21 14:52:19,297 INFO org.apache.hadoop.ipc.Server: Starting SocketReader 2011-05-21 14:52:19,302 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort50020 registered. 2011-05-21 14:52:19,302 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort50020 registered. 2011-05-21 14:52:19,306 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(localhost.localdomain:50010, storageID=DS-2090647581-127.0.0.1-50010-1305899294955, infoPort=50075, ipcPort=50020) 2011-05-21 14:52:19,313 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.0.3:50010, storageID=DS-2090647581-127.0.0.1-50010-1305899294955, infoPort=50075, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/usr/local/hadoop/temp/dfs/data/current'} 2011-05-21 14:52:19,318 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting 2011-05-21 14:52:19,319 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2011-05-21 14:52:19,320 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50020: starting 2011-05-21 14:52:19,320 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50020: starting 2011-05-21 14:52:19,321 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: using BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec 2011-05-21 14:52:19,322 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 50020: starting 2011-05-21 14:52:19,382 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 82 blocks took 28 msec to generate and 28 msecs for RPC and NN processing 2011-05-21 14:52:19,398 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic block scanner. 2011-05-21 14:52:19,618 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_4373949598800422770_1077 2011-05-21 14:52:50,780 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_2896425868153669032_1195 src: /192.168.0.2:55716 dest: /192.168.0.3:50010 2011-05-21 14:52:50,791 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received block blk_2896425868153669032_1195 src: /192.168.0.2:55716 dest: /192.168.0.3:50010 of size 4 2011-05-21 14:52:52,351 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_-7881936686607865616_1185 file /usr/local/hadoop/temp/dfs/data/current/subdir49/blk_-7881936686607865616 2011-05-21 14:52:56,510 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-1831692506900310290_1196 src: /192.168.0.2:55720 dest: /192.168.0.3:50010 2011-05-21 14:52:56,526 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:55720, dest: /192.168.0.3:50010, bytes: 4, op: HDFS_WRITE, cliID: DFSClient_1360734531, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-1831692506900310290_1196, duration: 8514035 2011-05-21 14:52:56,528 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-1831692506900310290_1196 terminating 2011-05-21 14:53:01,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_2896425868153669032_1195 file /usr/local/hadoop/temp/dfs/data/current/blk_2896425868153669032 2011-05-21 14:54:14,549 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_2060505820672246947_1197 src: /192.168.0.2:55744 dest: /192.168.0.3:50010 2011-05-21 14:54:14,561 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:55744, dest: /192.168.0.3:50010, bytes: 4, op: HDFS_WRITE, cliID: DFSClient_1190799726, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_2060505820672246947_1197, duration: 9593850 2011-05-21 14:54:14,564 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_2060505820672246947_1197 terminating 2011-05-21 14:54:19,364 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_-1831692506900310290_1196 file /usr/local/hadoop/temp/dfs/data/current/subdir49/blk_-1831692506900310290 2011-05-21 14:55:33,272 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-4725654367568120_1198 src: /192.168.0.2:55754 dest: /192.168.0.3:50010 2011-05-21 14:55:33,300 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:55754, dest: /192.168.0.3:50010, bytes: 142469, op: HDFS_WRITE, cliID: DFSClient_1913125612, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-4725654367568120_1198, duration: 24745044 2011-05-21 14:55:33,300 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-4725654367568120_1198 terminating 2011-05-21 14:55:33,429 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_1918463932323568925_1199 src: /192.168.0.2:55756 dest: /192.168.0.3:50010 2011-05-21 14:55:33,434 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:55756, dest: /192.168.0.3:50010, bytes: 791, op: HDFS_WRITE, cliID: DFSClient_1913125612, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_1918463932323568925_1199, duration: 2167018 2011-05-21 14:55:33,435 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_1918463932323568925_1199 terminating 2011-05-21 14:55:33,484 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_2744064770254320717_1200 src: /192.168.0.2:55758 dest: /192.168.0.3:50010 2011-05-21 14:55:33,490 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:55758, dest: /192.168.0.3:50010, bytes: 369, op: HDFS_WRITE, cliID: DFSClient_1913125612, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_2744064770254320717_1200, duration: 2426221 2011-05-21 14:55:33,490 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_2744064770254320717_1200 terminating 2011-05-21 14:55:33,710 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-3932363914994255324_1201 src: /192.168.0.2:55760 dest: /192.168.0.3:50010 2011-05-21 14:55:33,716 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:55760, dest: /192.168.0.3:50010, bytes: 19846, op: HDFS_WRITE, cliID: DFSClient_1913125612, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-3932363914994255324_1201, duration: 3138333 2011-05-21 14:55:33,716 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-3932363914994255324_1201 terminating 2011-05-21 14:55:34,101 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_5629827063427894174_1203 src: /192.168.0.2:55764 dest: /192.168.0.3:50010 2011-05-21 14:55:34,108 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:55764, dest: /192.168.0.3:50010, bytes: 19911, op: HDFS_WRITE, cliID: DFSClient_-50509214, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_5629827063427894174_1203, duration: 3419793 2011-05-21 14:55:34,108 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_5629827063427894174_1203 terminating 2011-05-21 14:55:34,146 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_6759089594327057366_1204 src: /192.168.0.2:55767 dest: /192.168.0.3:50010 2011-05-21 14:55:34,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:55767, dest: /192.168.0.3:50010, bytes: 106, op: HDFS_WRITE, cliID: DFSClient_1190799726, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_6759089594327057366_1204, duration: 4919279 2011-05-21 14:55:34,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_6759089594327057366_1204 terminating 2011-05-21 14:55:35,907 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52348, bytes: 110, op: HDFS_READ, cliID: DFSClient_-392503985, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_6759089594327057366_1204, duration: 1851036 2011-05-21 14:55:35,938 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52350, bytes: 20002, op: HDFS_READ, cliID: DFSClient_-1754713703, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-3932363914994255324_1201, duration: 1656543 2011-05-21 14:55:36,302 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52351, bytes: 143585, op: HDFS_READ, cliID: DFSClient_-1754713703, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-4725654367568120_1198, duration: 1939723 2011-05-21 14:55:44,668 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52362, bytes: 799, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000001_0, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_1918463932323568925_1199, duration: 781346 2011-05-21 14:55:44,675 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52363, bytes: 799, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000000_0, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_1918463932323568925_1199, duration: 7756815 2011-05-21 14:55:47,135 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52364, bytes: 1585404, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000001_0, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_5121398209261530714_1098, duration: 1833944199 2011-05-21 14:55:47,339 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52365, bytes: 1961090, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000000_0, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-6942899193516253617_1104, duration: 2078356978 2011-05-21 14:55:55,079 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52375, bytes: 799, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000004_0, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_1918463932323568925_1199, duration: 634643 2011-05-21 14:55:55,116 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52376, bytes: 283, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000005_0, offset: 512, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_1918463932323568925_1199, duration: 674162 2011-05-21 14:55:56,093 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52378, bytes: 397043, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000005_0, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_5248378979986262228_1101, duration: 443392379 2011-05-21 14:55:56,322 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52377, bytes: 599581, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000004_0, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_3160357645036143342_1102, duration: 709084214 2011-05-21 14:56:16,550 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52387, bytes: 799, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000003_1, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_1918463932323568925_1199, duration: 620951 2011-05-21 14:56:17,252 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52388, bytes: 679838, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000003_1, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_6496412950325448690_1100, duration: 386966343 2011-05-21 14:56:40,541 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52396, bytes: 799, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000002_1, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_1918463932323568925_1199, duration: 568824 2011-05-21 14:56:41,700 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52397, bytes: 1434925, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000002_1, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-8189473700879317678_1103, duration: 850349648 2011-05-21 14:57:01,576 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52405, bytes: 283, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000006_1, offset: 512, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_1918463932323568925_1199, duration: 556909 2011-05-21 14:57:02,109 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:50010, dest: /192.168.0.3:52407, bytes: 346383, op: HDFS_READ, cliID: DFSClient_attempt_201105211454_0001_m_000006_1, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-3455740601511534912_1099, duration: 207043050 2011-05-21 14:57:12,601 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_5930509450165114159_1205 src: /192.168.0.3:52412 dest: /192.168.0.3:50010 2011-05-21 14:57:13,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.3:52412, dest: /192.168.0.3:50010, bytes: 1412473, op: HDFS_WRITE, cliID: DFSClient_attempt_201105211454_0001_r_000000_0, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_5930509450165114159_1205, duration: 728785961 2011-05-21 14:57:13,339 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_5930509450165114159_1205 terminating 2011-05-21 14:57:24,127 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-4547735481832873182_1206 src: /192.168.0.2:34371 dest: /192.168.0.3:50010 2011-05-21 14:57:24,140 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.0.2:34371, dest: /192.168.0.3:50010, bytes: 33519, op: HDFS_WRITE, cliID: DFSClient_-50509214, offset: 0, srvID: DS-2090647581-127.0.0.1-50010-1305899294955, blockid: blk_-4547735481832873182_1206, duration: 9465149 2011-05-21 14:57:24,141 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-4547735481832873182_1206 terminating 2011-05-21 14:57:25,426 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_-3932363914994255324_1201 file /usr/local/hadoop/temp/dfs/data/current/subdir49/blk_-3932363914994255324 2011-05-21 14:57:25,427 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_-4725654367568120_1198 file /usr/local/hadoop/temp/dfs/data/current/subdir49/blk_-4725654367568120 2011-05-21 14:57:25,427 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_1918463932323568925_1199 file /usr/local/hadoop/temp/dfs/data/current/subdir49/blk_1918463932323568925 2011-05-21 14:57:25,427 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_2744064770254320717_1200 file /usr/local/hadoop/temp/dfs/data/current/subdir49/blk_2744064770254320717 2011-05-21 14:57:31,428 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_6759089594327057366_1204 file /usr/local/hadoop/temp/dfs/data/current/subdir49/blk_6759089594327057366 2011-05-21 15:02:19,733 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_5423469481465401267_1040 2011-05-21 15:12:19,820 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-5388130277041196252_1152 2011-05-21 15:14:37,799 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 85 blocks took 23 msec to generate and 7 msecs for RPC and NN processing