API tools faq
paste
Advertisement
Guest User

Updated Region Server log

a guest
Feb 27th, 2014
100
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 12.84 KB | None | 0 0
raw download clone embed print report
  1. 2014-02-21 13:34:22,497 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":12619,"call":"next(-6030489538921985429, 500), rpc version=1, client version=29, methodsFingerPrint=-56040613","client":"10.0.0.96:37210","starttimems":1393007649877,"queuetimems":0,"class":"HRegionServer","responsesize":6,"method":"next"}
  2. 2014-02-21 13:36:05,009 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.14 GB, free=326.76 MB, max=1.46 GB, blocks=11981, accesses=38164394, hits=36651135, hitRatio=96.03%, cachingAccesses=35917193, cachingHits=35592736, cachingHitsRatio=99.09%, evictions=154, evicted=312476, evictedPerRun=2029.06494140625
  3. 2014-02-21 13:36:14,213 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":28227,"call":"next(220282580609584272, 1), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.0.0.91:52602","starttimems":1393007745976,"queuetimems":0,"class":"HRegionServer","responsesize":6,"method":"next"}
  4. 2014-02-21 13:36:16,071 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":28654,"call":"next(-5619091692443609352, 1), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.0.0.91:52603","starttimems":1393007747416,"queuetimems":27,"class":"HRegionServer","responsesize":6,"method":"next"}
  5. 2014-02-21 13:36:18,359 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":30886,"call":"next(8085229352464661555, 1), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.0.0.91:52604","starttimems":1393007747472,"queuetimems":0,"class":"HRegionServer","responsesize":6,"method":"next"}
  6. 2014-02-21 13:36:27,496 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41236,"call":"next(-8680499896692404689, 1), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.0.0.96:46618","starttimems":1393007746259,"queuetimems":0,"class":"HRegionServer","responsesize":6,"method":"next"}
  7. 2014-02-21 13:41:00,272 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 10193644ms instead of 10000000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
  8. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sd because one (major) compacted file only and oldestTime 130726976ms is < ttl=9223372036854775807
  9. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of doc because one (major) compacted file only and oldestTime 11711480440ms is < ttl=9223372036854775807
  10. 2014-02-21 13:41:00,273 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 260409ms for sessionid 0x34432befe5417d2, closing socket connection and attempting reconnect
  11. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of pt because one (major) compacted file only and oldestTime 11693284375ms is < ttl=9223372036854775807
  12. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sc because one (major) compacted file only and oldestTime 11711480440ms is < ttl=9223372036854775807
  13. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of docd because one (major) compacted file only and oldestTime 11694603255ms is < ttl=9223372036854775807
  14. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sd because one (major) compacted file only and oldestTime 11711485527ms is < ttl=9223372036854775807
  15. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of docd because one (major) compacted file only and oldestTime 11694606250ms is < ttl=9223372036854775807
  16. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of docd because one (major) compacted file only and oldestTime 11694606169ms is < ttl=9223372036854775807
  17. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of doc because one (major) compacted file only and oldestTime 11711485510ms is < ttl=9223372036854775807
  18. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of pt because one (major) compacted file only and oldestTime 11692247337ms is < ttl=9223372036854775807
  19. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sc because one (major) compacted file only and oldestTime 11711485510ms is < ttl=9223372036854775807
  20. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of docd because one (major) compacted file only and oldestTime 11694604988ms is < ttl=9223372036854775807
  21. 2014-02-21 13:41:00,273 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of nds because one (major) compacted file only and oldestTime 10981063307ms is < ttl=9223372036854775807
  22. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sd because one (major) compacted file only and oldestTime 11689589012ms is < ttl=9223372036854775807
  23. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of doc because one (major) compacted file only and oldestTime 11711481592ms is < ttl=9223372036854775807
  24. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of pt because one (major) compacted file only and oldestTime 11692822789ms is < ttl=9223372036854775807
  25. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sc because one (major) compacted file only and oldestTime 11711481592ms is < ttl=9223372036854775807
  26. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of docd because one (major) compacted file only and oldestTime 11694599683ms is < ttl=9223372036854775807
  27. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of doc because one (major) compacted file only and oldestTime 11711486153ms is < ttl=9223372036854775807
  28. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of pt because one (major) compacted file only and oldestTime 11693278669ms is < ttl=9223372036854775807
  29. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sc because one (major) compacted file only and oldestTime 11711486153ms is < ttl=9223372036854775807
  30. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of c because one (major) compacted file only and oldestTime 11694961271ms is < ttl=9223372036854775807
  31. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sg because one (major) compacted file only and oldestTime 11694961271ms is < ttl=9223372036854775807
  32. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of docd because one (major) compacted file only and oldestTime 11694603437ms is < ttl=9223372036854775807
  33. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of docd because one (major) compacted file only and oldestTime 11694598767ms is < ttl=9223372036854775807
  34. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sd because one (major) compacted file only and oldestTime 11689577359ms is < ttl=9223372036854775807
  35. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of docd because one (major) compacted file only and oldestTime 11694602251ms is < ttl=9223372036854775807
  36. 2014-02-21 13:41:00,274 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":257565,"call":"next(8725763673633941339, 1), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.0.0.108:54755","starttimems":1393007802708,"queuetimems":4,"class":"HRegionServer","responsesize":274131,"method":"next"}
  37. 2014-02-21 13:41:00,274 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of sd because one (major) compacted file only and oldestTime 11711483303ms is < ttl=9223372036854775807
  38. 2014-02-21 13:41:00,274 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1886644171203631409 lease expired on region metadata_810091,10025-810091-0000000000000547166,1393003740325.01abf9aa5dc2a0054885d3e6a4956563.
  39. .
  40. .
  41. .
  42.  
  43. 2014-02-21 13:41:00,308 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -8722095645979020016 lease expired on region metadata_810091,10025-810091-0000000000000477722,1393003740325.7a7b4b2889d72297c72792bfc302d1f6.
  44. 2014-02-21 13:41:00,298 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":257640,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@690d127b), rpc version=1, client version=29, methodsFingerPrint=-56040613","client":"10.0.0.151:52329","starttimems":1393007802656,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  45. 2014-02-21 13:41:00,283 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":257647,"client":"10.0.0.108:57247","timeRange":[0,9223372036854775807],"starttimems":1393007802634,"responsesize":0,"class":"HRegionServer","table":"content_810091","cacheBlocks":true,"families":{"pt":["ALL"],"doc":["ALL"],"sc":["ALL"]},"row":"10025-810091-E181134FC2789E085B0483E209EE11F3","queuetimems":0,"method":"exists","totalColumns":3,"maxVersions":1}
  46. 2014-02-21 13:41:00,277 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-6695300470410774365_837638java.io.EOFException
  47. at java.io.DataInputStream.readFully(DataInputStream.java:180)
  48. at java.io.DataInputStream.readLong(DataInputStream.java:399)
  49. at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:122)
  50. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2547)
  51.  
  52. .
  53. .
  54. .
  55. 2014-02-21 13:41:00,277 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 259713ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
  56. 2014-02-21 13:41:00,276 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 295028ms for sessionid 0x243fbf83f6031da, closing socket connection and attempting reconnect
  57. 2014-02-21 13:41:00,332 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6695300470410774365_837638 bad datanode[0] 10.0.0.151:50010
  58. 2014-02-21 13:41:00,310 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -5629389063728817567 lease expired on region metadata_810091,,1392992886436.a0ea2ab244301b2ba5c377653435ca2b.
  59. 2014-02-21 13:41:00,334 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6695300470410774365_837638 in pipeline 10.0.0.151:50010, 10.0.0.108:50010, 10.0.0.96:50010: bad datanode 10.0.0.151:50010
  60. 2014-02-21 13:41:00,340 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -1844654193444347130 lease expired on region metadata_810091,10025-810091-0000000000000477722,1393003740325.7a7b4b2889d72297c72792bfc302d1f6.
  61. 2014-02-21 13:41:00,350 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server Tapi7,60020,1392967864961: Unhandled exception: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing Tapi7,60020,1392967864961 as dead server
  62. org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing Tapi7,60020,1392967864961 as dead server
  63. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  64. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  65. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  66. at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  67. at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
  68. at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)
  69. at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:852)
  70. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:723)
  71. at java.lang.Thread.run(Thread.java:662)
  72. Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing Tapi7,60020,1392967864961 as dead server
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!