API tools faq
paste
Advertisement
Guest User

Untitled

a guest
May 6th, 2011
310
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 71.98 KB | None | 0 0
raw download clone embed print report
  1. [root@scale-192-168-98-111 ~]# cfuse -m 192.168.98.111:6789 /mnt --log-file logfile --debug-client 20 --debug-ms 20 --debug-objectcacher 20
  2. ** WARNING: Ceph is still under heavy development, and is only suitable for **
  3. ** testing and review. Do not trust it with important data. **
  4. cfuse[19605]: starting ceph client
  5. cfuse[19605]: ceph mount failed with Connection timed out
  6. cfuse[19604]: mount failed: Connection timed out
  7. [root@scale-192-168-98-111 ~]# cat logfile
  8. 2011-05-06 21:39:41.446797 7f9a74e4a720 ceph version 0.27.1-301-gabdf5c8.commit: abdf5c86abbac1874c4244130afa25f4b52534b9. process: cfuse. pid: 19604
  9. 2011-05-06 21:39:41.447131 7f9a74e4a720 -- :/0 register_entity client?
  10. 2011-05-06 21:39:41.447149 7f9a74e4a720 -- :/0 register_entity client? at :/0
  11. 2011-05-06 21:39:41.447753 7f9a74e4a720 -- :/0 messenger.start
  12. 2011-05-06 21:39:41.448013 7f9a731ed710 -- :/19605 reaper_entry start
  13. 2011-05-06 21:39:41.448036 7f9a731ed710 -- :/19605 reaper
  14. 2011-05-06 21:39:41.448045 7f9a731ed710 -- :/19605 reaper done
  15. 2011-05-06 21:39:41.448096 7f9a74e4a720 -- :/19605 ready :/19605
  16. 2011-05-06 21:39:41.448126 7f9a721eb710 client?.objectcacher flusher start
  17. 2011-05-06 21:39:41.448317 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  18. 2011-05-06 21:39:41.448377 7f9a74e4a720 -- :/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x1c623a0
  19. 2011-05-06 21:39:41.448399 7f9a74e4a720 -- :/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  20. 2011-05-06 21:39:41.448410 7f9a74e4a720 -- :/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  21. 2011-05-06 21:39:41.448467 7f9a74e4a720 -- :/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=-1 pgs=0 cs=0 l=0).register_pipe
  22. 2011-05-06 21:39:41.448491 7f9a74e46710 -- :/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  23. 2011-05-06 21:39:41.448506 7f9a74e46710 -- :/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=-1 pgs=0 cs=0 l=0).connect 0
  24. 2011-05-06 21:39:41.448559 7f9a74e46710 -- :/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  25. 2011-05-06 21:39:41.448853 7f9a74e46710 -- :/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 4
  26. 2011-05-06 21:39:41.448875 7f9a74e46710 -- :/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38118/0
  27. 2011-05-06 21:39:41.448885 7f9a74e46710 -- 192.168.98.111:0/19605 learned my addr 192.168.98.111:0/19605
  28. 2011-05-06 21:39:41.448935 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  29. 2011-05-06 21:39:41.448952 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=0 cs=0 l=0).connect sending gseq=1 cseq=0 proto=15
  30. 2011-05-06 21:39:41.448980 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  31. 2011-05-06 21:39:41.448998 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 55 proto 15 flags 1
  32. 2011-05-06 21:39:41.449010 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).connect success 1, lossy = 1, features 506
  33. 2011-05-06 21:39:41.449035 7f9a74e46710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  34. 2011-05-06 21:39:41.449061 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).connect starting reader
  35. 2011-05-06 21:39:41.449093 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  36. 2011-05-06 21:39:41.449111 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  37. 2011-05-06 21:39:41.449127 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).writer: state = 2 policy.server=0
  38. 2011-05-06 21:39:41.449141 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).writer encoding 1 0x1c623a0 auth(proto 0 30 bytes) v1
  39. 2011-05-06 21:39:41.449164 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).writer sending 1 0x1c623a0
  40. 2011-05-06 21:39:41.449176 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).write_message 0x1c623a0
  41. 2011-05-06 21:39:41.449246 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).writer: state = 2 policy.server=0
  42. 2011-05-06 21:39:41.449259 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).writer sleeping
  43. 2011-05-06 21:39:41.449288 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).reader reading tag...
  44. 2011-05-06 21:39:41.654576 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).reader got ACK
  45. 2011-05-06 21:39:41.654617 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).reader reading tag...
  46. 2011-05-06 21:39:42.448412 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  47. 2011-05-06 21:39:43.448537 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  48. 2011-05-06 21:39:44.448311 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x1c625b0
  49. 2011-05-06 21:39:44.448354 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).unregister_pipe
  50. 2011-05-06 21:39:44.448366 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).stop
  51. 2011-05-06 21:39:44.448484 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a640008c0
  52. 2011-05-06 21:39:44.448507 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  53. 2011-05-06 21:39:44.448534 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  54. 2011-05-06 21:39:44.448618 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=-1 pgs=0 cs=0 l=0).register_pipe
  55. 2011-05-06 21:39:44.448682 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).reader couldn't read tag, Success
  56. 2011-05-06 21:39:44.448703 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).fault 0: Success
  57. 2011-05-06 21:39:44.448716 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).fault already closed|closing
  58. 2011-05-06 21:39:44.448742 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  59. 2011-05-06 21:39:44.448757 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=-1 pgs=0 cs=0 l=0).connect 0
  60. 2011-05-06 21:39:44.448771 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).reader done
  61. 2011-05-06 21:39:44.448788 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).writer finishing
  62. 2011-05-06 21:39:44.448806 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  63. 2011-05-06 21:39:44.448822 7f9a74e46710 -- 192.168.98.111:0/19605 queue_reap 0x1c625b0
  64. 2011-05-06 21:39:44.448843 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).writer done
  65. 2011-05-06 21:39:44.448884 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  66. 2011-05-06 21:39:44.448966 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  67. 2011-05-06 21:39:44.448987 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x1c625b0 192.168.98.111:6789/0
  68. 2011-05-06 21:39:44.448999 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).discard_queue
  69. 2011-05-06 21:39:44.449009 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1). dequeued pipe
  70. 2011-05-06 21:39:44.449045 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x1c625b0 sd=4 pgs=55 cs=1 l=1).unregister_pipe - not registered
  71. 2011-05-06 21:39:44.449078 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x1c625b0 192.168.98.111:6789/0
  72. 2011-05-06 21:39:44.449094 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x1c625b0
  73. 2011-05-06 21:39:44.449102 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  74. 2011-05-06 21:39:44.449178 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 6
  75. 2011-05-06 21:39:44.449193 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38119/0
  76. 2011-05-06 21:39:44.449226 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  77. 2011-05-06 21:39:44.449239 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=0 cs=0 l=0).connect sending gseq=2 cseq=0 proto=15
  78. 2011-05-06 21:39:44.449266 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  79. 2011-05-06 21:39:44.449283 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 56 proto 15 flags 1
  80. 2011-05-06 21:39:44.449295 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).connect success 1, lossy = 1, features 506
  81. 2011-05-06 21:39:44.449314 7f9a708e7710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  82. 2011-05-06 21:39:44.449333 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).connect starting reader
  83. 2011-05-06 21:39:44.449351 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  84. 2011-05-06 21:39:44.449367 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  85. 2011-05-06 21:39:44.449389 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).writer: state = 2 policy.server=0
  86. 2011-05-06 21:39:44.449404 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).writer encoding 1 0x7f9a640008c0 auth(proto 0 30 bytes) v1
  87. 2011-05-06 21:39:44.449421 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).writer sending 1 0x7f9a640008c0
  88. 2011-05-06 21:39:44.449433 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).write_message 0x7f9a640008c0
  89. 2011-05-06 21:39:44.449471 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).writer: state = 2 policy.server=0
  90. 2011-05-06 21:39:44.449483 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).writer sleeping
  91. 2011-05-06 21:39:44.449501 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).reader reading tag...
  92. 2011-05-06 21:39:44.656575 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).reader got ACK
  93. 2011-05-06 21:39:44.656614 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).reader reading tag...
  94. 2011-05-06 21:39:45.448974 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  95. 2011-05-06 21:39:46.466890 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  96. 2011-05-06 21:39:47.448876 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x7f9a64000b00
  97. 2011-05-06 21:39:47.448956 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).unregister_pipe
  98. 2011-05-06 21:39:47.448969 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).stop
  99. 2011-05-06 21:39:47.449056 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a640008c0
  100. 2011-05-06 21:39:47.449073 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  101. 2011-05-06 21:39:47.449083 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  102. 2011-05-06 21:39:47.449151 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=-1 pgs=0 cs=0 l=0).register_pipe
  103. 2011-05-06 21:39:47.449185 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).writer finishing
  104. 2011-05-06 21:39:47.449199 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).writer done
  105. 2011-05-06 21:39:47.449227 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).reader couldn't read tag, Success
  106. 2011-05-06 21:39:47.449247 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).fault 0: Success
  107. 2011-05-06 21:39:47.449263 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).fault already closed|closing
  108. 2011-05-06 21:39:47.449282 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  109. 2011-05-06 21:39:47.449298 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=-1 pgs=0 cs=0 l=0).connect 0
  110. 2011-05-06 21:39:47.449310 7f9a709e8710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a64000b00
  111. 2011-05-06 21:39:47.449332 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  112. 2011-05-06 21:39:47.449348 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).reader done
  113. 2011-05-06 21:39:47.449415 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  114. 2011-05-06 21:39:47.449430 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a64000b00 192.168.98.111:6789/0
  115. 2011-05-06 21:39:47.449443 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).discard_queue
  116. 2011-05-06 21:39:47.449453 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1). dequeued pipe
  117. 2011-05-06 21:39:47.449466 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64000b00 sd=6 pgs=56 cs=1 l=1).unregister_pipe - not registered
  118. 2011-05-06 21:39:47.449493 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a64000b00 192.168.98.111:6789/0
  119. 2011-05-06 21:39:47.449505 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a64000b00
  120. 2011-05-06 21:39:47.449513 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  121. 2011-05-06 21:39:47.449596 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 4
  122. 2011-05-06 21:39:47.449611 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38120/0
  123. 2011-05-06 21:39:47.449645 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  124. 2011-05-06 21:39:47.449658 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=0 cs=0 l=0).connect sending gseq=3 cseq=0 proto=15
  125. 2011-05-06 21:39:47.449706 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  126. 2011-05-06 21:39:47.449724 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 57 proto 15 flags 1
  127. 2011-05-06 21:39:47.449736 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).connect success 1, lossy = 1, features 506
  128. 2011-05-06 21:39:47.449749 7f9a74e46710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  129. 2011-05-06 21:39:47.449767 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).connect starting reader
  130. 2011-05-06 21:39:47.449785 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  131. 2011-05-06 21:39:47.449803 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).writer: state = 2 policy.server=0
  132. 2011-05-06 21:39:47.449819 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  133. 2011-05-06 21:39:47.449837 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).writer encoding 1 0x7f9a640008c0 auth(proto 0 30 bytes) v1
  134. 2011-05-06 21:39:47.449854 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).reader reading tag...
  135. 2011-05-06 21:39:47.449870 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).writer sending 1 0x7f9a640008c0
  136. 2011-05-06 21:39:47.449884 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).write_message 0x7f9a640008c0
  137. 2011-05-06 21:39:47.449924 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).writer: state = 2 policy.server=0
  138. 2011-05-06 21:39:47.449936 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).writer sleeping
  139. 2011-05-06 21:39:47.467005 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  140. 2011-05-06 21:39:47.656579 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).reader got ACK
  141. 2011-05-06 21:39:47.656616 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).reader reading tag...
  142. 2011-05-06 21:39:48.467100 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  143. 2011-05-06 21:39:49.467235 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  144. 2011-05-06 21:39:50.449257 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x7f9a640023c0
  145. 2011-05-06 21:39:50.449302 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).unregister_pipe
  146. 2011-05-06 21:39:50.510507 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).stop
  147. 2011-05-06 21:39:50.510549 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  148. 2011-05-06 21:39:50.510602 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a640008c0
  149. 2011-05-06 21:39:50.510619 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  150. 2011-05-06 21:39:50.510629 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  151. 2011-05-06 21:39:50.510696 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=-1 pgs=0 cs=0 l=0).register_pipe
  152. 2011-05-06 21:39:50.510764 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).writer finishing
  153. 2011-05-06 21:39:50.510780 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).writer done
  154. 2011-05-06 21:39:50.510858 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).reader couldn't read tag, Success
  155. 2011-05-06 21:39:50.510883 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).fault 0: Success
  156. 2011-05-06 21:39:50.510896 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).fault already closed|closing
  157. 2011-05-06 21:39:50.510914 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  158. 2011-05-06 21:39:50.510931 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=-1 pgs=0 cs=0 l=0).connect 0
  159. 2011-05-06 21:39:50.510956 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  160. 2011-05-06 21:39:50.510971 7f9a709e8710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a640023c0
  161. 2011-05-06 21:39:50.510993 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).reader done
  162. 2011-05-06 21:39:50.511050 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  163. 2011-05-06 21:39:50.511065 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a640023c0 192.168.98.111:6789/0
  164. 2011-05-06 21:39:50.511076 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).discard_queue
  165. 2011-05-06 21:39:50.511088 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1). dequeued pipe
  166. 2011-05-06 21:39:50.511101 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a640023c0 sd=4 pgs=57 cs=1 l=1).unregister_pipe - not registered
  167. 2011-05-06 21:39:50.511129 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a640023c0 192.168.98.111:6789/0
  168. 2011-05-06 21:39:50.511141 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a640023c0
  169. 2011-05-06 21:39:50.511149 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  170. 2011-05-06 21:39:50.511218 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 6
  171. 2011-05-06 21:39:50.511233 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38121/0
  172. 2011-05-06 21:39:50.511266 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  173. 2011-05-06 21:39:50.511279 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=0 cs=0 l=0).connect sending gseq=4 cseq=0 proto=15
  174. 2011-05-06 21:39:50.511304 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  175. 2011-05-06 21:39:50.511321 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 58 proto 15 flags 1
  176. 2011-05-06 21:39:50.511333 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).connect success 1, lossy = 1, features 506
  177. 2011-05-06 21:39:50.511346 7f9a708e7710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  178. 2011-05-06 21:39:50.511363 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).connect starting reader
  179. 2011-05-06 21:39:50.511381 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  180. 2011-05-06 21:39:50.511427 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).writer: state = 2 policy.server=0
  181. 2011-05-06 21:39:50.511443 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  182. 2011-05-06 21:39:50.511460 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).writer encoding 1 0x7f9a640008c0 auth(proto 0 30 bytes) v1
  183. 2011-05-06 21:39:50.511476 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).reader reading tag...
  184. 2011-05-06 21:39:50.511492 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).writer sending 1 0x7f9a640008c0
  185. 2011-05-06 21:39:50.511505 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).write_message 0x7f9a640008c0
  186. 2011-05-06 21:39:50.511562 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).writer: state = 2 policy.server=0
  187. 2011-05-06 21:39:50.511577 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).writer sleeping
  188. 2011-05-06 21:39:50.718578 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).reader got ACK
  189. 2011-05-06 21:39:50.718617 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).reader reading tag...
  190. 2011-05-06 21:39:51.510659 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  191. 2011-05-06 21:39:52.510778 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  192. 2011-05-06 21:39:53.510835 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x7f9a64002140
  193. 2011-05-06 21:39:53.510881 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).unregister_pipe
  194. 2011-05-06 21:39:53.510897 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  195. 2011-05-06 21:39:53.510920 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).stop
  196. 2011-05-06 21:39:53.510986 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a640008c0
  197. 2011-05-06 21:39:53.511000 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  198. 2011-05-06 21:39:53.511010 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  199. 2011-05-06 21:39:53.511090 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=-1 pgs=0 cs=0 l=0).register_pipe
  200. 2011-05-06 21:39:53.511124 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).writer finishing
  201. 2011-05-06 21:39:53.511146 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).writer done
  202. 2011-05-06 21:39:53.511170 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).reader couldn't read tag, Success
  203. 2011-05-06 21:39:53.511189 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).fault 0: Success
  204. 2011-05-06 21:39:53.511200 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).fault already closed|closing
  205. 2011-05-06 21:39:53.511211 7f9a709e8710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a64002140
  206. 2011-05-06 21:39:53.511235 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).reader done
  207. 2011-05-06 21:39:53.511253 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  208. 2011-05-06 21:39:53.511296 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a64002140 192.168.98.111:6789/0
  209. 2011-05-06 21:39:53.511310 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).discard_queue
  210. 2011-05-06 21:39:53.511323 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1). dequeued pipe
  211. 2011-05-06 21:39:53.511339 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  212. 2011-05-06 21:39:53.511354 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=-1 pgs=0 cs=0 l=0).connect 0
  213. 2011-05-06 21:39:53.511369 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002140 sd=6 pgs=58 cs=1 l=1).unregister_pipe - not registered
  214. 2011-05-06 21:39:53.511386 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  215. 2011-05-06 21:39:53.511403 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a64002140 192.168.98.111:6789/0
  216. 2011-05-06 21:39:53.511416 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a64002140
  217. 2011-05-06 21:39:53.511424 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  218. 2011-05-06 21:39:53.511583 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 4
  219. 2011-05-06 21:39:53.511601 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38122/0
  220. 2011-05-06 21:39:53.511639 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  221. 2011-05-06 21:39:53.511652 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=0 cs=0 l=0).connect sending gseq=5 cseq=0 proto=15
  222. 2011-05-06 21:39:53.511679 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  223. 2011-05-06 21:39:53.511695 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 59 proto 15 flags 1
  224. 2011-05-06 21:39:53.511707 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).connect success 1, lossy = 1, features 506
  225. 2011-05-06 21:39:53.511720 7f9a74e46710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  226. 2011-05-06 21:39:53.511738 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).connect starting reader
  227. 2011-05-06 21:39:53.511756 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  228. 2011-05-06 21:39:53.511773 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).writer: state = 2 policy.server=0
  229. 2011-05-06 21:39:53.511790 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).writer encoding 1 0x7f9a640008c0 auth(proto 0 30 bytes) v1
  230. 2011-05-06 21:39:53.511802 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  231. 2011-05-06 21:39:53.511820 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).writer sending 1 0x7f9a640008c0
  232. 2011-05-06 21:39:53.511837 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).reader reading tag...
  233. 2011-05-06 21:39:53.511852 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).write_message 0x7f9a640008c0
  234. 2011-05-06 21:39:53.511890 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).writer: state = 2 policy.server=0
  235. 2011-05-06 21:39:53.511925 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).writer sleeping
  236. 2011-05-06 21:39:53.718573 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).reader got ACK
  237. 2011-05-06 21:39:53.718613 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).reader reading tag...
  238. 2011-05-06 21:39:54.510993 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  239. 2011-05-06 21:39:55.511108 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  240. 2011-05-06 21:39:56.511205 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x7f9a64002ad0
  241. 2011-05-06 21:39:56.511260 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).unregister_pipe
  242. 2011-05-06 21:39:56.511276 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  243. 2011-05-06 21:39:56.511299 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).stop
  244. 2011-05-06 21:39:56.511366 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a64002140
  245. 2011-05-06 21:39:56.511382 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  246. 2011-05-06 21:39:56.511392 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  247. 2011-05-06 21:39:56.511462 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=-1 pgs=0 cs=0 l=0).register_pipe
  248. 2011-05-06 21:39:56.511497 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).writer finishing
  249. 2011-05-06 21:39:56.511514 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).writer done
  250. 2011-05-06 21:39:56.511555 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).reader couldn't read tag, Success
  251. 2011-05-06 21:39:56.511570 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).fault 0: Success
  252. 2011-05-06 21:39:56.511581 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).fault already closed|closing
  253. 2011-05-06 21:39:56.511590 7f9a709e8710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a64002ad0
  254. 2011-05-06 21:39:56.511616 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  255. 2011-05-06 21:39:56.511633 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=-1 pgs=0 cs=0 l=0).connect 0
  256. 2011-05-06 21:39:56.511649 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  257. 2011-05-06 21:39:56.511664 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a64002ad0 192.168.98.111:6789/0
  258. 2011-05-06 21:39:56.511677 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).discard_queue
  259. 2011-05-06 21:39:56.511690 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1). dequeued pipe
  260. 2011-05-06 21:39:56.511705 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  261. 2011-05-06 21:39:56.511721 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).reader done
  262. 2011-05-06 21:39:56.511738 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002ad0 sd=4 pgs=59 cs=1 l=1).unregister_pipe - not registered
  263. 2011-05-06 21:39:56.511788 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a64002ad0 192.168.98.111:6789/0
  264. 2011-05-06 21:39:56.511844 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a64002ad0
  265. 2011-05-06 21:39:56.511853 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  266. 2011-05-06 21:39:56.511927 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 6
  267. 2011-05-06 21:39:56.511943 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38123/0
  268. 2011-05-06 21:39:56.511975 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  269. 2011-05-06 21:39:56.511988 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=0 cs=0 l=0).connect sending gseq=6 cseq=0 proto=15
  270. 2011-05-06 21:39:56.512010 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  271. 2011-05-06 21:39:56.512038 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 60 proto 15 flags 1
  272. 2011-05-06 21:39:56.512052 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).connect success 1, lossy = 1, features 506
  273. 2011-05-06 21:39:56.512065 7f9a708e7710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  274. 2011-05-06 21:39:56.512082 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).connect starting reader
  275. 2011-05-06 21:39:56.512101 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  276. 2011-05-06 21:39:56.512119 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).writer: state = 2 policy.server=0
  277. 2011-05-06 21:39:56.512135 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  278. 2011-05-06 21:39:56.512152 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).writer encoding 1 0x7f9a64002140 auth(proto 0 30 bytes) v1
  279. 2011-05-06 21:39:56.512169 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).reader reading tag...
  280. 2011-05-06 21:39:56.512184 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).writer sending 1 0x7f9a64002140
  281. 2011-05-06 21:39:56.512199 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).write_message 0x7f9a64002140
  282. 2011-05-06 21:39:56.512238 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).writer: state = 2 policy.server=0
  283. 2011-05-06 21:39:56.512251 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).writer sleeping
  284. 2011-05-06 21:39:56.718571 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).reader got ACK
  285. 2011-05-06 21:39:56.718612 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).reader reading tag...
  286. 2011-05-06 21:39:57.511370 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  287. 2011-05-06 21:39:58.511496 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  288. 2011-05-06 21:39:59.511582 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x7f9a64002350
  289. 2011-05-06 21:39:59.511634 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  290. 2011-05-06 21:39:59.511657 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).unregister_pipe
  291. 2011-05-06 21:39:59.511699 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).stop
  292. 2011-05-06 21:39:59.511765 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a64002140
  293. 2011-05-06 21:39:59.511780 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  294. 2011-05-06 21:39:59.511790 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  295. 2011-05-06 21:39:59.511868 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=-1 pgs=0 cs=0 l=0).register_pipe
  296. 2011-05-06 21:39:59.511923 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).writer finishing
  297. 2011-05-06 21:39:59.511949 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  298. 2011-05-06 21:39:59.511967 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=-1 pgs=0 cs=0 l=0).connect 0
  299. 2011-05-06 21:39:59.511991 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).reader couldn't read tag, Success
  300. 2011-05-06 21:39:59.512007 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).fault 0: Success
  301. 2011-05-06 21:39:59.512019 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).fault already closed|closing
  302. 2011-05-06 21:39:59.512034 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).writer done
  303. 2011-05-06 21:39:59.512051 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  304. 2011-05-06 21:39:59.512066 7f9a709e8710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a64002350
  305. 2011-05-06 21:39:59.512085 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).reader done
  306. 2011-05-06 21:39:59.512137 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  307. 2011-05-06 21:39:59.512152 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a64002350 192.168.98.111:6789/0
  308. 2011-05-06 21:39:59.512164 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).discard_queue
  309. 2011-05-06 21:39:59.512175 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1). dequeued pipe
  310. 2011-05-06 21:39:59.512189 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002350 sd=6 pgs=60 cs=1 l=1).unregister_pipe - not registered
  311. 2011-05-06 21:39:59.512216 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a64002350 192.168.98.111:6789/0
  312. 2011-05-06 21:39:59.512228 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a64002350
  313. 2011-05-06 21:39:59.512235 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  314. 2011-05-06 21:39:59.512305 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 4
  315. 2011-05-06 21:39:59.512319 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38124/0
  316. 2011-05-06 21:39:59.512351 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  317. 2011-05-06 21:39:59.512365 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=0 cs=0 l=0).connect sending gseq=7 cseq=0 proto=15
  318. 2011-05-06 21:39:59.512391 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  319. 2011-05-06 21:39:59.512433 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 61 proto 15 flags 1
  320. 2011-05-06 21:39:59.512446 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).connect success 1, lossy = 1, features 506
  321. 2011-05-06 21:39:59.512459 7f9a74e46710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  322. 2011-05-06 21:39:59.512495 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  323. 2011-05-06 21:39:59.512509 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  324. 2011-05-06 21:39:59.512565 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).connect starting reader
  325. 2011-05-06 21:39:59.512912 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).writer: state = 2 policy.server=0
  326. 2011-05-06 21:39:59.512942 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).writer encoding 1 0x7f9a64002140 auth(proto 0 30 bytes) v1
  327. 2011-05-06 21:39:59.512961 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).writer sending 1 0x7f9a64002140
  328. 2011-05-06 21:39:59.512974 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).write_message 0x7f9a64002140
  329. 2011-05-06 21:39:59.513047 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).writer: state = 2 policy.server=0
  330. 2011-05-06 21:39:59.513063 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).writer sleeping
  331. 2011-05-06 21:39:59.513106 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).reader reading tag...
  332. 2011-05-06 21:39:59.719559 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).reader got ACK
  333. 2011-05-06 21:39:59.719590 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).reader reading tag...
  334. 2011-05-06 21:40:00.511725 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  335. 2011-05-06 21:40:01.511840 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  336. 2011-05-06 21:40:02.511966 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  337. 2011-05-06 21:40:02.512025 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x7f9a64002870
  338. 2011-05-06 21:40:02.512044 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).unregister_pipe
  339. 2011-05-06 21:40:02.512057 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).stop
  340. 2011-05-06 21:40:02.512120 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a64002140
  341. 2011-05-06 21:40:02.512136 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  342. 2011-05-06 21:40:02.512145 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  343. 2011-05-06 21:40:02.512222 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=-1 pgs=0 cs=0 l=0).register_pipe
  344. 2011-05-06 21:40:02.512297 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).reader couldn't read tag, Success
  345. 2011-05-06 21:40:02.512318 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).fault 0: Success
  346. 2011-05-06 21:40:02.512368 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).fault already closed|closing
  347. 2011-05-06 21:40:02.512382 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).reader done
  348. 2011-05-06 21:40:02.512400 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).writer finishing
  349. 2011-05-06 21:40:02.512419 7f9a74e46710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a64002870
  350. 2011-05-06 21:40:02.512436 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  351. 2011-05-06 21:40:02.512452 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=-1 pgs=0 cs=0 l=0).connect 0
  352. 2011-05-06 21:40:02.512467 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  353. 2011-05-06 21:40:02.512482 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a64002870 192.168.98.111:6789/0
  354. 2011-05-06 21:40:02.512495 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).discard_queue
  355. 2011-05-06 21:40:02.512508 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1). dequeued pipe
  356. 2011-05-06 21:40:02.512533 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  357. 2011-05-06 21:40:02.512552 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).writer done
  358. 2011-05-06 21:40:02.512595 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002870 sd=4 pgs=61 cs=1 l=1).unregister_pipe - not registered
  359. 2011-05-06 21:40:02.512624 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a64002870 192.168.98.111:6789/0
  360. 2011-05-06 21:40:02.512637 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a64002870
  361. 2011-05-06 21:40:02.512645 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  362. 2011-05-06 21:40:02.512747 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 6
  363. 2011-05-06 21:40:02.512765 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38125/0
  364. 2011-05-06 21:40:02.512797 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  365. 2011-05-06 21:40:02.512810 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=0 cs=0 l=0).connect sending gseq=8 cseq=0 proto=15
  366. 2011-05-06 21:40:02.512831 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  367. 2011-05-06 21:40:02.512859 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 62 proto 15 flags 1
  368. 2011-05-06 21:40:02.512873 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).connect success 1, lossy = 1, features 506
  369. 2011-05-06 21:40:02.512886 7f9a708e7710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  370. 2011-05-06 21:40:02.512904 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).connect starting reader
  371. 2011-05-06 21:40:02.512921 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  372. 2011-05-06 21:40:02.512940 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).writer: state = 2 policy.server=0
  373. 2011-05-06 21:40:02.512956 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  374. 2011-05-06 21:40:02.512999 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).writer encoding 1 0x7f9a64002140 auth(proto 0 30 bytes) v1
  375. 2011-05-06 21:40:02.513016 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).reader reading tag...
  376. 2011-05-06 21:40:02.513032 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).writer sending 1 0x7f9a64002140
  377. 2011-05-06 21:40:02.513047 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).write_message 0x7f9a64002140
  378. 2011-05-06 21:40:02.513086 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).writer: state = 2 policy.server=0
  379. 2011-05-06 21:40:02.513098 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).writer sleeping
  380. 2011-05-06 21:40:02.719572 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).reader got ACK
  381. 2011-05-06 21:40:02.719612 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).reader reading tag...
  382. 2011-05-06 21:40:03.512091 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  383. 2011-05-06 21:40:04.512208 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  384. 2011-05-06 21:40:05.512334 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x7f9a64002c50
  385. 2011-05-06 21:40:05.512377 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  386. 2011-05-06 21:40:05.512400 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).unregister_pipe
  387. 2011-05-06 21:40:05.512414 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).stop
  388. 2011-05-06 21:40:05.512478 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a64002140
  389. 2011-05-06 21:40:05.512493 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  390. 2011-05-06 21:40:05.512502 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  391. 2011-05-06 21:40:05.512589 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=-1 pgs=0 cs=0 l=0).register_pipe
  392. 2011-05-06 21:40:05.512614 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  393. 2011-05-06 21:40:05.512673 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=-1 pgs=0 cs=0 l=0).connect 0
  394. 2011-05-06 21:40:05.512693 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).writer finishing
  395. 2011-05-06 21:40:05.512712 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  396. 2011-05-06 21:40:05.512735 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).writer done
  397. 2011-05-06 21:40:05.512780 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).reader couldn't read tag, Success
  398. 2011-05-06 21:40:05.512796 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).fault 0: Success
  399. 2011-05-06 21:40:05.512807 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).fault already closed|closing
  400. 2011-05-06 21:40:05.512818 7f9a709e8710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a64002c50
  401. 2011-05-06 21:40:05.512835 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).reader done
  402. 2011-05-06 21:40:05.512896 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  403. 2011-05-06 21:40:05.512911 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a64002c50 192.168.98.111:6789/0
  404. 2011-05-06 21:40:05.512922 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).discard_queue
  405. 2011-05-06 21:40:05.512933 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1). dequeued pipe
  406. 2011-05-06 21:40:05.512955 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 4
  407. 2011-05-06 21:40:05.512974 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38126/0
  408. 2011-05-06 21:40:05.512989 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002c50 sd=6 pgs=62 cs=1 l=1).unregister_pipe - not registered
  409. 2011-05-06 21:40:05.513005 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  410. 2011-05-06 21:40:05.513022 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a64002c50 192.168.98.111:6789/0
  411. 2011-05-06 21:40:05.513042 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=0 cs=0 l=0).connect sending gseq=9 cseq=0 proto=15
  412. 2011-05-06 21:40:05.513058 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a64002c50
  413. 2011-05-06 21:40:05.513070 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  414. 2011-05-06 21:40:05.513084 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  415. 2011-05-06 21:40:05.513122 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 63 proto 15 flags 1
  416. 2011-05-06 21:40:05.513136 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).connect success 1, lossy = 1, features 506
  417. 2011-05-06 21:40:05.513149 7f9a74e46710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  418. 2011-05-06 21:40:05.513167 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).connect starting reader
  419. 2011-05-06 21:40:05.513184 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  420. 2011-05-06 21:40:05.513200 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  421. 2011-05-06 21:40:05.513217 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).writer: state = 2 policy.server=0
  422. 2011-05-06 21:40:05.513234 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).writer encoding 1 0x7f9a64002140 auth(proto 0 30 bytes) v1
  423. 2011-05-06 21:40:05.513249 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).reader reading tag...
  424. 2011-05-06 21:40:05.513265 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).writer sending 1 0x7f9a64002140
  425. 2011-05-06 21:40:05.513278 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).write_message 0x7f9a64002140
  426. 2011-05-06 21:40:05.513319 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).writer: state = 2 policy.server=0
  427. 2011-05-06 21:40:05.513331 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).writer sleeping
  428. 2011-05-06 21:40:05.720567 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).reader got ACK
  429. 2011-05-06 21:40:05.720641 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).reader reading tag...
  430. 2011-05-06 21:40:06.512472 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  431. 2011-05-06 21:40:07.512589 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  432. 2011-05-06 21:40:08.512717 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  433. 2011-05-06 21:40:08.512805 7f9a711e9710 -- 192.168.98.111:0/19605 mark_down 192.168.98.111:6789/0 -- 0x7f9a64002490
  434. 2011-05-06 21:40:08.512823 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).unregister_pipe
  435. 2011-05-06 21:40:08.512834 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).stop
  436. 2011-05-06 21:40:08.512902 7f9a711e9710 -- 192.168.98.111:0/19605 --> mon0 192.168.98.111:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f9a64002140
  437. 2011-05-06 21:40:08.512916 7f9a711e9710 -- 192.168.98.111:0/19605 submit_message auth(proto 0 30 bytes) v1 remote, 192.168.98.111:6789/0, new pipe.
  438. 2011-05-06 21:40:08.512926 7f9a711e9710 -- 192.168.98.111:0/19605 connect_rank to 192.168.98.111:6789/0, creating pipe and registering
  439. 2011-05-06 21:40:08.513003 7f9a711e9710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=-1 pgs=0 cs=0 l=0).register_pipe
  440. 2011-05-06 21:40:08.513035 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).writer finishing
  441. 2011-05-06 21:40:08.513049 7f9a74e46710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).writer done
  442. 2011-05-06 21:40:08.513083 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).reader couldn't read tag, Success
  443. 2011-05-06 21:40:08.513101 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).fault 0: Success
  444. 2011-05-06 21:40:08.513115 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).fault already closed|closing
  445. 2011-05-06 21:40:08.513133 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
  446. 2011-05-06 21:40:08.513149 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=-1 pgs=0 cs=0 l=0).connect 0
  447. 2011-05-06 21:40:08.513160 7f9a709e8710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a64002490
  448. 2011-05-06 21:40:08.513182 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=0 cs=0 l=0).connecting to 192.168.98.111:6789/0
  449. 2011-05-06 21:40:08.513198 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).reader done
  450. 2011-05-06 21:40:08.513274 7f9a731ed710 -- 192.168.98.111:0/19605 reaper
  451. 2011-05-06 21:40:08.513291 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a64002490 192.168.98.111:6789/0
  452. 2011-05-06 21:40:08.513302 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).discard_queue
  453. 2011-05-06 21:40:08.513314 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1). dequeued pipe
  454. 2011-05-06 21:40:08.513327 7f9a731ed710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002490 sd=4 pgs=63 cs=1 l=1).unregister_pipe - not registered
  455. 2011-05-06 21:40:08.513353 7f9a731ed710 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a64002490 192.168.98.111:6789/0
  456. 2011-05-06 21:40:08.513365 7f9a731ed710 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a64002490
  457. 2011-05-06 21:40:08.513372 7f9a731ed710 -- 192.168.98.111:0/19605 reaper done
  458. 2011-05-06 21:40:08.513442 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=0 cs=0 l=0).connect read peer addr 192.168.98.111:6789/0 on socket 6
  459. 2011-05-06 21:40:08.513487 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.98.111:38127/0
  460. 2011-05-06 21:40:08.513536 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=0 cs=0 l=0).connect sent my addr 192.168.98.111:0/19605
  461. 2011-05-06 21:40:08.513551 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=0 cs=0 l=0).connect sending gseq=10 cseq=0 proto=15
  462. 2011-05-06 21:40:08.513580 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
  463. 2011-05-06 21:40:08.513597 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 64 proto 15 flags 1
  464. 2011-05-06 21:40:08.513610 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).connect success 1, lossy = 1, features 506
  465. 2011-05-06 21:40:08.513622 7f9a708e7710 -- 192.168.98.111:0/19605 >> :/0 pipe(0x1c5e390 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
  466. 2011-05-06 21:40:08.513640 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).connect starting reader
  467. 2011-05-06 21:40:08.513658 7f9a719ea710 -- 192.168.98.111:0/19605 dispatch_entry pipe 0x1c5e390 dequeued 0
  468. 2011-05-06 21:40:08.513677 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).writer: state = 2 policy.server=0
  469. 2011-05-06 21:40:08.513691 7f9a719ea710 client-1 ms_handle_connect on 192.168.98.111:6789/0
  470. 2011-05-06 21:40:08.513709 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).writer encoding 1 0x7f9a64002140 auth(proto 0 30 bytes) v1
  471. 2011-05-06 21:40:08.513727 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).reader reading tag...
  472. 2011-05-06 21:40:08.513743 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).writer sending 1 0x7f9a64002140
  473. 2011-05-06 21:40:08.513757 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).write_message 0x7f9a64002140
  474. 2011-05-06 21:40:08.513796 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).writer: state = 2 policy.server=0
  475. 2011-05-06 21:40:08.513808 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).writer sleeping
  476. 2011-05-06 21:40:08.721582 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).reader got ACK
  477. 2011-05-06 21:40:08.721625 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).reader reading tag...
  478. 2011-05-06 21:40:09.512842 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  479. 2011-05-06 21:40:10.512959 7f9a721eb710 client?.objectcacher flusher 0 / 209715200: 0 tx, 0 rx, 0 clean, 0 dirty (8388608 target, 104857600 max)
  480. 2011-05-06 21:40:11.448803 7f9a74e4a720 monclient(hunting): authenticate timed out after 30
  481. 2011-05-06 21:40:11.448891 7f9a74e4a720 client-1 shutdown
  482. 2011-05-06 21:40:11.448915 7f9a721eb710 client?.objectcacher flusher finish
  483. 2011-05-06 21:40:11.449017 7f9a74e4a720 -- 192.168.98.111:0/19605 shutdown 192.168.98.111:0/19605
  484. 2011-05-06 21:40:11.449028 7f9a74e4a720 -- 192.168.98.111:0/19605 shutdown i am not dispatch, setting stop flag and joining thread.
  485. 2011-05-06 21:40:11.449116 7f9a74e4a720 -- 192.168.98.111:0/19605 wait: everything stopped
  486. 2011-05-06 21:40:11.449127 7f9a74e4a720 -- 192.168.98.111:0/19605 wait: stopping reaper thread
  487. 2011-05-06 21:40:11.449173 7f9a731ed710 -- 192.168.98.111:0/19605 reaper_entry done
  488. 2011-05-06 21:40:11.449198 7f9a74e4a720 -- 192.168.98.111:0/19605 wait: stopped reaper thread
  489. 2011-05-06 21:40:11.449207 7f9a74e4a720 -- 192.168.98.111:0/19605 wait: closing pipes
  490. 2011-05-06 21:40:11.449220 7f9a74e4a720 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).unregister_pipe
  491. 2011-05-06 21:40:11.449232 7f9a74e4a720 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).stop
  492. 2011-05-06 21:40:11.449287 7f9a74e4a720 -- 192.168.98.111:0/19605 reaper
  493. 2011-05-06 21:40:11.449297 7f9a74e4a720 -- 192.168.98.111:0/19605 reaper done
  494. 2011-05-06 21:40:11.449305 7f9a74e4a720 -- 192.168.98.111:0/19605 wait: waiting for pipes 0x7f9a64002a80 to close
  495. 2011-05-06 21:40:11.449321 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).writer finishing
  496. 2011-05-06 21:40:11.449332 7f9a708e7710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).writer done
  497. 2011-05-06 21:40:11.449401 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).reader couldn't read tag, Success
  498. 2011-05-06 21:40:11.449421 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).fault 0: Success
  499. 2011-05-06 21:40:11.449432 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).fault already closed|closing
  500. 2011-05-06 21:40:11.449441 7f9a709e8710 -- 192.168.98.111:0/19605 queue_reap 0x7f9a64002a80
  501. 2011-05-06 21:40:11.449456 7f9a709e8710 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).reader done
  502. 2011-05-06 21:40:11.449470 7f9a74e4a720 -- 192.168.98.111:0/19605 reaper
  503. 2011-05-06 21:40:11.449484 7f9a74e4a720 -- 192.168.98.111:0/19605 reaper reaping pipe 0x7f9a64002a80 192.168.98.111:6789/0
  504. 2011-05-06 21:40:11.449495 7f9a74e4a720 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).discard_queue
  505. 2011-05-06 21:40:11.449505 7f9a74e4a720 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1). dequeued pipe
  506. 2011-05-06 21:40:11.449517 7f9a74e4a720 -- 192.168.98.111:0/19605 >> 192.168.98.111:6789/0 pipe(0x7f9a64002a80 sd=6 pgs=64 cs=1 l=1).unregister_pipe - not registered
  507. 2011-05-06 21:40:11.449557 7f9a74e4a720 -- 192.168.98.111:0/19605 reaper reaped pipe 0x7f9a64002a80 192.168.98.111:6789/0
  508. 2011-05-06 21:40:11.449568 7f9a74e4a720 -- 192.168.98.111:0/19605 reaper deleted pipe 0x7f9a64002a80
  509. 2011-05-06 21:40:11.449576 7f9a74e4a720 -- 192.168.98.111:0/19605 reaper done
  510. 2011-05-06 21:40:11.449584 7f9a74e4a720 -- 192.168.98.111:0/19605 wait: done.
  511. 2011-05-06 21:40:11.449591 7f9a74e4a720 -- 192.168.98.111:0/19605 shutdown complete.
  512. You have new mail in /var/spool/mail/root
  513. [root@scale-192-168-98-111 ~]#
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!