Fri Oct 5 10:09:16 [conn793737] waiting till out of critical section Fri Oct 5 10:09:16 [conn808344] waiting till out of critical section Fri Oct 5 10:09:16 [conn806377] waiting till out of critical section Fri Oct 5 10:09:16 [conn808452] waiting till out of critical section Fri Oct 5 10:09:16 [conn732381] waiting till out of critical section Fri Oct 5 10:09:16 [conn796239] waiting till out of critical section Fri Oct 5 10:09:16 [conn799080] waiting till out of critical section Fri Oct 5 10:09:16 [conn809152] waiting till out of critical section Fri Oct 5 10:09:16 [conn717991] waiting till out of critical section Fri Oct 5 10:09:16 [conn797115] waiting till out of critical section Fri Oct 5 10:09:16 [conn734286] waiting till out of critical section Fri Oct 5 10:09:16 [conn810526] waiting till out of critical section Fri Oct 5 10:09:16 [conn719097] waiting till out of critical section Fri Oct 5 10:09:16 [conn802213] waiting till out of critical section Fri Oct 5 10:09:16 [conn805144] waiting till out of critical section Fri Oct 5 10:09:16 [conn796168] waiting till out of critical section Fri Oct 5 10:09:16 [conn794799] waiting till out of critical section Fri Oct 5 10:09:16 [conn793279] waiting till out of critical section Fri Oct 5 10:09:16 [conn740129] waiting till out of critical section Fri Oct 5 10:09:16 [conn795987] waiting till out of critical section Fri Oct 5 10:09:16 [conn803180] waiting till out of critical section Fri Oct 5 10:09:16 [conn808509] waiting till out of critical section Fri Oct 5 10:09:16 [conn797143] waiting till out of critical section Fri Oct 5 10:09:16 [conn801380] waiting till out of critical section Fri Oct 5 10:09:16 [conn804995] waiting till out of critical section Fri Oct 5 10:09:16 [conn804207] waiting till out of critical section Fri Oct 5 10:09:16 [conn807029] waiting till out of critical section Fri Oct 5 10:09:16 [conn732381] waiting till out of critical section Fri Oct 5 10:09:16 [conn806502] waiting till out of critical section Fri Oct 5 10:09:16 [interruptThread] now exiting Fri Oct 5 10:09:16 dbexit: Fri Oct 5 10:09:16 [conn810152] waiting till out of critical section Fri Oct 5 10:09:16 [conn796211] waiting till out of critical section Fri Oct 5 10:09:16 [conn794212] waiting till out of critical section Fri Oct 5 10:09:16 [conn731888] waiting till out of critical section Fri Oct 5 10:09:16 [conn800882] waiting till out of critical section Fri Oct 5 10:09:16 [conn795324] waiting till out of critical section Fri Oct 5 10:09:16 [conn799602] waiting till out of critical section Fri Oct 5 10:09:16 [conn728877] waiting till out of critical section Fri Oct 5 10:09:16 [conn804522] waiting till out of critical section Fri Oct 5 10:09:16 [conn799559] waiting till out of critical section Fri Oct 5 10:09:16 [conn809205] waiting till out of critical section Fri Oct 5 10:09:16 [interruptThread] shutdown: going to close listening sockets... Fri Oct 5 10:09:16 [interruptThread] closing listening socket: 5 Fri Oct 5 10:09:16 [interruptThread] closing listening socket: 6 Fri Oct 5 10:09:16 [interruptThread] closing listening socket: 7 Fri Oct 5 10:09:16 [interruptThread] removing socket file: /tmp/mongodb-27018.sock Fri Oct 5 10:09:16 [interruptThread] shutdown: going to flush diaglog... Fri Oct 5 10:09:16 [interruptThread] shutdown: going to close sockets... Fri Oct 5 10:09:16 [interruptThread] shutdown: waiting for fs preallocator... Fri Oct 5 10:09:16 [interruptThread] shutdown: closing all files... Fri Oct 5 10:09:16 [conn1] end connection 127.0.1.1:52491 (18795 connections now open) Fri Oct 5 10:09:16 ERROR: Client::shutdown not called: slaveTracking Fri Oct 5 10:09:16 ERROR: Client::shutdown not called: rsHealthPoll Fri Oct 5 10:09:17 [conn824267] end connection 172.16.64.39:51053 (18794 connections now open) Fri Oct 5 10:09:17 [conn271] end connection 172.16.64.143:42015 (18793 connections now open) Fri Oct 5 10:09:17 ERROR: Client::shutdown not called: rsHealthPoll Fri Oct 5 10:09:17 [conn824267] end connection 172.16.64.39:51053 (18794 connections now open) Fri Oct 5 10:09:17 [conn271] end connection 172.16.64.143:42015 (18793 connections now open) Fri Oct 5 10:09:17 ERROR: Client::shutdown not called: rsHealthPoll Fri Oct 5 10:09:18 [conn824265] end connection 172.16.64.140:58972 (18792 connections now open) Fri Oct 5 10:09:18 [interruptThread] 60/93 64% Fri Oct 5 10:09:18 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 172.16.65.71:27018 Fri Oct 5 10:09:18 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 172.16.64.139:27018 Fri Oct 5 10:09:18 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 172.16.64.172:27018 Fri Oct 5 10:09:18 [ReplicaSetMonitorWatcher] trying reconnect to s136:27018 Fri Oct 5 10:09:18 [ReplicaSetMonitorWatcher] reconnect s136:27018 ok Fri Oct 5 10:09:18 [ReplicaSetMonitorWatcher] Assertion failure ! inShutdown() src/mongo/client/connpool.cpp 158 0xade6e1 0x803dfd 0x5eba35 0x5ec20b 0x95f2e0 0x7915ac 0x7967cd 0x797900 0x7991f1 0x79950a 0x7a682b 0x595fce 0x597a84 0x7c3659 0x7f8153d768ba 0x7f815312a02d Fri Oct 5 10:09:18 [interruptThread] closeAllFiles() finished Fri Oct 5 10:09:18 [interruptThread] shutdown: removing fs lock... Fri Oct 5 10:09:18 dbexit: really exiting now ***** SERVER RESTARTED ***** Fri Oct 5 10:09:25 [initandlisten] MongoDB starting : pid=23061 port=27018 dbpath=/data/mongo/db 64-bit host=s117 Fri Oct 5 10:09:25 [initandlisten] db version v2.2.0, pdfile version 4.5 Fri Oct 5 10:09:25 [initandlisten] git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207 Fri Oct 5 10:09:25 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Fri Oct 5 10:09:25 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/data/mongo/db", directoryperdb: "true", keyFile: "/data/mongo/keyfile.txt", logappend: "true", logpath: "/data/mongo/log/mongodb.log", nojournal: "true", port: 27018, repairpath: "/data_local/mongo/db", replSet: "offerStoreDE3", rest: "true", shardsvr: "true" } Fri Oct 5 10:09:25 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/data/mongo/db/journal" Fri Oct 5 10:09:25 [initandlisten] waiting for connections on port 27018 Fri Oct 5 10:09:25 [websvr] admin web console waiting for connections on port 28018 Fri Oct 5 10:09:25 [initandlisten] connection accepted from 127.0.1.1:58183 #1 (1 connection now open) Fri Oct 5 10:09:25 [conn1] authenticate db: local { authenticate: 1, nonce: "b22c6258066ffab2", user: "__system", key: "58520e3dca893894bb0109a6f0ccb6f4" } Fri Oct 5 10:09:25 [rsStart] replSet I am s117:27018 Fri Oct 5 10:09:25 [rsStart] replSet STARTUP2 Fri Oct 5 10:09:25 [rsHealthPoll] replSet member s129:27018 is up Fri Oct 5 10:09:25 [rsHealthPoll] replSet member s124:27018 is up Fri Oct 5 10:09:25 [rsHealthPoll] replSet member s129:27018 is now in state PRIMARY Fri Oct 5 10:09:25 [rsHealthPoll] replSet member s124:27018 is now in state SECONDARY Fri Oct 5 10:09:25 [initandlisten] connection accepted from 172.16.64.39:51065 #2 (2 connections now open) Fri Oct 5 10:09:25 [conn2] authenticate db: local { authenticate: 1, nonce: "9a6730fd3f508e7e", user: "__system", key: "f2815b6f2d30ae7897eada0cdd8fdd41" } Fri Oct 5 10:09:26 [initandlisten] connection accepted from 172.16.64.140:58977 #3 (3 connections now open) Fri Oct 5 10:09:26 [conn3] authenticate db: local { authenticate: 1, nonce: "a1916c9c5434d1d8", user: "__system", key: "c20bba8641e302f7df0408825e40f56d", $auth: { local: { __system: 2 } } } Fri Oct 5 10:09:26 [rsSync] replSet SECONDARY Fri Oct 5 10:09:26 [initandlisten] connection accepted from 172.16.64.143:34398 #4 (4 connections now open) Fri Oct 5 10:09:26 [conn4] authenticate db: local { authenticate: 1, nonce: "96a2ef8b3f62d2c1", user: "__system", key: "a6b3a8f9f9ccd80d346c9a9a1c66090a" } Fri Oct 5 10:09:27 [initandlisten] connection accepted from 172.16.64.39:51067 #5 (5 connections now open) Fri Oct 5 10:09:27 [conn5] authenticate db: local { authenticate: 1, nonce: "9fa54ec6bae0a298", user: "__system", key: "57f1550aef2e9f6a0ed70b26941dfdbe", $auth: { local: { __system: 2 } } } Fri Oct 5 10:09:31 [initandlisten] connection accepted from 172.16.64.143:34399 #6 (6 connections now open) Fri Oct 5 10:09:31 [initandlisten] connection accepted from 172.16.64.143:34400 #7 (7 connections now open) Fri Oct 5 10:09:31 [conn7] authenticate db: local { authenticate: 1, nonce: "e08537cd239ef3ba", user: "__system", key: "64d467a4f2665df8de92e7ead3c0bbc6" } Fri Oct 5 10:09:32 [rsBackgroundSync] replSet syncing to: s129:27018 Fri Oct 5 10:09:33 [rsSyncNotifier] replset setting oplog notifier to s129:27018 Fri Oct 5 10:09:34 [initandlisten] connection accepted from 172.16.64.164:38038 #9 (8 connections now open) Fri Oct 5 10:09:35 [initandlisten] connection accepted from 172.16.64.172:56387 #10 (9 connections now open) Fri Oct 5 10:09:35 [initandlisten] connection accepted from 172.16.64.36:43276 #12 (10 connections now open) Fri Oct 5 10:09:35 [conn12] authenticate db: admin { authenticate: 1, user: "admin", nonce: "c535424bdb7b1316", key: "1ce059089f4fbb2604e738291d704ff5" }