API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Dec 11th, 2023
34
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 145.71 KB | None | 0 0
raw download clone embed print report
  1. sh: 1: cannot create /opt/mapr/pid/warden.pid.tmp: Permission denied
  2. 2023-12-11 22:38:51,355 INFO WardenMain [main]: Log dir: /opt/mapr/hadoop/hadoop-2.7.6/logs
  3. 2023-12-11 22:38:51,402 INFO MaprRPCContext [main]: init MAPRContext
  4. 2023-12-11 22:38:51,404 INFO MaprRPCContext [main]: init MAPRContextHS
  5. 2023-12-11 22:38:51,410 WARN MaprRPCContext [main]: Error while trying to get correct Ticket with errorCode: 2
  6. 2023-12-11 22:38:51,467 INFO WardenMain [main]: My pid: 6961
  7. 2023-12-11 22:38:51,486 INFO WardenManager [maprUserTicketGetCheckThread]: maprUserTicketExpiration: 922337203685477
  8. 2023-12-11 22:38:51,783 ERROR CLDBRpcCommonUtils [main-SendThread(172.17.0.2:5181)]: No data returned in RPC: 2345.190 from CLDB Ips: 172.17.0.2-, Port: 7222. Continue searching for correct CLDB
  9. 2023-12-11 22:38:51,784 INFO CLDBRpcCommonUtils [main-SendThread(172.17.0.2:5181)]: Bad CLDB credentials removed: CLDB Ips: 172.17.0.2-, Port: 7222
  10. 2023-12-11 22:38:51,784 WARN MaprSecurityLoginModule [main-SendThread(172.17.0.2:5181)]: Unable to generate the server key.
  11. 2023-12-11 22:38:52,046 INFO ZookeeperClientSessionManagement [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40000 local:/172.17.0.2:33964 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  12. 2023-12-11 22:38:52,069 INFO ZookeeperClientSessionManagement [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40000 local:/172.17.0.2:33964 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  13. 2023-12-11 22:38:52,128 INFO ZookeeperClientSessionManagement [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  14. 2023-12-11 22:38:52,180 INFO WardenManager [main]: Configured services: [drill-bits:all:cldb, s3server:all:cldb, kvstore:all, cldb:all:kvstore, nfs:all:cldb, gateway:all:cldb, hoststats:all:kvstore, data-access-gateway:all:cldb, apiserver:all:cldb]
  15. 2023-12-11 22:38:52,181 INFO WardenManager [main]: warden.conf content:
  16. {service.command.nfs4.start=/opt/mapr/initscripts/mapr-nfs4server start, service.command.zk.heapsize.max=1500, hs.rpcon=true, service.command.webserver.heapsize.min=128, services.retries=3, isM7=0, hs.port=1111, service.command.kvstore.monitorcommand=/opt/mapr/initscripts/mapr-mfs status, service.command.mfs.monitor=server/mfs, service.command.nfs.type=BACKGROUND, service.command.webserver.heapsize.max=128, services.retryinterval.time.sec=1800, currentHost=maprdemo.mapr.io, nodeTotalCpuCores=2, nodeMfsCpuCores=2, service.command.mfs.type=BACKGROUND, service.command.cldb.heapsize.max=256, service.command.os.heapsize.percent=10, service.command.hoststats.start=/opt/mapr/initscripts/mapr-hoststats start, service.command.warden.heapsize.min=64, service.command.os.heapsize.min=256, isDB=false, pollcentralconfig.interval.seconds=300, nodeTotalFreeMemory=13914956, service.command.mfs.heapsize.min=512, service.command.nfs4.heapsize.max=12288, service.command.nfs.monitor=server/nfsserver, service.command.kvstore.type=BACKGROUND, service.command.cldb.heapsize.min=256, nodes.mincount=1, service.command.hoststats.type=BACKGROUND, service.command.mfs.heapsize.maxpercent=85, service.command.nfs.heapsize.percent=3, cldb.port=7222, nodeTotalDisks=6, service.command.mfs.heapsize.percent=15, nodeTotalMemory=6291456, service.command.warden.heapsize.max=750, service.command.nfs4.heapsize.min=2048, service.command.zk.heapsize.min=256, currentHostID=6e74c1c26dc11480, service.command.webserver.stop=/opt/mapr/adminuiapp/webserver stop, service.command.cldb.monitor=com.mapr.fs.cldb.CLDB, service.command.nfs4.stop=/opt/mapr/initscripts/mapr-nfs4server stop, service.command.nfs.start=/opt/mapr/initscripts/mapr-nfsserver start, service.command.cldb.type=BACKGROUND, service.command.cldb.start=/opt/mapr/initscripts/mapr-cldb start, service.command.nfs4.monitor=bin/nfs4server, service.command.cldb.monitorcommand=/opt/mapr/initscripts/mapr-cldb status, service.command.hoststats.monitorcommand=/opt/mapr/initscripts/mapr-hoststats status, service.command.nfs.stop=/opt/mapr/initscripts/mapr-nfsserver stop, centralconfig.enabled=true, service.command.mfs.stop=/opt/mapr/initscripts/mapr-mfs stop, mapr.home.dir=/opt/mapr, service.command.nfs.heapsize.min=64, service.nice.value=-10, hs.host=localhost, service.command.cldb.retryinterval.time.sec=600, service.command.webserver.monitorcommand=/opt/mapr/adminuiapp/webserver status, warden.enable.jmxremote=false, service.command.cldb.heapsize.percent=8, service.command.kvstore.start=/opt/mapr/initscripts/mapr-mfs start, service.command.warden.heapsize.percent=1, service.command.nfs.monitorcommand=/opt/mapr/initscripts/mapr-nfsserver status, service.command.nfs.heapsize.max=64, mfs.port=5660, service.command.webserver.type=BACKGROUND, services.memoryallocation.alarm.threshold=97, zookeeper.servers=172.17.0.2:5181, enable.overcommit=true, service.command.nfs4.monitorcommand=/opt/mapr/initscripts/mapr-nfs4server status, services=nfs:all:cldb;kvstore:all;cldb:all:kvstore;hoststats:all:kvstore, service.command.mfs.monitorcommand=/opt/mapr/initscripts/mapr-mfs status, log.retention.exceptions=cldbaudit*,authaudit*,pbsaudit*,cldbmirroraudit*,cldbs3serveraudit*,cldbaudit*,authaudit*,pbsaudit*,cldbmirroraudit*,cldb*.log,hoststats.log,configure.log,mfs.log-*,nfsserver.log*,moss.log,moss.fileclient.log, service.command.webserver.heapsize.percent=3, service.command.zk.heapsize.percent=1, hoststats.port=5660, nodeMfsDisks=0, kvstore.port=5660, service.command.nfs4.type=BACKGROUND, service.command.cldb.stop=/opt/mapr/initscripts/mapr-cldb stop, service.command.hoststats.stop=/opt/mapr/initscripts/mapr-hoststats stop, service.command.kvstore.monitor=server/mfs, service.command.os.heapsize.max=4000, service.command.webserver.start=/opt/mapr/adminuiapp/webserver start, service.command.mfs.start=/opt/mapr/initscripts/mapr-mfs start, service.command.nfs4.heapsize.percent=4, service.command.kvstore.stop=/opt/mapr/initscripts/mapr-mfs stop, rpc.drop=false}
  17. 2023-12-11 22:38:52,375 INFO Service [main]: Initing service: drill-bits CurrentHost: maprdemo.mapr.io
  18. 2023-12-11 22:38:52,398 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  19. 2023-12-11 22:38:52,398 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  20. 2023-12-11 22:38:52,399 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  21. 2023-12-11 22:38:52,402 INFO WardenManager [main]: Memory required for service: drill-bits is 1024.0
  22. 2023-12-11 22:38:52,403 INFO WardenManager [main]: Total memory required for services so far is: 1958
  23. 2023-12-11 22:38:52,403 INFO Service [main]: Initing service: s3server CurrentHost: maprdemo.mapr.io
  24. 2023-12-11 22:38:52,411 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  25. 2023-12-11 22:38:52,427 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  26. 2023-12-11 22:38:52,427 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  27. 2023-12-11 22:38:52,427 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  28. 2023-12-11 22:38:52,429 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  29. 2023-12-11 22:38:52,431 INFO WardenManager [main]: Memory required for service: s3server is 2048.0
  30. 2023-12-11 22:38:52,431 INFO WardenManager [main]: Total memory required for services so far is: 4006
  31. 2023-12-11 22:38:52,433 INFO Service [main]: Initing service: kvstore CurrentHost: maprdemo.mapr.io
  32. 2023-12-11 22:38:52,465 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  33. 2023-12-11 22:38:52,465 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40003 local:/172.17.0.2:34454 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  34. 2023-12-11 22:38:52,465 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40003 local:/172.17.0.2:34454 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  35. 2023-12-11 22:38:52,467 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  36. 2023-12-11 22:38:52,470 INFO KVStoreService [main]: Before adjustment, memory for KVStore = 922.0
  37. 2023-12-11 22:38:52,470 INFO WardenManager [main]: Memory required for service: kvstore is 922.0
  38. 2023-12-11 22:38:52,470 INFO KVStoreService [main]: Before adjustment, memory for KVStore = 922.0
  39. 2023-12-11 22:38:52,471 INFO WardenManager [main]: Total memory required for services so far is: 4928
  40. 2023-12-11 22:38:52,473 INFO Service [main]: Initing service: cldb CurrentHost: maprdemo.mapr.io
  41. 2023-12-11 22:38:52,489 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  42. 2023-12-11 22:38:52,489 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:3 queuedpkts:0 pendingresp:0 queuedevents:1
  43. 2023-12-11 22:38:52,490 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:3 queuedpkts:0 pendingresp:0 queuedevents:1
  44. 2023-12-11 22:38:52,490 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  45. 2023-12-11 22:38:52,499 INFO WardenManager [main]: Memory required for service: cldb is 256.0
  46. 2023-12-11 22:38:52,500 INFO WardenManager [main]: Total memory required for services so far is: 5184
  47. 2023-12-11 22:38:52,500 INFO Service [main]: Initing service: nfs CurrentHost: maprdemo.mapr.io
  48. 2023-12-11 22:38:52,522 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  49. 2023-12-11 22:38:52,523 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40005 local:/172.17.0.2:34476 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:2 queuedpkts:0 pendingresp:0 queuedevents:0
  50. 2023-12-11 22:38:52,523 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40005 local:/172.17.0.2:34476 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:3 queuedpkts:0 pendingresp:0 queuedevents:1
  51. 2023-12-11 22:38:52,524 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  52. 2023-12-11 22:38:52,533 INFO WardenManager [main]: Memory required for service: nfs is 64.0
  53. 2023-12-11 22:38:52,533 INFO WardenManager [main]: Total memory required for services so far is: 5248
  54. 2023-12-11 22:38:52,534 INFO Service [main]: Initing service: gateway CurrentHost: maprdemo.mapr.io
  55. 2023-12-11 22:38:52,551 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  56. 2023-12-11 22:38:52,551 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  57. 2023-12-11 22:38:52,551 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  58. 2023-12-11 22:38:52,552 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  59. 2023-12-11 22:38:52,553 INFO WardenManager [main]: Memory required for service: gateway is 64.0
  60. 2023-12-11 22:38:52,553 INFO WardenManager [main]: Total memory required for services so far is: 5312
  61. 2023-12-11 22:38:52,554 INFO Service [main]: Initing service: hoststats CurrentHost: maprdemo.mapr.io
  62. 2023-12-11 22:38:52,575 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  63. 2023-12-11 22:38:52,575 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  64. 2023-12-11 22:38:52,575 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  65. 2023-12-11 22:38:52,576 INFO WardenManager [main]: Memory required for service: hoststats is 0.0
  66. 2023-12-11 22:38:52,576 INFO WardenManager [main]: Total memory required for services so far is: 5312
  67. 2023-12-11 22:38:52,577 INFO Service [main]: Initing service: data-access-gateway CurrentHost: maprdemo.mapr.io
  68. 2023-12-11 22:38:52,578 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  69. 2023-12-11 22:38:52,586 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  70. 2023-12-11 22:38:52,586 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40008 local:/172.17.0.2:34514 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:3 queuedpkts:0 pendingresp:0 queuedevents:1
  71. 2023-12-11 22:38:52,587 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40008 local:/172.17.0.2:34514 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:3 queuedpkts:0 pendingresp:0 queuedevents:1
  72. 2023-12-11 22:38:52,588 INFO WardenManager [main]: Memory required for service: data-access-gateway is 512.0
  73. 2023-12-11 22:38:52,588 INFO WardenManager [main]: Total memory required for services so far is: 5824
  74. 2023-12-11 22:38:52,588 INFO Service [main]: Initing service: apiserver CurrentHost: maprdemo.mapr.io
  75. 2023-12-11 22:38:52,587 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  76. 2023-12-11 22:38:52,595 INFO Service [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  77. 2023-12-11 22:38:52,596 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40009 local:/172.17.0.2:34524 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:2 queuedpkts:0 pendingresp:0 queuedevents:0
  78. 2023-12-11 22:38:52,596 INFO Service [main]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40009 local:/172.17.0.2:34524 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:2 queuedpkts:0 pendingresp:0 queuedevents:0
  79. 2023-12-11 22:38:52,596 INFO Service [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  80. 2023-12-11 22:38:52,596 INFO WardenManager [main]: Memory required for service: apiserver is 1000.0
  81. 2023-12-11 22:38:52,597 INFO WardenManager [main]: Total memory required for services so far is: 6824
  82. 2023-12-11 22:38:52,597 INFO WardenManager [main]: Total node memory: 6291456
  83. 2023-12-11 22:38:52,602 WARN WardenManager [main]:
  84. --------------- Total memory required to run all configured services is exceeding available memory on the box.
  85. --------------- Total memory needed: 6824MB where Total node memory is: 6144MB. Overcommitting memory.
  86. 2023-12-11 22:38:52,604 INFO WardenManager [main]: Raised NODE_ALARM_MEMORY_ALLOCATION_EXCEEDED alarm on node maprdemo.mapr.io
  87. 2023-12-11 22:38:52,604 INFO WardenManager [main]: Total consumed memory: 6824
  88. 2023-12-11 22:38:52,604 INFO WardenManager [main]: Still available free memory: 0
  89. 2023-12-11 22:38:52,605 INFO WardenManager [main]: Total node cpu cores : 2
  90. 2023-12-11 22:38:52,608 INFO WardenManager [main]: Total node mfs cpu cores : 2
  91. 2023-12-11 22:38:52,609 INFO WardenManager [main]: Total available cpu cores : 0
  92. 2023-12-11 22:38:52,609 INFO WardenManager [main]: Total node disks : 6
  93. 2023-12-11 22:38:52,611 INFO WardenManager [main]: Total node mfs disks : 0
  94. 2023-12-11 22:38:52,611 INFO WardenManager [main]: WardenManager: started service drill-bits
  95. 2023-12-11 22:38:52,612 INFO WardenManager [main]: WardenManager: started service s3server
  96. 2023-12-11 22:38:52,612 INFO WardenManager [main]: WardenManager: started service kvstore
  97. 2023-12-11 22:38:52,613 INFO WardenManager [main]: WardenManager: started service cldb
  98. 2023-12-11 22:38:52,613 INFO WardenManager [main]: WardenManager: started service nfs
  99. 2023-12-11 22:38:52,614 INFO WardenManager [main]: WardenManager: started service gateway
  100. 2023-12-11 22:38:52,614 INFO WardenManager [main]: WardenManager: started service hoststats
  101. 2023-12-11 22:38:52,615 INFO WardenManager [main]: WardenManager: started service data-access-gateway
  102. 2023-12-11 22:38:52,619 INFO WardenManager [main]: WardenManager: started service apiserver
  103. 2023-12-11 22:38:52,620 INFO Service [main]: Process delayed alarms: 1
  104. 2023-12-11 22:38:52,634 ERROR Service [Thread-17]: Need to recreate alarm to add: NODE_ALARM_SERVICE_DRILL-BITS_DOWN
  105. 2023-12-11 22:38:52,634 INFO Service [Thread-17]: Alarm add being set to delayed command: NODE_ALARM_SERVICE_DRILL-BITS_DOWN
  106. 2023-12-11 22:38:52,638 ERROR Service [Thread-19]: Need to recreate alarm to add: NODE_ALARM_SERVICE_FILESERVER_DOWN
  107. 2023-12-11 22:38:52,638 ERROR Service [Thread-20]: Need to recreate alarm to add: NODE_ALARM_SERVICE_CLDB_DOWN
  108. 2023-12-11 22:38:52,641 ERROR Service [Thread-21]: Need to recreate alarm to add: NODE_ALARM_SERVICE_NFS_DOWN
  109. 2023-12-11 22:38:52,642 ERROR Service [Thread-18]: Need to recreate alarm to add: NODE_ALARM_SERVICE_S3SERVER_DOWN
  110. 2023-12-11 22:38:52,642 INFO Service [Thread-18]: Alarm add being set to delayed command: NODE_ALARM_SERVICE_S3SERVER_DOWN
  111. 2023-12-11 22:38:52,642 ERROR Service [Thread-23]: Need to recreate alarm to add: NODE_ALARM_SERVICE_HOSTSTATS_DOWN
  112. 2023-12-11 22:38:52,642 ERROR Service [Thread-22]: Need to recreate alarm to add: NODE_ALARM_SERVICE_GATEWAY_DOWN
  113. 2023-12-11 22:38:52,643 INFO Service [Thread-22]: Alarm add being set to delayed command: NODE_ALARM_SERVICE_GATEWAY_DOWN
  114. 2023-12-11 22:38:52,649 INFO Service [Thread-22]: Node: /nodes/maprdemo.mapr.io/services/gateway does not exist yet
  115. 2023-12-11 22:38:52,650 INFO Service [Thread-23]: Node: /nodes/maprdemo.mapr.io/services/hoststats does not exist yet
  116. 2023-12-11 22:38:52,651 INFO Service [Thread-18]: Node: /nodes/maprdemo.mapr.io/services/s3server does not exist yet
  117. 2023-12-11 22:38:52,652 INFO Service [Thread-19]: Node: /nodes/maprdemo.mapr.io/services/kvstore does not exist yet
  118. 2023-12-11 22:38:52,652 INFO Service [Thread-17]: Node: /nodes/maprdemo.mapr.io/services/drill-bits does not exist yet
  119. 2023-12-11 22:38:52,652 INFO Service [Thread-21]: Node: /nodes/maprdemo.mapr.io/services/nfs does not exist yet
  120. 2023-12-11 22:38:52,653 ERROR Service [Thread-24]: Need to recreate alarm to add: NODE_ALARM_SERVICE_DATA-ACCESS-GATEWAY_DOWN
  121. 2023-12-11 22:38:52,653 INFO Service [Thread-24]: Alarm add being set to delayed command: NODE_ALARM_SERVICE_DATA-ACCESS-GATEWAY_DOWN
  122. 2023-12-11 22:38:52,654 ERROR Service [Thread-25]: Need to recreate alarm to add: NODE_ALARM_SERVICE_APISERVER_DOWN
  123. 2023-12-11 22:38:52,654 INFO Service [Thread-25]: Alarm add being set to delayed command: NODE_ALARM_SERVICE_APISERVER_DOWN
  124. 2023-12-11 22:38:52,674 INFO Service [Thread-20]: Node: /nodes/maprdemo.mapr.io/services/cldb does not exist yet
  125. 2023-12-11 22:38:52,697 INFO Service [Thread-25]: Node: /nodes/maprdemo.mapr.io/services/apiserver does not exist yet
  126. 2023-12-11 22:38:52,698 INFO Service [Thread-24]: Node: /nodes/maprdemo.mapr.io/services/data-access-gateway does not exist yet
  127. 2023-12-11 22:38:52,708 INFO Service [Thread-22]: addServiceToZK: for node /services/gateway created zknode /services/gateway
  128. 2023-12-11 22:38:52,709 INFO Service [Thread-23]: addServiceToZK: for node /services/hoststats created zknode /services/hoststats
  129. 2023-12-11 22:38:52,711 INFO Service [Thread-18]: addServiceToZK: for node /services/s3server created zknode /services/s3server
  130. 2023-12-11 22:38:52,714 INFO Service [Thread-17]: addServiceToZK: for node /services/drill-bits created zknode /services/drill-bits
  131. 2023-12-11 22:38:52,714 INFO Service [Thread-21]: addServiceToZK: for node /services/nfs created zknode /services/nfs
  132. 2023-12-11 22:38:52,715 INFO Service [Thread-20]: addServiceToZK: for node /services/cldb created zknode /services/cldb
  133. 2023-12-11 22:38:52,715 INFO Service [Thread-19]: addServiceToZK: for node /services/kvstore created zknode /services/kvstore
  134. 2023-12-11 22:38:52,719 INFO Service [Thread-24]: addServiceToZK: for node /services/data-access-gateway created zknode /services/data-access-gateway
  135. 2023-12-11 22:38:52,723 ERROR DependentService [Thread-23]: For service: hoststats, dep service: kvstore, dependent zknode /services/kvstore/maprdemo.mapr.io doesn't exist
  136. 2023-12-11 22:38:52,723 ERROR DependentService [Thread-20]: For service: cldb, dep service: kvstore, dependent zknode /services/kvstore/maprdemo.mapr.io doesn't exist
  137. 2023-12-11 22:38:52,725 ERROR DependentService [Thread-17]: For service: drill-bits, dep service: cldb, dependent zknode /services/cldb/master doesn't exist
  138. 2023-12-11 22:38:52,725 INFO Service [Thread-25]: addServiceToZK: for node /services/apiserver created zknode /services/apiserver
  139. 2023-12-11 22:38:52,726 ERROR DependentService [Thread-24]: For service: data-access-gateway, dep service: cldb, dependent zknode /services/cldb/master doesn't exist
  140. 2023-12-11 22:38:52,725 ERROR DependentService [Thread-18]: For service: s3server, dep service: cldb, dependent zknode /services/cldb/master doesn't exist
  141. 2023-12-11 22:38:52,729 ERROR DependentService [Thread-21]: For service: nfs, dep service: cldb, dependent zknode /services/cldb/master doesn't exist
  142. 2023-12-11 22:38:52,730 ERROR DependentService [Thread-25]: For service: apiserver, dep service: cldb, dependent zknode /services/cldb/master doesn't exist
  143. 2023-12-11 22:38:52,748 ERROR DependentService [Thread-22]: For service: gateway, dep service: cldb, dependent zknode /services/cldb/master doesn't exist
  144. 2023-12-11 22:38:52,751 INFO DependentService [Thread-23]: createIPChild: svc:hoststats isRestartNeeded:false isDepServiceRunning:false stats:null
  145. 2023-12-11 22:38:52,753 INFO DependentService [Thread-22]: createIPChild: svc:gateway isRestartNeeded:false isDepServiceRunning:false stats:null
  146. 2023-12-11 22:38:52,755 INFO DependentService [Thread-22]: createIPChild: svc:gateway, wait for cldb to start.
  147. 2023-12-11 22:38:52,755 INFO DependentService [Thread-24]: createIPChild: svc:data-access-gateway isRestartNeeded:false isDepServiceRunning:false stats:null
  148. 2023-12-11 22:38:52,755 INFO DependentService [Thread-24]: createIPChild: svc:data-access-gateway, wait for cldb to start.
  149. 2023-12-11 22:38:52,755 INFO DependentService [Thread-21]: createIPChild: svc:nfs isRestartNeeded:false isDepServiceRunning:false stats:null
  150. 2023-12-11 22:38:52,755 INFO DependentService [Thread-21]: createIPChild: svc:nfs, wait for cldb to start.
  151. 2023-12-11 22:38:52,756 INFO DependentService [Thread-25]: createIPChild: svc:apiserver isRestartNeeded:false isDepServiceRunning:false stats:null
  152. 2023-12-11 22:38:52,756 INFO DependentService [Thread-25]: createIPChild: svc:apiserver, wait for cldb to start.
  153. 2023-12-11 22:38:52,756 INFO DependentService [Thread-17]: createIPChild: svc:drill-bits isRestartNeeded:false isDepServiceRunning:false stats:null
  154. 2023-12-11 22:38:52,756 INFO DependentService [Thread-17]: createIPChild: svc:drill-bits, wait for cldb to start.
  155. 2023-12-11 22:38:52,758 INFO DependentService [Thread-23]: createIPChild: svc:hoststats, wait for kvstore to start.
  156. 2023-12-11 22:38:52,759 INFO DependentService [Thread-18]: createIPChild: svc:s3server isRestartNeeded:false isDepServiceRunning:false stats:null
  157. 2023-12-11 22:38:52,759 INFO DependentService [Thread-18]: createIPChild: svc:s3server, wait for cldb to start.
  158. 2023-12-11 22:38:52,760 INFO DependentService [Thread-20]: createIPChild: svc:cldb isRestartNeeded:false isDepServiceRunning:false stats:null
  159. 2023-12-11 22:38:52,760 INFO DependentService [Thread-20]: createIPChild: svc:cldb, wait for kvstore to start.
  160. 2023-12-11 22:38:52,786 INFO Service [main-EventThread]: Process path: /services/kvstore/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  161. 2023-12-11 22:38:52,786 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40003 local:/172.17.0.2:34454 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:59 xid:9 sent:7 recv:11 queuedpkts:0 pendingresp:0 queuedevents:1
  162. 2023-12-11 22:38:52,787 INFO Service [Thread-19]: createBaseIPChild: created zknode: /services/kvstore/maprdemo.mapr.io
  163. 2023-12-11 22:38:52,794 INFO Service [Thread-19]: -------------Service is starting for: kvstore
  164. 2023-12-11 22:38:52,796 INFO ServiceMonitorRun [Thread-19]: Command: [/opt/mapr/initscripts/mapr-mfs, status], Directory: /opt/mapr/initscripts/
  165. 2023-12-11 22:38:52,804 INFO DependentService [main-EventThread]: DependentWatcher for service: cldb Process path: /services/kvstore/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  166. 2023-12-11 22:38:52,816 INFO Service [main-EventThread]: Thread: 33, NodeCreated: /services/kvstore/maprdemo.mapr.io
  167. 2023-12-11 22:38:52,826 INFO Service [kvstore_monitor]: Need delayed alarm clearing for: NODE_ALARM_SERVICE_FILESERVER_DOWN
  168. 2023-12-11 22:38:52,827 INFO Service [main-EventThread]: Process path: /services/kvstore. Event state: SyncConnected. Event type: NodeChildrenChanged
  169. 2023-12-11 22:38:52,827 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40003 local:/172.17.0.2:34454 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:62 xid:12 sent:10 recv:15 queuedpkts:0 pendingresp:0 queuedevents:1
  170. 2023-12-11 22:38:52,827 INFO Service [main-EventThread]: Process path: /services/kvstore/master. Event state: SyncConnected. Event type: NodeCreated
  171. 2023-12-11 22:38:52,827 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40003 local:/172.17.0.2:34454 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:62 xid:12 sent:10 recv:15 queuedpkts:0 pendingresp:0 queuedevents:0
  172. 2023-12-11 22:38:52,827 INFO Service [main-EventThread]: Thread: 33, NodeCreated: /services/kvstore/master
  173. 2023-12-11 22:38:52,826 INFO DependentService [main-EventThread]: DependentWatcher for service: hoststats Process path: /services/kvstore/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  174. 2023-12-11 22:38:52,837 INFO Service [main-EventThread]: NodeCreated: Thread: 33, MasterIP: maprdemo.mapr.io
  175. 2023-12-11 22:38:52,841 INFO DependentService [main-EventThread]: NodeCreated for service cldb Process path: /services/kvstore/maprdemo.mapr.io found zknodeData length 21
  176. 2023-12-11 22:38:52,846 ERROR DependentService [main-EventThread]: For service: cldb, dep service: kvstore, dependent zknode /services/kvstore/maprdemo.mapr.io service data does not have running status information
  177. 2023-12-11 22:38:52,848 INFO DependentService [main-EventThread]: createIPChild: svc:cldb isRestartNeeded:false isDepServiceRunning:false stats:null
  178. 2023-12-11 22:38:52,849 INFO DependentService [main-EventThread]: createIPChild: svc:cldb, wait for kvstore to start.
  179. 2023-12-11 22:38:52,849 INFO DependentService [main-EventThread]: NodeCreated for service hoststats Process path: /services/kvstore/maprdemo.mapr.io found zknodeData length 21
  180. 2023-12-11 22:38:52,852 ERROR DependentService [main-EventThread]: For service: hoststats, dep service: kvstore, dependent zknode /services/kvstore/maprdemo.mapr.io service data does not have running status information
  181. 2023-12-11 22:38:52,854 INFO DependentService [main-EventThread]: createIPChild: svc:hoststats isRestartNeeded:false isDepServiceRunning:false stats:null
  182. 2023-12-11 22:38:52,854 INFO DependentService [main-EventThread]: createIPChild: svc:hoststats, wait for kvstore to start.
  183. 2023-12-11 22:38:53,687 INFO ServiceRun [kvstore_monitor]: Command: [/opt/mapr/initscripts/mapr-mfs, start], Directory: /opt/mapr/initscripts
  184. 2023-12-11 22:38:53,688 INFO KVStoreService [kvstore_monitor]: Before adjustment, memory for KVStore = 922.0
  185. 2023-12-11 22:38:53,688 INFO KVStoreService [kvstore_monitor]: Before adjustment, memory for KVStore = 922.0
  186. 2023-12-11 22:38:55,274 INFO ServiceRun [kvstore_monitor]: INFO: Fileserver not configured for MapR-DB
  187.  
  188. 2023-12-11 22:38:57,622 ERROR Service [main]: Error while running command: [/opt/mapr/bin/maprcli, alarm, raise, -alarm, NODE_ALARM_MEMORY_ALLOCATION_EXCEEDED, -entity, maprdemo.mapr.io, -description, Memory allocated for services on this node is exceeding 97% of total memory available.]
  189. 2023-12-11 22:38:57,623 ERROR Service [main]: alarm raise
  190. [ -cluster cluster_name ]
  191. -alarm alarm
  192. [ -entity entity (hostname OR volume name OR Ae name) ]
  193. [ -description brief description ]
  194.  
  195. 2023-12-11 22:38:57,623 WARN Service [main]: Delayed alarm processing failed for alarm: com.mapr.warden.service.baseservice.Service$AlarmKeyEntity@6ad0e48c. Process later
  196. 2023-12-11 22:38:57,674 INFO WardenRequestBuilder [main]: [e_SERV_CONF, hostName, ma_host, ma_process]
  197. 2023-12-11 22:38:57,674 INFO WardenRequestBuilder [main]: []
  198. 2023-12-11 22:38:57,769 ERROR MaprRPCContext [Thread-4]: Response is null. Most likely hoststats is not accepting requests or it is down.
  199. 2023-12-11 22:39:05,274 INFO WardenRequestBuilder [kvstore_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  200. 2023-12-11 22:39:05,274 INFO WardenRequestBuilder [kvstore_monitor]: []
  201. 2023-12-11 22:39:05,283 INFO Service [kvstore_monitor]: setCurrentServiceUp: setting /services/kvstore/maprdemo.mapr.io to UP, version: 0
  202. 2023-12-11 22:39:05,287 INFO DependentService [main-EventThread]: DependentWatcher for service: cldb Process path: /services/kvstore/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  203. 2023-12-11 22:39:05,289 INFO DependentService [main-EventThread]: DependentWatcher for service: hoststats Process path: /services/kvstore/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  204. 2023-12-11 22:39:05,294 INFO Service [main-EventThread]: Process path: /services/kvstore/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  205. 2023-12-11 22:39:05,294 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40003 local:/172.17.0.2:34454 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:63 xid:16 sent:15 recv:21 queuedpkts:0 pendingresp:0 queuedevents:0
  206. 2023-12-11 22:39:05,295 INFO Service [kvstore_monitor]: Marking service is UP for kvstore on node /services/kvstore/maprdemo.mapr.io
  207. 2023-12-11 22:39:05,296 ERROR DependentService [main-EventThread]: For service: cldb, dep service: kvstore, dependent zknode /services/kvstore/maprdemo.mapr.io is running
  208. 2023-12-11 22:39:05,297 ERROR DependentService [main-EventThread]: For service: hoststats, dep service: kvstore, dependent zknode /services/kvstore/maprdemo.mapr.io is running
  209. 2023-12-11 22:39:05,299 INFO Service [kvstore_monitor]: setCurrentServiceUp: setting /services/kvstore/master to UP, version: 0
  210. 2023-12-11 22:39:05,300 INFO DependentService [main-EventThread]: createIPChild: svc:hoststats isRestartNeeded:false isDepServiceRunning:true stats:null
  211. 2023-12-11 22:39:05,302 INFO Service [main-EventThread]: Process path: /services/kvstore/master. Event state: SyncConnected. Event type: NodeDataChanged
  212. 2023-12-11 22:39:05,302 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40003 local:/172.17.0.2:34454 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:64 xid:19 sent:18 recv:25 queuedpkts:0 pendingresp:0 queuedevents:0
  213. 2023-12-11 22:39:05,302 INFO Service [kvstore_monitor]: Marking service is UP for kvstore on node /services/kvstore/master
  214. 2023-12-11 22:39:05,303 INFO DependentService [main-EventThread]: createIPChild: svc:cldb isRestartNeeded:false isDepServiceRunning:true stats:null
  215. 2023-12-11 22:39:05,307 INFO Service [main-EventThread]: createBaseIPChild: created zknode: /services/hoststats/maprdemo.mapr.io
  216. 2023-12-11 22:39:05,309 INFO Service [main-EventThread]: createBaseIPChild: created zknode: /services/cldb/maprdemo.mapr.io
  217. 2023-12-11 22:39:05,318 INFO DependentService [main-EventThread]: DependentWatcher for service: data-access-gateway Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeCreated
  218. 2023-12-11 22:39:05,319 INFO DependentService [main-EventThread]: DependentWatcher for service: apiserver Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeCreated
  219. 2023-12-11 22:39:05,319 INFO DependentService [main-EventThread]: DependentWatcher for service: gateway Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeCreated
  220. 2023-12-11 22:39:05,319 INFO DependentService [main-EventThread]: DependentWatcher for service: nfs Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeCreated
  221. 2023-12-11 22:39:05,319 INFO DependentService [main-EventThread]: DependentWatcher for service: drill-bits Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeCreated
  222. 2023-12-11 22:39:05,320 INFO DependentService [main-EventThread]: DependentWatcher for service: s3server Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeCreated
  223. 2023-12-11 22:39:05,321 INFO DependentService [main-EventThread]: NodeCreated for service s3server Process path: /services/cldb/master found zknodeData length 21
  224. 2023-12-11 22:39:05,321 INFO DependentService [main-EventThread]: NodeCreated for service gateway Process path: /services/cldb/master found zknodeData length 21
  225. 2023-12-11 22:39:05,321 INFO DependentService [main-EventThread]: NodeCreated for service drill-bits Process path: /services/cldb/master found zknodeData length 21
  226. 2023-12-11 22:39:05,322 INFO DependentService [main-EventThread]: NodeCreated for service apiserver Process path: /services/cldb/master found zknodeData length 21
  227. 2023-12-11 22:39:05,322 ERROR DependentService [main-EventThread]: For service: hoststats, dep service: kvstore, dependent zknode /services/kvstore/maprdemo.mapr.io is running
  228. 2023-12-11 22:39:05,322 INFO Service [main-EventThread]: -------------Service is starting for: hoststats
  229. 2023-12-11 22:39:05,322 INFO DependentService [main-EventThread]: NodeCreated for service nfs Process path: /services/cldb/master found zknodeData length 21
  230. 2023-12-11 22:39:05,323 INFO DependentService [main-EventThread]: NodeCreated for service data-access-gateway Process path: /services/cldb/master found zknodeData length 21
  231. 2023-12-11 22:39:05,330 ERROR DependentService [main-EventThread]: For service: nfs, dep service: cldb, dependent zknode /services/cldb/master service data does not have running status information
  232. 2023-12-11 22:39:05,330 ERROR DependentService [main-EventThread]: For service: cldb, dep service: kvstore, dependent zknode /services/kvstore/maprdemo.mapr.io is running
  233. 2023-12-11 22:39:05,330 INFO Service [main-EventThread]: -------------Service is starting for: cldb
  234. 2023-12-11 22:39:05,330 INFO ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/initscripts/mapr-cldb, status], Directory: /opt/mapr/initscripts/
  235. 2023-12-11 22:39:05,330 ERROR DependentService [main-EventThread]: For service: gateway, dep service: cldb, dependent zknode /services/cldb/master service data does not have running status information
  236. 2023-12-11 22:39:05,330 INFO ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/initscripts/mapr-hoststats, status], Directory: /opt/mapr/initscripts/
  237. 2023-12-11 22:39:05,330 ERROR DependentService [main-EventThread]: For service: drill-bits, dep service: cldb, dependent zknode /services/cldb/master service data does not have running status information
  238. 2023-12-11 22:39:05,331 ERROR DependentService [main-EventThread]: For service: s3server, dep service: cldb, dependent zknode /services/cldb/master service data does not have running status information
  239. 2023-12-11 22:39:05,331 ERROR DependentService [main-EventThread]: For service: apiserver, dep service: cldb, dependent zknode /services/cldb/master service data does not have running status information
  240. 2023-12-11 22:39:05,332 ERROR DependentService [main-EventThread]: For service: data-access-gateway, dep service: cldb, dependent zknode /services/cldb/master service data does not have running status information
  241. 2023-12-11 22:39:05,335 INFO DependentService [main-EventThread]: createIPChild: svc:gateway isRestartNeeded:false isDepServiceRunning:false stats:null
  242. 2023-12-11 22:39:05,335 INFO DependentService [main-EventThread]: createIPChild: svc:gateway, wait for cldb to start.
  243. 2023-12-11 22:39:05,335 INFO DependentService [main-EventThread]: createIPChild: svc:s3server isRestartNeeded:false isDepServiceRunning:false stats:null
  244. 2023-12-11 22:39:05,335 INFO DependentService [main-EventThread]: createIPChild: svc:s3server, wait for cldb to start.
  245. 2023-12-11 22:39:05,335 INFO DependentService [main-EventThread]: createIPChild: svc:apiserver isRestartNeeded:false isDepServiceRunning:false stats:null
  246. 2023-12-11 22:39:05,335 INFO DependentService [main-EventThread]: createIPChild: svc:apiserver, wait for cldb to start.
  247. 2023-12-11 22:39:05,337 INFO DependentService [main-EventThread]: createIPChild: svc:data-access-gateway isRestartNeeded:false isDepServiceRunning:false stats:null
  248. 2023-12-11 22:39:05,337 INFO DependentService [main-EventThread]: createIPChild: svc:data-access-gateway, wait for cldb to start.
  249. 2023-12-11 22:39:05,338 INFO DependentService [main-EventThread]: createIPChild: svc:drill-bits isRestartNeeded:false isDepServiceRunning:false stats:null
  250. 2023-12-11 22:39:05,338 INFO DependentService [main-EventThread]: createIPChild: svc:drill-bits, wait for cldb to start.
  251. 2023-12-11 22:39:05,338 INFO DependentService [main-EventThread]: createIPChild: svc:nfs isRestartNeeded:false isDepServiceRunning:false stats:null
  252. 2023-12-11 22:39:05,338 INFO DependentService [main-EventThread]: createIPChild: svc:nfs, wait for cldb to start.
  253. 2023-12-11 22:39:05,340 INFO Service [main-EventThread]: Process path: /services/cldb/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  254. 2023-12-11 22:39:05,340 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:68 xid:27 sent:26 recv:33 queuedpkts:0 pendingresp:0 queuedevents:2
  255. 2023-12-11 22:39:05,340 INFO Service [main-EventThread]: Thread: 36, NodeCreated: /services/cldb/maprdemo.mapr.io
  256. 2023-12-11 22:39:05,341 INFO Service [main-EventThread]: Process path: /services/cldb. Event state: SyncConnected. Event type: NodeChildrenChanged
  257. 2023-12-11 22:39:05,341 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:68 xid:28 sent:27 recv:34 queuedpkts:0 pendingresp:0 queuedevents:1
  258. 2023-12-11 22:39:05,341 INFO Service [main-EventThread]: Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeCreated
  259. 2023-12-11 22:39:05,341 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:68 xid:28 sent:27 recv:34 queuedpkts:0 pendingresp:0 queuedevents:0
  260. 2023-12-11 22:39:05,341 INFO Service [main-EventThread]: Thread: 36, NodeCreated: /services/cldb/master
  261. 2023-12-11 22:39:05,342 INFO Service [main-EventThread]: NodeCreated: Thread: 36, MasterIP: maprdemo.mapr.io
  262. 2023-12-11 22:39:05,346 INFO Service [main-EventThread]: Process path: /services/hoststats/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  263. 2023-12-11 22:39:05,346 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:68 xid:25 sent:24 recv:31 queuedpkts:0 pendingresp:0 queuedevents:2
  264. 2023-12-11 22:39:05,346 INFO Service [main-EventThread]: Thread: 45, NodeCreated: /services/hoststats/maprdemo.mapr.io
  265. 2023-12-11 22:39:05,353 INFO Service [main-EventThread]: Process path: /services/hoststats. Event state: SyncConnected. Event type: NodeChildrenChanged
  266. 2023-12-11 22:39:05,353 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:69 xid:26 sent:25 recv:32 queuedpkts:0 pendingresp:0 queuedevents:1
  267. 2023-12-11 22:39:05,354 INFO Service [main-EventThread]: Process path: /services/hoststats/master. Event state: SyncConnected. Event type: NodeCreated
  268. 2023-12-11 22:39:05,354 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:69 xid:26 sent:25 recv:32 queuedpkts:0 pendingresp:0 queuedevents:0
  269. 2023-12-11 22:39:05,354 INFO Service [main-EventThread]: Thread: 45, NodeCreated: /services/hoststats/master
  270. 2023-12-11 22:39:05,362 INFO Service [cldb_monitor]: Need delayed alarm clearing for: NODE_ALARM_SERVICE_CLDB_DOWN
  271. 2023-12-11 22:39:05,362 INFO Service [main-EventThread]: NodeCreated: Thread: 45, MasterIP: maprdemo.mapr.io
  272. 2023-12-11 22:39:05,365 INFO Service [hoststats_monitor]: Need delayed alarm clearing for: NODE_ALARM_SERVICE_HOSTSTATS_DOWN
  273. 2023-12-11 22:39:06,270 INFO ServiceRun [hoststats_monitor]: Command: [/opt/mapr/initscripts/mapr-hoststats, start], Directory: /opt/mapr/initscripts
  274. 2023-12-11 22:39:06,480 INFO ServiceRun [hoststats_monitor]:
  275. 2023-12-11 22:39:06,506 ERROR CLDBService [cldb_monitor]: Exception while trying to get children of: /datacenter/controlnodes/cldb/active/CLDBNodes
  276. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /datacenter/controlnodes/cldb/active/CLDBNodes
  277. at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[zookeeper-3.5.6.0-mapr-2112.jar:3.5.6.0-mapr-2112]
  278. at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.5.6.0-mapr-2112.jar:3.5.6.0-mapr-2112]
  279. at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2624) ~[zookeeper-3.5.6.0-mapr-2112.jar:3.5.6.0-mapr-2112]
  280. at com.mapr.warden.service.baseservice.common.ZKUtilsLocking.getZkNodeChildren(ZKUtilsLocking.java:90) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  281. at com.mapr.warden.service.CLDBService.isToStartNow(CLDBService.java:321) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  282. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2130) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  283. at java.lang.Thread.run(Thread.java:829) [?:?]
  284. 2023-12-11 22:39:06,506 INFO ServiceRun [cldb_monitor]: Command: [/opt/mapr/initscripts/mapr-cldb, start], Directory: /opt/mapr/initscripts
  285. 2023-12-11 22:39:08,955 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  286. 2023-12-11 22:39:08,955 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  287. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  288. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  289.  
  290. 2023-12-11 22:39:11,678 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  291. 2023-12-11 22:39:11,678 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  292. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  293. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  294.  
  295. 2023-12-11 22:39:16,481 INFO WardenRequestBuilder [hoststats_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  296. 2023-12-11 22:39:16,481 INFO WardenRequestBuilder [hoststats_monitor]: []
  297. 2023-12-11 22:39:16,499 INFO Service [hoststats_monitor]: setCurrentServiceUp: setting /services/hoststats/maprdemo.mapr.io to UP, version: 0
  298. 2023-12-11 22:39:16,503 INFO Service [main-EventThread]: Process path: /services/hoststats/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  299. 2023-12-11 22:39:16,503 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:124 xid:30 sent:30 recv:38 queuedpkts:0 pendingresp:0 queuedevents:0
  300. 2023-12-11 22:39:16,504 INFO Service [hoststats_monitor]: Marking service is UP for hoststats on node /services/hoststats/maprdemo.mapr.io
  301. 2023-12-11 22:39:16,504 INFO Service [hoststats_monitor]: setCurrentServiceUp: setting /services/hoststats/master to UP, version: 0
  302. 2023-12-11 22:39:16,508 INFO Service [main-EventThread]: Process path: /services/hoststats/master. Event state: SyncConnected. Event type: NodeDataChanged
  303. 2023-12-11 22:39:16,508 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:125 xid:33 sent:33 recv:42 queuedpkts:0 pendingresp:0 queuedevents:0
  304. 2023-12-11 22:39:16,508 INFO Service [hoststats_monitor]: Marking service is UP for hoststats on node /services/hoststats/master
  305. 2023-12-11 22:39:17,246 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  306. 2023-12-11 22:39:17,246 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  307. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  308. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  309.  
  310. 2023-12-11 22:39:18,374 ERROR ServiceMonitorRun [hoststats_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-hoststats, status]can not determine if service: hoststats is running. Retrying. Retrial #1. Total retries count is: 3
  311. 2023-12-11 22:39:18,374 ERROR ServiceMonitorRun [hoststats_monitor]: * hoststats is not running
  312.  
  313. 2023-12-11 22:39:18,374 INFO ServiceRun [hoststats_monitor]: Command: [/opt/mapr/initscripts/mapr-hoststats, start], Directory: /opt/mapr/initscripts
  314. kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
  315. Warden start failed
  316. Error: warden can not be started. See /opt/mapr/logs/warden.log for details
  317. 2023-12-11 22:39:20,508 INFO ServiceRun [hoststats_monitor]:
  318. 2023-12-11 22:39:23,578 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  319. 2023-12-11 22:39:23,578 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  320. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  321. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  322.  
  323. 2023-12-11 22:39:30,509 INFO WardenRequestBuilder [hoststats_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  324. 2023-12-11 22:39:30,509 INFO WardenRequestBuilder [hoststats_monitor]: []
  325. 2023-12-11 22:39:32,159 ERROR ServiceMonitorRun [hoststats_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-hoststats, status]can not determine if service: hoststats is running. Retrying. Retrial #2. Total retries count is: 3
  326. 2023-12-11 22:39:32,159 ERROR ServiceMonitorRun [hoststats_monitor]: * hoststats is not running
  327.  
  328. 2023-12-11 22:39:32,159 INFO ServiceRun [hoststats_monitor]: Command: [/opt/mapr/initscripts/mapr-hoststats, start], Directory: /opt/mapr/initscripts
  329. 2023-12-11 22:39:33,579 INFO WardenRequestBuilder [cldb_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  330. 2023-12-11 22:39:33,579 INFO WardenRequestBuilder [cldb_monitor]: []
  331. 2023-12-11 22:39:33,588 INFO Service [cldb_monitor]: Could not find PID file for service: CLDB doing pgrep instead
  332. 2023-12-11 22:39:33,953 INFO ServiceRun [hoststats_monitor]:
  333. 2023-12-11 22:39:35,116 INFO Service [cldb_monitor]: setCurrentServiceUp: setting /services/cldb/maprdemo.mapr.io to UP, version: 0
  334. 2023-12-11 22:39:35,120 INFO Service [main-EventThread]: Process path: /services/cldb/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  335. 2023-12-11 22:39:35,120 INFO Service [cldb_monitor]: Marking service is UP for cldb on node /services/cldb/maprdemo.mapr.io
  336. 2023-12-11 22:39:35,120 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:188 xid:33 sent:34 recv:42 queuedpkts:0 pendingresp:0 queuedevents:0
  337. 2023-12-11 22:39:37,684 ERROR Service [cldb_monitor]: Error while running command: [/opt/mapr/server/maprexecute, adjustoom, -17, 9649]
  338. 2023-12-11 22:39:37,685 ERROR Service [cldb_monitor]:
  339. 2023-12-11 22:39:37,690 ERROR Service [cldb_monitor]: reniceProcess fetching child PIDs for 9649 failed: 2
  340. 2023-12-11 22:39:37,690 ERROR ServiceMonitorRun [cldb_monitor]: Could not renice process: 9649 to -10
  341. 2023-12-11 22:39:39,132 ERROR Service [cldb_monitor]: reniceProcessExec failed for child 10444 of parent 10381
  342. 2023-12-11 22:39:39,132 ERROR ServiceMonitorRun [cldb_monitor]: Could not renice process: 10381 to -10
  343. 2023-12-11 22:39:41,576 INFO CLDBService [main-EventThread]: Process path: /datacenter/controlnodes/cldb/active/CLDBRunningMaster. Event state: SyncConnected. Event type: NodeCreated
  344. 2023-12-11 22:39:41,577 INFO WardenManager [main-EventThread]: Process path: /datacenter/controlnodes/cldb/active/CLDBRunningMaster. Event state: SyncConnected. Event type: NodeCreated
  345. 2023-12-11 22:39:41,592 INFO WardenManager [maprUserTicketGetCheckThread]: maprUserTicketExpiration: 922337203685477
  346. 2023-12-11 22:39:41,591 INFO Service [main-EventThread]: Process delayed alarms: 9
  347. 2023-12-11 22:39:41,609 INFO Service [main-EventThread]: setCurrentServiceUp: setting /services/cldb/master to UP, version: 0
  348. 2023-12-11 22:39:41,616 INFO DependentService [main-EventThread]: DependentWatcher for service: nfs Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDataChanged
  349. 2023-12-11 22:39:41,616 INFO DependentService [main-EventThread]: DependentWatcher for service: drill-bits Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDataChanged
  350. 2023-12-11 22:39:41,616 INFO DependentService [main-EventThread]: DependentWatcher for service: apiserver Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDataChanged
  351. 2023-12-11 22:39:41,617 INFO DependentService [main-EventThread]: DependentWatcher for service: gateway Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDataChanged
  352. 2023-12-11 22:39:41,617 INFO DependentService [main-EventThread]: DependentWatcher for service: s3server Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDataChanged
  353. 2023-12-11 22:39:41,618 INFO DependentService [main-EventThread]: DependentWatcher for service: data-access-gateway Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDataChanged
  354. 2023-12-11 22:39:41,620 ERROR DependentService [main-EventThread]: For service: gateway, dep service: cldb, dependent zknode /services/cldb/master is running
  355. 2023-12-11 22:39:41,620 ERROR DependentService [main-EventThread]: For service: drill-bits, dep service: cldb, dependent zknode /services/cldb/master is running
  356. 2023-12-11 22:39:41,620 ERROR DependentService [main-EventThread]: For service: apiserver, dep service: cldb, dependent zknode /services/cldb/master is running
  357. 2023-12-11 22:39:41,621 ERROR DependentService [main-EventThread]: For service: data-access-gateway, dep service: cldb, dependent zknode /services/cldb/master is running
  358. 2023-12-11 22:39:41,621 ERROR DependentService [main-EventThread]: For service: s3server, dep service: cldb, dependent zknode /services/cldb/master is running
  359. 2023-12-11 22:39:41,622 ERROR DependentService [main-EventThread]: For service: nfs, dep service: cldb, dependent zknode /services/cldb/master is running
  360. 2023-12-11 22:39:41,624 INFO DependentService [main-EventThread]: createIPChild: svc:gateway isRestartNeeded:false isDepServiceRunning:true stats:null
  361. 2023-12-11 22:39:41,638 INFO DependentService [main-EventThread]: createIPChild: svc:data-access-gateway isRestartNeeded:false isDepServiceRunning:true stats:null
  362. 2023-12-11 22:39:41,638 INFO DependentService [main-EventThread]: createIPChild: svc:drill-bits isRestartNeeded:false isDepServiceRunning:true stats:null
  363. 2023-12-11 22:39:41,638 INFO DependentService [main-EventThread]: createIPChild: svc:apiserver isRestartNeeded:false isDepServiceRunning:true stats:null
  364. 2023-12-11 22:39:41,639 INFO DependentService [main-EventThread]: createIPChild: svc:nfs isRestartNeeded:false isDepServiceRunning:true stats:null
  365. 2023-12-11 22:39:41,639 INFO DependentService [main-EventThread]: createIPChild: svc:s3server isRestartNeeded:false isDepServiceRunning:true stats:null
  366. 2023-12-11 22:39:41,647 INFO Service [main-EventThread]: createBaseIPChild: created zknode: /services/gateway/maprdemo.mapr.io
  367. 2023-12-11 22:39:41,648 INFO Service [main-EventThread]: createBaseIPChild: created zknode: /services/drill-bits/maprdemo.mapr.io
  368. 2023-12-11 22:39:41,650 INFO Service [main-EventThread]: createBaseIPChild: created zknode: /services/apiserver/maprdemo.mapr.io
  369. 2023-12-11 22:39:41,656 INFO Service [main-EventThread]: createBaseIPChild: created zknode: /services/s3server/maprdemo.mapr.io
  370. 2023-12-11 22:39:41,661 INFO Service [main-EventThread]: createBaseIPChild: created zknode: /services/nfs/maprdemo.mapr.io
  371. 2023-12-11 22:39:41,663 INFO Service [main-EventThread]: createBaseIPChild: created zknode: /services/data-access-gateway/maprdemo.mapr.io
  372. 2023-12-11 22:39:41,681 INFO Service [main-EventThread]: Marking service is UP for cldb on node /services/cldb/master
  373. 2023-12-11 22:39:41,682 INFO CLDBService [main-EventThread]: NodeCreated: setting cldbRunningMasterName:maprdemo.mapr.io
  374. 2023-12-11 22:39:41,682 INFO Service [main-EventThread]: Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDataChanged
  375. 2023-12-11 22:39:41,682 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:193 xid:38 sent:39 recv:49 queuedpkts:0 pendingresp:0 queuedevents:0
  376. 2023-12-11 22:39:41,688 ERROR DependentService [main-EventThread]: For service: s3server, dep service: cldb, dependent zknode /services/cldb/master is running
  377. 2023-12-11 22:39:41,688 INFO Service [main-EventThread]: -------------Service is starting for: s3server
  378. 2023-12-11 22:39:41,689 ERROR DependentService [main-EventThread]: For service: gateway, dep service: cldb, dependent zknode /services/cldb/master is running
  379. 2023-12-11 22:39:41,689 INFO Service [main-EventThread]: -------------Service is starting for: gateway
  380. 2023-12-11 22:39:41,689 INFO ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/initscripts/mapr-gateway, status], Directory: /opt/mapr/initscripts/
  381. 2023-12-11 22:39:41,702 ERROR DependentService [main-EventThread]: For service: apiserver, dep service: cldb, dependent zknode /services/cldb/master is running
  382. 2023-12-11 22:39:41,702 INFO Service [main-EventThread]: -------------Service is starting for: apiserver
  383. 2023-12-11 22:39:41,702 INFO ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/apiserver/bin/mapr-apiserver.sh, status], Directory: /opt/mapr/apiserver/bin/
  384. 2023-12-11 22:39:41,709 INFO ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/initscripts/mapr-s3server, status], Directory: /opt/mapr/initscripts/
  385. 2023-12-11 22:39:41,718 ERROR DependentService [main-EventThread]: For service: drill-bits, dep service: cldb, dependent zknode /services/cldb/master is running
  386. 2023-12-11 22:39:41,724 ERROR DependentService [main-EventThread]: For service: nfs, dep service: cldb, dependent zknode /services/cldb/master is running
  387. 2023-12-11 22:39:41,724 INFO Service [main-EventThread]: -------------Service is starting for: nfs
  388. 2023-12-11 22:39:41,724 INFO ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/initscripts/mapr-nfsserver, status], Directory: /opt/mapr/initscripts/
  389. 2023-12-11 22:39:41,735 INFO Service [main-EventThread]: -------------Service is starting for: drill-bits
  390. 2023-12-11 22:39:41,736 INFO ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status], Directory: /opt/mapr/drill/drill-1.14.0/bin/
  391. 2023-12-11 22:39:41,767 ERROR DependentService [main-EventThread]: For service: data-access-gateway, dep service: cldb, dependent zknode /services/cldb/master is running
  392. 2023-12-11 22:39:41,767 INFO Service [main-EventThread]: -------------Service is starting for: data-access-gateway
  393. 2023-12-11 22:39:41,767 INFO ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/data-access-gateway/bin/mapr-data-access-gateway.sh, status], Directory: /opt/mapr/data-access-gateway/bin/
  394. 2023-12-11 22:39:41,772 INFO Service [main-EventThread]: Process path: /services/gateway/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  395. 2023-12-11 22:39:41,772 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:205 xid:25 sent:27 recv:34 queuedpkts:0 pendingresp:0 queuedevents:2
  396. 2023-12-11 22:39:41,773 INFO Service [main-EventThread]: Thread: 42, NodeCreated: /services/gateway/maprdemo.mapr.io
  397. 2023-12-11 22:39:41,774 INFO Service [main-EventThread]: Process path: /services/gateway. Event state: SyncConnected. Event type: NodeChildrenChanged
  398. 2023-12-11 22:39:41,774 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:205 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:1
  399. 2023-12-11 22:39:41,774 INFO Service [main-EventThread]: Process path: /services/gateway/master. Event state: SyncConnected. Event type: NodeCreated
  400. 2023-12-11 22:39:41,774 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:205 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:0
  401. 2023-12-11 22:39:41,774 INFO Service [main-EventThread]: Thread: 42, NodeCreated: /services/gateway/master
  402. 2023-12-11 22:39:41,775 INFO Service [main-EventThread]: NodeCreated: Thread: 42, MasterIP: maprdemo.mapr.io
  403. 2023-12-11 22:39:41,811 INFO Service [gateway_monitor]: Adding pluggable alarm NODE_ALARM_SERVICE_GATEWAY_DOWN to service: gateway
  404. 2023-12-11 22:39:41,848 INFO Service [main-EventThread]: Process path: /services/data-access-gateway/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  405. 2023-12-11 22:39:41,848 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40008 local:/172.17.0.2:34514 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:205 xid:25 sent:27 recv:34 queuedpkts:0 pendingresp:0 queuedevents:2
  406. 2023-12-11 22:39:41,848 INFO Service [main-EventThread]: Thread: 48, NodeCreated: /services/data-access-gateway/maprdemo.mapr.io
  407. 2023-12-11 22:39:41,849 INFO Service [main-EventThread]: Process path: /services/data-access-gateway. Event state: SyncConnected. Event type: NodeChildrenChanged
  408. 2023-12-11 22:39:41,850 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40008 local:/172.17.0.2:34514 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:206 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:1
  409. 2023-12-11 22:39:41,850 INFO Service [main-EventThread]: Process path: /services/data-access-gateway/master. Event state: SyncConnected. Event type: NodeCreated
  410. 2023-12-11 22:39:41,850 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40008 local:/172.17.0.2:34514 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:206 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:0
  411. 2023-12-11 22:39:41,850 INFO Service [main-EventThread]: Thread: 48, NodeCreated: /services/data-access-gateway/master
  412. 2023-12-11 22:39:41,851 INFO Service [main-EventThread]: NodeCreated: Thread: 48, MasterIP: maprdemo.mapr.io
  413. 2023-12-11 22:39:41,879 INFO Service [main-EventThread]: Process path: /services/drill-bits/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  414. 2023-12-11 22:39:41,880 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:205 xid:25 sent:27 recv:34 queuedpkts:0 pendingresp:0 queuedevents:2
  415. 2023-12-11 22:39:41,880 INFO Service [main-EventThread]: Thread: 27, NodeCreated: /services/drill-bits/maprdemo.mapr.io
  416. 2023-12-11 22:39:41,882 INFO Service [main-EventThread]: Process path: /services/drill-bits. Event state: SyncConnected. Event type: NodeChildrenChanged
  417. 2023-12-11 22:39:41,882 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:207 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:1
  418. 2023-12-11 22:39:41,882 INFO Service [main-EventThread]: Process path: /services/drill-bits/master. Event state: SyncConnected. Event type: NodeCreated
  419. 2023-12-11 22:39:41,882 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:207 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:0
  420. 2023-12-11 22:39:41,883 INFO Service [main-EventThread]: Thread: 27, NodeCreated: /services/drill-bits/master
  421. 2023-12-11 22:39:41,889 INFO Service [main-EventThread]: NodeCreated: Thread: 27, MasterIP: maprdemo.mapr.io
  422. 2023-12-11 22:39:41,894 INFO Service [data-access-gateway_monitor]: Adding pluggable alarm NODE_ALARM_SERVICE_DATA-ACCESS-GATEWAY_DOWN to service: data-access-gateway
  423. 2023-12-11 22:39:41,900 INFO Service [drill-bits_monitor]: Adding pluggable alarm NODE_ALARM_SERVICE_DRILL-BITS_DOWN to service: drill-bits
  424. 2023-12-11 22:39:41,925 INFO Service [main-EventThread]: Process path: /services/nfs/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  425. 2023-12-11 22:39:41,925 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40005 local:/172.17.0.2:34476 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:205 xid:25 sent:27 recv:34 queuedpkts:0 pendingresp:0 queuedevents:2
  426. 2023-12-11 22:39:41,925 INFO Service [main-EventThread]: Thread: 39, NodeCreated: /services/nfs/maprdemo.mapr.io
  427. 2023-12-11 22:39:41,927 INFO Service [main-EventThread]: Process path: /services/nfs. Event state: SyncConnected. Event type: NodeChildrenChanged
  428. 2023-12-11 22:39:41,927 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40005 local:/172.17.0.2:34476 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:208 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:1
  429. 2023-12-11 22:39:41,927 INFO Service [main-EventThread]: Process path: /services/nfs/master. Event state: SyncConnected. Event type: NodeCreated
  430. 2023-12-11 22:39:41,927 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40005 local:/172.17.0.2:34476 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:208 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:0
  431. 2023-12-11 22:39:41,927 INFO Service [main-EventThread]: Thread: 39, NodeCreated: /services/nfs/master
  432. 2023-12-11 22:39:41,928 INFO Service [main-EventThread]: NodeCreated: Thread: 39, MasterIP: maprdemo.mapr.io
  433. 2023-12-11 22:39:41,962 INFO Service [nfs_monitor]: Alarm clearing command: [/opt/mapr/bin/maprcli, alarm, clear, -alarm, NODE_ALARM_SERVICE_NFS_DOWN, -entity, maprdemo.mapr.io]
  434. 2023-12-11 22:39:41,972 INFO Service [main-EventThread]: Process path: /services/s3server/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  435. 2023-12-11 22:39:41,973 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:205 xid:25 sent:27 recv:34 queuedpkts:0 pendingresp:0 queuedevents:2
  436. 2023-12-11 22:39:41,973 INFO Service [main-EventThread]: Thread: 30, NodeCreated: /services/s3server/maprdemo.mapr.io
  437. 2023-12-11 22:39:41,974 INFO Service [main-EventThread]: Process path: /services/s3server. Event state: SyncConnected. Event type: NodeChildrenChanged
  438. 2023-12-11 22:39:41,974 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:210 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:1
  439. 2023-12-11 22:39:41,974 INFO Service [main-EventThread]: Process path: /services/s3server/master. Event state: SyncConnected. Event type: NodeCreated
  440. 2023-12-11 22:39:41,974 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:210 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:0
  441. 2023-12-11 22:39:41,974 INFO Service [main-EventThread]: Thread: 30, NodeCreated: /services/s3server/master
  442. 2023-12-11 22:39:41,976 INFO Service [main-EventThread]: NodeCreated: Thread: 30, MasterIP: maprdemo.mapr.io
  443. 2023-12-11 22:39:42,013 INFO Service [s3server_monitor]: Adding pluggable alarm NODE_ALARM_SERVICE_S3SERVER_DOWN to service: s3server
  444. 2023-12-11 22:39:42,026 INFO Service [main-EventThread]: Process path: /services/apiserver/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeCreated
  445. 2023-12-11 22:39:42,027 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40009 local:/172.17.0.2:34524 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:205 xid:25 sent:27 recv:34 queuedpkts:0 pendingresp:0 queuedevents:2
  446. 2023-12-11 22:39:42,027 INFO Service [main-EventThread]: Thread: 51, NodeCreated: /services/apiserver/maprdemo.mapr.io
  447. 2023-12-11 22:39:42,028 INFO Service [main-EventThread]: Process path: /services/apiserver. Event state: SyncConnected. Event type: NodeChildrenChanged
  448. 2023-12-11 22:39:42,028 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40009 local:/172.17.0.2:34524 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:211 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:1
  449. 2023-12-11 22:39:42,028 INFO Service [main-EventThread]: Process path: /services/apiserver/master. Event state: SyncConnected. Event type: NodeCreated
  450. 2023-12-11 22:39:42,028 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40009 local:/172.17.0.2:34524 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:211 xid:26 sent:28 recv:35 queuedpkts:0 pendingresp:0 queuedevents:0
  451. 2023-12-11 22:39:42,028 INFO Service [main-EventThread]: Thread: 51, NodeCreated: /services/apiserver/master
  452. 2023-12-11 22:39:42,029 INFO Service [main-EventThread]: NodeCreated: Thread: 51, MasterIP: maprdemo.mapr.io
  453. 2023-12-11 22:39:42,061 INFO Service [apiserver_monitor]: Adding pluggable alarm NODE_ALARM_SERVICE_APISERVER_DOWN to service: apiserver
  454. 2023-12-11 22:39:43,953 INFO WardenRequestBuilder [hoststats_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  455. 2023-12-11 22:39:43,954 INFO WardenRequestBuilder [hoststats_monitor]: []
  456. 2023-12-11 22:39:47,903 ERROR ServiceMonitorRun [cldb_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-cldb, status]can not determine if service: cldb is running. Retrying. Retrial #1. Total retries count is: 3
  457. 2023-12-11 22:39:47,903 ERROR ServiceMonitorRun [cldb_monitor]: CLDB not running.
  458.  
  459. 2023-12-11 22:39:49,690 ERROR ServiceMonitorRun [hoststats_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-hoststats, status]cannot determine if service: hoststats is running. Number of retrials exceeded. Closing Zookeeper
  460. 2023-12-11 22:39:49,716 INFO Service [hoststats_monitor]: 67 about to close zk for service: hoststats
  461. 2023-12-11 22:39:49,721 INFO Service [main-EventThread]: Process path: /services/hoststats/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDeleted
  462. 2023-12-11 22:39:49,721 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:125 xid:36 sent:39 recv:49 queuedpkts:0 pendingresp:1 queuedevents:1
  463. 2023-12-11 22:39:49,723 ERROR ZKUtilsLocking [main-EventThread]: Lost connection to ZK while trying to checkZKNodeForExistence of: /services/kvstore/maprdemo.mapr.io. Retrying...
  464. 2023-12-11 22:39:49,832 INFO WardenRequestBuilder [hoststats_monitor]: [e_SERV_FAIL, hostName, ma_host, ma_process]
  465. 2023-12-11 22:39:49,832 INFO WardenRequestBuilder [hoststats_monitor]: []
  466. 2023-12-11 22:39:49,833 INFO Service [hoststats_monitor]: Alarm raising command: [/opt/mapr/bin/maprcli, alarm, raise, -alarm, NODE_ALARM_SERVICE_HOSTSTATS_DOWN, -entity, maprdemo.mapr.io, -description, Can not determine if service: hoststats is running. Check logs at: /opt/mapr/logs/hoststats.log]
  467. 2023-12-11 22:39:50,224 ERROR ZKUtilsLocking [main-EventThread]: SessionExpiredException while trying to checkZKNodeForExistence of: /services/kvstore/maprdemo.mapr.io
  468. 2023-12-11 22:39:50,224 ERROR DependentService [main-EventThread]: ZK Session was either or closed or expired for service: hoststats
  469. 2023-12-11 22:39:50,224 ERROR ZKUtilsLocking [main-EventThread]: SessionExpiredException while trying to checkZKNodeForExistence of: /services/hoststats/maprdemo.mapr.io
  470. 2023-12-11 22:39:50,224 ERROR Service [main-EventThread]: ZK Session was either or closed or expired for service: hoststats
  471. 2023-12-11 22:39:50,224 INFO Service [main-EventThread]: Process path: /services/hoststats/master. Event state: SyncConnected. Event type: NodeDeleted
  472. 2023-12-11 22:39:50,224 INFO Service [main-EventThread]: ZK Connect state:State:CLOSED sessionid:0x1000010ecd40007 local:/172.17.0.2:34502 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:213 xid:36 sent:39 recv:50 queuedpkts:0 pendingresp:0 queuedevents:2
  473. 2023-12-11 22:39:50,225 INFO Service [main-EventThread]: ZK is closed for service: hoststats
  474. 2023-12-11 22:39:50,225 INFO DependentService [main-EventThread]: DependentWatcher for service: hoststats Process path: null. Event state: Closed. Event type: None
  475. 2023-12-11 22:39:50,225 INFO DependentService [main-EventThread]: ZK is closed for service: hoststats
  476. 2023-12-11 22:39:50,225 INFO Service [main-EventThread]: Process path: null. Event state: Closed. Event type: None
  477. 2023-12-11 22:39:50,225 INFO Service [main-EventThread]: ZK is closed for service: hoststats
  478. 2023-12-11 22:40:00,922 INFO ServiceRun [cldb_monitor]: Command: [/opt/mapr/initscripts/mapr-cldb, start], Directory: /opt/mapr/initscripts
  479. 2023-12-11 22:40:10,785 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  480. 2023-12-11 22:40:10,785 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  481. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  482. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  483.  
  484. 2023-12-11 22:40:10,952 ERROR Service [data-access-gateway_monitor]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_DATA-ACCESS-GATEWAY_DOWN, -terse, dataaccesssrvd, -service, data-access-gateway, -displayName, DataAccessGatewayDown, -baseService, 1, -serviceName, DataAccessGateway]
  485. 2023-12-11 22:40:10,953 ERROR Service [data-access-gateway_monitor]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  486.  
  487. 2023-12-11 22:40:10,953 WARN Service [data-access-gateway_monitor]: Unable to add alarm. NODE_ALARM_SERVICE_DATA-ACCESS-GATEWAY_DOWN. Adding to delayed condition
  488. 2023-12-11 22:40:10,962 INFO Service [data-access-gateway_monitor]: Alarm clearing command: [/opt/mapr/bin/maprcli, alarm, clear, -alarm, NODE_ALARM_SERVICE_DATA-ACCESS-GATEWAY_DOWN, -entity, maprdemo.mapr.io]
  489. 2023-12-11 22:40:11,011 ERROR Service [drill-bits_monitor]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_DRILL-BITS_DOWN, -terse, nadrill, -service, drill-bits, -displayName, DRILLDOWNALARM, -serviceName, Drillbit]
  490. 2023-12-11 22:40:11,014 ERROR Service [drill-bits_monitor]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  491.  
  492. 2023-12-11 22:40:11,014 WARN Service [drill-bits_monitor]: Unable to add alarm. NODE_ALARM_SERVICE_DRILL-BITS_DOWN. Adding to delayed condition
  493. 2023-12-11 22:40:11,021 INFO Service [drill-bits_monitor]: Alarm clearing command: [/opt/mapr/bin/maprcli, alarm, clear, -alarm, NODE_ALARM_SERVICE_DRILL-BITS_DOWN, -entity, maprdemo.mapr.io]
  494. 2023-12-11 22:40:12,061 ERROR Service [apiserver_monitor]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_APISERVER_DOWN, -terse, naaapisrvd, -service, apiserver, -displayName, ApiServerDown, -baseService, 1, -serviceName, ApiServer]
  495. 2023-12-11 22:40:12,061 ERROR Service [apiserver_monitor]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  496.  
  497. 2023-12-11 22:40:12,062 WARN Service [apiserver_monitor]: Unable to add alarm. NODE_ALARM_SERVICE_APISERVER_DOWN. Adding to delayed condition
  498. 2023-12-11 22:40:12,067 INFO Service [apiserver_monitor]: Alarm clearing command: [/opt/mapr/bin/maprcli, alarm, clear, -alarm, NODE_ALARM_SERVICE_APISERVER_DOWN, -entity, maprdemo.mapr.io]
  499. 2023-12-11 22:40:12,078 ERROR Service [s3server_monitor]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_S3SERVER_DOWN, -terse, nas3sd, -service, s3server, -displayName, S3ServerServiceDown, -baseService, 1, -serviceName, S3ServerService]
  500. 2023-12-11 22:40:12,079 ERROR Service [s3server_monitor]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  501.  
  502. 2023-12-11 22:40:12,079 WARN Service [s3server_monitor]: Unable to add alarm. NODE_ALARM_SERVICE_S3SERVER_DOWN. Adding to delayed condition
  503. 2023-12-11 22:40:12,094 INFO Service [s3server_monitor]: Alarm clearing command: [/opt/mapr/bin/maprcli, alarm, clear, -alarm, NODE_ALARM_SERVICE_S3SERVER_DOWN, -entity, maprdemo.mapr.io]
  504. 2023-12-11 22:40:13,040 ERROR Service [main-EventThread]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_DATA-ACCESS-GATEWAY_DOWN, -terse, dataaccesssrvd, -service, data-access-gateway, -displayName, DataAccessGatewayDown, -baseService, 1, -serviceName, DataAccessGateway]
  505. 2023-12-11 22:40:13,041 ERROR Service [main-EventThread]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  506.  
  507. 2023-12-11 22:40:13,041 WARN Service [main-EventThread]: Delayed alarm processing failed for alarm: com.mapr.warden.service.baseservice.Service$AlarmKeyEntity@818b005a. Process later
  508. 2023-12-11 22:40:13,206 ERROR Service [gateway_monitor]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_GATEWAY_DOWN, -terse, nagwsd, -service, gateway, -displayName, GatewayServiceDown, -baseService, 1, -serviceName, GatewayService]
  509. 2023-12-11 22:40:13,209 ERROR Service [gateway_monitor]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  510.  
  511. 2023-12-11 22:40:13,209 WARN Service [gateway_monitor]: Unable to add alarm. NODE_ALARM_SERVICE_GATEWAY_DOWN. Adding to delayed condition
  512. 2023-12-11 22:40:13,216 INFO Service [gateway_monitor]: Alarm clearing command: [/opt/mapr/bin/maprcli, alarm, clear, -alarm, NODE_ALARM_SERVICE_GATEWAY_DOWN, -entity, maprdemo.mapr.io]
  513. 2023-12-11 22:40:15,229 INFO ServiceRun [nfs_monitor]: Command: [/opt/mapr/initscripts/mapr-nfsserver, start], Directory: /opt/mapr/initscripts
  514. 2023-12-11 22:40:18,403 INFO ServiceRun [nfs_monitor]:
  515. 2023-12-11 22:40:19,890 INFO Service [Thread-14-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  516. 2023-12-11 22:40:19,890 INFO Service [Thread-14]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd4000e local:/172.17.0.2:37342 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  517. 2023-12-11 22:40:19,904 INFO Service [Thread-14]: Service reinitThread: zk expired: false reconnected to zk, lastAction: null
  518. 2023-12-11 22:40:19,905 INFO Service [Thread-14-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd4000e local:/172.17.0.2:37342 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:1 pendingresp:0 queuedevents:0
  519. 2023-12-11 22:40:19,923 INFO Service [Thread-14-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  520. 2023-12-11 22:40:19,926 INFO Service [Thread-14]: Node: /nodes/maprdemo.mapr.io/services/hoststats does not exist yet
  521. 2023-12-11 22:40:19,938 INFO Service [Thread-14]: addServiceToZK: zknode /services/hoststats already exists
  522. 2023-12-11 22:40:19,939 INFO Service [Thread-14]: addServiceToZK: after remove master, children node size 0
  523. 2023-12-11 22:40:20,407 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  524. 2023-12-11 22:40:20,408 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  525. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  526. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  527.  
  528. 2023-12-11 22:40:28,405 INFO WardenRequestBuilder [nfs_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  529. 2023-12-11 22:40:28,405 INFO WardenRequestBuilder [nfs_monitor]: []
  530. 2023-12-11 22:40:28,507 INFO Service [nfs_monitor]: setCurrentServiceUp: setting /services/nfs/maprdemo.mapr.io to UP, version: 0
  531. 2023-12-11 22:40:28,511 INFO Service [main-EventThread]: Process path: /services/nfs/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  532. 2023-12-11 22:40:28,511 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40005 local:/172.17.0.2:34476 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:226 xid:30 sent:36 recv:44 queuedpkts:0 pendingresp:0 queuedevents:0
  533. 2023-12-11 22:40:28,511 INFO Service [nfs_monitor]: Marking service is UP for nfs on node /services/nfs/maprdemo.mapr.io
  534. 2023-12-11 22:40:28,512 INFO Service [nfs_monitor]: setCurrentServiceUp: setting /services/nfs/master to UP, version: 0
  535. 2023-12-11 22:40:28,521 INFO Service [main-EventThread]: Process path: /services/nfs/master. Event state: SyncConnected. Event type: NodeDataChanged
  536. 2023-12-11 22:40:28,521 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40005 local:/172.17.0.2:34476 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:227 xid:33 sent:39 recv:48 queuedpkts:0 pendingresp:0 queuedevents:0
  537. 2023-12-11 22:40:28,521 INFO Service [nfs_monitor]: Marking service is UP for nfs on node /services/nfs/master
  538. 2023-12-11 22:40:30,299 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  539. 2023-12-11 22:40:30,299 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  540. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  541. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  542.  
  543. 2023-12-11 22:40:32,500 ERROR ServiceMonitorRun [nfs_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-nfsserver, status]can not determine if service: nfs is running. Retrying. Retrial #1. Total retries count is: 3
  544. 2023-12-11 22:40:32,501 ERROR ServiceMonitorRun [nfs_monitor]: * nfsserver is not running
  545.  
  546. 2023-12-11 22:40:32,501 INFO ServiceRun [nfs_monitor]: Command: [/opt/mapr/initscripts/mapr-nfsserver, start], Directory: /opt/mapr/initscripts
  547. 2023-12-11 22:40:36,359 INFO ServiceRun [nfs_monitor]:
  548. 2023-12-11 22:40:40,995 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  549. 2023-12-11 22:40:40,995 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  550. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  551. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  552.  
  553. 2023-12-11 22:40:44,147 ERROR ServiceMonitorRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status] firing
  554. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  555. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  556. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  557. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:1994) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  558. at java.lang.Thread.run(Thread.java:829) [?:?]
  559. Caused by: java.io.IOException: error=2, No such file or directory
  560. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  561. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  562. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  563. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  564. ... 3 more
  565. 2023-12-11 22:40:44,147 INFO ServiceRun [drill-bits_monitor]: Command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start], Directory: /opt/mapr/drill/drill-1.14.0/bin
  566. 2023-12-11 22:40:44,178 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  567. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  568. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  569. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  570. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  571. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  572. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  573. at java.lang.Thread.run(Thread.java:829) [?:?]
  574. Caused by: java.io.IOException: error=2, No such file or directory
  575. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  576. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  577. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  578. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  579. ... 5 more
  580. 2023-12-11 22:40:44,178 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  581. 2023-12-11 22:40:44,178 ERROR ServiceRun [drill-bits_monitor]:
  582. 2023-12-11 22:40:44,213 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  583. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  584. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  585. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  586. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  587. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  588. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  589. at java.lang.Thread.run(Thread.java:829) [?:?]
  590. Caused by: java.io.IOException: error=2, No such file or directory
  591. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  592. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  593. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  594. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  595. ... 5 more
  596. 2023-12-11 22:40:44,213 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  597. 2023-12-11 22:40:44,214 ERROR ServiceRun [drill-bits_monitor]:
  598. 2023-12-11 22:40:44,247 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  599. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  600. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  601. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  602. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  603. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  604. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  605. at java.lang.Thread.run(Thread.java:829) [?:?]
  606. Caused by: java.io.IOException: error=2, No such file or directory
  607. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  608. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  609. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  610. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  611. ... 5 more
  612. 2023-12-11 22:40:44,247 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  613. 2023-12-11 22:40:44,247 ERROR ServiceRun [drill-bits_monitor]:
  614. 2023-12-11 22:40:46,359 INFO WardenRequestBuilder [nfs_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  615. 2023-12-11 22:40:46,359 INFO WardenRequestBuilder [nfs_monitor]: []
  616. 2023-12-11 22:40:47,098 ERROR Service [main-EventThread]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_GATEWAY_DOWN, -terse, nagwsd, -service, gateway, -displayName, GatewayServiceDown, -baseService, 1, -serviceName, GatewayService]
  617. 2023-12-11 22:40:47,099 ERROR Service [main-EventThread]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  618.  
  619. 2023-12-11 22:40:47,102 WARN Service [main-EventThread]: Delayed alarm processing failed for alarm: com.mapr.warden.service.baseservice.Service$AlarmKeyEntity@1e073fb4. Process later
  620. 2023-12-11 22:40:47,251 INFO ServiceRun [s3server_monitor]: Command: [/opt/mapr/initscripts/mapr-s3server, start], Directory: /opt/mapr/initscripts
  621. 2023-12-11 22:40:47,252 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  622. 2023-12-11 22:40:49,623 INFO ServiceRun [gateway_monitor]: Command: [/opt/mapr/initscripts/mapr-gateway, start], Directory: /opt/mapr/initscripts
  623. 2023-12-11 22:40:50,995 INFO WardenRequestBuilder [cldb_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  624. 2023-12-11 22:40:50,995 INFO WardenRequestBuilder [cldb_monitor]: []
  625. 2023-12-11 22:40:50,996 INFO Service [cldb_monitor]: Could not find PID file for service: CLDB doing pgrep instead
  626. 2023-12-11 22:40:51,612 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  627. 2023-12-11 22:40:51,612 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  628. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  629.  
  630. 2023-12-11 22:40:51,612 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  631. 2023-12-11 22:40:52,194 INFO ServiceRun [apiserver_monitor]: Command: [/opt/mapr/apiserver/bin/mapr-apiserver.sh, start], Directory: /opt/mapr/apiserver/bin
  632. 2023-12-11 22:40:54,247 INFO WardenRequestBuilder [drill-bits_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  633. 2023-12-11 22:40:54,248 INFO WardenRequestBuilder [drill-bits_monitor]: []
  634. 2023-12-11 22:40:54,251 INFO Service [drill-bits_monitor]: setCurrentServiceUp: setting /services/drill-bits/maprdemo.mapr.io to UP, version: 0
  635. 2023-12-11 22:40:54,260 INFO Service [main-EventThread]: Process path: /services/drill-bits/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  636. 2023-12-11 22:40:54,260 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:278 xid:30 sent:39 recv:47 queuedpkts:0 pendingresp:0 queuedevents:0
  637. 2023-12-11 22:40:54,260 INFO Service [drill-bits_monitor]: Marking service is UP for drill-bits on node /services/drill-bits/maprdemo.mapr.io
  638. 2023-12-11 22:40:54,262 INFO Service [drill-bits_monitor]: setCurrentServiceUp: setting /services/drill-bits/master to UP, version: 0
  639. 2023-12-11 22:40:54,265 INFO Service [main-EventThread]: Process path: /services/drill-bits/master. Event state: SyncConnected. Event type: NodeDataChanged
  640. 2023-12-11 22:40:54,265 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:279 xid:33 sent:42 recv:51 queuedpkts:0 pendingresp:0 queuedevents:0
  641. 2023-12-11 22:40:54,265 INFO Service [drill-bits_monitor]: Marking service is UP for drill-bits on node /services/drill-bits/master
  642. 2023-12-11 22:40:54,313 ERROR ServiceMonitorRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status] firing
  643. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  644. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  645. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  646. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:1994) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  647. at java.lang.Thread.run(Thread.java:829) [?:?]
  648. Caused by: java.io.IOException: error=2, No such file or directory
  649. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  650. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  651. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  652. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  653. ... 3 more
  654. 2023-12-11 22:40:54,313 ERROR ServiceMonitorRun [drill-bits_monitor]: Monitor command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status]can not determine if service: drill-bits is running. Retrying. Retrial #1. Total retries count is: 3
  655. 2023-12-11 22:40:54,313 ERROR ServiceMonitorRun [drill-bits_monitor]:
  656. 2023-12-11 22:40:54,313 INFO ServiceRun [drill-bits_monitor]: Command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start], Directory: /opt/mapr/drill/drill-1.14.0/bin
  657. 2023-12-11 22:40:54,338 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  658. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  659. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  660. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  661. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  662. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  663. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  664. at java.lang.Thread.run(Thread.java:829) [?:?]
  665. Caused by: java.io.IOException: error=2, No such file or directory
  666. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  667. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  668. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  669. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  670. ... 5 more
  671. 2023-12-11 22:40:54,338 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  672. 2023-12-11 22:40:54,338 ERROR ServiceRun [drill-bits_monitor]:
  673. 2023-12-11 22:40:54,380 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  674. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  675. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  676. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  677. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  678. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  679. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  680. at java.lang.Thread.run(Thread.java:829) [?:?]
  681. Caused by: java.io.IOException: error=2, No such file or directory
  682. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  683. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  684. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  685. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  686. ... 5 more
  687. 2023-12-11 22:40:54,380 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  688. 2023-12-11 22:40:54,380 ERROR ServiceRun [drill-bits_monitor]:
  689. 2023-12-11 22:40:54,419 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  690. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  691. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  692. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  693. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  694. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  695. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  696. at java.lang.Thread.run(Thread.java:829) [?:?]
  697. Caused by: java.io.IOException: error=2, No such file or directory
  698. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  699. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  700. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  701. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  702. ... 5 more
  703. 2023-12-11 22:40:54,419 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  704. 2023-12-11 22:40:54,419 ERROR ServiceRun [drill-bits_monitor]:
  705. 2023-12-11 22:40:55,698 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  706. 2023-12-11 22:40:55,699 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  707. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  708. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  709.  
  710. 2023-12-11 22:40:56,518 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  711. 2023-12-11 22:40:56,519 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  712. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  713.  
  714. 2023-12-11 22:40:56,519 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  715. 2023-12-11 22:41:01,443 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  716. 2023-12-11 22:41:01,443 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  717. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  718.  
  719. 2023-12-11 22:41:01,443 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  720. 2023-12-11 22:41:02,038 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  721. 2023-12-11 22:41:02,038 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  722. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  723. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  724.  
  725. 2023-12-11 22:41:04,420 INFO WardenRequestBuilder [drill-bits_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  726. 2023-12-11 22:41:04,420 INFO WardenRequestBuilder [drill-bits_monitor]: []
  727. 2023-12-11 22:41:04,435 ERROR ServiceMonitorRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status] firing
  728. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  729. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  730. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  731. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:1994) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  732. at java.lang.Thread.run(Thread.java:829) [?:?]
  733. Caused by: java.io.IOException: error=2, No such file or directory
  734. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  735. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  736. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  737. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  738. ... 3 more
  739. 2023-12-11 22:41:04,435 ERROR ServiceMonitorRun [drill-bits_monitor]: Monitor command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status]can not determine if service: drill-bits is running. Retrying. Retrial #2. Total retries count is: 3
  740. 2023-12-11 22:41:04,435 ERROR ServiceMonitorRun [drill-bits_monitor]:
  741. 2023-12-11 22:41:04,435 INFO ServiceRun [drill-bits_monitor]: Command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start], Directory: /opt/mapr/drill/drill-1.14.0/bin
  742. 2023-12-11 22:41:04,461 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  743. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  744. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  745. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  746. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  747. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  748. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  749. at java.lang.Thread.run(Thread.java:829) [?:?]
  750. Caused by: java.io.IOException: error=2, No such file or directory
  751. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  752. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  753. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  754. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  755. ... 5 more
  756. 2023-12-11 22:41:04,461 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  757. 2023-12-11 22:41:04,461 ERROR ServiceRun [drill-bits_monitor]:
  758. 2023-12-11 22:41:04,488 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  759. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  760. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  761. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  762. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  763. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  764. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  765. at java.lang.Thread.run(Thread.java:829) [?:?]
  766. Caused by: java.io.IOException: error=2, No such file or directory
  767. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  768. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  769. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  770. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  771. ... 5 more
  772. 2023-12-11 22:41:04,488 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  773. 2023-12-11 22:41:04,488 ERROR ServiceRun [drill-bits_monitor]:
  774. 2023-12-11 22:41:04,502 ERROR ServiceRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start] firing
  775. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  776. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  777. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  778. at com.mapr.warden.service.baseservice.Service$ServiceRun.run(Service.java:2287) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  779. at java.lang.Thread.run(Thread.java:829) ~[?:?]
  780. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:2141) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  781. at java.lang.Thread.run(Thread.java:829) [?:?]
  782. Caused by: java.io.IOException: error=2, No such file or directory
  783. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  784. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  785. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  786. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  787. ... 5 more
  788. 2023-12-11 22:41:04,502 ERROR ServiceRun [drill-bits_monitor]: Error while running command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, start]
  789. 2023-12-11 22:41:04,503 ERROR ServiceRun [drill-bits_monitor]:
  790. 2023-12-11 22:41:06,954 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  791. 2023-12-11 22:41:06,954 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  792. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  793.  
  794. 2023-12-11 22:41:08,942 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  795. 2023-12-11 22:41:08,943 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  796. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  797. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  798.  
  799. 2023-12-11 22:41:13,812 ERROR Service [cldb_monitor]: reniceProcessExec failed for child 20420 of parent 16006
  800. 2023-12-11 22:41:13,812 ERROR ServiceMonitorRun [cldb_monitor]: Could not renice process: 16006 to -10
  801. 2023-12-11 22:41:14,509 INFO WardenRequestBuilder [drill-bits_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  802. 2023-12-11 22:41:14,509 INFO WardenRequestBuilder [drill-bits_monitor]: []
  803. 2023-12-11 22:41:14,572 ERROR ServiceMonitorRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status] firing
  804. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  805. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  806. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  807. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:1994) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  808. at java.lang.Thread.run(Thread.java:829) [?:?]
  809. Caused by: java.io.IOException: error=2, No such file or directory
  810. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  811. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  812. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  813. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  814. ... 3 more
  815. 2023-12-11 22:41:14,677 ERROR ServiceMonitorRun [drill-bits_monitor]: IOException during command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status] firing
  816. java.io.IOException: Cannot run program "/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh" (in directory "/opt/mapr/drill/drill-1.14.0/bin"): error=2, No such file or directory
  817. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
  818. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
  819. at com.mapr.warden.service.baseservice.Service$ServiceMonitorRun.run(Service.java:1994) ~[warden-7.0.0.0-mapr.jar:7.0.0.0-mapr]
  820. at java.lang.Thread.run(Thread.java:829) [?:?]
  821. Caused by: java.io.IOException: error=2, No such file or directory
  822. at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
  823. at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
  824. at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
  825. at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
  826. ... 3 more
  827. 2023-12-11 22:41:14,678 ERROR ServiceMonitorRun [drill-bits_monitor]: Monitor command: [/opt/mapr/drill/drill-1.14.0/bin/drillbit.sh, status]cannot determine if service: drill-bits is running. Number of retrials exceeded. Closing Zookeeper
  828. 2023-12-11 22:41:14,678 INFO Service [drill-bits_monitor]: 77 about to close zk for service: drill-bits
  829. 2023-12-11 22:41:14,684 INFO Service [main-EventThread]: Process path: /services/drill-bits/master. Event state: SyncConnected. Event type: NodeDeleted
  830. 2023-12-11 22:41:14,684 INFO Service [main-EventThread]: ZK Connect state:State:CLOSED sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:305 xid:36 sent:47 recv:58 queuedpkts:0 pendingresp:0 queuedevents:1
  831. 2023-12-11 22:41:14,684 INFO Service [main-EventThread]: ZK is closed for service: drill-bits
  832. 2023-12-11 22:41:14,684 INFO Service [main-EventThread]: Process path: /services/drill-bits/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDeleted
  833. 2023-12-11 22:41:14,684 INFO Service [main-EventThread]: ZK Connect state:State:CLOSED sessionid:0x1000010ecd40001 local:/172.17.0.2:34440 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:305 xid:36 sent:47 recv:58 queuedpkts:0 pendingresp:0 queuedevents:0
  834. 2023-12-11 22:41:14,684 INFO Service [main-EventThread]: ZK is closed for service: drill-bits
  835. 2023-12-11 22:41:14,787 INFO Service [main-EventThread]: Process path: null. Event state: Closed. Event type: None
  836. 2023-12-11 22:41:14,787 INFO Service [main-EventThread]: ZK is closed for service: drill-bits
  837. 2023-12-11 22:41:14,787 INFO DependentService [main-EventThread]: DependentWatcher for service: drill-bits Process path: null. Event state: Closed. Event type: None
  838. 2023-12-11 22:41:14,787 INFO DependentService [main-EventThread]: ZK is closed for service: drill-bits
  839. 2023-12-11 22:41:14,792 INFO WardenRequestBuilder [drill-bits_monitor]: [e_SERV_FAIL, hostName, ma_host, ma_process]
  840. 2023-12-11 22:41:14,792 INFO WardenRequestBuilder [drill-bits_monitor]: []
  841. 2023-12-11 22:41:14,793 INFO Service [drill-bits_monitor]: Alarm raising command: [/opt/mapr/bin/maprcli, alarm, raise, -alarm, NODE_ALARM_SERVICE_DRILL-BITS_DOWN, -entity, maprdemo.mapr.io, -description, Can not determine if service: drill-bits is running. Check logs at: /opt/mapr/drill/drill-1.14.0/logs/]
  842. 2023-12-11 22:41:16,895 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  843. 2023-12-11 22:41:16,895 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  844. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  845. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  846.  
  847. 2023-12-11 22:41:16,954 INFO WardenRequestBuilder [s3server_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  848. 2023-12-11 22:41:16,955 INFO WardenRequestBuilder [s3server_monitor]: []
  849. 2023-12-11 22:41:16,955 INFO Service [s3server_monitor]: Could not find PID file for service: S3SERVER doing pgrep instead
  850. 2023-12-11 22:41:18,089 INFO Service [s3server_monitor]: setCurrentServiceUp: setting /services/s3server/maprdemo.mapr.io to UP, version: 0
  851. 2023-12-11 22:41:18,098 INFO Service [main-EventThread]: Process path: /services/s3server/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  852. 2023-12-11 22:41:18,099 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:307 xid:30 sent:41 recv:49 queuedpkts:0 pendingresp:0 queuedevents:0
  853. 2023-12-11 22:41:18,099 INFO Service [s3server_monitor]: Marking service is UP for s3server on node /services/s3server/maprdemo.mapr.io
  854. 2023-12-11 22:41:18,100 INFO Service [s3server_monitor]: setCurrentServiceUp: setting /services/s3server/master to UP, version: 0
  855. 2023-12-11 22:41:18,103 INFO Service [main-EventThread]: Process path: /services/s3server/master. Event state: SyncConnected. Event type: NodeDataChanged
  856. 2023-12-11 22:41:18,103 INFO Service [s3server_monitor]: Marking service is UP for s3server on node /services/s3server/master
  857. 2023-12-11 22:41:18,103 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:308 xid:33 sent:44 recv:53 queuedpkts:0 pendingresp:0 queuedevents:0
  858. 2023-12-11 22:41:18,239 ERROR Service [cldb_monitor]: reniceProcessExec failed for child 21785 of parent 18052
  859. 2023-12-11 22:41:18,240 ERROR ServiceMonitorRun [cldb_monitor]: Could not renice process: 18052 to -10
  860. 2023-12-11 22:41:20,415 ERROR Service [s3server_monitor]: reniceProcessExec failed for child 23121 of parent 23052
  861. 2023-12-11 22:41:20,415 ERROR ServiceMonitorRun [s3server_monitor]: Could not renice process: 23052 to -8
  862. 2023-12-11 22:41:20,472 ERROR Service [main-EventThread]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_S3SERVER_DOWN, -terse, nas3sd, -service, s3server, -displayName, S3ServerServiceDown, -baseService, 1, -serviceName, S3ServerService]
  863. 2023-12-11 22:41:20,477 ERROR Service [main-EventThread]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  864.  
  865. 2023-12-11 22:41:20,477 WARN Service [main-EventThread]: Delayed alarm processing failed for alarm: com.mapr.warden.service.baseservice.Service$AlarmKeyEntity@3e3ee0ab. Process later
  866. 2023-12-11 22:41:25,429 ERROR ServiceMonitorRun [s3server_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-s3server, status]can not determine if service: s3server is running. Retrying. Retrial #1. Total retries count is: 3
  867. 2023-12-11 22:41:25,429 ERROR ServiceMonitorRun [s3server_monitor]: /opt/mapr/server/s3mon not running
  868.  
  869. 2023-12-11 22:41:25,429 INFO ServiceRun [s3server_monitor]: Command: [/opt/mapr/initscripts/mapr-s3server, start], Directory: /opt/mapr/initscripts
  870. 2023-12-11 22:41:25,430 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  871. 2023-12-11 22:41:26,896 INFO WardenRequestBuilder [gateway_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  872. 2023-12-11 22:41:26,896 INFO WardenRequestBuilder [gateway_monitor]: []
  873. 2023-12-11 22:41:26,897 INFO Service [gateway_monitor]: setCurrentServiceUp: setting /services/gateway/maprdemo.mapr.io to UP, version: 0
  874. 2023-12-11 22:41:26,901 INFO Service [main-EventThread]: Process path: /services/gateway/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  875. 2023-12-11 22:41:26,901 INFO Service [gateway_monitor]: Marking service is UP for gateway on node /services/gateway/maprdemo.mapr.io
  876. 2023-12-11 22:41:26,901 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:334 xid:30 sent:42 recv:50 queuedpkts:1 pendingresp:0 queuedevents:0
  877. 2023-12-11 22:41:26,902 INFO Service [gateway_monitor]: setCurrentServiceUp: setting /services/gateway/master to UP, version: 0
  878. 2023-12-11 22:41:26,904 INFO Service [main-EventThread]: Process path: /services/gateway/master. Event state: SyncConnected. Event type: NodeDataChanged
  879. 2023-12-11 22:41:26,904 INFO Service [gateway_monitor]: Marking service is UP for gateway on node /services/gateway/master
  880. 2023-12-11 22:41:26,904 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:335 xid:33 sent:45 recv:54 queuedpkts:0 pendingresp:0 queuedevents:0
  881. 2023-12-11 22:41:27,223 INFO ServiceRun [data-access-gateway_monitor]: Command: [/opt/mapr/data-access-gateway/bin/mapr-data-access-gateway.sh, start], Directory: /opt/mapr/data-access-gateway/bin
  882. 2023-12-11 22:41:31,679 ERROR ServiceMonitorRun [gateway_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-gateway, status]can not determine if service: gateway is running. Retrying. Retrial #1. Total retries count is: 3
  883. 2023-12-11 22:41:31,679 ERROR ServiceMonitorRun [gateway_monitor]: GATEWAY not running.
  884.  
  885. 2023-12-11 22:41:31,679 INFO ServiceRun [gateway_monitor]: Command: [/opt/mapr/initscripts/mapr-gateway, start], Directory: /opt/mapr/initscripts
  886. 2023-12-11 22:41:31,892 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  887. 2023-12-11 22:41:31,892 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  888. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  889.  
  890. 2023-12-11 22:41:31,895 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  891. 2023-12-11 22:41:36,151 ERROR ServiceMonitorRun [cldb_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-cldb, status]can not determine if service: cldb is running. Retrying. Retrial #2. Total retries count is: 3
  892. 2023-12-11 22:41:36,151 ERROR ServiceMonitorRun [cldb_monitor]: CLDB not running.
  893.  
  894. 2023-12-11 22:41:36,315 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  895. 2023-12-11 22:41:36,315 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  896. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  897.  
  898. 2023-12-11 22:41:36,315 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  899. 2023-12-11 22:41:37,169 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  900. 2023-12-11 22:41:37,169 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  901. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  902. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  903.  
  904. 2023-12-11 22:41:39,886 ERROR ServiceRun [apiserver_monitor]: Error while running command: [/opt/mapr/apiserver/bin/mapr-apiserver.sh, start]
  905. 2023-12-11 22:41:39,886 ERROR ServiceRun [apiserver_monitor]: apiserver (23152) start failed (143). See /opt/mapr/apiserver/logs/apiserver.log for details
  906.  
  907. 2023-12-11 22:41:40,193 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  908. 2023-12-11 22:41:40,193 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  909. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  910.  
  911. 2023-12-11 22:41:40,194 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  912. 2023-12-11 22:41:42,046 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  913. 2023-12-11 22:41:42,046 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  914. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  915. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  916.  
  917. 2023-12-11 22:41:44,200 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  918. 2023-12-11 22:41:44,200 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  919. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  920.  
  921. 2023-12-11 22:41:44,892 INFO Service [Thread-8-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  922. 2023-12-11 22:41:44,892 INFO Service [Thread-8]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40013 local:/172.17.0.2:57160 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:3 queuedpkts:0 pendingresp:0 queuedevents:1
  923. 2023-12-11 22:41:44,892 INFO Service [Thread-8]: Service reinitThread: zk expired: false reconnected to zk, lastAction: null
  924. 2023-12-11 22:41:44,892 INFO Service [Thread-8-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40013 local:/172.17.0.2:57160 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:3 queuedpkts:1 pendingresp:0 queuedevents:1
  925. 2023-12-11 22:41:44,892 INFO Service [Thread-8-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  926. 2023-12-11 22:41:44,904 INFO Service [Thread-8]: Node: /nodes/maprdemo.mapr.io/services/drill-bits does not exist yet
  927. 2023-12-11 22:41:44,911 INFO Service [Thread-8]: addServiceToZK: zknode /services/drill-bits already exists
  928. 2023-12-11 22:41:44,914 INFO Service [Thread-8]: addServiceToZK: after remove master, children node size 0
  929. 2023-12-11 22:41:48,041 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  930. 2023-12-11 22:41:48,041 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  931. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  932. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  933.  
  934. 2023-12-11 22:41:53,427 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  935. 2023-12-11 22:41:53,427 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  936. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  937. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  938.  
  939. 2023-12-11 22:41:54,200 INFO WardenRequestBuilder [s3server_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  940. 2023-12-11 22:41:54,201 INFO WardenRequestBuilder [s3server_monitor]: []
  941. 2023-12-11 22:41:54,201 INFO Service [s3server_monitor]: Could not find PID file for service: S3SERVER doing pgrep instead
  942. 2023-12-11 22:41:56,782 ERROR ServiceMonitorRun [s3server_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-s3server, status]can not determine if service: s3server is running. Retrying. Retrial #2. Total retries count is: 3
  943. 2023-12-11 22:41:56,782 ERROR ServiceMonitorRun [s3server_monitor]: /opt/mapr/server/s3mon not running
  944.  
  945. 2023-12-11 22:41:56,782 INFO ServiceRun [s3server_monitor]: Command: [/opt/mapr/initscripts/mapr-s3server, start], Directory: /opt/mapr/initscripts
  946. 2023-12-11 22:41:56,782 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  947. 2023-12-11 22:42:00,839 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  948. 2023-12-11 22:42:00,839 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  949. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  950.  
  951. 2023-12-11 22:42:00,840 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  952. 2023-12-11 22:42:01,180 INFO ServiceRun [cldb_monitor]: Command: [/opt/mapr/initscripts/mapr-cldb, start], Directory: /opt/mapr/initscripts
  953. 2023-12-11 22:42:03,427 INFO WardenRequestBuilder [gateway_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  954. 2023-12-11 22:42:03,427 INFO WardenRequestBuilder [gateway_monitor]: []
  955. 2023-12-11 22:42:05,188 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  956. 2023-12-11 22:42:05,188 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  957. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  958.  
  959. 2023-12-11 22:42:05,188 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  960. 2023-12-11 22:42:06,284 ERROR ServiceMonitorRun [gateway_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-gateway, status]can not determine if service: gateway is running. Retrying. Retrial #2. Total retries count is: 3
  961. 2023-12-11 22:42:06,284 ERROR ServiceMonitorRun [gateway_monitor]: GATEWAY not running.
  962.  
  963. 2023-12-11 22:42:06,284 INFO ServiceRun [gateway_monitor]: Command: [/opt/mapr/initscripts/mapr-gateway, start], Directory: /opt/mapr/initscripts
  964. 2023-12-11 22:42:09,676 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  965. 2023-12-11 22:42:09,676 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  966. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  967.  
  968. 2023-12-11 22:42:09,676 INFO S3ServerService [s3server_monitor]: service.sharedmemory.size 2147483648
  969. 2023-12-11 22:42:09,848 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  970. 2023-12-11 22:42:09,848 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  971. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  972. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  973.  
  974. 2023-12-11 22:42:11,781 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  975. 2023-12-11 22:42:11,781 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  976. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  977. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  978.  
  979. 2023-12-11 22:42:12,374 INFO ServiceRun [apiserver_monitor]: apiserver (29308) started with log /opt/mapr/apiserver/logs/apiserver.log
  980. ln: failed to create symbolic link '/opt/mapr/pid/apiserver.pid': Permission denied
  981.  
  982. 2023-12-11 22:42:13,940 ERROR ServiceRun [s3server_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-s3server, start]
  983. 2023-12-11 22:42:13,940 ERROR ServiceRun [s3server_monitor]: cat: /opt/mapr/pid/s3server.pid: No such file or directory
  984. Error: /opt/mapr/server/s3mon can not be started. See /opt/mapr/logs/ for details
  985.  
  986. 2023-12-11 22:42:14,447 ERROR Service [main-EventThread]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_APISERVER_DOWN, -terse, naaapisrvd, -service, apiserver, -displayName, ApiServerDown, -baseService, 1, -serviceName, ApiServer]
  987. 2023-12-11 22:42:14,447 ERROR Service [main-EventThread]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  988.  
  989. 2023-12-11 22:42:14,447 WARN Service [main-EventThread]: Delayed alarm processing failed for alarm: com.mapr.warden.service.baseservice.Service$AlarmKeyEntity@e1ddfefb. Process later
  990. 2023-12-11 22:42:17,224 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  991. 2023-12-11 22:42:17,224 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  992. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  993. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  994.  
  995. 2023-12-11 22:42:18,316 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  996. 2023-12-11 22:42:18,316 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  997. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  998. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  999.  
  1000. 2023-12-11 22:42:22,374 INFO WardenRequestBuilder [apiserver_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  1001. 2023-12-11 22:42:22,375 INFO WardenRequestBuilder [apiserver_monitor]: []
  1002. 2023-12-11 22:42:22,378 INFO Service [apiserver_monitor]: setCurrentServiceUp: setting /services/apiserver/maprdemo.mapr.io to UP, version: 0
  1003. 2023-12-11 22:42:22,382 INFO Service [apiserver_monitor]: Marking service is UP for apiserver on node /services/apiserver/maprdemo.mapr.io
  1004. 2023-12-11 22:42:22,384 INFO Service [main-EventThread]: Process path: /services/apiserver/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  1005. 2023-12-11 22:42:22,384 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40009 local:/172.17.0.2:34524 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:343 xid:31 sent:49 recv:56 queuedpkts:0 pendingresp:1 queuedevents:0
  1006. 2023-12-11 22:42:22,385 INFO Service [apiserver_monitor]: setCurrentServiceUp: setting /services/apiserver/master to UP, version: 0
  1007. 2023-12-11 22:42:22,390 INFO Service [main-EventThread]: Process path: /services/apiserver/master. Event state: SyncConnected. Event type: NodeDataChanged
  1008. 2023-12-11 22:42:22,390 INFO Service [apiserver_monitor]: Marking service is UP for apiserver on node /services/apiserver/master
  1009. 2023-12-11 22:42:22,390 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40009 local:/172.17.0.2:34524 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:344 xid:33 sent:51 recv:60 queuedpkts:0 pendingresp:0 queuedevents:0
  1010. 2023-12-11 22:42:22,981 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  1011. 2023-12-11 22:42:22,981 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  1012. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  1013. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  1014.  
  1015. 2023-12-11 22:42:23,940 INFO WardenRequestBuilder [s3server_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  1016. 2023-12-11 22:42:23,940 INFO WardenRequestBuilder [s3server_monitor]: []
  1017. 2023-12-11 22:42:23,940 INFO Service [s3server_monitor]: Could not find PID file for service: S3SERVER doing pgrep instead
  1018. 2023-12-11 22:42:27,769 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  1019. 2023-12-11 22:42:27,769 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  1020. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  1021. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  1022.  
  1023. 2023-12-11 22:42:29,500 ERROR ServiceRun [gateway_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-gateway, start]
  1024. 2023-12-11 22:42:29,500 ERROR ServiceRun [gateway_monitor]: /opt/mapr/initscripts/mapr-gateway: line 299: /opt/mapr/pid/gateway.pid: Permission denied
  1025. cat: /opt/mapr/pid/gateway.pid: No such file or directory
  1026. Error: gateway can not be started. See /opt/mapr/logs/gatewayinit.log for details
  1027.  
  1028. 2023-12-11 22:42:30,875 ERROR ServiceMonitorRun [s3server_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-s3server, status]cannot determine if service: s3server is running. Number of retrials exceeded. Closing Zookeeper
  1029. 2023-12-11 22:42:30,875 INFO Service [s3server_monitor]: 75 about to close zk for service: s3server
  1030. 2023-12-11 22:42:30,883 INFO Service [main-EventThread]: Process path: /services/s3server/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDeleted
  1031. 2023-12-11 22:42:30,883 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:345 xid:36 sent:54 recv:65 queuedpkts:0 pendingresp:0 queuedevents:1
  1032. 2023-12-11 22:42:30,983 ERROR ZKUtilsLocking [main-EventThread]: SessionExpiredException while trying to checkZKNodeForExistence of: /services/cldb/master
  1033. 2023-12-11 22:42:30,983 ERROR DependentService [main-EventThread]: ZK Session was either or closed or expired for service: s3server
  1034. 2023-12-11 22:42:30,983 ERROR ZKUtilsLocking [main-EventThread]: SessionExpiredException while trying to checkZKNodeForExistence of: /services/s3server/maprdemo.mapr.io
  1035. 2023-12-11 22:42:30,983 ERROR Service [main-EventThread]: ZK Session was either or closed or expired for service: s3server
  1036. 2023-12-11 22:42:30,983 INFO Service [main-EventThread]: Process path: /services/s3server/master. Event state: SyncConnected. Event type: NodeDeleted
  1037. 2023-12-11 22:42:30,984 INFO Service [main-EventThread]: ZK Connect state:State:CLOSED sessionid:0x1000010ecd40002 local:/172.17.0.2:34442 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:345 xid:36 sent:54 recv:65 queuedpkts:0 pendingresp:0 queuedevents:1
  1038. 2023-12-11 22:42:30,984 INFO Service [main-EventThread]: ZK is closed for service: s3server
  1039. 2023-12-11 22:42:30,984 INFO DependentService [main-EventThread]: DependentWatcher for service: s3server Process path: null. Event state: Closed. Event type: None
  1040. 2023-12-11 22:42:30,984 INFO DependentService [main-EventThread]: ZK is closed for service: s3server
  1041. 2023-12-11 22:42:30,984 INFO Service [main-EventThread]: Process path: null. Event state: Closed. Event type: None
  1042. 2023-12-11 22:42:30,984 INFO Service [main-EventThread]: ZK is closed for service: s3server
  1043. 2023-12-11 22:42:30,991 INFO WardenRequestBuilder [s3server_monitor]: [e_SERV_FAIL, hostName, ma_host, ma_process]
  1044. 2023-12-11 22:42:30,991 INFO WardenRequestBuilder [s3server_monitor]: []
  1045. 2023-12-11 22:42:30,991 INFO Service [s3server_monitor]: Alarm raising command: [/opt/mapr/bin/maprcli, alarm, raise, -alarm, NODE_ALARM_SERVICE_S3SERVER_DOWN, -entity, maprdemo.mapr.io, -description, Can not determine if service: s3server is running. Check logs at: /opt/mapr/logs/s3server.log]
  1046. 2023-12-11 22:42:35,898 INFO ServiceRun [data-access-gateway_monitor]: JMX disabled by user request
  1047. data-access-gateway (31465) started with log /opt/mapr/data-access-gateway/logs/data-access-gateway.log
  1048. ln: failed to create symbolic link '/opt/mapr/pid/data-access-gateway.pid': Permission denied
  1049.  
  1050. 2023-12-11 22:42:37,839 ERROR ServiceRun [cldb_monitor]: Error while running command: [/opt/mapr/initscripts/mapr-cldb, start]
  1051. 2023-12-11 22:42:37,839 ERROR ServiceRun [cldb_monitor]: /opt/mapr/initscripts/mapr-cldb: line 351: /opt/mapr/pid/cldb.pid: Permission denied
  1052. cat: /opt/mapr/pid/cldb.pid: No such file or directory
  1053. Error: cldb can not be started. See /opt/mapr/logs/cldb.log for details
  1054.  
  1055. 2023-12-11 22:42:39,500 INFO WardenRequestBuilder [gateway_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  1056. 2023-12-11 22:42:39,501 INFO WardenRequestBuilder [gateway_monitor]: []
  1057. 2023-12-11 22:42:45,256 ERROR Service [main-EventThread]: Error while running command: [maprcli, alarm, add, -alarm, NODE_ALARM_SERVICE_DRILL-BITS_DOWN, -terse, nadrill, -service, drill-bits, -displayName, DRILLDOWNALARM, -serviceName, Drillbit]
  1058. 2023-12-11 22:42:45,257 ERROR Service [main-EventThread]: ERROR (17) - Alarm name already exist. Cannot add duplicate alarm name.
  1059.  
  1060. 2023-12-11 22:42:45,257 WARN Service [main-EventThread]: Delayed alarm processing failed for alarm: com.mapr.warden.service.baseservice.Service$AlarmKeyEntity@b7ae3bb6. Process later
  1061. 2023-12-11 22:42:45,898 INFO WardenRequestBuilder [data-access-gateway_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  1062. 2023-12-11 22:42:45,898 INFO WardenRequestBuilder [data-access-gateway_monitor]: []
  1063. 2023-12-11 22:42:45,901 INFO Service [data-access-gateway_monitor]: setCurrentServiceUp: setting /services/data-access-gateway/maprdemo.mapr.io to UP, version: 0
  1064. 2023-12-11 22:42:45,907 INFO Service [main-EventThread]: Process path: /services/data-access-gateway/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDataChanged
  1065. 2023-12-11 22:42:45,908 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40008 local:/172.17.0.2:34514 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:373 xid:30 sent:50 recv:58 queuedpkts:0 pendingresp:0 queuedevents:0
  1066. 2023-12-11 22:42:45,908 INFO Service [data-access-gateway_monitor]: Marking service is UP for data-access-gateway on node /services/data-access-gateway/maprdemo.mapr.io
  1067. 2023-12-11 22:42:45,908 INFO Service [data-access-gateway_monitor]: setCurrentServiceUp: setting /services/data-access-gateway/master to UP, version: 0
  1068. 2023-12-11 22:42:45,916 INFO Service [main-EventThread]: Process path: /services/data-access-gateway/master. Event state: SyncConnected. Event type: NodeDataChanged
  1069. 2023-12-11 22:42:45,916 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40008 local:/172.17.0.2:34514 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:374 xid:33 sent:53 recv:62 queuedpkts:0 pendingresp:0 queuedevents:0
  1070. 2023-12-11 22:42:45,916 INFO Service [data-access-gateway_monitor]: Marking service is UP for data-access-gateway on node /services/data-access-gateway/master
  1071. 2023-12-11 22:42:46,500 ERROR ServiceMonitorRun [gateway_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-gateway, status]cannot determine if service: gateway is running. Number of retrials exceeded. Closing Zookeeper
  1072. 2023-12-11 22:42:46,500 INFO Service [gateway_monitor]: 73 about to close zk for service: gateway
  1073. 2023-12-11 22:42:46,507 INFO Service [main-EventThread]: Process path: /services/gateway/master. Event state: SyncConnected. Event type: NodeDeleted
  1074. 2023-12-11 22:42:46,507 INFO Service [main-EventThread]: ZK Connect state:State:CLOSED sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:375 xid:36 sent:55 recv:66 queuedpkts:0 pendingresp:0 queuedevents:1
  1075. 2023-12-11 22:42:46,507 INFO Service [main-EventThread]: ZK is closed for service: gateway
  1076. 2023-12-11 22:42:46,507 INFO Service [main-EventThread]: Process path: /services/gateway/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDeleted
  1077. 2023-12-11 22:42:46,507 INFO Service [main-EventThread]: ZK Connect state:State:CLOSED sessionid:0x1000010ecd40006 local:/172.17.0.2:34486 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:375 xid:36 sent:55 recv:66 queuedpkts:0 pendingresp:0 queuedevents:0
  1078. 2023-12-11 22:42:46,507 INFO Service [main-EventThread]: ZK is closed for service: gateway
  1079. 2023-12-11 22:42:46,607 INFO Service [main-EventThread]: Process path: null. Event state: Closed. Event type: None
  1080. 2023-12-11 22:42:46,607 INFO Service [main-EventThread]: ZK is closed for service: gateway
  1081. 2023-12-11 22:42:46,607 INFO DependentService [main-EventThread]: DependentWatcher for service: gateway Process path: null. Event state: Closed. Event type: None
  1082. 2023-12-11 22:42:46,607 INFO DependentService [main-EventThread]: ZK is closed for service: gateway
  1083. 2023-12-11 22:42:46,616 INFO WardenRequestBuilder [gateway_monitor]: [e_SERV_FAIL, hostName, ma_host, ma_process]
  1084. 2023-12-11 22:42:46,616 INFO WardenRequestBuilder [gateway_monitor]: []
  1085. 2023-12-11 22:42:46,617 INFO Service [gateway_monitor]: Alarm raising command: [/opt/mapr/bin/maprcli, alarm, raise, -alarm, NODE_ALARM_SERVICE_GATEWAY_DOWN, -entity, maprdemo.mapr.io, -description, Can not determine if service: gateway is running. Check logs at: /opt/mapr/logs/gateway.log]
  1086. 2023-12-11 22:42:47,839 INFO WardenRequestBuilder [cldb_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
  1087. 2023-12-11 22:42:47,839 INFO WardenRequestBuilder [cldb_monitor]: []
  1088. 2023-12-11 22:42:47,839 INFO Service [cldb_monitor]: Could not find PID file for service: CLDB doing pgrep instead
  1089. 2023-12-11 22:43:01,032 INFO Service [Thread-9-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  1090. 2023-12-11 22:43:01,033 INFO Service [Thread-9]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40019 local:/172.17.0.2:52278 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  1091. 2023-12-11 22:43:01,033 INFO Service [Thread-9]: Service reinitThread: zk expired: false reconnected to zk, lastAction: null
  1092. 2023-12-11 22:43:01,037 INFO Service [Thread-9-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40019 local:/172.17.0.2:52278 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:1 pendingresp:0 queuedevents:0
  1093. 2023-12-11 22:43:01,038 INFO Service [Thread-9-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  1094. 2023-12-11 22:43:01,045 INFO Service [Thread-9]: Node: /nodes/maprdemo.mapr.io/services/s3server does not exist yet
  1095. 2023-12-11 22:43:01,053 INFO Service [Thread-9]: addServiceToZK: zknode /services/s3server already exists
  1096. 2023-12-11 22:43:01,057 INFO Service [Thread-9]: addServiceToZK: after remove master, children node size 0
  1097. 2023-12-11 22:43:08,366 ERROR Service [cldb_monitor]: reniceProcessExec failed for child 33053 of parent 32574
  1098. 2023-12-11 22:43:08,366 ERROR ServiceMonitorRun [cldb_monitor]: Could not renice process: 32574 to -10
  1099. 2023-12-11 22:43:12,233 ERROR Service [cldb_monitor]: reniceProcessExec failed for child 36826 of parent 33458
  1100. 2023-12-11 22:43:12,234 ERROR ServiceMonitorRun [cldb_monitor]: Could not renice process: 33458 to -10
  1101. 2023-12-11 22:43:16,650 INFO Service [Thread-13-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  1102. 2023-12-11 22:43:16,651 INFO Service [Thread-13]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd4001b local:/172.17.0.2:49438 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
  1103. 2023-12-11 22:43:16,651 INFO Service [Thread-13]: Service reinitThread: zk expired: false reconnected to zk, lastAction: null
  1104. 2023-12-11 22:43:16,651 INFO Service [Thread-13-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd4001b local:/172.17.0.2:49438 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:2 sent:1 recv:1 queuedpkts:1 pendingresp:0 queuedevents:0
  1105. 2023-12-11 22:43:16,653 INFO Service [Thread-13-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  1106. 2023-12-11 22:43:16,655 INFO Service [Thread-13]: Node: /nodes/maprdemo.mapr.io/services/gateway does not exist yet
  1107. 2023-12-11 22:43:16,662 INFO Service [Thread-13]: addServiceToZK: zknode /services/gateway already exists
  1108. 2023-12-11 22:43:16,664 INFO Service [Thread-13]: addServiceToZK: after remove master, children node size 0
  1109. 2023-12-11 22:43:28,375 ERROR ServiceMonitorRun [cldb_monitor]: Monitor command: [/opt/mapr/initscripts/mapr-cldb, status]cannot determine if service: cldb is running. Number of retrials exceeded. Closing Zookeeper
  1110. 2023-12-11 22:43:28,375 INFO Service [cldb_monitor]: 66 about to close zk for service: cldb
  1111. 2023-12-11 22:43:28,384 INFO Service [main-EventThread]: Process path: /services/cldb/maprdemo.mapr.io. Event state: SyncConnected. Event type: NodeDeleted
  1112. 2023-12-11 22:43:28,384 INFO DependentService [main-EventThread]: DependentWatcher for service: nfs Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDeleted
  1113. 2023-12-11 22:43:28,384 INFO Service [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:464 xid:81 sent:99 recv:111 queuedpkts:0 pendingresp:0 queuedevents:1
  1114. 2023-12-11 22:43:28,384 INFO DependentService [main-EventThread]: DependentWatcher for service: apiserver Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDeleted
  1115. 2023-12-11 22:43:28,385 INFO DependentService [main-EventThread]: DependentWatcher for service: data-access-gateway Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDeleted
  1116. 2023-12-11 22:43:28,485 ERROR ZKUtilsLocking [main-EventThread]: SessionExpiredException while trying to checkZKNodeForExistence of: /services/kvstore/maprdemo.mapr.io
  1117. 2023-12-11 22:43:28,485 ERROR DependentService [main-EventThread]: ZK Session was either or closed or expired for service: cldb
  1118. 2023-12-11 22:43:28,485 ERROR ZKUtilsLocking [main-EventThread]: SessionExpiredException while trying to checkZKNodeForExistence of: /services/cldb/maprdemo.mapr.io
  1119. 2023-12-11 22:43:28,485 ERROR Service [main-EventThread]: ZK Session was either or closed or expired for service: cldb
  1120. 2023-12-11 22:43:28,485 INFO Service [main-EventThread]: Process path: /services/cldb/master. Event state: SyncConnected. Event type: NodeDeleted
  1121. 2023-12-11 22:43:28,485 INFO Service [main-EventThread]: ZK Connect state:State:CLOSED sessionid:0x1000010ecd40004 local:/172.17.0.2:34462 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:464 xid:81 sent:99 recv:111 queuedpkts:0 pendingresp:0 queuedevents:2
  1122. 2023-12-11 22:43:28,485 INFO Service [main-EventThread]: ZK is closed for service: cldb
  1123. 2023-12-11 22:43:28,485 INFO Service [main-EventThread]: Process path: null. Event state: Closed. Event type: None
  1124. 2023-12-11 22:43:28,485 INFO Service [main-EventThread]: ZK is closed for service: cldb
  1125. 2023-12-11 22:43:28,485 INFO DependentService [main-EventThread]: DependentWatcher for service: cldb Process path: null. Event state: Closed. Event type: None
  1126. 2023-12-11 22:43:28,485 INFO DependentService [main-EventThread]: ZK is closed for service: cldb
  1127. 2023-12-11 22:43:28,485 INFO CLDBService [main-EventThread]: Process path: null. Event state: Closed. Event type: None
  1128. 2023-12-11 22:43:28,490 INFO CLDBService [main-EventThread]: ZK is closed for service: cldb
  1129. 2023-12-11 22:43:28,491 INFO WardenRequestBuilder [cldb_monitor]: [e_SERV_FAIL, hostName, ma_host, ma_process]
  1130. 2023-12-11 22:43:28,491 INFO WardenRequestBuilder [cldb_monitor]: []
  1131. 2023-12-11 22:43:28,491 INFO Service [cldb_monitor]: Alarm raising command: [/opt/mapr/bin/maprcli, alarm, raise, -alarm, NODE_ALARM_SERVICE_CLDB_DOWN, -entity, maprdemo.mapr.io, -description, Can not determine if service: cldb is running. Check logs at: /opt/mapr/logs/cldb.log]
  1132. 2023-12-11 22:43:32,416 ERROR Service [main-EventThread]: Error while running command: [/opt/mapr/bin/maprcli, alarm, raise, -alarm, NODE_ALARM_MEMORY_ALLOCATION_EXCEEDED, -entity, maprdemo.mapr.io, -description, Memory allocated for services on this node is exceeding 97% of total memory available.]
  1133. 2023-12-11 22:43:32,416 ERROR Service [main-EventThread]: alarm raise
  1134. [ -cluster cluster_name ]
  1135. -alarm alarm
  1136. [ -entity entity (hostname OR volume name OR Ae name) ]
  1137. [ -description brief description ]
  1138.  
  1139. 2023-12-11 22:43:32,416 WARN Service [main-EventThread]: Delayed alarm processing failed for alarm: com.mapr.warden.service.baseservice.Service$AlarmKeyEntity@6ad0e48c. Process later
  1140. 2023-12-11 22:43:51,365 INFO PullCentralConfigTaskScheduler [PullCentralConfigTask]: Launching a separate process to execute /opt/mapr/server/pullcentralconfig
  1141. 2023-12-11 22:43:54,662 INFO PullCentralConfigTaskScheduler [PullCentralConfigTask]: /opt/mapr/server/pullcentralconfig process terminated with status: 0
  1142. 2023-12-11 22:43:57,780 ERROR MaprRPCContext [Thread-4]: Response is null. Most likely hoststats is not accepting requests or it is down.
  1143. 2023-12-11 22:43:58,494 INFO Service [Thread-11-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
  1144. 2023-12-11 22:43:58,495 INFO Service [Thread-11]: Connected to ZK: 172.17.0.2:5181With State: State:CONNECTED Timeout:30000 sessionid:0x1000010ecd4001c local:/172.17.0.2:45556 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:2 queuedpkts:0 pendingresp:0 queuedevents:1
  1145. 2023-12-11 22:43:58,495 INFO Service [Thread-11]: Service reinitThread: zk expired: false reconnected to zk, lastAction: null
  1146. 2023-12-11 22:43:58,495 INFO Service [Thread-11-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x1000010ecd4001c local:/172.17.0.2:45556 remoteserver:maprdemo.mapr.io/172.17.0.2:5181 lastZxid:0 xid:3 sent:1 recv:2 queuedpkts:1 pendingresp:0 queuedevents:1
  1147. 2023-12-11 22:43:58,495 INFO Service [Thread-11-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
  1148. 2023-12-11 22:43:58,501 INFO Service [Thread-11]: Node: /nodes/maprdemo.mapr.io/services/cldb does not exist yet
  1149. 2023-12-11 22:43:58,506 INFO Service [Thread-11]: addServiceToZK: zknode /services/cldb already exists
  1150. 2023-12-11 22:43:58,506 INFO Service [Thread-11]: addServiceToZK: after remove master, children node size 0
  1151.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!