Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
- WARN [PluginLoader] Plugin directory /plugin does not exist, not loading plugins.
- INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
- INFO [cluster] Adding discovered server graylog-01:27017 to client view of cluster
- INFO [cluster] Adding discovered server graylog-02:27017 to client view of cluster
- INFO [cluster] Adding discovered server graylog-03:27017 to client view of cluster
- INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/21.0.10+7-LTS"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6d67f5eb, com.mongodb.Jep395RecordCodecProvider@4f654cee, com.mongodb.KotlinCodecProvider@3e1fd62b, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[graylog-01:27017, graylog-02:27017, graylog-03:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs0', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
- INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/21.0.10+7-LTS"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6d67f5eb, com.mongodb.Jep395RecordCodecProvider@4f654cee, com.mongodb.KotlinCodecProvider@3e1fd62b, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[graylog-01:27017, graylog-02:27017, graylog-03:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs0', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
- INFO [cluster] Waiting for server to become available for operation { ping: 1 } with ID 5. Remaining time: 29995 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=graylog-03:27017, type=UNKNOWN, state=CONNECTING}, {address=graylog-02:27017, type=UNKNOWN, state=CONNECTING}, {address=graylog-01:27017, type=UNKNOWN, state=CONNECTING}].
- INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=graylog-03:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17327907, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=graylog-03:27017, hosts=[graylog-03:27017, graylog-01.mydomain.com:27017, graylog-02:27017], passives=[], arbiters=[], primary='graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5736d190a4b6ade386cca, counter=5}, lastWriteDate=Fri Apr 03 12:36:29 PDT 2026, lastUpdateTimeNanos=36985000645774682}
- INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=graylog-01:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17328709, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=graylog-01.mydomain.com:27017, hosts=[graylog-03:27017, graylog-01.mydomain.com:27017, graylog-02:27017], passives=[], arbiters=[], primary='graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5729ebb9e82ecc9da539c, counter=10}, lastWriteDate=Fri Apr 03 12:36:29 PDT 2026, lastUpdateTimeNanos=36985000645775584}
- INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=graylog-02:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17240508, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=graylog-02:27017, hosts=[graylog-03:27017, graylog-01.mydomain.com:27017, graylog-02:27017], passives=[], arbiters=[], primary='graylog-02:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000a2, setVersion=5, topologyVersion=TopologyVersion{processId=69c5734b7911307697c7fb33, counter=7}, lastWriteDate=Fri Apr 03 12:36:29 PDT 2026, lastUpdateTimeNanos=36985000645769743}
- INFO [cluster] Adding discovered server graylog-01.mydomain.com:27017 to client view of cluster
- INFO [cluster] Canonical address graylog-01.mydomain.com:27017 does not match server address. Removing graylog-01:27017 from client view of cluster
- INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=graylog-01.mydomain.com:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=893940, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=graylog-01.mydomain.com:27017, hosts=[graylog-03:27017, graylog-01.mydomain.com:27017, graylog-02:27017], passives=[], arbiters=[], primary='graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5729ebb9e82ecc9da539c, counter=10}, lastWriteDate=Fri Apr 03 12:36:29 PDT 2026, lastUpdateTimeNanos=36985000662460728}
- INFO [cluster] Discovered replica set primary graylog-02:27017 with max election id 7fffffff00000000000000a2 and max set version 5
- INFO [DatanodeDirectories] Opensearch of the node 076ad60c-1d4d-4128-8568-39bd4eba8891 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/mnt/opensearch', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
- INFO [cluster] Adding discovered server graylog-01:27017 to client view of cluster
- INFO [cluster] Adding discovered server graylog-02:27017 to client view of cluster
- INFO [cluster] Adding discovered server graylog-03:27017 to client view of cluster
- INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/21.0.10+7-LTS"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6d67f5eb, com.mongodb.Jep395RecordCodecProvider@4f654cee, com.mongodb.KotlinCodecProvider@3e1fd62b, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[graylog-01:27017, graylog-02:27017, graylog-03:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs0', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
- INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/21.0.10+7-LTS"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6d67f5eb, com.mongodb.Jep395RecordCodecProvider@4f654cee, com.mongodb.KotlinCodecProvider@3e1fd62b, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[graylog-01:27017, graylog-02:27017, graylog-03:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs0', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
- INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=graylog-01:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1074359, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=graylog-01.mydomain.com:27017, hosts=[graylog-03:27017, graylog-01.mydomain.com:27017, graylog-02:27017], passives=[], arbiters=[], primary='graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5729ebb9e82ecc9da539c, counter=10}, lastWriteDate=Fri Apr 03 12:36:29 PDT 2026, lastUpdateTimeNanos=36985000752602221}
- INFO [cluster] Waiting for server to become available for operation { ping: 1 } with ID 20. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=graylog-03:27017, type=UNKNOWN, state=CONNECTING}, {address=graylog-02:27017, type=UNKNOWN, state=CONNECTING}, {address=graylog-01:27017, type=UNKNOWN, state=CONNECTING}].
- INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=graylog-02:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1282201, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=graylog-02:27017, hosts=[graylog-03:27017, graylog-01.mydomain.com:27017, graylog-02:27017], passives=[], arbiters=[], primary='graylog-02:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000a2, setVersion=5, topologyVersion=TopologyVersion{processId=69c5734b7911307697c7fb33, counter=7}, lastWriteDate=Fri Apr 03 12:36:29 PDT 2026, lastUpdateTimeNanos=36985000752999530}
- INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=graylog-03:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1179993, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=graylog-03:27017, hosts=[graylog-03:27017, graylog-01.mydomain.com:27017, graylog-02:27017], passives=[], arbiters=[], primary='graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5736d190a4b6ade386cca, counter=5}, lastWriteDate=Fri Apr 03 12:36:29 PDT 2026, lastUpdateTimeNanos=36985000753133769}
- INFO [cluster] Adding discovered server graylog-01.mydomain.com:27017 to client view of cluster
- INFO [cluster] Canonical address graylog-01.mydomain.com:27017 does not match server address. Removing graylog-01:27017 from client view of cluster
- INFO [cluster] Discovered replica set primary graylog-02:27017 with max election id 7fffffff00000000000000a2 and max set version 5
- INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=graylog-01.mydomain.com:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=829845, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=graylog-01.mydomain.com:27017, hosts=[graylog-03:27017, graylog-01.mydomain.com:27017, graylog-02:27017], passives=[], arbiters=[], primary='graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5729ebb9e82ecc9da539c, counter=10}, lastWriteDate=Fri Apr 03 12:36:29 PDT 2026, lastUpdateTimeNanos=36985000755293288}
- INFO [MongoDBPreflightCheck] Connected to MongoDB version 8.2.5
- INFO [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64]
- INFO [MemorySegmentIndexInputProvider] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
- WARN [VectorizationProvider] Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
- INFO [OpensearchDataDirCompatibilityCheck] Found 31 indices and all of them are valid with current opensearch version 2.19.3
- INFO [DatanodeDirectories] Opensearch of the node 076ad60c-1d4d-4128-8568-39bd4eba8891 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/mnt/opensearch', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
- INFO [DatanodeBootstrap] Graylog Data Node 7.0.6+711d207 starting up (command: datanode)
- INFO [DatanodeBootstrap] JRE: Eclipse Adoptium 21.0.10 on Linux 4.18.0-553.el8_10.x86_64
- INFO [DatanodeBootstrap] Deployment: rpm
- INFO [DatanodeBootstrap] OS: Rocky Linux 8.10 (Green Obsidian) (rocky)
- INFO [DatanodeBootstrap] Arch: amd64
- INFO [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [2s].
- INFO [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
- INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
- INFO [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
- INFO [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
- INFO [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
- INFO [SearchableSnapshotsConfigurationBean] Opensearch snapshots not configured, skipping search role and cache configuration.
- INFO [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
- INFO [TruststoreCreator] Adding key certificate chain of alias datanode to the truststore
- INFO [TruststoreCreator] Adding key certificate chain of alias datanode to the truststore
- INFO [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64]
- INFO [DatanodeBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, OpensearchProcessService [RUNNING]=0, PeriodicalsService [RUNNING]=13, OpensearchConfigurationService [RUNNING]=74}
- INFO [DatanodeBootstrap] Graylog DataNode datanode up and running.
- INFO [OpensearchProcessService] OpenSearch starting up
- INFO [OpensearchCommandLineProcess] Creating opensearch keystore
- INFO [JerseyService] Starting Data node REST API
- INFO [Version] HV000001: Hibernate Validator 9.0.1.Final
- INFO [NetworkListener] Started listener bound to [10.10.0.238:8999]
- INFO [HttpServer] [HttpServer] Started.
- INFO [JerseyService] Started REST API at <10.10.0.238:8999>
- INFO [OpensearchCommandLineProcess] Created opensearch keystore in /var/lib/graylog-datanode/opensearch/config/opensearch10504328495040196591/opensearch.keystore
- INFO [OpensearchCommandLineProcess] Added 4 keystore items
- INFO [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/bin/opensearch
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from UNCONFIGURED to STARTING (reason: PROCESS_STARTED)
- WARN [AbstractNodeService] Did not find meta info of this node. Re-registering.
- WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3117464 unavailable. Cause: Unrecognized SSL message, plaintext connection?
- WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3117464 unavailable. Cause: Unrecognized SSL message, plaintext connection?
- WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3117464 unavailable. Cause: Unrecognized SSL message, plaintext connection?
- WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/lib/opensearch-2.19.3.jar)
- WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
- WARN [OpensearchProcessImpl] Apr 03, 2026 12:37:07 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
- WARN [OpensearchProcessImpl] WARNING: COMPAT locale provider will be removed in a future release
- WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/lib/opensearch-2.19.3.jar)
- WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:07,397][INFO ][o.o.n.Node ] [opensrch-02] version[2.19.3], pid[3117464], build[tar/a90f864b8524bc75570a8461ccb569d2a4bfed42/2025-07-21T22:34:18.003652598Z], OS[Linux/4.18.0-553.el8_10.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.7/21.0.7+6-LTS]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:07,399][INFO ][o.o.n.Node ] [opensrch-02] JVM home [/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/jdk], using bundled JDK/JRE [true]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:07,399][INFO ][o.o.n.Node ] [opensrch-02] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5426530526821111649, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/tmp/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.security.policy=file:///var/lib/graylog-datanode/opensearch/config/opensearch10504328495040196591/opensearch.policy, -Xms31g, -Xmx31g, -Dopensearch.transport.cname_in_publish_address=true, -Djavax.net.ssl.trustStore=datanode-truststore.p12, -Djavax.net.ssl.trustStorePassword=moZYKgDrxViXqlprhNonuufGHgaAAvTquXpXzLQnxNxAALWrQgFDWuIpAfjGBqCpnIXEmyIQbijszUfzhzSnpsPHRTggSIZuPwjBskMtxFWQcnsiAEmmnViORUGYqpmNAReKTwGpgUVADfYvhUOvnkytxSubPuxBNHzMPmcxwvwWgxAKuLOXRklKBOJefCkXTbpSxkpREoWkiXBnrqBXETVMzltPWnWHMwLVvqkETLtAJKTsXEeNQVakcxjeFxGR, -Djavax.net.ssl.trustStoreType=pkcs12, -XX:MaxDirectMemorySize=16642998272, -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64, -Dopensearch.path.conf=/var/lib/graylog-datanode/opensearch/config/opensearch10504328495040196591, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:07,518][WARN ][o.a.l.i.v.VectorizationProvider] [opensrch-02] Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,111][INFO ][o.o.s.s.t.SSLConfig ] [opensrch-02] SSL dual mode is disabled
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,111][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensrch-02] OpenSearch Config path is /var/lib/graylog-datanode/opensearch/config/opensearch10504328495040196591
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,335][WARN ][o.o.s.s.c.SslCertificatesLoader] [opensrch-02] Setting [plugins.security.ssl.http.keystore_keypassword] has a secure counterpart [plugins.security.ssl.http.keystore_keypassword_secure] which should be used instead - allowing for legacy SSL setups
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,339][INFO ][o.o.s.s.SslSettingsManager] [opensrch-02] TLS HTTP Provider : JDK
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,339][INFO ][o.o.s.s.SslSettingsManager] [opensrch-02] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,340][WARN ][o.o.s.s.c.SslCertificatesLoader] [opensrch-02] Setting [plugins.security.ssl.transport.keystore_keypassword] has a secure counterpart [plugins.security.ssl.transport.keystore_keypassword_secure] which should be used instead - allowing for legacy SSL setups
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,340][INFO ][o.o.s.s.SslSettingsManager] [opensrch-02] TLS Transport Client Provider : JDK
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,340][INFO ][o.o.s.s.SslSettingsManager] [opensrch-02] TLS Transport Server Provider : JDK
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,340][INFO ][o.o.s.s.SslSettingsManager] [opensrch-02] Enabled TLS protocols for Transport layer : [TLSv1.3, TLSv1.2]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,528][INFO ][o.o.s.s.c.KeyStoreUtils ] [opensrch-02] Skipping validation for C=DE,O=Atos,CN=Atos TrustedRoot 2011
- <skipping HUGE list of skipped validations>
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,810][INFO ][o.o.s.s.c.KeyStoreUtils ] [opensrch-02] Skipping validation for CN=opensrch-02.mydomain.com
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,810][INFO ][o.o.s.s.c.KeyStoreUtils ] [opensrch-02] Skipping validation for CN=opensrch-02.mydomain.com
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:08,835][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensrch-02] Clustername: graylog-search
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,240][INFO ][o.o.i.r.ReindexPlugin ] [opensrch-02] ReindexPlugin reloadSPI called
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,241][INFO ][o.o.i.r.ReindexPlugin ] [opensrch-02] Unable to find any implementation for RemoteReindexExtension
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,247][INFO ][o.o.j.JobSchedulerPlugin ] [opensrch-02] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,248][INFO ][o.o.j.JobSchedulerPlugin ] [opensrch-02] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,250][INFO ][o.o.j.JobSchedulerPlugin ] [opensrch-02] Loaded scheduler extension: checkBatchJobTaskStatus, index: .ml_commons_task_polling_job
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,261][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [aggs-matrix-stats]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [analysis-common]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [cache-common]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [geo]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [ingest-common]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [ingest-geoip]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [ingest-user-agent]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [lang-expression]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [lang-mustache]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [lang-painless]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [mapper-extras]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [opensearch-dashboards]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [parent-join]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [percolator]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [rank-eval]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [reindex]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [repository-url]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [search-pipeline-common]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [systemd]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded module [transport-netty4]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-anomaly-detection]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-asynchronous-search]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-cross-cluster-replication]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,262][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-flow-framework]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-index-management]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-job-scheduler]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-ltr]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-ml]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-security]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-skills]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [opensearch-system-templates]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [repository-gcs]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [repository-hdfs]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,263][INFO ][o.o.p.PluginsService ] [opensrch-02] loaded plugin [repository-s3]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,275][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensrch-02] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
- WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3117464 unavailable. Cause: Unrecognized SSL message, plaintext connection?
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,297][WARN ][stderr ] [opensrch-02] WARNING: A restricted method in java.lang.foreign.Linker has been called
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,297][WARN ][stderr ] [opensrch-02] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,297][WARN ][stderr ] [opensrch-02] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,352][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [opensrch-02] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,357][INFO ][o.o.e.NodeEnvironment ] [opensrch-02] using [1] data paths, mounts [[/mnt/opensearch (/dev/mapper/vg_opensearch-lv_opensearch)]], net usable_space [3.2tb], net total_space [3.4tb], types [xfs]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,357][INFO ][o.o.e.NodeEnvironment ] [opensrch-02] heap size [31gb], compressed ordinary object pointers [true]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,412][INFO ][o.o.n.Node ] [opensrch-02] node name [opensrch-02], node ID [sszJShvqS_eg4_BvMC6Bkg], cluster name [graylog-search], roles [data, cluster_manager]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:10,642][INFO ][o.o.e.ExtensionsManager ] [opensrch-02] ExtensionsManager initialized
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,464][WARN ][o.o.s.c.Salt ] [opensrch-02] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,489][ERROR][o.o.s.a.s.SinkProvider ] [opensrch-02] Default endpoint could not be created, auditlog will not work properly.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,490][WARN ][o.o.s.a.r.AuditMessageRouter] [opensrch-02] No default storage available, audit log may not work properly. Please check configuration.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,490][INFO ][o.o.s.a.i.AuditLogImpl ] [opensrch-02] Message routing enabled: false
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,513][INFO ][o.o.s.f.SecurityFilter ] [opensrch-02] <NONE> indices are made immutable.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,605][INFO ][o.o.s.s.SslSettingsManager] [opensrch-02] Added SSL configuration change listener for: /var/lib/graylog-datanode/opensearch/config/opensearch10504328495040196591
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,607][INFO ][o.o.l.b.LTRCircuitBreakerService] [opensrch-02] Registered memory breaker.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,703][INFO ][o.o.t.b.CircuitBreakerService] [opensrch-02] Registered memory breaker.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:12,915][INFO ][o.o.r.m.c.i.SdkClientFactory] [opensrch-02] Using local opensearch cluster as metadata store.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,048][INFO ][o.o.r.m.c.i.SdkClientFactory] [opensrch-02] Using local opensearch cluster as metadata store.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,062][INFO ][o.o.m.b.MLCircuitBreakerService] [opensrch-02] Registered ML memory breaker.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,062][INFO ][o.o.m.b.MLCircuitBreakerService] [opensrch-02] Registered ML disk breaker.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,062][INFO ][o.o.m.b.MLCircuitBreakerService] [opensrch-02] Registered ML native memory breaker.
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,133][INFO ][o.r.Reflections ] [opensrch-02] Reflections took 39 ms to scan 1 urls, producing 27 keys and 67 values
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,147][INFO ][o.r.Reflections ] [opensrch-02] Reflections took 1 ms to scan 1 urls, producing 3 keys and 5 values
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,374][INFO ][o.o.t.NettyAllocator ] [opensrch-02] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,379][INFO ][o.o.s.s.t.SSLConfig ] [opensrch-02] SSL dual mode is disabled
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,451][INFO ][o.o.d.DiscoveryModule ] [opensrch-02] using discovery type [zen] and seed hosts providers [settings]
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:13,867][WARN ][o.o.g.DanglingIndicesState] [opensrch-02] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:14,252][INFO ][o.o.n.Node ] [opensrch-02] initialized
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:14,339][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensrch-02] uncaught exception in thread [main]
- INFO [OpensearchProcessImpl] org.opensearch.bootstrap.StartupException: BindTransportException[Failed to bind to 10.10.0.238:9300]; nested: BindException[Address already in use];
- WARN [OpensearchProcessImpl] uncaught exception in thread [main]
- INFO [OpensearchProcessImpl] Caused by: org.opensearch.transport.BindTransportException: Failed to bind to 10.10.0.238:9300
- INFO [OpensearchProcessImpl] Caused by: java.net.BindException: Address already in use
- WARN [OpensearchProcessImpl] BindTransportException[Failed to bind to 10.10.0.238:9300]; nested: BindException[Address already in use];
- WARN [OpensearchProcessImpl] Likely root cause: java.net.BindException: Address already in use
- INFO [OpensearchProcessImpl] Suppressed: java.lang.RuntimeException: Rethrowing promise failure cause
- WARN [OpensearchProcessImpl] For complete error details, refer to the log at /var/log/graylog-datanode/opensearch/graylog-search.log
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:15,276][INFO ][o.o.s.a.r.AuditMessageRouter] [opensrch-02] Closing AuditMessageRouter
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:15,277][INFO ][o.o.s.a.s.SinkProvider ] [opensrch-02] Closing DebugSink
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:15,278][INFO ][o.o.n.Node ] [opensrch-02] stopped
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:15,281][INFO ][o.o.s.a.i.AuditLogImpl ] [opensrch-02] Closing AuditLogImpl
- INFO [OpensearchProcessImpl] [2026-04-03T12:37:15,287][INFO ][o.o.n.Node ] [opensrch-02] closed
- WARN [OpensearchProcessImpl] Opensearch process failed
- Process exited with an error: 1 (Exit value: 1)
- INFO [OpensearchCommandLineProcess] Process 3117464 still alive, waiting for termination. Retry #1
- INFO [OpensearchCommandLineProcess] Process 3117464 successfully terminated.
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from STARTING to UNAVAILABLE (reason: PROCESS_TERMINATED)
- INFO [OpensearchWatchdog] Detected terminated process, restarting. Attempt #1
- INFO [OpensearchCommandLineProcess] Creating opensearch keystore
- INFO [OpensearchCommandLineProcess] Created opensearch keystore in /var/lib/graylog-datanode/opensearch/config/opensearch10504328495040196591/opensearch.keystore
- ERROR [OpensearchNodeHeartbeat] Uncaught exception in Periodical
- Request execution cancelled
- com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedPeriodicRunnable.run(InstrumentedScheduledExecutorService.java:264)
- by: java.util.concurrent.CancellationException: Request execution cancelled
- INFO [OpensearchCommandLineProcess] Added 4 keystore items
- INFO [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/bin/opensearch
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from UNAVAILABLE to STARTING (reason: PROCESS_STARTED)
- WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/lib/opensearch-2.19.3.jar)
- WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
- WARN [OpensearchProcessImpl] Apr 03, 2026 12:37:24 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
- WARN [OpensearchProcessImpl] WARNING: COMPAT locale provider will be removed in a future release
- WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/lib/opensearch-2.19.3.jar)
- WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
- INFO [OpensearchProcessImpl] 12:37:28.582 [main] ERROR org.opensearch.security.auditlog.sink.SinkProvider - Default endpoint could not be created, auditlog will not work properly.
- INFO [OpensearchProcessImpl] 12:37:30.190 [main] ERROR org.opensearch.bootstrap.OpenSearchUncaughtExceptionHandler - uncaught exception in thread [main]
- WARN [OpensearchProcessImpl] uncaught exception in thread [main]
- INFO [OpensearchProcessImpl] org.opensearch.bootstrap.StartupException: BindTransportException[Failed to bind to 10.10.0.238:9300]; nested: BindException[Address already in use];
- INFO [OpensearchProcessImpl] Caused by: org.opensearch.transport.BindTransportException: Failed to bind to 10.10.0.238:9300
- INFO [OpensearchProcessImpl] Caused by: java.net.BindException: Address already in use
- WARN [OpensearchProcessImpl] BindTransportException[Failed to bind to 10.10.0.238:9300]; nested: BindException[Address already in use];
- WARN [OpensearchProcessImpl] Likely root cause: java.net.BindException: Address already in use
- INFO [OpensearchProcessImpl] Suppressed: java.lang.RuntimeException: Rethrowing promise failure cause
- WARN [OpensearchProcessImpl] For complete error details, refer to the log at /var/log/graylog-datanode/opensearch/graylog-search.log
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from STARTING to UNAVAILABLE (reason: HEALTH_CHECK_FAILED)
- WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3118738 unavailable. Cause: Unrecognized SSL message, plaintext connection?
- WARN [OpensearchProcessImpl] Opensearch process failed
- Process exited with an error: 1 (Exit value: 1)
- INFO [OpensearchCommandLineProcess] Process 3118738 still alive, waiting for termination. Retry #1
- INFO [OpensearchCommandLineProcess] Process 3118738 successfully terminated.
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from UNAVAILABLE to UNAVAILABLE (reason: PROCESS_TERMINATED)
- INFO [OpensearchWatchdog] Detected terminated process, restarting. Attempt #2
- INFO [OpensearchCommandLineProcess] Creating opensearch keystore
- INFO [OpensearchCommandLineProcess] Created opensearch keystore in /var/lib/graylog-datanode/opensearch/config/opensearch10504328495040196591/opensearch.keystore
- INFO [OpensearchCommandLineProcess] Added 4 keystore items
- INFO [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/bin/opensearch
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from UNAVAILABLE to STARTING (reason: PROCESS_STARTED)
- WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/lib/opensearch-2.19.3.jar)
- WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
- WARN [OpensearchProcessImpl] Apr 03, 2026 12:37:39 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
- WARN [OpensearchProcessImpl] WARNING: COMPAT locale provider will be removed in a future release
- WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/lib/opensearch-2.19.3.jar)
- WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from STARTING to UNAVAILABLE (reason: HEALTH_CHECK_FAILED)
- WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3119780 unavailable. Cause: Unrecognized SSL message, plaintext connection?
- INFO [OpensearchProcessImpl] 12:37:43.229 [main] ERROR org.opensearch.security.auditlog.sink.SinkProvider - Default endpoint could not be created, auditlog will not work properly.
- INFO [OpensearchProcessImpl] 12:37:44.817 [main] ERROR org.opensearch.bootstrap.OpenSearchUncaughtExceptionHandler - uncaught exception in thread [main]
- WARN [OpensearchProcessImpl] uncaught exception in thread [main]
- INFO [OpensearchProcessImpl] org.opensearch.bootstrap.StartupException: BindTransportException[Failed to bind to 10.10.0.238:9300]; nested: BindException[Address already in use];
- INFO [OpensearchProcessImpl] Caused by: org.opensearch.transport.BindTransportException: Failed to bind to 10.10.0.238:9300
- INFO [OpensearchProcessImpl] Caused by: java.net.BindException: Address already in use
- WARN [OpensearchProcessImpl] BindTransportException[Failed to bind to 10.10.0.238:9300]; nested: BindException[Address already in use];
- WARN [OpensearchProcessImpl] Likely root cause: java.net.BindException: Address already in use
- INFO [OpensearchProcessImpl] Suppressed: java.lang.RuntimeException: Rethrowing promise failure cause
- WARN [OpensearchProcessImpl] For complete error details, refer to the log at /var/log/graylog-datanode/opensearch/graylog-search.log
- WARN [OpensearchProcessImpl] Opensearch process failed
- Process exited with an error: 1 (Exit value: 1)
- INFO [OpensearchCommandLineProcess] Process 3119780 still alive, waiting for termination. Retry #1
- INFO [OpensearchCommandLineProcess] Process 3119780 successfully terminated.
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from UNAVAILABLE to UNAVAILABLE (reason: PROCESS_TERMINATED)
- INFO [OpensearchWatchdog] Detected terminated process, restarting. Attempt #3
- INFO [OpensearchCommandLineProcess] Creating opensearch keystore
- INFO [OpensearchCommandLineProcess] Created opensearch keystore in /var/lib/graylog-datanode/opensearch/config/opensearch10504328495040196591/opensearch.keystore
- ERROR [OpensearchNodeHeartbeat] Uncaught exception in Periodical
- Request execution cancelled
- com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedPeriodicRunnable.run(InstrumentedScheduledExecutorService.java:264)
- by: java.util.concurrent.CancellationException: Request execution cancelled
- INFO [OpensearchCommandLineProcess] Added 4 keystore items
- INFO [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/bin/opensearch
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from UNAVAILABLE to STARTING (reason: PROCESS_STARTED)
- WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/lib/opensearch-2.19.3.jar)
- WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
- WARN [OpensearchProcessImpl] Apr 03, 2026 12:37:53 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
- WARN [OpensearchProcessImpl] WARNING: COMPAT locale provider will be removed in a future release
- WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/graylog-datanode/dist/opensearch-2.19.3-linux-x64/lib/opensearch-2.19.3.jar)
- WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
- WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
- INFO [OpensearchProcessImpl] 12:37:57.722 [main] ERROR org.opensearch.security.auditlog.sink.SinkProvider - Default endpoint could not be created, auditlog will not work properly.
- INFO [OpensearchProcessImpl] 12:37:59.315 [main] ERROR org.opensearch.bootstrap.OpenSearchUncaughtExceptionHandler - uncaught exception in thread [main]
- WARN [OpensearchProcessImpl] uncaught exception in thread [main]
- INFO [OpensearchProcessImpl] org.opensearch.bootstrap.StartupException: BindTransportException[Failed to bind to 10.10.0.238:9300]; nested: BindException[Address already in use];
- INFO [OpensearchProcessImpl] Caused by: org.opensearch.transport.BindTransportException: Failed to bind to 10.10.0.238:9300
- INFO [OpensearchProcessImpl] Caused by: java.net.BindException: Address already in use
- WARN [OpensearchProcessImpl] BindTransportException[Failed to bind to 10.10.0.238:9300]; nested: BindException[Address already in use];
- WARN [OpensearchProcessImpl] Likely root cause: java.net.BindException: Address already in use
- INFO [OpensearchProcessImpl] Suppressed: java.lang.RuntimeException: Rethrowing promise failure cause
- WARN [OpensearchProcessImpl] For complete error details, refer to the log at /var/log/graylog-datanode/opensearch/graylog-search.log
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from STARTING to UNAVAILABLE (reason: HEALTH_CHECK_FAILED)
- WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3120664 unavailable. Cause: Unrecognized SSL message, plaintext connection?
- WARN [OpensearchProcessImpl] Opensearch process failed
- Process exited with an error: 1 (Exit value: 1)
- INFO [OpensearchCommandLineProcess] Process 3120664 still alive, waiting for termination. Retry #1
- INFO [OpensearchCommandLineProcess] Process 3120664 successfully terminated.
- INFO [ClusterNodeStateTracer] Updating cluster node 076ad60c-1d4d-4128-8568-39bd4eba8891 from UNAVAILABLE to UNAVAILABLE (reason: PROCESS_TERMINATED)
- WARN [OpensearchWatchdog] Process watchdog terminated after too many restart attempts
- INFO [Datanode] SIGNAL received. Shutting down.
- INFO [GracefulShutdown] Graceful shutdown initiated.
- INFO [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical].
- INFO [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical].
- INFO [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
- INFO [GracefulShutdown] Goodbye.
Advertisement