Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- [2025-06-12T13:29:26,351][WARN ][stderr ] [node1] Jun 12, 2025 1:29:26 PM org.opensearch.javaagent.bootstrap.AgentPolicy setPolicy
- [2025-06-12T13:29:26,355][WARN ][stderr ] [node1] INFO: Policy attached successfully: org.opensearch.bootstrap.OpenSearchPolicy@687e4c93
- [2025-06-12T13:29:26,373][INFO ][o.o.n.Node ] [node1] version[3.0.0], pid[2347], build[deb/dc4efa821904cc2d7ea7ef61c0f577d3fc0d8be9/2025-05-03T06:23:34.992456558Z], OS[Linux/6.1.0-37-amd64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.7/21.0.7+6-LTS]
- [2025-06-12T13:29:26,374][INFO ][o.o.n.Node ] [node1] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
- [2025-06-12T13:29:26,374][INFO ][o.o.n.Node ] [node1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,CLDR, -Xms4g, -Xmx4g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-13871182874840855876, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/opensearch, -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, --add-modules=jdk.incubator.vector, -javaagent:agent/opensearch-agent.jar, --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=2147483648, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
- [2025-06-12T13:29:26,652][WARN ][stderr ] [node1] Jun 12, 2025 1:29:26 PM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>
- [2025-06-12T13:29:26,653][WARN ][stderr ] [node1] INFO: Java vector incubator API enabled; uses preferredBitSize=128; floating-point vectors only
- [2025-06-12T13:29:27,932][INFO ][o.o.s.s.t.SSLConfig ] [node1] SSL dual mode is disabled
- [2025-06-12T13:29:27,932][INFO ][o.o.s.OpenSearchSecurityPlugin] [node1] OpenSearch Config path is /etc/opensearch
- [2025-06-12T13:29:28,725][INFO ][o.o.s.s.SslSettingsManager] [node1] TLS HTTP Provider : JDK
- [2025-06-12T13:29:28,727][INFO ][o.o.s.s.SslSettingsManager] [node1] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
- [2025-06-12T13:29:28,728][INFO ][o.o.s.s.SslSettingsManager] [node1] TLS Transport Client Provider : JDK
- [2025-06-12T13:29:28,728][INFO ][o.o.s.s.SslSettingsManager] [node1] TLS Transport Server Provider : JDK
- [2025-06-12T13:29:28,728][INFO ][o.o.s.s.SslSettingsManager] [node1] Enabled TLS protocols for Transport layer : [TLSv1.3, TLSv1.2]
- [2025-06-12T13:29:30,022][INFO ][o.o.s.OpenSearchSecurityPlugin] [node1] Clustername: logsystem
- [2025-06-12T13:29:30,028][WARN ][o.o.s.OpenSearchSecurityPlugin] [node1] File /etc/opensearch/jvm.options.d/test.option has insecure file permissions (should be 0600)
- [2025-06-12T13:29:30,029][WARN ][o.o.s.OpenSearchSecurityPlugin] [node1] File /etc/opensearch/securityadmin_demo.sh has insecure file permissions (should be 0600)
- [2025-06-12T13:29:30,029][WARN ][o.o.s.OpenSearchSecurityPlugin] [node1] File /etc/opensearch/esnode.pem has insecure file permissions (should be 0600)
- [2025-06-12T13:29:30,030][WARN ][o.o.s.OpenSearchSecurityPlugin] [node1] File /etc/opensearch/root-ca.pem has insecure file permissions (should be 0600)
- [2025-06-12T13:29:30,030][WARN ][o.o.s.OpenSearchSecurityPlugin] [node1] Directory /etc/opensearch/pem_arch has insecure file permissions (should be 0700)
- [2025-06-12T13:29:31,393][ERROR][o.o.p.c.c.PluginSettings ] [node1] Loading config file /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties failed with error: org.opensearch.performanceanalyzer.commons.config.ConfigFileException: java.io.FileNotFoundException: /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties (No such file or directory). Disabling plugin.
- [2025-06-12T13:29:31,394][INFO ][o.o.p.c.c.PluginSettings ] [node1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: false, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
- [2025-06-12T13:29:31,738][ERROR][o.o.p.c.c.StatsCollector ] [node1] Error in loading metadata for folderLocation: /etc/opensearch/opensearch-performance-analyzer/, fileLocation: plugin-stats-metadata
- java.io.FileNotFoundException: /etc/opensearch/opensearch-performance-analyzer/plugin-stats-metadata (No such file or directory)
- at java.base/java.io.FileInputStream.open0(Native Method) ~[?:?]
- at java.base/java.io.FileInputStream.open(FileInputStream.java:213) ~[?:?]
- at java.base/java.io.FileInputStream.<init>(FileInputStream.java:152) ~[?:?]
- at java.base/java.io.FileInputStream.<init>(FileInputStream.java:106) ~[?:?]
- at org.opensearch.performanceanalyzer.commons.collectors.StatsCollector.loadMetadata(StatsCollector.java:158) ~[performance-analyzer-commons-2.0.0.jar:?]
- at org.opensearch.performanceanalyzer.commons.collectors.StatsCollector.instance(StatsCollector.java:73) ~[performance-analyzer-commons-2.0.0.jar:?]
- at org.opensearch.performanceanalyzer.PerformanceAnalyzerPlugin.scheduleRcaCollectors(PerformanceAnalyzerPlugin.java:264) ~[opensearch-performance-analyzer-3.0.0.0.jar:3.0.0.0]
- at org.opensearch.performanceanalyzer.PerformanceAnalyzerPlugin.<init>(PerformanceAnalyzerPlugin.java:213) ~[opensearch-performance-analyzer-3.0.0.0.jar:3.0.0.0]
- at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) [?:?]
- at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) [?:?]
- at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) [?:?]
- at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:840) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:788) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:582) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.node.Node.<init>(Node.java:533) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.node.Node.<init>(Node.java:460) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:243) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:243) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:405) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-3.0.0.jar:3.0.0]
- at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-3.0.0.jar:3.0.0]
- at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.0.0.jar:3.0.0]
- [2025-06-12T13:29:31,749][INFO ][o.o.p.c.SearchBackPressureStatsCollector] [node1] SearchBackPressureStatsCollector started
- [2025-06-12T13:29:32,357][INFO ][o.o.i.r.ReindexModulePlugin] [node1] ReindexPlugin reloadSPI called
- [2025-06-12T13:29:32,358][INFO ][o.o.i.r.ReindexModulePlugin] [node1] Unable to find any implementation for RemoteReindexExtension
- [2025-06-12T13:29:32,467][INFO ][o.o.j.JobSchedulerPlugin ] [node1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
- [2025-06-12T13:29:32,469][INFO ][o.o.j.JobSchedulerPlugin ] [node1] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
- [2025-06-12T13:29:32,471][INFO ][o.o.j.JobSchedulerPlugin ] [node1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
- [2025-06-12T13:29:32,475][INFO ][o.o.j.JobSchedulerPlugin ] [node1] Loaded scheduler extension: checkBatchJobTaskStatus, index: .ml_commons_task_polling_job
- [2025-06-12T13:29:32,476][INFO ][o.o.j.JobSchedulerPlugin ] [node1] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
- [2025-06-12T13:29:32,478][INFO ][o.o.j.JobSchedulerPlugin ] [node1] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
- [2025-06-12T13:29:32,480][INFO ][o.o.j.JobSchedulerPlugin ] [node1] Loaded scheduler extension: async-query-scheduler, index: .async-query-scheduler
- [2025-06-12T13:29:32,519][INFO ][o.o.p.PluginsService ] [node1] loaded module [aggs-matrix-stats]
- [2025-06-12T13:29:32,520][INFO ][o.o.p.PluginsService ] [node1] loaded module [analysis-common]
- [2025-06-12T13:29:32,520][INFO ][o.o.p.PluginsService ] [node1] loaded module [cache-common]
- [2025-06-12T13:29:32,520][INFO ][o.o.p.PluginsService ] [node1] loaded module [geo]
- [2025-06-12T13:29:32,520][INFO ][o.o.p.PluginsService ] [node1] loaded module [ingest-common]
- [2025-06-12T13:29:32,521][INFO ][o.o.p.PluginsService ] [node1] loaded module [ingest-geoip]
- [2025-06-12T13:29:32,521][INFO ][o.o.p.PluginsService ] [node1] loaded module [ingest-user-agent]
- [2025-06-12T13:29:32,521][INFO ][o.o.p.PluginsService ] [node1] loaded module [lang-expression]
- [2025-06-12T13:29:32,521][INFO ][o.o.p.PluginsService ] [node1] loaded module [lang-mustache]
- [2025-06-12T13:29:32,522][INFO ][o.o.p.PluginsService ] [node1] loaded module [lang-painless]
- [2025-06-12T13:29:32,522][INFO ][o.o.p.PluginsService ] [node1] loaded module [mapper-extras]
- [2025-06-12T13:29:32,522][INFO ][o.o.p.PluginsService ] [node1] loaded module [opensearch-dashboards]
- [2025-06-12T13:29:32,522][INFO ][o.o.p.PluginsService ] [node1] loaded module [parent-join]
- [2025-06-12T13:29:32,523][INFO ][o.o.p.PluginsService ] [node1] loaded module [percolator]
- [2025-06-12T13:29:32,523][INFO ][o.o.p.PluginsService ] [node1] loaded module [rank-eval]
- [2025-06-12T13:29:32,523][INFO ][o.o.p.PluginsService ] [node1] loaded module [reindex]
- [2025-06-12T13:29:32,523][INFO ][o.o.p.PluginsService ] [node1] loaded module [repository-url]
- [2025-06-12T13:29:32,524][INFO ][o.o.p.PluginsService ] [node1] loaded module [search-pipeline-common]
- [2025-06-12T13:29:32,524][INFO ][o.o.p.PluginsService ] [node1] loaded module [systemd]
- [2025-06-12T13:29:32,524][INFO ][o.o.p.PluginsService ] [node1] loaded module [transport-netty4]
- [2025-06-12T13:29:32,525][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-alerting]
- [2025-06-12T13:29:32,525][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-anomaly-detection]
- [2025-06-12T13:29:32,525][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-asynchronous-search]
- [2025-06-12T13:29:32,525][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-cross-cluster-replication]
- [2025-06-12T13:29:32,526][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-custom-codecs]
- [2025-06-12T13:29:32,526][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-flow-framework]
- [2025-06-12T13:29:32,526][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-geospatial]
- [2025-06-12T13:29:32,526][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-index-management]
- [2025-06-12T13:29:32,527][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-job-scheduler]
- [2025-06-12T13:29:32,527][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-knn]
- [2025-06-12T13:29:32,527][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-ltr]
- [2025-06-12T13:29:32,527][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-ml]
- [2025-06-12T13:29:32,528][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-neural-search]
- [2025-06-12T13:29:32,528][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-notifications]
- [2025-06-12T13:29:32,528][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-notifications-core]
- [2025-06-12T13:29:32,528][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-observability]
- [2025-06-12T13:29:32,528][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-performance-analyzer]
- [2025-06-12T13:29:32,529][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-reports-scheduler]
- [2025-06-12T13:29:32,529][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-security]
- [2025-06-12T13:29:32,529][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-security-analytics]
- [2025-06-12T13:29:32,530][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-skills]
- [2025-06-12T13:29:32,530][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-sql]
- [2025-06-12T13:29:32,530][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [opensearch-system-templates]
- [2025-06-12T13:29:32,530][INFO ][o.o.p.PluginsService ] [node1] loaded plugin [query-insights]
- [2025-06-12T13:29:32,557][INFO ][o.o.s.OpenSearchSecurityPlugin] [node1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
- [2025-06-12T13:29:32,606][WARN ][stderr ] [node1] WARNING: A restricted method in java.lang.foreign.Linker has been called
- [2025-06-12T13:29:32,606][WARN ][stderr ] [node1] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
- [2025-06-12T13:29:32,606][WARN ][stderr ] [node1] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
- [2025-06-12T13:29:32,769][INFO ][o.o.e.NodeEnvironment ] [node1] using [1] data paths, mounts [[/ (/dev/mapper/vg-lv)]], net usable_space [84.2gb], net total_space [96gb], types [ext4]
- [2025-06-12T13:29:32,770][INFO ][o.o.e.NodeEnvironment ] [node1] heap size [4gb], compressed ordinary object pointers [true]
- [2025-06-12T13:29:32,980][INFO ][o.o.n.Node ] [node1] node name [node1], node ID [9UpzsqELQKmAGc_cnqGSIg], cluster name [logsystem], roles [ingest, remote_cluster_client, data, cluster_manager]
- [2025-06-12T13:29:33,109][INFO ][o.o.e.ExtensionsManager ] [node1] ExtensionsManager initialized
- [2025-06-12T13:29:36,797][ERROR][o.o.p.c.j.GCMetrics ] [node1] MX bean missing: G1 Concurrent GC
- [2025-06-12T13:29:39,024][WARN ][stderr ] [node1] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
- [2025-06-12T13:29:39,025][WARN ][stderr ] [node1] SLF4J: Defaulting to no-operation (NOP) logger implementation
- [2025-06-12T13:29:39,026][WARN ][stderr ] [node1] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
- [2025-06-12T13:29:39,155][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Message routing enabled: true
- [2025-06-12T13:29:39,256][WARN ][o.o.s.c.Salt ] [node1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
- [2025-06-12T13:29:39,258][WARN ][o.o.s.c.Salt ] [node1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
- [2025-06-12T13:29:39,263][INFO ][o.o.s.f.SecurityFilter ] [node1] <NONE> indices are made immutable.
- [2025-06-12T13:29:39,755][INFO ][o.o.l.b.LTRCircuitBreakerService] [node1] Registered memory breaker.
- [2025-06-12T13:29:39,867][INFO ][o.o.r.m.c.i.SdkClientFactory] [node1] Using local opensearch cluster as metadata store.
- [2025-06-12T13:29:40,240][INFO ][o.o.t.b.CircuitBreakerService] [node1] Registered memory breaker.
- [2025-06-12T13:29:41,514][INFO ][o.o.r.m.c.i.SdkClientFactory] [node1] Using local opensearch cluster as metadata store.
- [2025-06-12T13:29:41,549][INFO ][o.o.m.b.MLCircuitBreakerService] [node1] Registered ML memory breaker.
- [2025-06-12T13:29:41,550][INFO ][o.o.m.b.MLCircuitBreakerService] [node1] Registered ML disk breaker.
- [2025-06-12T13:29:41,550][INFO ][o.o.m.b.MLCircuitBreakerService] [node1] Registered ML native memory breaker.
- [2025-06-12T13:29:41,724][INFO ][o.r.Reflections ] [node1] Reflections took 104 ms to scan 1 urls, producing 31 keys and 81 values
- [2025-06-12T13:29:41,766][INFO ][o.r.Reflections ] [node1] Reflections took 4 ms to scan 1 urls, producing 3 keys and 5 values
- [2025-06-12T13:29:42,435][WARN ][o.o.s.p.SQLPlugin ] [node1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
- [2025-06-12T13:29:43,125][INFO ][o.o.p.i.c.s.QueryInsightsService] [node1] Setting query insights index template priority to [1847]
- [2025-06-12T13:29:44,464][INFO ][o.o.t.NettyAllocator ] [node1] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=512kb, suggested_max_allocation_size=512kb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=2mb}]
- [2025-06-12T13:29:44,475][INFO ][o.o.s.s.t.SSLConfig ] [node1] SSL dual mode is disabled
- [2025-06-12T13:29:44,746][INFO ][o.o.d.DiscoveryModule ] [node1] using discovery type [zen] and seed hosts providers [settings]
- [2025-06-12T13:29:45,995][WARN ][o.o.g.DanglingIndicesState] [node1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
- [2025-06-12T13:29:46,777][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,781][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,781][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,781][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,782][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,782][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,782][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,783][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,787][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,787][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,788][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,788][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,788][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,789][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,789][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,789][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,790][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,790][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,791][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,791][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,791][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,792][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,797][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,797][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,798][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,798][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,799][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,799][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,799][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,800][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,800][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,800][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,801][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,801][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,801][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,802][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,802][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,802][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,802][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,805][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,805][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,806][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,806][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,806][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,807][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,807][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,807][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,808][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,808][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,808][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:46,809][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node1] Fail to read queue capacity via reflection
- [2025-06-12T13:29:47,703][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [node1] PerformanceAnalyzer Enabled: true
- [2025-06-12T13:29:47,810][INFO ][o.o.n.Node ] [node1] initialized
- [2025-06-12T13:29:47,810][INFO ][o.o.n.Node ] [node1] starting ...
- [2025-06-12T13:29:47,894][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [windows_logtype.json] log type
- [2025-06-12T13:29:47,895][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [waf_logtype.json] log type
- [2025-06-12T13:29:47,895][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [vpcflow_logtype.json] log type
- [2025-06-12T13:29:47,896][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [test_windows_logtype.json] log type
- [2025-06-12T13:29:47,896][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [s3_logtype.json] log type
- [2025-06-12T13:29:47,897][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [others_web_logtype.json] log type
- [2025-06-12T13:29:47,897][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [others_proxy_logtype.json] log type
- [2025-06-12T13:29:47,898][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [others_macos_logtype.json] log type
- [2025-06-12T13:29:47,898][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [others_compliance_logtype.json] log type
- [2025-06-12T13:29:47,899][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [others_cloud_logtype.json] log type
- [2025-06-12T13:29:47,899][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [others_apt_logtype.json] log type
- [2025-06-12T13:29:47,900][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [others_application_logtype.json] log type
- [2025-06-12T13:29:47,900][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [okta_logtype.json] log type
- [2025-06-12T13:29:47,901][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [network_logtype.json] log type
- [2025-06-12T13:29:47,901][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [netflow_logtype.json] log type
- [2025-06-12T13:29:47,901][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [m365_logtype.json] log type
- [2025-06-12T13:29:47,902][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [linux_logtype.json] log type
- [2025-06-12T13:29:47,902][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [gworkspace_logtype.json] log type
- [2025-06-12T13:29:47,903][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [github_logtype.json] log type
- [2025-06-12T13:29:47,903][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [dns_logtype.json] log type
- [2025-06-12T13:29:47,904][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [cloudtrail_logtype.json] log type
- [2025-06-12T13:29:47,904][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [azure_logtype.json] log type
- [2025-06-12T13:29:47,905][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [apache_access_logtype.json] log type
- [2025-06-12T13:29:47,905][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node1] Loaded [ad_ldap_logtype.json] log type
- [2025-06-12T13:29:48,103][INFO ][o.o.t.TransportService ] [node1] publish_address {192.168.100.9:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {192.168.100.9:9300}
- [2025-06-12T13:29:48,106][INFO ][o.o.t.TransportService ] [node1] Remote clusters initialized successfully.
- [2025-06-12T13:29:49,705][INFO ][o.o.b.BootstrapChecks ] [node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
- [2025-06-12T13:29:49,724][INFO ][o.o.c.c.Coordinator ] [node1] cluster UUID [CgVl5LKQQnO4jQSlFu2uaQ]
- [2025-06-12T13:29:50,253][WARN ][o.o.d.HandshakingTransportAddressConnector] [node1] handshake failed for [connectToRemoteMasterNode[192.168.100.10:9300]]
- org.opensearch.transport.RemoteTransportException: [node2][192.168.100.10:9300][internal:transport/handshake]
- Caused by: org.opensearch.OpenSearchException: java.security.cert.CertificateParsingException: URI name must include scheme:*.local.arpa
- at org.opensearch.security.transport.DefaultInterClusterRequestEvaluator.isInterClusterRequest(DefaultInterClusterRequestEvaluator.java:174) ~[?:?]
- at org.opensearch.security.transport.SecurityRequestHandler.addAdditionalContextValues(SecurityRequestHandler.java:358) ~[?:?]
- at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:140) ~[?:?]
- at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:869) ~[?:?]
- at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:120) ~[?:?]
- at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:44) ~[?:?]
- at org.opensearch.performanceanalyzer.transport.RTFPerformanceAnalyzerTransportRequestHandler.messageReceived(RTFPerformanceAnalyzerTransportRequestHandler.java:63) ~[?:?]
- at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
- at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
- at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
- at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1515) ~[?:?]
- at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378) ~[?:?]
- at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427) ~[?:?]
- at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
- at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
- at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
- at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[?:?]
- at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
- at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[?:?]
- at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[?:?]
- at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[?:?]
- at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
- at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
- at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
- at java.lang.Thread.run(Thread.java:1583) [?:?]
- Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: certificate_parsing_exception: URI name must include scheme:*.local.arpa
- at sun.security.x509.X509CertImpl.getExtensionIfParseable(X509CertImpl.java:1154) ~[?:?]
- at sun.security.x509.X509CertImpl.getSubjectAlternativeNames(X509CertImpl.java:1316) ~[?:?]
- at org.opensearch.security.transport.DefaultInterClusterRequestEvaluator.isInterClusterRequest(DefaultInterClusterRequestEvaluator.java:123) ~[?:?]
- at org.opensearch.security.transport.SecurityRequestHandler.addAdditionalContextValues(SecurityRequestHandler.java:358) ~[?:?]
- at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:140) ~[?:?]
- at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:869) ~[?:?]
- at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:120) ~[?:?]
- at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:44) ~[?:?]
- at org.opensearch.performanceanalyzer.transport.RTFPerformanceAnalyzerTransportRequestHandler.messageReceived(RTFPerformanceAnalyzerTransportRequestHandler.java:63) ~[?:?]
- at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-3.0.0.jar:3.0.0]
- at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
- at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
- at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
- at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1515) ~[?:?]
- at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378) ~[?:?]
- at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427) ~[?:?]
- at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
- at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
- at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
- at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
- at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[?:?]
- at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
- at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[?:?]
- at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[?:?]
- at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[?:?]
- at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
- at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
- at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
- at java.lang.Thread.run(Thread.java:1583) [?:?]
- [2025-06-12T13:29:50,340][INFO ][o.o.c.s.ClusterManagerService] [node1] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper, count:3 and sample tasks: elected-as-cluster-manager ([1] nodes joined)[{node1}{9UpzsqELQKmAGc_cnqGSIg}{rZJyxp2zQnSSeRBNXQW4xA}{192.168.100.9}{192.168.100.9:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 12, version: 107, delta: cluster-manager node changed {previous [], current [{node1}{9UpzsqELQKmAGc_cnqGSIg}{rZJyxp2zQnSSeRBNXQW4xA}{192.168.100.9}{192.168.100.9:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
- [2025-06-12T13:29:51,381][INFO ][o.o.c.s.ClusterApplierService] [node1] cluster-manager node changed {previous [], current [{node1}{9UpzsqELQKmAGc_cnqGSIg}{rZJyxp2zQnSSeRBNXQW4xA}{192.168.100.9}{192.168.100.9:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 12, version: 107, reason: Publication{term=12, version=107}
- [2025-06-12T13:29:51,409][INFO ][o.o.t.i.IndexManagement ] [node1] Candidate custom result indices are empty.
- [2025-06-12T13:29:51,411][INFO ][o.o.t.i.IndexManagement ] [node1] Candidate custom result indices are empty.
- [2025-06-12T13:29:51,412][INFO ][o.o.t.c.ClusterEventListener] [node1] Cluster is not recovered yet.
- [2025-06-12T13:29:51,571][INFO ][o.o.i.i.ManagedIndexCoordinator] [node1] Cache cluster manager node onClusterManager time: 1749749391571
- [2025-06-12T13:29:51,595][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node1] Config override setting update called with empty string. Ignoring.
- [2025-06-12T13:29:51,636][INFO ][o.o.d.PeerFinder ] [node1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node1}{9UpzsqELQKmAGc_cnqGSIg}{rZJyxp2zQnSSeRBNXQW4xA}{192.168.100.9}{192.168.100.9:9300}{dimr}{shard_indexing_pressure_enabled=true}]
- [2025-06-12T13:29:51,654][WARN ][o.o.c.r.a.AllocationService] [node1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
- [2025-06-12T13:29:51,658][INFO ][o.o.h.AbstractHttpServerTransport] [node1] publish_address {192.168.100.9:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {192.168.100.9:9200}
- [2025-06-12T13:29:51,659][INFO ][o.o.n.Node ] [node1] started
- [2025-06-12T13:29:51,678][INFO ][o.o.s.c.ConfigurationRepository] [node1] Will attempt to create index .opendistro_security and default configs if they are absent
- [2025-06-12T13:29:51,684][INFO ][o.o.s.OpenSearchSecurityPlugin] [node1] 0 OpenSearch Security modules loaded so far: []
- [2025-06-12T13:29:51,687][INFO ][o.o.s.c.ConfigurationRepository] [node1] Background init thread started. Install default config?: true
- [2025-06-12T13:29:51,688][INFO ][o.o.s.c.ConfigurationRepository] [node1] Wait for cluster to be available ...
- [2025-06-12T13:29:51,712][WARN ][o.o.c.r.a.AllocationService] [node1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
- [2025-06-12T13:29:52,240][INFO ][o.o.t.c.HashRing ] [node1] Node added: [9UpzsqELQKmAGc_cnqGSIg]
- [2025-06-12T13:29:52,250][INFO ][o.o.t.c.HashRing ] [node1] Add data node to version hash ring: 9UpzsqELQKmAGc_cnqGSIg
- [2025-06-12T13:29:52,256][INFO ][o.o.t.c.HashRing ] [node1] All nodes with known version: {9UpzsqELQKmAGc_cnqGSIg=ADNodeInfo{version=3.0.0, isEligibleDataNode=true}}
- [2025-06-12T13:29:52,257][INFO ][o.o.t.c.HashRing ] [node1] Rebuild hash ring for realtime with cooldown, nodeChangeEvents size 0
- [2025-06-12T13:29:52,257][INFO ][o.o.t.c.HashRing ] [node1] Build version hash ring successfully
- [2025-06-12T13:29:52,260][INFO ][o.o.t.c.ADDataMigrator ] [node1] Start migrating AD data
- [2025-06-12T13:29:52,260][INFO ][o.o.t.c.ADDataMigrator ] [node1] AD job index doesn't exist, no need to migrate
- [2025-06-12T13:29:52,261][INFO ][o.o.t.c.ClusterEventListener] [node1] Init version hash ring successfully
- [2025-06-12T13:29:52,293][INFO ][o.o.c.s.ClusterManagerTaskThrottler] [node1] Starting cluster manager throttling as all nodes are higher than or equal to 2.5.0
- [2025-06-12T13:29:52,294][INFO ][o.o.g.GatewayService ] [node1] recovered [6] indices into cluster_state
- [2025-06-12T13:29:52,298][INFO ][o.o.s.l.LogTypeService ] [node1] Loading builtin types!
- [2025-06-12T13:29:52,302][INFO ][o.o.s.l.LogTypeService ] [node1] Indexing [432] fieldMappingDocs from logTypes: 24
- [2025-06-12T13:29:52,353][WARN ][o.o.c.r.a.AllocationService] [node1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
- [2025-06-12T13:29:52,363][WARN ][o.o.s.SecurityAnalyticsPlugin] [node1] Failed to initialize LogType config index and builtin log types
- [2025-06-12T13:29:52,603][INFO ][o.o.s.l.LogTypeService ] [node1] Loading builtin types!
- [2025-06-12T13:29:52,606][INFO ][o.o.s.l.LogTypeService ] [node1] Indexing [432] fieldMappingDocs from logTypes: 24
- [2025-06-12T13:29:52,610][INFO ][o.o.s.l.LogTypeService ] [node1] Loading builtin types!
- [2025-06-12T13:29:52,611][INFO ][o.o.s.l.LogTypeService ] [node1] Indexing [432] fieldMappingDocs from logTypes: 24
- [2025-06-12T13:29:52,616][INFO ][o.o.s.i.DetectorIndexManagementService] [node1] info deleteOldIndices
- [2025-06-12T13:29:52,624][INFO ][o.o.s.i.DetectorIndexManagementService] [node1] No Old Correlation Indices to delete
- [2025-06-12T13:29:52,627][INFO ][o.o.s.i.DetectorIndexManagementService] [node1] info deleteOldIndices
- [2025-06-12T13:29:52,628][INFO ][o.o.s.i.DetectorIndexManagementService] [node1] No Old IOC Findings Indices to delete
- [2025-06-12T13:29:53,519][ERROR][o.o.s.a.BackendRegistry ] [node1] Not yet initialized (you may need to run securityadmin)
- [2025-06-12T13:29:53,712][ERROR][o.o.s.a.BackendRegistry ] [node1] Not yet initialized (you may need to run securityadmin)
- [2025-06-12T13:29:53,723][ERROR][o.o.s.a.BackendRegistry ] [node1] Not yet initialized (you may need to run securityadmin)
- [2025-06-12T13:29:53,733][ERROR][o.o.s.a.BackendRegistry ] [node1] Not yet initialized (you may need to run securityadmin)
- [2025-06-12T13:29:53,906][INFO ][o.o.p.PluginsService ] [node1] PluginService:onIndexModule index:[.opendistro_security/yTD1r-h1T1SGcewPecxQhw]
- [2025-06-12T13:29:54,346][INFO ][o.o.p.PluginsService ] [node1] PluginService:onIndexModule index:[top_queries-2025.06.12-11036/gwVF0buTTP68VhGIP1JULw]
- [2025-06-12T13:29:54,415][INFO ][o.o.p.PluginsService ] [node1] PluginService:onIndexModule index:[.plugins-ml-config/VgcuIgm_S_CNE8WAENdPog]
- [2025-06-12T13:29:54,474][INFO ][o.o.p.PluginsService ] [node1] PluginService:onIndexModule index:[.kibana_1/FxdBh_B_TTWLG6R-lnZ7yw]
- [2025-06-12T13:29:55,258][INFO ][o.o.s.c.ConfigurationRepository] [node1] Index .opendistro_security already exists
- [2025-06-12T13:29:55,259][INFO ][o.o.s.c.ConfigurationRepository] [node1] Node started, try to initialize it. Wait for at least yellow cluster state....
- [2025-06-12T13:29:55,669][WARN ][o.o.c.r.a.AllocationService] [node1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
- [2025-06-12T13:29:55,675][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'config' with /etc/opensearch/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
- [2025-06-12T13:29:55,704][ERROR][o.o.s.a.BackendRegistry ] [node1] Not yet initialized (you may need to run securityadmin)
- [2025-06-12T13:29:55,708][ERROR][o.o.s.a.BackendRegistry ] [node1] Not yet initialized (you may need to run securityadmin)
- [2025-06-12T13:29:55,711][ERROR][o.o.s.a.BackendRegistry ] [node1] Not yet initialized (you may need to run securityadmin)
- [2025-06-12T13:29:55,714][ERROR][o.o.s.a.BackendRegistry ] [node1] Not yet initialized (you may need to run securityadmin)
- [2025-06-12T13:29:55,821][INFO ][o.o.p.PluginsService ] [node1] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/mkUT769SQv6GxWKpapqRdw]
- [2025-06-12T13:29:55,866][INFO ][o.o.p.PluginsService ] [node1] PluginService:onIndexModule index:[top_queries-2025.06.04-11007/uZbqC9KtQx21A0KHxhURAw]
- [2025-06-12T13:29:55,912][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id config, skipping update.
- [2025-06-12T13:29:55,912][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'roles' with /etc/opensearch/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
- [2025-06-12T13:29:55,953][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id roles, skipping update.
- [2025-06-12T13:29:55,954][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'rolesmapping' with /etc/opensearch/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
- [2025-06-12T13:29:55,967][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id rolesmapping, skipping update.
- [2025-06-12T13:29:55,967][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'internalusers' with /etc/opensearch/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
- [2025-06-12T13:29:55,979][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id internalusers, skipping update.
- [2025-06-12T13:29:55,980][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'actiongroups' with /etc/opensearch/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
- [2025-06-12T13:29:55,986][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id actiongroups, skipping update.
- [2025-06-12T13:29:55,986][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'tenants' with /etc/opensearch/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
- [2025-06-12T13:29:55,990][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id tenants, skipping update.
- [2025-06-12T13:29:55,991][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'nodesdn' with /etc/opensearch/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
- [2025-06-12T13:29:55,998][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id nodesdn, skipping update.
- [2025-06-12T13:29:55,998][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'allowlist' with /etc/opensearch/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
- [2025-06-12T13:29:56,001][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id allowlist, skipping update.
- [2025-06-12T13:29:56,002][INFO ][o.o.s.s.ConfigHelper ] [node1] Will update 'audit' with /etc/opensearch/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
- [2025-06-12T13:29:56,031][INFO ][o.o.s.s.ConfigHelper ] [node1] Index .opendistro_security already contains doc with id audit, skipping update.
- [2025-06-12T13:29:56,221][WARN ][o.o.c.r.a.AllocationService] [node1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
- [2025-06-12T13:29:56,289][INFO ][o.o.c.r.a.AllocationService] [node1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[top_queries-2025.06.04-11007][0]]]).
- [2025-06-12T13:29:56,407][WARN ][o.o.c.r.a.AllocationService] [node1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
- [2025-06-12T13:29:56,445][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing on REST API is enabled.
- [2025-06-12T13:29:56,446][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
- [2025-06-12T13:29:56,446][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing on Transport API is enabled.
- [2025-06-12T13:29:56,447][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
- [2025-06-12T13:29:56,447][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing of request body is enabled.
- [2025-06-12T13:29:56,447][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Bulk requests resolution is disabled during request auditing.
- [2025-06-12T13:29:56,447][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Index resolution is enabled during request auditing.
- [2025-06-12T13:29:56,447][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Sensitive headers exclusion from auditing is enabled.
- [2025-06-12T13:29:56,447][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing requests from kibanaserver users is disabled.
- [2025-06-12T13:29:56,448][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing request headers <NONE> is disabled.
- [2025-06-12T13:29:56,448][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing request url params <NONE> is disabled.
- [2025-06-12T13:29:56,453][WARN ][o.o.s.a.r.AuditMessageRouter] [node1] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
- [2025-06-12T13:29:56,453][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing of external configuration is disabled.
- [2025-06-12T13:29:56,453][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing of internal configuration is enabled.
- [2025-06-12T13:29:56,453][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing only metadata information for read request is enabled.
- [2025-06-12T13:29:56,453][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing will watch {} for read requests.
- [2025-06-12T13:29:56,454][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing read operation requests from kibanaserver users is disabled.
- [2025-06-12T13:29:56,454][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing only metadata information for write request is enabled.
- [2025-06-12T13:29:56,454][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing diffs for write requests is disabled.
- [2025-06-12T13:29:56,454][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing write operation requests from kibanaserver users is disabled.
- [2025-06-12T13:29:56,454][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Auditing will watch <NONE> for write requests.
- [2025-06-12T13:29:56,455][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] .opendistro_security is used as internal security index.
- [2025-06-12T13:29:56,455][INFO ][o.o.s.a.i.AuditLogImpl ] [node1] Internal index used for posting audit logs is null
- [2025-06-12T13:29:56,455][INFO ][o.o.s.c.ConfigurationRepository] [node1] Hot-reloading of audit configuration is enabled
- [2025-06-12T13:29:56,456][INFO ][o.o.s.c.ConfigurationRepository] [node1] Node 'node1' initialized
- [2025-06-12T13:30:01,650][INFO ][o.o.m.a.MLModelAutoReDeployer] [node1] Index not found, not performing auto reloading!
- [2025-06-12T13:30:01,652][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [node1] Starting ML sync up job...
- [2025-06-12T13:30:11,692][INFO ][o.o.m.c.MLSyncUpCron ] [node1] Skipping sync up job - ML model index not found
- [2025-06-12T13:30:11,701][INFO ][o.o.m.c.MLSyncUpCron ] [node1] ML configuration already initialized, no action needed
- [2025-06-12T13:30:21,696][INFO ][o.o.m.c.MLSyncUpCron ] [node1] Skipping sync up job - ML model index not found
- [2025-06-12T13:30:31,698][INFO ][o.o.m.c.MLSyncUpCron ] [node1] Skipping sync up job - ML model index not found
- [2025-06-12T13:30:41,701][INFO ][o.o.m.c.MLSyncUpCron ] [node1] Skipping sync up job - ML model index not found
- [2025-06-12T13:30:51,705][INFO ][o.o.m.c.MLSyncUpCron ] [node1] Skipping sync up job - ML model index not found
- [2025-06-12T13:31:01,709][INFO ][o.o.m.c.MLSyncUpCron ] [node1] Skipping sync up job - ML model index not found
Advertisement
Add Comment
Please, Sign In to add comment