Elasticsearch node crashing
One of the elasticsearch pods is in a crash cycle. Stack trace:
{"timestamp": "2021-10-21T16:27:52+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"type": "server", "timestamp": "2021-10-21T16:27:56,909Z", "level": "WARN", "component": "o.e.c.s.SettingsModule", "cluster.name": "condor", "node.name": "condor-es-elasticsearch-1", "message": "\n*************************************************************************************\nFound index level settings on node level configuration.\n\nSince elasticsearch 5.x index level settings can NOT be set on the nodes \nconfiguration like the elasticsearch.yaml, in system properties or command line \narguments.In order to upgrade all indices the settings must be updated via the \n/${index}/_settings API. Unless all settings are dynamic all indices must be closed \nin order to apply the upgradeIndices created in the future should use index templates \nto set default values. \n\nPlease ensure all required values are updated on all indices by executing: \n\ncurl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{\n \"index.default_pipeline\" : \"htcondor-igwn\"\n}'\n*************************************************************************************\n" }
{"type": "server", "timestamp": "2021-10-21T16:27:56,913Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "condor", "node.name": "condor-es-elasticsearch-1", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: node settings must not contain any index level settings",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.12.0.jar:7.12.0]",
"at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.12.0.jar:7.12.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.12.0.jar:7.12.0]",
"Caused by: java.lang.IllegalArgumentException: node settings must not contain any index level settings",
"at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:132) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:396) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.12.0.jar:7.12.0]",
"... 6 more"] }
uncaught exception in thread [main]
java.lang.IllegalArgumentException: node settings must not contain any index level settings
at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:132)
at org.elasticsearch.node.Node.<init>(Node.java:396)
at org.elasticsearch.node.Node.<init>(Node.java:278)
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
at org.elasticsearch.cli.Command.main(Command.java:79)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/condor.log
Forum posts clarify what is meant by:
java.lang.IllegalArgumentException: node settings must not contain any index level settings
The problem is, I don't explicitly have anything like that. Here's the node configuration from the deployment yaml:
config:
indices.query.bool.max_clause_count: "8192"
search.max_buckets: "100000"
node.master: true
node.data: true
node.ingest: true
node.ml: true
The only place I'm explicitly doing anything with this is in the index settings for htcondor-igwn
to specify the ingest node pipeline:
{
"index.blocks.read_only_allow_delete": "false",
"index.priority": "1",
"index.query.default_field": [
"*"
],
"index.refresh_interval": "1s",
"index.write.wait_for_active_shards": "1",
"index.routing.allocation.include._tier_preference": "data_content",
"index.default_pipeline": "htcondor-igwn",
"index.number_of_replicas": "1"
}
where I added the line
"index.default_pipeline": "htcondor-igwn",
and all other settings are defaults.
But I don't understand why that would be a node-level configuration. Complicating matters, I cannot seem to get changes to that configuration to save.