# Kibana Insertion of Sensitive Information into Log File (ESA-2023-27) An issue was discovered by Elastic whereby sensitive information may be recorded in Kibana logs in the event of an error or in the event where debug level logging is enabled in Kibana. Elastic has released Kibana 8.11.2 which resolves this issue. The messages recorded in the log may contain: * Account credentials for the `kibana_system` user, API Keys, and credentials of Kibana end-users. * Elastic Security package policy objects which can contain private keys, bearer token, and sessions of 3rd-party integrations. * Authorization headers, client secrets, local file paths, and stack traces The issue may occur in any Kibana instance running an affected version (which are listed below) that could potentially receive an unexpected error when communicating to Elasticsearch causing it to include sensitive data into Kibana error logs. It could also occur under specific circumstances when debug level logging is enabled in Kibana. **Note:** It was found that the fix for ESA-2023-25 in Kibana 8.11.1 for a similar issue was incomplete. ## Affected Versions: Kibana versions on or after 7.13.0 and before 7.17.16. Kibana versions on or after 8.0.0 and before 8.11.2. ## Solutions and Mitigations: The issue is resolved in Kibana 8.11.2 and 7.17.16 ### Elastic Cloud The following mitigations have been implemented in Elastic Cloud: * We have purged sensitive data that was logged from our monitoring environment. * We have adjusted the rulesets of our redaction solution so that no new instances of sensitive information are logged in our monitoring environment and in customer’s monitoring clusters. For Elastic Cloud customers with **self-managed monitoring clusters** , affected logs should be reviewed for any potentially sensitive data and if deemed necessary, follow up actions such as purging sensitive data from logs and rotating any potentially exposed credentials should be performed. As additional mitigation, Elastic Cloud customers on affected versions of Kibana are advised to upgrade to 8.11.3. ### Self-Managed Users on affected versions of Kibana in self-managed, ECE, or ECK, are advised to upgrade to 8.11.2 or later. Affected logs should be reviewed for any potentially sensitive data and if deemed necessary, follow up actions such as purging sensitive data from logs and rotating any potentially exposed credentials should be performed. For users that cannot upgrade, see the section _Preventing Ingest of Sensitive Information_ for mitigation actions that can be applied. ## Reviewing Logs for Sensitive Information This section describes how to review logs to identify instances of potentially sensitive information in your logs. ### Elastic Cloud customers with self-managed monitoring clusters Affected log lines can be identified with the use of the queries described in _Appendix A. Search Queries._ against the data views that contain Kibana logs on the monitoring cluster ### Self-Managed with Elastic Stack Monitoring If you are using Elastic Stack to ingest Kibana logs, affected log lines can be identified with the use of the queries that are defined in _Appendix A. Search Queries_ against the data views that contain Kibana logs on the monitoring cluster: * If Elastic Agent Collection is configured: `logs-kibana.*-default` * If Filebeat Collection is configured: `filebeat-{version}` , where `{version}` is the installed Kibana version. ### Self-Managed without Elastic Stack Monitoring If you are not ingesting Kibana logs in the Elastic Stack, you can search the log files on disk for occurrences where the following terms are present: * “MaxListenersExceededWarning”, “IncomingMessage” * Any of the following message substrings * "policy info exists as" * "package policy exists as" * "policy responses exists as" * "endpoint metadata exists as" * "Trusted Apps" * "EP Exceptions" * "EP Event Filters" * "Value List Meta Data" * "Timeline Events" * "Preview of field's value", "--token", ("plugins.securitySolution.ruleExecution" or "plugins.ruleRegistry”) * “error thrown getting the access token” * Any of the following fields * "kibana.usageCounters.results.meta.meta.request.options.headers.authorization" * "kibana.usageCounters.results.meta.meta.request.options.headers.X-Elastic-App-Auth" * "kibana.usageCounters.results.meta.meta.request.options.headers.x-elastic-app-auth" * "kibana.usageCounters.results.meta.meta.request.options.headers.es-client-authentication" * "kibana.usageCounters.results.meta.meta.request.options.headers.ES-Client-Authentication" * "kibana.usageCounters.results.meta.meta.request.params.headers.authorization" * "kibana.usageCounters.results.meta.meta.request.params.headers.X-Elastic-App-Auth" * "kibana.usageCounters.results.meta.meta.request.params.headers.x-elastic-app-auth" * "kibana.usageCounters.results.meta.meta.request.params.headers.es-client-authentication" * "kibana.usageCounters.results.meta.meta.request.params.headers.ES-Client-Authentication" * "http.request.headers.authorization" * "http.request.headers.x-elastic-app-auth" * "http.request.headers.X-Elastic-App-Auth" * "http.request.headers.es-client-authentication" * "http.request.headers.ES-Client-Authentication" * "error.cause.meta.meta.request.options.headers.authorization" * "error.cause.meta.meta.request.options.headers.x-elastic-app-auth" * "error.cause.meta.meta.request.options.headers.X-Elastic-App-Auth" * "error.cause.meta.meta.request.options.headers.es-client-authentication" * "error.cause.meta.meta.request.options.headers.ES-Client-Authentication" * "meta.meta.request.params.headers.authorization" * "meta.meta.request.params.headers.x-elastic-app-auth" * "meta.meta.request.params.headers.X-Elastic-App-Auth" * "meta.meta.request.params.headers.es-client-authentication" * "meta.meta.request.params.headers.ES-Client-Authentication" * "meta.meta.request.options.headers.authorization" * "meta.meta.request.options.headers.x-elastic-app-auth" * "meta.meta.request.options.headers.X-Elastic-App-Auth" * "meta.meta.request.options.headers.es-client-authentication" * "meta.meta.request.options.headers.ES-Client-Authentication" * "left.error.meta.meta.request.params.headers.authorization" * "left.error.meta.meta.request.params.headers.x-elastic-app-auth" * "left.error.meta.meta.request.params.headers.X-Elastic-App-Auth" * "left.error.meta.meta.request.params.headers.es-client-authentication" * "left.error.meta.meta.request.params.headers.ES-Client-Authentication" * "left.error.meta.meta.request.options.headers.authorization" * "left.error.meta.meta.request.options.headers.x-elastic-app-auth" * "left.error.meta.meta.request.options.headers.X-Elastic-App-Auth" * "left.error.meta.meta.request.options.headers.es-client-authentication" * "left.error.meta.meta.request.options.headers.ES-Client-Authentication" * "error.originalError.meta.meta.request.params.headers.authorization" * "error.originalError.meta.meta.request.params.headers.x-elastic-app-auth" * "error.originalError.meta.meta.request.params.headers.X-Elastic-App-Auth" * "error.originalError.meta.meta.request.params.headers.es-client-authentication" * "error.originalError.meta.meta.request.params.headers.ES-Client-Authentication" * "error.originalError.meta.meta.request.options.headers.authorization" * "error.originalError.meta.meta.request.options.headers.x-elastic-app-auth" * "error.originalError.meta.meta.request.options.headers.X-Elastic-App-Auth" * "error.originalError.meta.meta.request.options.headers.es-client-authentication" * "error.originalError.meta.meta.request.options.headers.ES-Client-Authentication" * "error.meta.meta.request.params.headers.authorization" * "error.meta.meta.request.params.headers.x-elastic-app-auth" * "error.meta.meta.request.params.headers.X-Elastic-App-Auth" * "error.meta.meta.request.params.headers.es-client-authentication" * "error.meta.meta.request.params.headers.ES-Client-Authentication" * "error.meta.meta.request.options.headers.authorization" * "error.meta.meta.request.options.headers.x-elastic-app-auth" * "error.meta.meta.request.options.headers.X-Elastic-App-Auth" * "error.meta.meta.request.options.headers.es-client-authentication" * "error.meta.meta.request.options.headers.ES-Client-Authentication" * "error.meta.meta.meta.request.params.headers.authorization" * "error.meta.meta.meta.request.params.headers.x-elastic-app-auth" * "error.meta.meta.meta.request.params.headers.X-Elastic-App-Auth" * "error.meta.meta.meta.request.params.headers.es-client-authentication" * "error.meta.meta.meta.request.params.headers.ES-Client-Authentication" * "error.meta.meta.meta.request.options.headers.authorization" * "error.meta.meta.meta.request.options.headers.x-elastic-app-auth" * "error.meta.meta.meta.request.options.headers.X-Elastic-App-Auth" * "error.meta.meta.meta.request.options.headers.es-client-authentication" * "error.meta.meta.meta.request.options.headers.ES-Client-Authentication" * "error.cause.meta.meta.request.params.headers.authorization" * "error.cause.meta.meta.request.params.headers.x-elastic-app-auth" * "error.cause.meta.meta.request.params.headers.X-Elastic-App-Auth" * "error.cause.meta.meta.request.params.headers.es-client-authentication" * "error.cause.meta.meta.request.params.headers.ES-Client-Authentication" ### ECE On ECE, affected log lines can be identified with the use of the queries that are defined in _Appendix A. Search Queries_ against the `cluster-logs-*` indices in the logging and metrics cluster. ### ECK If Stack Monitoring is enabled, affected log lines can be identified with the use of the queries that are defined in _Appendix A. Search Queries_ against the data views that contain Kibana logs on the monitoring cluster: * If Elastic Agent Collection is configured: `logs-kibana.*-default` * If Filebeat Collection is configured: `filebeat-{version}` , where `{version}` is the installed Kibana version. ## Remediating Sensitive Information If the review reveals that credentials have been included in the logs, the following remediation actions are recommended Installation Type | Remediation Actions ---|--- Elastic Cloud | `found-internal-kibana4-server credentials` credentials are being rotated by Elastic. End user credentials can be changed using the Management > Users UI in Kibana or the Change Password API. Archive Docker RPM/DEB | kibana_system credentials can be rotated using one of the available methods if a native user is used. If service account tokens are used, you need to delete the logged token and create a new token End user credentials can be changed using the Management > Users UI in Kibana or the Change Password API. ECE | `found-internal-kibana4-server` credentials can be rotated using advanced editor: 1\. Go to advanced editor 2\. Find ".resources.kibana[0].plan.cluster_topology[0].kibana.system_settings" 3\. Set "elasticsearch_password" to a random string 4\. submit End user credentials can be changed using the Management > Users UI in Kibana or the Change Password API. ECK | Kibana system user credentials can be rotated by either of the following options: \- Delete `$KIBANA_NAME-kibana-user` secret in the Kibana K8s namespace and restart the ECK operator. \- Delete `$KIBANA_NAME-kibana-user` and `$NAMESPACE-$KIBANA_NAME-kibana-user` secrets in the Kibana K8s namespace End user credentials can be changed using the Management > Users UI in Kibana or the Change Password API. ## Preventing Ingest of Sensitive Information in Logs For users that cannot upgrade, the following mitigation actions can be applied to prevent ingestion of sensitive information into an elasticsearch logging cluster. The following mitigations are for customers who have enabled the monitoring features for their clusters as outlined in this guide. If you are using Elastic Stack to ingest Kibana logs, use ECE, or use ECK with stack monitoring enabled , you can use an Ingest Pipeline to redact sensitive information from being ingested in a logging cluster. 1. Create the remediation pipeline: PUT _ingest/pipeline/redact-esa–2023-27 { "description": "Prevent Insertion of Sensitive Information into Log File (ESA-2023-27)", "processors": [ { "set": { "description": "Redact credentials in MaxListenersExceededWarning errors message field", "if": "ctx.message != null && ctx.message.startsWith('MaxListenersExceededWarning') && ctx.message.contains('IncomingMessage')", "field": "message", "value": "MaxListenersExceededWarning: Possible EventEmitter memory leak detected. There are aborted listeners added to [IncomingMessage]. Use emitter.setMaxListeners() to increase limit", "ignore_empty_value": true } }, { "set": { "description": "Redact credentials in MaxListenersExceededWarning errors event.original field", "if": "ctx.event.original != null && ctx.event.original.startsWith('MaxListenersExceededWarning') && ctx.event.original.contains('IncomingMessage')", "field": "event.original", "value": "MaxListenersExceededWarning: Possible EventEmitter memory leak detected. There are aborted listeners added to [IncomingMessage]. Use emitter.setMaxListeners() to increase limit", "ignore_empty_value": true } }, { "remove": { "description": "Kibana log record metadata with sensitive HTTP headers", "field": [ "http.request.headers.authorization", "http.request.headers.x-elastic-app-auth", "http.request.headers.X-Elastic-App-Auth", "http.request.headers.es-client-authentication", "http.request.headers.ES-Client-Authentication", "error.cause.meta.meta.request.options.headers.authorization", "error.cause.meta.meta.request.options.headers.x-elastic-app-auth", "error.cause.meta.meta.request.options.headers.X-Elastic-App-Auth", "error.cause.meta.meta.request.options.headers.es-client-authentication", "error.cause.meta.meta.request.options.headers.ES-Client-Authentication", "meta.meta.request.params.headers.authorization", "meta.meta.request.params.headers.x-elastic-app-auth", "meta.meta.request.params.headers.X-Elastic-App-Auth", "meta.meta.request.params.headers.es-client-authentication", "meta.meta.request.params.headers.ES-Client-Authentication", "meta.meta.request.options.headers.authorization", "meta.meta.request.options.headers.x-elastic-app-auth", "meta.meta.request.options.headers.X-Elastic-App-Auth", "meta.meta.request.options.headers.es-client-authentication", "meta.meta.request.options.headers.ES-Client-Authentication", "left.error.meta.meta.request.params.headers.authorization", "left.error.meta.meta.request.params.headers.x-elastic-app-auth", "left.error.meta.meta.request.params.headers.X-Elastic-App-Auth", "left.error.meta.meta.request.params.headers.es-client-authentication", "left.error.meta.meta.request.params.headers.ES-Client-Authentication", "left.error.meta.meta.request.options.headers.authorization", "left.error.meta.meta.request.options.headers.x-elastic-app-auth", "left.error.meta.meta.request.options.headers.X-Elastic-App-Auth", "left.error.meta.meta.request.options.headers.es-client-authentication", "left.error.meta.meta.request.options.headers.ES-Client-Authentication", "error.originalError.meta.meta.request.params.headers.authorization", "error.originalError.meta.meta.request.params.headers.x-elastic-app-auth", "error.originalError.meta.meta.request.params.headers.X-Elastic-App-Auth", "error.originalError.meta.meta.request.params.headers.es-client-authentication", "error.originalError.meta.meta.request.params.headers.ES-Client-Authentication", "error.originalError.meta.meta.request.options.headers.authorization", "error.originalError.meta.meta.request.options.headers.x-elastic-app-auth", "error.originalError.meta.meta.request.options.headers.X-Elastic-App-Auth", "error.originalError.meta.meta.request.options.headers.es-client-authentication", "error.originalError.meta.meta.request.options.headers.ES-Client-Authentication", "error.meta.meta.request.params.headers.authorization", "error.meta.meta.request.params.headers.x-elastic-app-auth", "error.meta.meta.request.params.headers.X-Elastic-App-Auth", "error.meta.meta.request.params.headers.es-client-authentication", "error.meta.meta.request.params.headers.ES-Client-Authentication", "error.meta.meta.request.options.headers.authorization", "error.meta.meta.request.options.headers.x-elastic-app-auth", "error.meta.meta.request.options.headers.X-Elastic-App-Auth", "error.meta.meta.request.options.headers.es-client-authentication", "error.meta.meta.request.options.headers.ES-Client-Authentication", "error.meta.meta.meta.request.params.headers.authorization", "error.meta.meta.meta.request.params.headers.x-elastic-app-auth", "error.meta.meta.meta.request.params.headers.X-Elastic-App-Auth", "error.meta.meta.meta.request.params.headers.es-client-authentication", "error.meta.meta.meta.request.params.headers.ES-Client-Authentication", "error.meta.meta.meta.request.options.headers.authorization", "error.meta.meta.meta.request.options.headers.x-elastic-app-auth", "error.meta.meta.meta.request.options.headers.X-Elastic-App-Auth", "error.meta.meta.meta.request.options.headers.es-client-authentication", "error.meta.meta.meta.request.options.headers.ES-Client-Authentication", "error.cause.meta.meta.request.params.headers.authorization", "error.cause.meta.meta.request.params.headers.x-elastic-app-auth", "error.cause.meta.meta.request.params.headers.X-Elastic-App-Auth", "error.cause.meta.meta.request.params.headers.es-client-authentication", "error.cause.meta.meta.request.params.headers.ES-Client-Authentication" ], "ignore_missing": true } }, { "foreach": { "field": "kibana.usageCounters.results", "processor": { "remove": { "field": [ "_ingest._value.meta.meta.request.options.headers.authorization", "_ingest._value.meta.meta.request.options.headers.X-Elastic-App-Auth", "_ingest._value.meta.meta.request.options.headers.x-elastic-app-auth", "_ingest._value.meta.meta.request.options.headers.es-client-authentication", "_ingest._value.meta.meta.request.options.headers.ES-Client-Authentication", "_ingest._value.meta.meta.request.params.headers.authorization", "_ingest._value.meta.meta.request.params.headers.X-Elastic-App-Auth", "_ingest._value.meta.meta.request.params.headers.x-elastic-app-auth", "_ingest._value.meta.meta.request.params.headers.es-client-authentication", "_ingest._value.meta.meta.request.params.headers.ES-Client-Authentication" ], "ignore_missing": true } }, "ignore_missing": true } }, { "dot_expander": { "description": "Expand 'event.original'", "field": "event.original" } }, { "set": { "field": "message", "value": "policy info exists as [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('policy info exists as')" } }, { "set": { "field": "event.original", "value": "policy info exists as [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('policy info exists as')" } }, { "set": { "field": "message", "value": "package policy exists as [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('package policy exists as')" } }, { "set": { "field": "event.original", "value": "package policy exists as [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('package policy exists as')" } }, { "set": { "field": "message", "value": "policy responses exists as [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('policy responses exists as')" } }, { "set": { "field": "event.original", "value": "policy responses exists as [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('policy responses exists as')" } }, { "set": { "field": "message", "value": "endpoint metadata exists as [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('endpoint metadata exists as')" } }, { "set": { "field": "event.original", "value": "endpoint metadata exists as [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('endpoint metadata exists as')" } }, { "set": { "field": "message", "value": "Trusted Apps: [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('trusted apps:')" } }, { "set": { "field": "event.original", "value": "Trusted Apps: [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('trusted apps:')" } }, { "set": { "field": "message", "value": "EP Exceptions: [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('ep exceptions:')" } }, { "set": { "field": "event.original", "value": "EP Exceptions: [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('ep exceptions:')" } }, { "set": { "field": "message", "value": "EP Event Filters: [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('ep event filters:')" } }, { "set": { "field": "event.original", "value": "EP Event Filters: [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('ep event filters:')" } }, { "set": { "field": "message", "value": "Value List Meta Data: [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('value list meta data:')" } }, { "set": { "field": "event.original", "value": "Value List Meta Data: [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('value list meta data:')" } }, { "set": { "field": "message", "value": "Timeline Events: [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('timeline events:')" } }, { "set": { "field": "event.original", "value": "Timeline Events: [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('timeline events:')" } }, { "set": { "field": "message", "value": "Error decoding JSON: [redacted]", "if": "ctx.message != null && ctx.error != null && ctx.error.message != null && ctx.error.message.toLowerCase().contains('error decoding json') && ctx.message.toLowerCase().contains('plugins.securitysolution.telemetry_events')" } }, { "set": { "field": "event.original", "value": "Error decoding JSON: [redacted]", "if": "ctx.event.original != null && ctx.error != null && ctx.error.message != null && ctx.error.message.toLowerCase().contains('error decoding json') && ctx.event.original.toLowerCase().contains('plugins.securitysolution.telemetry_events')" } }, { "set": { "description": "Redacting sensitive info from errors in message field", "field": "message", "value": "error thrown getting the access token [redacted]", "if": "ctx.message != null && ctx.message.toLowerCase().contains('error thrown getting the access token')" } }, { "set": { "description": "Redacting sensitive info from errors in event.original field", "field": "event.original", "value": "error thrown getting the access token [redacted]", "if": "ctx.event.original != null && ctx.event.original.toLowerCase().contains('error thrown getting the access token')" } }, { "set": { "description": "Redacting sensitive info from message field in Security Solution RuleExecution or RuleRegistry docs", "field": "message", "value": "[redacted]", "if": "ctx.log.logger != null && ctx.message != null && (ctx.log.logger.toLowerCase() == 'plugins.securitySolution.ruleexecution' || ctx.log.logger.toLowerCase() == 'plugins.ruleregistry') && ctx.message.toLowerCase().contains('preview of field\\'s value')" } }, { "set": { "description": "Redacting sensitive info from event.original field in Security Solution RuleExecution or RuleRegistry docs", "field": "event.original", "value": "[redacted]", "if": "ctx.log.logger != null && ctx.event.original != null && (ctx.log.logger.toLowerCase() == 'plugins.securitySolution.ruleexecution' || ctx.log.logger.toLowerCase() == 'plugins.ruleregistry') && ctx.event.original.toLowerCase().contains('preview of field\\'s value')" } }, { "set": { "description": "Redacting sensitive info from error.stack_trace field in Security Solution RuleExecution or RuleRegistry docs", "field": "error.stack_trace", "value": "[redacted]", "if": "ctx.log.logger != null && ctx.error != null && ctx.error.stack_trace != null && (ctx.log.logger.toLowerCase() == 'plugins.securitySolution.ruleexecution' || ctx.log.logger.toLowerCase() == 'plugins.ruleregistry') && ctx.error.stack_trace.toLowerCase().contains('preview of field\\'s value')" } }, { "set": { "description": "Redacting sensitive info from error.message field in Security Solution RuleExecution or RuleRegistry docs", "field": "error.message", "value": "[redacted]", "if": "ctx.log.logger != null && ctx.error != null && ctx.error.message != null && (ctx.log.logger.toLowerCase() == 'plugins.securitySolution.ruleexecution' || ctx.log.logger.toLowerCase() == 'plugins.ruleregistry') && ctx.error.message.toLowerCase().contains('preview of field\\'s value')" } } ] } 2. Identify the indexes or datastreams that your Kibana logs are being ingested into. 3. Identify whether these indices have a default pipeline configured by checking for the index.default_pipeline setting on the indices themselves and the templates. a. If the indices DO NOT configure a default pipeline: * Configure the template to add the `index.default_pipeline` setting pointing to the example `redact-esa-2023-27` pipeline. * Update the settings of any existing indices to use `index.default_pipeline: redact-esa-2023-27`. b. If the indices DO configure a default pipeline: * Identify the name of the pipeline from the setting. * `GET /_ingest/pipeline/{pipeline_name}` * Add the processor `redact-esa-2023-27` as an additional pipeline processor to the body of the default pipeline * Use PUT _ingest/pipeline API to update the default pipeline to the new configuration ... { "pipeline": { "name": "redact-esa-2023-27" } } … The instructions above give API usage examples. Custom pipelines can also be edited using the Ingest Pipelines UI. See Tutorial: Transform data with custom ingest pipelines | Fleet and Elastic Agent Guide [8.11] | Elastic ### Self-Managed without Elastic Stack Monitoring If you are not ingesting Kibana logs in the Elastic Stack, you should limit access to the directory where Kibana log files are stored. ### ECE logging cluster instructions This assumes the above ingest pipeline (`redact-esa–2023-27`) has been created in the logging cluster as described above. By default the ECE logging cluster ships with a single template for the `cluster-logs-*` indices and: * It does not have a “settings.default_pipeline” * There are no other templates with `“GET”` You should confirm this remains the case for you (`“GET _template/cluster-logs-*”`, look for `“default_pipeline”`) and (`“GET _template/”`, compare the `“index_patterns”` fields) respectively, and contact support if there are. Otherwise, you will then create an overlapping template just specifying the ingest pipeline: PUT _template/cluster-logs-esa–2023-27-redaction { "index_patterns" : ["cluster-logs-*"], "order" : 99, "settings": {"default_pipeline": "redact-esa–2023-27"} } The next time the index rolls over, the message field will be replaced by `“[redacted]”` for offending messages. You should then redact sensitive information from already ingested logs as described below. ## Redacting Sensitive Information already ingested in logs The same Ingest Pipeline can be used in conjunction with the Update By Query API in order to redact already recorded sensitive information in logs. This is applicable for all deployments which includes: Elastic Cloud customers with self-managed monitoring clusters, Self-Managed, ECE, ECK. Repeat the following Updaye By Queries for all Kibana logging indices that contain sensitive information. ### Update Query #1 POST //_update_by_query?pipeline=redact-esa–2023-27&pretty=true&allow_no_indices=false&wait_for_completion=false&expand_wildcards=all&conflicts=proceed { "query": { "bool": { "must": [ {"match": {"message": "MaxListenersExceededWarning"}}, {"match": {"message": "IncomingMessage"}} ], "must_not" : [ { "term": { "_tier": { "value": "data_frozen" } } } ] } } } ###Update Query #2 POST //_update_by_query?pipeline=redact-esa–2023-27&pretty=true&allow_no_indices=false&wait_for_completion=false&expand_wildcards=all&conflicts=proceed { "query": { "bool": { "must_not" : [ { "term": { "_tier": { "value": "data_frozen" } } } ], "should": [ { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.authorization"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.x-elastic-app-auth"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.es-client-authentication"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.ES-Client-Authentication"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.authorization"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.x-elastic-app-auth"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.es-client-authentication"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.ES-Client-Authentication"}} ], "minimum_should_match": 1 } } } ### Update Query #3 POST //_update_by_query?pipeline=redact-esa–2023-27&pretty=true&allow_no_indices=false&wait_for_completion=false&expand_wildcards=all&conflicts=proceed { "query": { "bool": { "must_not" : [ { "term": { "_tier": { "value": "data_frozen" } } } ], "should": [ { "match": { "message": "policy info exists as"}}, { "match": { "message": "package policy exists as"}}, { "match": { "message": "policy responses exists as"}}, { "match": { "message": "endpoint metadata exists as"}}, { "match": { "message": "trusted apps:"}}, { "match": { "message": "ep exceptions:"}}, { "match": { "message": "ep event filters:"}}, { "match": { "message": "value list meta data:"}}, { "match": { "message": "timeline events:"}} ], "minimum_should_match": 1 } } } ### Update Query #4 POST //_update_by_query?pipeline=redact-esa–2023-27&pretty=true&allow_no_indices=false&wait_for_completion=false&expand_wildcards=all&conflicts=proceed { "query": { "bool": { "must_not" : [ { "term": { "_tier": { "value": "data_frozen" } } } ], "must": [ {"match": {"message": "Preview of field's value"}}, {"match": {"message": "--token"}} ], "should": [ {"match": {"log.logger": "plugins.securitySolution.ruleExecution"}}, {"match": {"log.logger": "plugins.ruleRegistry"}} ], "minimum_should_match": 1 } } } ### Update Query #5 POST //_update_by_query?pipeline=redact-esa–2023-27&pretty=true&allow_no_indices=false&wait_for_completion=false&expand_wildcards=all&conflicts=proceed { "query": { "bool": { "must_not" : [ { "term": { "_tier": { "value": "data_frozen" } } } ], "must": [ {"match_phrase": {"message": "error thrown getting the access token"}} ] } } } ### Update Query #6 POST //_update_by_query?pipeline=redact-esa–2023-27&pretty=true&allow_no_indices=false&wait_for_completion=false&expand_wildcards=all&conflicts=proceed { "query": { "bool": { "must_not" : [ { "term": { "_tier": { "value": "data_frozen" } } } ], "should": [ {"exists": {"field": "http.request.headers.authorization"}}, {"exists": {"field": "http.request.headers.x-elastic-app-auth"}}, {"exists": {"field": "http.request.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "http.request.headers.es-client-authentication"}}, {"exists": {"field": "http.request.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "error.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "error.meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "error.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "error.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.ES-Client-Authentication"}} ], "minimum_should_match": 1 } } } To check that the update_by_query task completes successfully: GET /_tasks/{taskId from above request} — **Severity:** CVSSv3.1, 8.0 (High) AV:A/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H **CVE ID:** CVE-2023-46675 ## Appendix ### A. Search Queries GET //_search { "query": { "bool": { "must": [ {"match": {"message": "MaxListenersExceededWarning"}}, {"match": {"message": "IncomingMessage"}} ] } } } GET //_search { "query": { "bool": { "should": [ { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.authorization"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.x-elastic-app-auth"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.es-client-authentication"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.options.headers.ES-Client-Authentication"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.authorization"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.x-elastic-app-auth"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.es-client-authentication"}}, { "exists": { "field": "kibana.usageCounters.results.meta.meta.request.params.headers.ES-Client-Authentication"}} ], "minimum_should_match": 1 } } } GET //_search { "query": { "bool": { "should": [ { "match": { "message": "policy info exists as"}}, { "match": { "message": "package policy exists as"}}, { "match": { "message": "policy responses exists as"}}, { "match": { "message": "endpoint metadata exists as"}}, { "match": { "message": "trusted apps:"}}, { "match": { "message": "ep exceptions:"}}, { "match": { "message": "ep event filters:"}}, { "match": { "message": "value list meta data:"}}, { "match": { "message": "timeline events:"}} ], "minimum_should_match": 1 } } } GET //_search { "query": { "bool": { "must": [ {"match": {"message": "Preview of field's value"}}, {"match": {"message": "--token"}} ], "should": [ {"match": {"log.logger": "plugins.securitySolution.ruleExecution"}}, {"match": {"log.logger": "plugins.ruleRegistry"}} ], "minimum_should_match": 1 } } } GET //_search { "query": { "bool": { "must": [ {"match": { "log.logger": "plugins.actions.email" }}, {"match": { "message": "token" }} ] } } } GET //_search { "query": { "bool": { "should": [ {"exists": {"field": "http.request.headers.authorization"}}, {"exists": {"field": "http.request.headers.x-elastic-app-auth"}}, {"exists": {"field": "http.request.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "http.request.headers.es-client-authentication"}}, {"exists": {"field": "http.request.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "error.cause.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "left.error.meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "left.error.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "error.originalError.meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "error.originalError.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "error.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "error.meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "error.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "error.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "error.meta.meta.meta.request.params.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.authorization"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.es-client-authentication"}}, {"exists": {"field": "error.meta.meta.meta.request.options.headers.ES-Client-Authentication"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.authorization"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.x-elastic-app-auth"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.X-Elastic-App-Auth"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.es-client-authentication"}}, {"exists": {"field": "error.cause.meta.meta.request.params.headers.ES-Client-Authentication"}} ], "minimum_should_match": 1 } } }