## Purpose Kasten now supports exporting metrics from the embedded Prometheus to external backends using Prometheus's remote_write capability. This feature supports the collection, aggregation, and visualization of cluster and multi-cluster metrics in monitoring tools like Grafana Cloud and Datadog. Enterprises managing complex Kubernetes environments with Kasten often require centralized observability across multiple clusters. A common challenge has been the inability to aggregate Prometheus metrics from Kastendeployments into external monitoring platforms running outside the clusters. To address this, enhancements have been introduced to expose Kasten's Prometheus metrics externally, enabling seamless integration with centralized visualization and alerting tools. This improvement simplifies multi-cluster monitoring and provides more granular operational insights. For more information on Grafana and Kasten, refer to: KB4635: How to Install Grafana with K10 Dashboard and DataSources Pre-provisioned ##### **Key Benefits** * Centralized Observability: Aggregate metrics from multiple clusters in a single backend. * Flexible Integration: Works with any Prometheus-compatible remote backend. * Cluster Identification: Metrics are labeled with a unique cluster identifier for filtering and aggregation. ## Solution ### **Enabling remote_write in Embedded Prometheus Instance** To facilitate monitoring for multiple clusters and enable customers to aggregate metrics in external backends like Grafana Cloud, we enabled Prometheus's remote_write capability in the Kasten deployment. This was accomplished via Helm values, allowing cluster administrators to specify remote endpoints, authentication details, and optional filtering or relabeling rules directly in their deployment configuration. By setting these values during installation or upgrade, users can ensure that their Prometheus instance forwards metrics to a centralized backend. A key aspect of this configuration is cluster labeling. Through Helm values, users will specify a unique identifier for each cluster (e.g., clusterID), which is then applied as a label to all exported metrics, via external_labels. This labeling ensures that when metrics from multiple clusters arrive at the remote backend, they can be filtered, grouped, and visualized by cluster. Additionally, Kasten automatically attempts to add a unique cluster UID label (cluster_uid) to each metric, providing an extra layer of identification and helping distinguish between clusters even if their names are similar. ### **Kasten End-User Guide:** Enable Prometheus Remote Write #### **Prerequisites** * kubectl and helm cli tools configured * From your preferred data visualization/aggregation service (Grafana Cloud, Datadog, etc.), obtain a remote_write url which will be the destination for where we send our Prometheus cluster metrics. You will also need to obtain authentication credentials including an ID (username) and password (API key). #### **Configuration Options** ##### **Option 1:** Basic Authentication _(Username/Password)_ Create a new file with the name: `**k10-remote-write-values.yaml**`. Paste in the code below and replace the placeholder values. Give the cluster a name of your choosing in the `**clusterName**` entry. clusterName: "" # REQUIRED: Enter a cluster name prometheus: server: remote_write: - url: basic_auth: username: # Remote backend ID password: # Remote backend API key Copy **Note:** `clusterName` is required when remote_write is enabled; the deployment will fail without it. clusterName will appear as the cluster_name on all exported metrics. A unique cluster_uid label is automatically added. ##### **Option 2:** Bearer Token File Authentication _(Recommended for Sensitive Credentials)_ For enhanced security, you can store credentials in a Kubernetes Secret and reference them using bearer_token_file: 1. Create a Kubernetes Secret with your bearer token: kubectl create secret generic prometheus-remote-write-token \ --from-literal=token= \ -n kasten-io Copy 2. Create `**k10-remote-write-values.yaml**` with `extraSecretMounts`: clusterName: "" # REQUIRED: Enter a cluster name prometheus: server: remote_write: - url: bearer_token_file: /etc/prometheus-secrets/token extraSecretMounts: - name: remote-write-token mountPath: /etc/prometheus-secrets secretName: prometheus-remote-write-token readOnly: true Copy **Alternative:** If your secret has multiple keys, use `subPath` _(note line 10)_ : clusterName: "" prometheus: server: remote_write: - url: bearer_token_file: /etc/prometheus-secrets/bearer-token extraSecretMounts: - name: remote-write-token mountPath: /etc/prometheus-secrets subPath: token # Mount specific key from secret secretName: prometheus-remote-write-token readOnly: true Copy #### **Optional Configurations** ##### Basic Metric Filtering Optionally add basic filtering like: prometheus: server: remote_write: - url: # ... auth configuration ... metricDrop: - k10_debug_.* Copy Advanced: Manual Cluster UID Override (optional): If you're using GitOps, helm template, or have RBAC restrictions that prevent namespace lookups, you can manually specify the cluster UID: clusterName: "" prometheus: server: clusterUIDOverride: "" # Optional: manual cluster UID remote_write: - url: basic_auth: username: password: Copy #### **Applying Configuration** ##### If Kasten is already installed: Apply the new remote_write configuration with the following helm command: helm upgrade k10 kasten/k10 -n kasten-io -f k10-remote-write-values.yaml Copy Or if using a local chart: helm upgrade k10 ./helm/k10 -n kasten-io -f k10-remote-write-values.yaml Copy ##### If installing Kasten for the first time: helm repo add kasten https://charts.kasten.io/ helm repo update # Replace "8.0.12" with your desired Kasten version helm install k10 kasten/k10 -n kasten-io \ -f k10-remote-write-values.yaml \ --set global.image.tag="8.0.12" \ --create-namespace Copy #### **Verification** 1. Once pods are ready, verify that the new values were applied correctly: helm get values k10 -n kasten-io Copy Look for your new remote_write configurations in the output: clusterName: prod-k10-demo prometheus: server: remote_write: - url: ... 2. Verify the Prometheus ConfigMap was updated: kubectl get configmap k10-k10-prometheus-config -n kasten-io -o yaml | less Copy Output should include: external_labels: cluster_uid: # Auto-detected from namespace UID (if lookup succeeded) cluster_name: remote_write: - url: https://... basic_auth: username: 3. Ensure Prometheus is running: kubectl get pods -n kasten-io | grep prometheus Copy Expected output: prometheus-server-xxxxx-xxxxx 2/2 Running 0 2m 4. Verify remote_write is working: kubectl logs -n kasten-io -l component=server,app=prometheus -c prometheus-server --tail=50 Copy Look for these indicators (may need to adjust tail value in kubectl command): * `**Starting WAL watcher**` with your Grafana URL * `**Starting scraped metadata watcher**` for your remote endpoint * `**Done replaying WAL**` (indicates historical data was sent) 5. Confirm that metrics are available in your remote backend system. You may need to wait 1-2 minutes for the data to appear. We can also query for it locally: curl http:///api/v1/series?match[]=k10_backup_duration_seconds Copy ## More Information #### **ConfigMap Editing Considerations** While it is technically possible to manually edit the Prometheus ConfigMap using kubectl edit, this approach is generally discouraged. Manual edits can be overwritten by subsequent Helm upgrades, leading to lost configuration and potential outages. Additionally, ConfigMap editing can be error-prone, especially when dealing with remote_write or relabeling rules. Instead, all configuration should flow through Helm values, ensuring consistency, safety, and easy rollback or upgrades. The end result of both options is the same. Both allow Prometheus to export metrics to a remote backend with proper labeling. kubectl edit is a manual, in-place edit of the live config. The Helm values approach is a declarative, template-driven config applied at install or upgrade. For quick, temporary, or emergency configuration changes, we might use kubectl edit; however, for production, it's recommended to use Helm values. Configs are stored in version control and are automatically applied and updated during every install/upgrade. #### **Frequently Asked Questions** * **Can I use any remote backend? ** Yes, any backend that supports Prometheus remote_write (Grafana Cloud, Datadog, etc.). * **Is manual ConfigMap editing supported? ** Technically possible but not recommended. Use Helm for safer, version-controlled configuration. * **What happens to old unlabeled metrics? ** Only new metrics will have cluster labels. For consistent querying, consider purging old data or accepting a transition period. * **How do I secure my remote_write credentials? ** Store sensitive credentials, such as API keys, in Kubernetes secrets and reference them in your Helm values file. * **Can I filter which metrics are exported via remote_write? ** Yes, you can use the metricDrop option in your Helm values to filter out specific metrics you don't want sent to the remote backend. * **Can I export metrics to multiple endpoints at once? ** Yes, you can specify multiple remote_write endpoints in your Helm values to export metrics to more than one remote backend simultaneously. * **What should I do if I see errors in Prometheus logs related to remote_write? ** Check your remote backend URL, credentials, and network connectivity. Refer to Prometheus log messages for specific error details. #### **Additional Resources** * Veeam KB4635: How to Install Grafana with K10 Dashboard and Data Sources Pre-provisioned * Grafana Documentation - prometheus.remote_write