How to Install Grafana with K10 Dashboard and Data Sources Pre-provisioned

Purpose Starting in Veeam Kasten for Kubernetes v7.5.0, Grafana will no longer be included by default. This in-depth technical guide will demonstrate the process of installing Grafana on Kubernetes and populating it with Kasten Data. It will allow you to quickly access pre-provisioned Kasten...

Troubleshooting apiservices Unavailability Due to Failed Discovery

Article Applicability This article is relevant for environments where Cilium CNI is in use. If the errors or behavior described below occur in an environment where Cilium CNI is not in use, please create a support case. Challenge In Kubernetes clusters that use Cilium for CNI Container Network...

How to Investigate 'Encrypted Data Event' from Malware Detection

Version Requirement This tool only works for malware inline detection events created by Veeam Backup & Replication 12.1.2 and newer. Previous versions of Veeam Backup & Replication are not supported. Protected Workload Guest OS Requirement This tool only supports investigating Windows-based...

Imagestream Export or Restore Fails With "error writing layer: no space left on device"

Challenge Export Action or Restore Action of OpenShift Imagestream fails with the following error: Failed to exec command in pod: command terminated with exit code 1.stdout: error writing layer: write/var/lib/image//blobs/sha256/: no space left on device Cause During the export or restore of an...

Linux Hardened Repository Unable to Update Immutability or Remove Restore Points due to SGID

Article Applicability The errors in this article may occur for other reasons, this article is specifically relevant to a situation in which these errors occur because the folder where the backup files are stored has an SGID Set-Group ID configured. Example: backupsvc@rhel9$ ls -l total 1 drwxr-s...

Veeam Kasten for Kubernetes - Used Ports

Purpose This article provides information regarding network ports that are used for different internal services of the Veeam Kasten for Kubernetes application. Solution The following information of Veeam Kasten for Kubernetes services and network ports associated with them will help with port...

Veeam Kasten for Kubernetes Instant Recovery with Veeam Backup & Replication vPower NFS datastore

Purpose K10 Instant Recovery uses the Veeam Backup and Replication vPower NFS datastore to provide vSphere First Class Disks for storage. This feature enables "instant" recovery of workloads with Kubernetes. Solution The vPower NFS datastore is a fully functional datastore in the vSphere cluster...

How to Make Veeam Kasten for Kubernetes Grafana Pod rootless

Purpose This article explains how to disable init-container in k10-grafana to run it as rootless. Cause Veeam Kasten for Kubernetes installation provides an instance of Grafana that is deployed automatically and can be used to query metrics from Kasten's Prometheus instance. This grafana pod is r...

There was a problem validating the profile: Repository not found.

Challenge Repository not found error while adding Veeam Repository in Veeam Kasten for Kubernetes despite verifying the repository in the Veeam Backup & Replication Console. Cause The 'Repository Not found' issue can happen in multiple scenarios. Most of the cases are due to repository Permission...

Security Context Constraints(SCC) capabilities setting for Generic Storage Backup and Restore with OCP 4.11 or higher

Purpose This article describes Security Context ConstraintsSCC capabilities that need to be added to use Generic Backup and Restore feature capabilities on OCP 4.11 and higher. Known Issues With OCP 4.11, default security context constraints SCC have changed. Due to this, when using Generic Stora...

How to Clean Up Veeam Kasten for Kubernetes Manual Backups

Purpose This article provides an overview of scenarios where out-of-schedule restorePoints are created and how to identify and remove them. Cause With Veeam Kasten for Kubernetes , there are multiple ways to take backups & exports. A manual backup/export from the application menu. A manual policy...

Report Generation Fails with "lookup prometheus-server-exp: no such host"

Challenge After enabling Veeam Kasten for Kubernetes reports, reports are not generated. When running an on-demand report policy, the following error message is observed within the executor logs: "message":"Post "http://prometheus-server-exp:80/k10/prometheus/api/v1/query": dial tcp: lookup...

Openshift Authentication - Failed to authenticate: oidc: failed to get token: oauth2: cannot fetch token

Challenge Users are unable to login to Veeam Kasten for Kubernetes Dashboard and receive the following error: Failed to authenticate: oidc: failed to get token: oauth2: cannot fetch token: 400 Bad Request Response: "error":"unauthorizedclient","errordescription": "The client is not authorized to...

Application Migration - Best Practices and Prerequisites

Purpose This article explains the prerequisites and best practices that must be considered for successful import during application migration. Solution Veeam Kasten for Kubernetes platform is built to support application migration across clusters, which is an extremely powerful feature that enabl...

Error: could not get determine if bucket supports versioning: Access Denied

Challenge Veeam Kasten for Kubernetes is unable to monitor or refresh the retention period of the data in the S3 bucket and cannot calculate the lock duration based on your retention setting because of this issue. This will show up as an error message in the notification bar: error: can't connect...

Veeam Kasten for Kubernetes Performance Troubleshooting Guide

Purpose This article documents two of the most common performance tuning settings for Veeam Kasten for Kubernetes. Veeam Kasten for Kubernetes uses helm parameters, which can be tuned to adjust performance. The default settings should suit most environments. However, in complex environments,...

Import Job Fails with "No kopia manifests found in the repository"

Challenge Import jobs fail with the error: No kopia manifests found in the repository Full error: "cause": "file": "kasten.io/k10/kio/collections/kopia/operations.go:144", "function": "kasten.io/k10/kio/collections/kopia.GetLatestSnapshot", "linenumber": 144, "message": "No kopia...

How to Exclude Applications from Dashboard & Compliance Reporting

Purpose This article provides steps to exclude system namespaces from the Veeam Kasten for Kubernetes dashboard and compliance reporting. Solution Explanation Veeam Kasten for Kubernetes automatically discovers all application namespaces of the cluster on which it is deployed. Each of these...

Option to Add Veeam Kasten for Kubernetes Does Not Appear in Veeam Backup & Replication

Challenge When attempting to add Veeam Kasten for Kubernetes to Veeam Backup & Replication, the option for Kasten does not appear in the Add Server options. Cause This issue may occur if: The Veeam Kubernetes Service is not running and needs to be started. The Kasten Plug-In is not installed. Thi...

Exports Don't Work After Veeam Kasten for Kubernetes Reinstall

Critical Warning The solution provided in this article results in data removal. These steps should only be followed when testing or evaluating Veeam Kasten for Kubernetes or if the existing data is no longer required. If you have any questions, please contact Support. Challenge Description: Jobs...

Clone Job Fails Due to Security Context Constraints (SCC) Issue

Challenge Cloned-restore jobs time out and eventually fail because an application's security context provides permissions in the original application namespace. This results in the application's pod being in a non-running state. Example: Namespace = quay-enterprise SCC during deployment = anyuid...

Overcoming Failed Snapshots Using Ignore Exceptions

Challenge This article demonstrates how to use the Ignore Exceptions feature to overcome failed snapshots due to workloads not in a ready state. Description: Job failed to be executed Veeam Kasten for Kubernetes has retry logic to overcome service interruptions during any job. If a Backup does no...

DR Restore - Internal error occurred: Could not retrieve artifacts for prefix

Challenge A Veeam Kasten for Kubernetes DR Restore fails with the following error: Internal error occurred: "message":"Could not retrieve artifacts for prefix...

Cloned-Restore Fails on OpenShift Clusters with ImagePull Error

Challenge Cloned-restore jobs time-out and eventually fail due to an ImagePullError. Upon further investigation, the root cause is that the default SA in the target namespace does not have ImageStreams for the pod to pull an image from a private registry. Failed to pull image...

PVC Export Fails Due to Timeout

Challenge The export action will fail if Veeam Kasten for Kubernetes takes more than 45 minutes to complete the job. Checking the export action details, the message below is the clue where Veeam Kasten for Kubernetes hits the timeout for the operation waitWithBackoffwithRetries with the duration...

How to Configure an Air-Gapped Veeam Kasten for Kubernetes Deployment Using JFrog Artifactory

Purpose This article provides a step-by-step approach to configuring a JFrog Artifactory server and installing Veeam Kasten for Kubernetes. This allows for creating an air-gapped installation using a private container registry to install Veeam Kasten for Kubernetes. While this can always be done...

XFS Volume Restore Fails to Mount

Challenge Description: XFS kernel module remembers the old UUID and refuses to mount Cloning an XFS volume and mounting the filesystem of the cloned volume on the same node as the source volume will throw a "duplicate UUID" error or similar errors. The cloning of the volume reflects everything fr...

No action was taken because the import artifacts already existed

Challenge Veeam Kasten for Kubernetes has smart logic built-in to perform intelligent actions. Assuming an Import job on the target cluster was successfully completed, unless there's a new restorepoint from the source cluster, the job won't run. This can happen if: there are no new restorepoints ...

Backup Failing With `Too many snapshots` When Using Longhorn as a Storage Provisioner

Challenge Veeam Kasten for Kubernetes backup action for longhorn volumes fails with the error message: too many snapshots created Cause When integrating with CSI-based volumes, Veeam Kasten for Kubernetes employs VolumeSnapshot resources to create snapshots during backup operations. With Longhorn...

How to Clean Up Veeam Kasten for Kubernetes Orphaned Volumes on GKE k8s 1.28.x

Purpose This guide provides instructions on how to clean up Veeam Kasten for Kubernetes provisioned PVs that are not deleted, causing orphaned volumes in the GKE k8s 1.28 release. It has been observed on GKE running k8s 1.28, Veeam Kasten for Kubernetes provisioned PVs using in-tree provisioner...

How to Configure Veeam Kasten for Kubernetes Dashboard to Use a New Certificate

Purpose This article documents how to troubleshoot and replace the certificate used by the Veeam Kasten for Kubernetes Dashboard. Scenario Example The Veeam Kasten for Kubernetes Dashboard becomes inaccessible due to an expired certificate, resulting in an HTTP ERROR 401 , indicating a lack of...

Veeam Kasten for Kubernetes - vSphere Block Mode Exports Failure With Error 14009

Challenge During vSphere block mode export, in certain situations, it might be observed that the folders/objects get created in the storage bucket. However, the export job remains stuck, and the following error message is seen in the debug logs: "Open virtual disk file failed. The error code is...

How to Patch Veeam Kasten for Kubernetes CR used by Red Hat Marketplace Operator

Purpose This article provides an easy way to patch the Veeam Kasten for Kubernetes operand and make changes to the Veeam Kasten for Kubernetes configuration in an operator-based installation. Solution Veeam Kasten for Kubernetes is available for installation in an OCP cluster through either helm...

Upgrading Veeam Kasten for Kubernetes Fails With Parse Error

Challenge When upgrading to Veeam Kasten for Kubernetes 6.5.3 or higher the upgrade fails with: parse error at k10/templates/v0services.yaml:128: function "continue" not defined Cause This issue is related to the Helm binary version that is installed. Solution To resolve this issue, download the...

Alternate Repository for Veeam Kasten for Kubernetes Container Images

Solution This article documents an alternate location where Kasten's images are stored. Kasten's container images are hosted on the Google Container Registry gcr.io/kasten-images; however, this location may be inaccessible due to country-specific or company-imposed firewalls. If allowing access t...

How to Add a Blueprint Target Using Execution Hooks From the Policy API

Purpose This article explains how to set resources other than namespaces as the hooks for blueprint subjects. By default, hooks sets the namespace as the target for the blueprint unless specified otherwise. Solution Veeam Kasten for Kubernetes supports kanister execution hooks through Policy API...

How to Configure AWS Application Loadbalancer (ALB) for Veeam Kasten for Kubernetes

Purpose This article provides an example of configuring AWS Application Loadbalancer ALB for accessing the Veeam Kasten for Kubernetes UI. Note: This documentation assumes that the ALB ingress controller has already been installed and configured. Solution What is an ALB An Application Load Balanc...

How to Configure Veeam Intelligent Diagnostics Log Location

Purpose This article documents how to change the location where Veeam Intelligent Diagnostics stores the logs it collects--allowing customers to configure both where those logs are stored temporarily on the Veeam Backup Server before being transferred to the Veeam ONE server and where those logs...

Resolving Pending State During Cluster Removal with k10multicluster Tool

Challenge This article helps in finding and deleting long-running finalizers that are waiting on other resources and can cause deletions to appear stuck in the terminating state. Description: When Kubernetes attempts to delete the cluster object that has finalizers "dist.kio.kasten.io/debootstrap...

How to Enable Veeam Kasten for Kubernetes Disaster Recovery using CLI

Purpose This article provides step-by-step instructions on how to enable Veeam Kasten for Kubernetes Disaster Recovery KDR using CLI. Veeam Kasten for Kubernetes Documentation - Disaster Recovery Solution 1. Replace in the following command and run it to create a DR secret with a passphrase: Note...

Veeam Recovery Orchestrator Vulnerability (CVE-2024-29855)

Veeam Software Security Commitment Veeam® is committed to ensuring its products protect customers from potential risks. As part of that commitment, we operate a Vulnerability Disclosure Program VDP for all Veeam products and perform extensive internal code audits. When a vulnerability is...

How to Enable Changed Block Tracking for Guest Cluster on vSphere with Tanzu

Purpose Changed Block Tracking is a VMware feature that tracks changes in virtual disks. Veeam Kasten for Kubernetes uses this feature in vSphere with Tanzu Guest Clusters to efficiently back up Persistent Volumes. Enabling Changed Block Tracking in each Supervisor Cluster where Veeam Kasten for...

Debugging Backups with Longhorn CSI

Challenge Veeam Kasten for Kubernetes backup job that doesn't complete while using Longhorn CSI drivers is never-ending even after the proper installation of CSI snapshotter components and controllers. No errors are noticed generally for this issue. The job waits for the volumesnapshot object in...

How to ‘Transform’ Multiple Resources with Regex

Purpose Veeam Kasten makes use of the 'Transforms' feature to migrate applications between different clusters, between different storage backends, and many other use cases. Transforms enable modifications to Kubernetes resources on restore. The ability to move an application across clusters is an...

How to Properly Off-Board a Namespace From Veeam Kasten for Kubernetes Backups

Purpose This article documents the procedure to properly off-board a namespace in Veeam Kasten for Kubernetes. Solution 1. Check all restorepointcontents for the policy using the following command: Replace with the actual policy name. kubectl get restorepointcontents -l k10.kasten.io/policyName=...

What Data Does Veeam Kasten for Kubernetes Include in the Phone Home Reports

Purpose This article documents what data is included in the phone home reports sent by Veeam Kasten for Kubernetes. Solution The data is divided into the following sections Basic Data Infrastructure Data Run Time Data for Billing Purposes Additional Metrics Profiles Artifact Metrics Basic Data...

Troubleshooting '401 - Unauthorized' or 'x509' Errors When Accessing the Veeam Kasten for Kubernetes Dashboard

Challenge When attempting to access the Veeam Kasten for Kubernetes dashboard the following error occurs despite using the cacertconfigmap.name helm value: 401 - Unauthorized This issue may also manifest as errors related to x509 in the gateway pod or auth-svc pod logs: x509: certificate signed b...

How to Override Volume storageClass During Export

Purpose This article documents how to configure a Veeam Kasten for Kubernetes export parameter to specify the storageClass that is created and used during exports. Solution The ExporterStorageClassName parameter can be used for namespaces that have PersistentVolumeClaims PVCs using a single...

How to Configure EKS Clusters to Use AWS IAM Users/Roles for Veeam Kasten for Kubernetes Access

Purpose Follow this guide to provide appropriate Veeam Kasten for Kubernetes role-based access using AWS IAM users or roles. Solution Description Veeam Kasten for Kubernetes integrates with whatever authentication mechanism customers use to access their Kubernetes clusters. Since EKS natively...

Job For .local Domain Fails When Using Ubuntu-base VMware Backup Proxy

Challenge Backup of VMware VMs fails when using an Ubuntu-based Linux VMware Proxy, and the VMware environment is added to Veeam with an FQDN ending in .local e.g., esxi.somedomain.local. In the Task Log, errors related to NFC connectivity can be found: Processing Error: NFC storage connection is...