There was a problem validating the profile: Repository not found.

Challenge Repository not found error while adding Veeam Repository in Veeam Kasten for Kubernetes despite verifying the repository in the Veeam Backup & Replication Console. Cause The 'Repository Not found' issue can happen in multiple scenarios. Most of the cases are due to repository Permission...

Openshift Authentication - Failed to authenticate: oidc: failed to get token: oauth2: cannot fetch token

Challenge Users are unable to login to Veeam Kasten for Kubernetes Dashboard and receive the following error: Failed to authenticate: oidc: failed to get token: oauth2: cannot fetch token: 400 Bad Request Response: "error":"unauthorizedclient","errordescription": "The client is not authorized to...

Report Generation Fails with "lookup prometheus-server-exp: no such host"

Challenge After enabling Veeam Kasten for Kubernetes reports, reports are not generated. When running an on-demand report policy, the following error message is observed within the executor logs: "message":"Post "http://prometheus-server-exp:80/k10/prometheus/api/v1/query": dial tcp: lookup...

How to Clean Up Veeam Kasten for Kubernetes Manual Backups

Purpose This article provides an overview of scenarios where out-of-schedule restorePoints are created and how to identify and remove them. Cause With Veeam Kasten for Kubernetes , there are multiple ways to take backups & exports. A manual backup/export from the application menu. A manual policy...

Error: could not get determine if bucket supports versioning: Access Denied

Challenge Veeam Kasten for Kubernetes is unable to monitor or refresh the retention period of the data in the S3 bucket and cannot calculate the lock duration based on your retention setting because of this issue. This will show up as an error message in the notification bar: error: can't connect...

Application Migration - Best Practices and Prerequisites

Purpose This article explains the prerequisites and best practices that must be considered for successful import during application migration. Solution Veeam Kasten for Kubernetes platform is built to support application migration across clusters, which is an extremely powerful feature that enabl...

Import Job Fails with "No kopia manifests found in the repository"

Challenge Import jobs fail with the error: No kopia manifests found in the repository Full error: "cause": "file": "kasten.io/k10/kio/collections/kopia/operations.go:144", "function": "kasten.io/k10/kio/collections/kopia.GetLatestSnapshot", "linenumber": 144, "message": "No kopia...

Veeam Kasten for Kubernetes Performance Troubleshooting Guide

Purpose This article documents two of the most common performance tuning settings for Veeam Kasten for Kubernetes. Veeam Kasten for Kubernetes uses helm parameters, which can be tuned to adjust performance. The default settings should suit most environments. However, in complex environments,...

How to Exclude Applications from Dashboard & Compliance Reporting

Purpose This article provides steps to exclude system namespaces from the Veeam Kasten for Kubernetes dashboard and compliance reporting. Solution Explanation Veeam Kasten for Kubernetes automatically discovers all application namespaces of the cluster on which it is deployed. Each of these...

Option to Add Veeam Kasten for Kubernetes Does Not Appear in Veeam Backup & Replication

Challenge When attempting to add Veeam Kasten for Kubernetes to Veeam Backup & Replication, the option for Kasten does not appear in the Add Server options. Cause This issue may occur if: The Veeam Kubernetes Service is not running and needs to be started. The Kasten Plug-In is not installed. Thi...

Exports Don't Work After Veeam Kasten for Kubernetes Reinstall

Critical Warning The solution provided in this article results in data removal. These steps should only be followed when testing or evaluating Veeam Kasten for Kubernetes or if the existing data is no longer required. If you have any questions, please contact Support. Challenge Description: Jobs...

Clone Job Fails Due to Security Context Constraints (SCC) Issue

Challenge Cloned-restore jobs time out and eventually fail because an application's security context provides permissions in the original application namespace. This results in the application's pod being in a non-running state. Example: Namespace = quay-enterprise SCC during deployment = anyuid...

Overcoming Failed Snapshots Using Ignore Exceptions

Challenge This article demonstrates how to use the Ignore Exceptions feature to overcome failed snapshots due to workloads not in a ready state. Description: Job failed to be executed Veeam Kasten for Kubernetes has retry logic to overcome service interruptions during any job. If a Backup does no...

DR Restore - Internal error occurred: Could not retrieve artifacts for prefix

Challenge A Veeam Kasten for Kubernetes DR Restore fails with the following error: Internal error occurred: "message":"Could not retrieve artifacts for prefix...

PVC Export Fails Due to Timeout

Challenge The export action will fail if Veeam Kasten for Kubernetes takes more than 45 minutes to complete the job. Checking the export action details, the message below is the clue where Veeam Kasten for Kubernetes hits the timeout for the operation waitWithBackoffwithRetries with the duration...

Cloned-Restore Fails on OpenShift Clusters with ImagePull Error

Challenge Cloned-restore jobs time-out and eventually fail due to an ImagePullError. Upon further investigation, the root cause is that the default SA in the target namespace does not have ImageStreams for the pod to pull an image from a private registry. Failed to pull image...

How to Configure an Air-Gapped Veeam Kasten for Kubernetes Deployment Using JFrog Artifactory

Purpose This article provides a step-by-step approach to configuring a JFrog Artifactory server and installing Veeam Kasten for Kubernetes. This allows for creating an air-gapped installation using a private container registry to install Veeam Kasten for Kubernetes. While this can always be done...

XFS Volume Restore Fails to Mount

Challenge Description: XFS kernel module remembers the old UUID and refuses to mount Cloning an XFS volume and mounting the filesystem of the cloned volume on the same node as the source volume will throw a "duplicate UUID" error or similar errors. The cloning of the volume reflects everything fr...

No action was taken because the import artifacts already existed

Challenge Veeam Kasten for Kubernetes has smart logic built-in to perform intelligent actions. Assuming an Import job on the target cluster was successfully completed, unless there's a new restorepoint from the source cluster, the job won't run. This can happen if: there are no new restorepoints ...

Backup Failing With `Too many snapshots` When Using Longhorn as a Storage Provisioner

Challenge Veeam Kasten for Kubernetes backup action for longhorn volumes fails with the error message: too many snapshots created Cause When integrating with CSI-based volumes, Veeam Kasten for Kubernetes employs VolumeSnapshot resources to create snapshots during backup operations. With Longhorn...

How to Clean Up Veeam Kasten for Kubernetes Orphaned Volumes on GKE k8s 1.28.x

Purpose This guide provides instructions on how to clean up Veeam Kasten for Kubernetes provisioned PVs that are not deleted, causing orphaned volumes in the GKE k8s 1.28 release. It has been observed on GKE running k8s 1.28, Veeam Kasten for Kubernetes provisioned PVs using in-tree provisioner...

Upgrading Veeam Kasten for Kubernetes Fails With Parse Error

Challenge When upgrading to Veeam Kasten for Kubernetes 6.5.3 or higher the upgrade fails with: parse error at k10/templates/v0services.yaml:128: function "continue" not defined Cause This issue is related to the Helm binary version that is installed. Solution To resolve this issue, download the...

Veeam Kasten for Kubernetes - vSphere Block Mode Exports Failure With Error 14009

Challenge During vSphere block mode export, in certain situations, it might be observed that the folders/objects get created in the storage bucket. However, the export job remains stuck, and the following error message is seen in the debug logs: "Open virtual disk file failed. The error code is...

How to Configure Veeam Kasten for Kubernetes Dashboard to Use a New Certificate

Purpose This article documents how to troubleshoot and replace the certificate used by the Veeam Kasten for Kubernetes Dashboard. Scenario Example The Veeam Kasten for Kubernetes Dashboard becomes inaccessible due to an expired certificate, resulting in an HTTP ERROR 401 , indicating a lack of...

How to Patch Veeam Kasten for Kubernetes CR used by Red Hat Marketplace Operator

Purpose This article provides an easy way to patch the Veeam Kasten for Kubernetes operand and make changes to the Veeam Kasten for Kubernetes configuration in an operator-based installation. Solution Veeam Kasten for Kubernetes is available for installation in an OCP cluster through either helm...

Alternate Repository for Veeam Kasten for Kubernetes Container Images

Solution This article documents an alternate location where Kasten's images are stored. Kasten's container images are hosted on the Google Container Registry gcr.io/kasten-images; however, this location may be inaccessible due to country-specific or company-imposed firewalls. If allowing access t...

How to Add a Blueprint Target Using Execution Hooks From the Policy API

Purpose This article explains how to set resources other than namespaces as the hooks for blueprint subjects. By default, hooks sets the namespace as the target for the blueprint unless specified otherwise. Solution Veeam Kasten for Kubernetes supports kanister execution hooks through Policy API...

Resolving Pending State During Cluster Removal with k10multicluster Tool

Challenge This article helps in finding and deleting long-running finalizers that are waiting on other resources and can cause deletions to appear stuck in the terminating state. Description: When Kubernetes attempts to delete the cluster object that has finalizers "dist.kio.kasten.io/debootstrap...

How to Configure Veeam Intelligent Diagnostics Log Location

Purpose This article documents how to change the location where Veeam Intelligent Diagnostics stores the logs it collects--allowing customers to configure both where those logs are stored temporarily on the Veeam Backup Server before being transferred to the Veeam ONE server and where those logs...

How to Configure AWS Application Loadbalancer (ALB) for Veeam Kasten for Kubernetes

Purpose This article provides an example of configuring AWS Application Loadbalancer ALB for accessing the Veeam Kasten for Kubernetes UI. Note: This documentation assumes that the ALB ingress controller has already been installed and configured. Solution What is an ALB An Application Load Balanc...

How to Enable Veeam Kasten for Kubernetes Disaster Recovery using CLI

Purpose This article provides step-by-step instructions on how to enable Veeam Kasten for Kubernetes Disaster Recovery KDR using CLI. Veeam Kasten for Kubernetes Documentation - Disaster Recovery Solution 1. Replace in the following command and run it to create a DR secret with a passphrase: Note...

Veeam Recovery Orchestrator Vulnerability (CVE-2024-29855)

Veeam Software Security Commitment Veeam® is committed to ensuring its products protect customers from potential risks. As part of that commitment, we operate a Vulnerability Disclosure Program VDP for all Veeam products and perform extensive internal code audits. When a vulnerability is...

How to Enable Changed Block Tracking for Guest Cluster on vSphere with Tanzu

Purpose Changed Block Tracking is a VMware feature that tracks changes in virtual disks. Veeam Kasten for Kubernetes uses this feature in vSphere with Tanzu Guest Clusters to efficiently back up Persistent Volumes. Enabling Changed Block Tracking in each Supervisor Cluster where Veeam Kasten for...

Debugging Backups with Longhorn CSI

Challenge Veeam Kasten for Kubernetes backup job that doesn't complete while using Longhorn CSI drivers is never-ending even after the proper installation of CSI snapshotter components and controllers. No errors are noticed generally for this issue. The job waits for the volumesnapshot object in...

How to Override Volume storageClass During Export

Purpose This article documents how to configure a Veeam Kasten for Kubernetes export parameter to specify the storageClass that is created and used during exports. Solution The ExporterStorageClassName parameter can be used for namespaces that have PersistentVolumeClaims PVCs using a single...

How to Properly Off-Board a Namespace From Veeam Kasten for Kubernetes Backups

Purpose This article documents the procedure to properly off-board a namespace in Veeam Kasten for Kubernetes. Solution 1. Check all restorepointcontents for the policy using the following command: Replace with the actual policy name. kubectl get restorepointcontents -l k10.kasten.io/policyName=...

Troubleshooting '401 - Unauthorized' or 'x509' Errors When Accessing the Veeam Kasten for Kubernetes Dashboard

Challenge When attempting to access the Veeam Kasten for Kubernetes dashboard the following error occurs despite using the cacertconfigmap.name helm value: 401 - Unauthorized This issue may also manifest as errors related to x509 in the gateway pod or auth-svc pod logs: x509: certificate signed b...

How to ‘Transform’ Multiple Resources with Regex

Purpose Veeam Kasten makes use of the 'Transforms' feature to migrate applications between different clusters, between different storage backends, and many other use cases. Transforms enable modifications to Kubernetes resources on restore. The ability to move an application across clusters is an...

What Data Does Veeam Kasten for Kubernetes Include in the Phone Home Reports

Purpose This article documents what data is included in the phone home reports sent by Veeam Kasten for Kubernetes. Solution The data is divided into the following sections Basic Data Infrastructure Data Run Time Data for Billing Purposes Additional Metrics Profiles Artifact Metrics Basic Data...

How to Configure EKS Clusters to Use AWS IAM Users/Roles for Veeam Kasten for Kubernetes Access

Purpose Follow this guide to provide appropriate Veeam Kasten for Kubernetes role-based access using AWS IAM users or roles. Solution Description Veeam Kasten for Kubernetes integrates with whatever authentication mechanism customers use to access their Kubernetes clusters. Since EKS natively...

Job For .local Domain Fails When Using Ubuntu-base VMware Backup Proxy

Challenge Backup of VMware VMs fails when using an Ubuntu-based Linux VMware Proxy, and the VMware environment is added to Veeam with an FQDN ending in .local e.g., esxi.somedomain.local. In the Task Log, errors related to NFC connectivity can be found: Processing Error: NFC storage connection is...

How to Configure PXE Booting of Veeam Agent for Linux Recovery Media

Veeam Agent for Linux Version Requirement Starting with Veeam Agent for Linux 6.1.2, the Veeam Recovery Media is designed to be capable of use in PXE booting.veeam-recovery iso released after 22-May-2024 Purpose This article documents how to configure Linux to PXE boot the Veeam Agent for Linux...

VSS Snapshot Creation Delay on Server with DFSR Enabled

Article Applicability The issue described in this article affects earlier versions of Veeam Agent for Microsoft Windows VAW, however, the solution provided will only work with VAW 6.1.2 and higher. Challenge When backing up a server with the DFS feature enabled, the VSS snapshot creation step in...

Backup fails with "Invalid argument Asynchronous request operation has failed. Failed to open storage for read/write access"

Challenge When attempting to write a restore point VBK or VIB to a Linux repository with a 4k logical block size, the task fails with the error: Invalid argument Asynchronous request operation has failed. Failed to open storage for read/write access Cause This error occurs when the storage device...

Veeam Data Cloud Authentication Error

Challenge On or after May 18th 2024, attempting to login to the Veeam Data Cloud VDC console using a previously functioning bookmark or outdated hyperlink may result in login errors, such as: Veeam Data Cloud encountered an error retrieving the user profile. Please try again and contact support i...

Veeam Agent for Microsoft Windows Vulnerability (CVE-2024-29853)

Veeam Software Security Commitment Veeam® is committed to ensuring its products protect customers from potential risks. As part of that commitment, we operate a Vulnerability Disclosure Program VDP for all Veeam products and perform extensive internal code audits. When a vulnerability is...

New Readiness Check - Multiple Veeam Backup Servers Are Required by One Recovery Plan

Article Applicability This article documents a new Plan Readiness Check parameter in Veeam Recovery Orchestrator 7.1. Purpose This article documents a new addition to the Plan Readiness Check in Veeam Recovery Orchestrator VRO v7.1. This new check will be part of the Plan Readiness Check for all...

Veeam Backup Enterprise Manager Vulnerabilities (CVE-2024-29849, CVE-2024-29850, CVE-2024-29851, CVE-2024-29852)

Veeam Software Security Commitment Veeam® is committed to ensuring its products protect customers from potential risks. As part of that commitment, we operate a Vulnerability Disclosure Program VDP for all Veeam products and perform extensive internal code audits. When a vulnerability is...

How to Configure WDS to PXE Boot Veeam Agent for Microsoft Windows Recovery Media

Purpose This article documents a method to configure Windows Deployment Services WDS to PXE boot the Veeam Agent for Microsoft Windows Recovery Media over a network. Solution General Guidance Disclaimer The information provided below is intended only as general guidance. The method and...

"HTTP protocol is not supported, please use HTTPS." Error When Adding Object Storage

Challenge When attempting to add an S3-compatible Object Storage Repository, the wizard displays the error: HTTP protocol is not supported, please use HTTPS. If the service point is then modified to change to HTTPS, the wizard then displays the error: Failed to retrieve certificate from...