1858 matches found
Hyper-V SureBackup Task Failure: "VM was created with the later version of Hyper-V than the one installed on host"
Challenge A task within a SureBackup job for a Hyper-V environment fails with the error: Error: VM was created with the later version of Hyper-V than the one installed on host Cause This error occurs when the Virtual Lab in use by the SureBackup job is located on a host that is running a version ...
Alternative Method for Migrating Backups to Hardened Linux Repository
Purpose This article documents an alternative method for migrating backups from a Scale-Out Backup Repository SOBR utilizing non-immutable performance tier extents to one using Hardened Linux Immutable performance tier extents. This method is useful in situations where a SOBR contains backup data...
How to Export Console and HAR File
Purpose This article documents how to generate a HAR file HTTP Archive to provide additional diagnostic information when investigating issues for web-based applications e.g., Veeam Software Appliance, Veeam Backup & Replication Web UI, Veeam Service Provider Console, Veeam Recovery Orchestrator,...
How to Install Grafana with K10 Dashboard and Data Sources Pre-provisioned
Purpose Starting in Veeam Kasten for Kubernetes v7.5.0, Grafana will no longer be included by default. This in-depth technical guide will demonstrate the process of installing Grafana on Kubernetes and populating it with Kasten Data. It will allow you to quickly access pre-provisioned Kasten...
Troubleshooting apiservices Unavailability Due to Failed Discovery
Challenge In Kubernetes clusters that use Cilium for CNI Container Network Interface, Veeam Kasten for Kubernetes aggregated apiservices are unavailable due to failure in discovery. This causes the Kasten UI not to load, or display empty null values. This causes crucial Kasten APIs like...
How to Investigate 'Encrypted Data Event' from Malware Detection
Purpose This article documents how to investigate which files are encrypted within a machine when the Malware Detection system flags a machine as having Encrypted data. This tool only works for malware inline detection events created by Veeam Backup & Replication 12.1.2 and newer. Previous versio...
Imagestream Export or Restore Fails With "error writing layer: no space left on device"
Challenge Export Action or Restore Action of OpenShift Imagestream fails with the following error: Failed to exec command in pod: command terminated with exit code 1.stdout: error writing layer: write/var/lib/image//blobs/sha256/: no space left on device Cause During the export or restore of an...
Linux Hardened Repository Unable to Update Immutability or Remove Restore Points due to SGID
Challenge A job targetting a Hardened Linux Repository Immutable fails with an error about being unable to remove restore points or update immutability examples below. Failed to delete file '/path/to/storage.vbk': Operation not permitted Failed to perform setting immutability for backup Error:...
How to Make Veeam Kasten for Kubernetes Grafana Pod rootless
Purpose This article explains how to disable init-container in k10-grafana to run it as rootless. Cause Veeam Kasten for Kubernetes installation provides an instance of Grafana that is deployed automatically and can be used to query metrics from Kasten's Prometheus instance. This grafana pod is r...
Veeam Kasten for Kubernetes - Used Ports
Purpose This article provides information regarding network ports that are used for different internal services of the Veeam Kasten for Kubernetes application. Solution The following information of Veeam Kasten for Kubernetes services and network ports associated with them will help with port...
Veeam Kasten for Kubernetes Instant Recovery with Veeam Backup & Replication vPower NFS datastore
Purpose K10 Instant Recovery uses the Veeam Backup and Replication vPower NFS datastore to provide vSphere First Class Disks for storage. This feature enables "instant" recovery of workloads with Kubernetes. Solution The vPower NFS datastore is a fully functional datastore in the vSphere cluster...
There was a problem validating the profile: Repository not found.
Challenge Repository not found error while adding Veeam Repository in Veeam Kasten for Kubernetes despite verifying the repository in the Veeam Backup & Replication Console. Cause The 'Repository Not found' issue can happen in multiple scenarios. Most of the cases are due to repository Permission...
Security Context Constraints(SCC) capabilities setting for Generic Storage Backup and Restore with OCP 4.11 or higher
Purpose This article describes Security Context ConstraintsSCC capabilities that need to be added to use Generic Backup and Restore feature capabilities on OCP 4.11 and higher. Known Issues With OCP 4.11, default security context constraints SCC have changed. Due to this, when using Generic Stora...
How to Clean Up Veeam Kasten for Kubernetes Manual Backups
Purpose This article provides an overview of scenarios where out-of-schedule restorePoints are created and how to identify and remove them. Cause With Veeam Kasten for Kubernetes , there are multiple ways to take backups & exports. A manual backup/export from the application menu. A manual policy...
Openshift Authentication - Failed to authenticate: oidc: failed to get token: oauth2: cannot fetch token
Challenge Users are unable to login to Veeam Kasten for Kubernetes Dashboard and receive the following error: Failed to authenticate: oidc: failed to get token: oauth2: cannot fetch token: 400 Bad Request Response: "error":"unauthorizedclient","errordescription": "The client is not authorized to...
Report Generation Fails with "lookup prometheus-server-exp: no such host"
Challenge After enabling Veeam Kasten for Kubernetes reports, reports are not generated. When running an on-demand report policy, the following error message is observed within the executor logs: "message":"Post "http://prometheus-server-exp:80/k10/prometheus/api/v1/query": dial tcp: lookup...
Application Migration - Best Practices and Prerequisites
Purpose This article explains the prerequisites and best practices that must be considered for successful import during application migration. Solution Veeam Kasten for Kubernetes platform is built to support application migration across clusters, which is an extremely powerful feature that enabl...
Error: could not get determine if bucket supports versioning: Access Denied
Challenge Veeam Kasten for Kubernetes is unable to monitor or refresh the retention period of the data in the S3 bucket and cannot calculate the lock duration based on your retention setting because of this issue. This will show up as an error message in the notification bar: error: can't connect...
Import Job Fails with "No kopia manifests found in the repository"
Challenge Import jobs fail with the error: No kopia manifests found in the repository Full error: "cause": "file": "kasten.io/k10/kio/collections/kopia/operations.go:144", "function": "kasten.io/k10/kio/collections/kopia.GetLatestSnapshot", "linenumber": 144, "message": "No kopia...
How to Exclude Applications from Dashboard & Compliance Reporting
Purpose This article provides steps to exclude system namespaces from the Veeam Kasten for Kubernetes dashboard and compliance reporting. Solution Explanation Veeam Kasten for Kubernetes automatically discovers all application namespaces of the cluster on which it is deployed. Each of these...
Veeam Kasten for Kubernetes Performance Troubleshooting Guide
Purpose This article documents two of the most common performance tuning settings for Veeam Kasten for Kubernetes. Veeam Kasten for Kubernetes uses helm parameters, which can be tuned to adjust performance. The default settings should suit most environments. However, in complex environments,...
Option to Add Veeam Kasten for Kubernetes Does Not Appear in Veeam Backup & Replication
Challenge When attempting to add Veeam Kasten for Kubernetes to Veeam Backup & Replication, the option for Kasten does not appear in the Add Server options. Cause This issue may occur if: The Veeam Kubernetes Service is not running and needs to be started. The Kasten Plug-In is not installed. Thi...
Overcoming Failed Snapshots Using Ignore Exceptions
Challenge This article demonstrates how to use the Ignore Exceptions feature to overcome failed snapshots due to workloads not in a ready state. Description: Job failed to be executed Veeam Kasten for Kubernetes has retry logic to overcome service interruptions during any job. If a Backup does no...
Exports Don't Work After Veeam Kasten for Kubernetes Reinstall
Challenge Description: Jobs fail to run after reinstalling Veeam Kasten for Kubernetes on the same cluster Veeam Kasten for Kubernetes backups can be transferred to any S3-compliant object store to keep secondary copies and for long-term retention based on compliance needs. Both metadata and data...
Clone Job Fails Due to Security Context Constraints (SCC) Issue
Challenge Cloned-restore jobs time out and eventually fail because an application's security context provides permissions in the original application namespace. This results in the application's pod being in a non-running state. Example: Namespace = quay-enterprise SCC during deployment = anyuid...
DR Restore - Internal error occurred: Could not retrieve artifacts for prefix
Challenge A Veeam Kasten for Kubernetes DR Restore fails with the following error: Internal error occurred: "message":"Could not retrieve artifacts for prefix...
PVC Export Fails Due to Timeout
Challenge The export action will fail if Veeam Kasten for Kubernetes takes more than 45 minutes to complete the job. Checking the export action details, the message below is the clue where Veeam Kasten for Kubernetes hits the timeout for the operation waitWithBackoffwithRetries with the duration...
No action was taken because the import artifacts already existed
Challenge Veeam Kasten for Kubernetes has smart logic built-in to perform intelligent actions. Assuming an Import job on the target cluster was successfully completed, unless there's a new restorepoint from the source cluster, the job won't run. This can happen if: there are no new restorepoints ...
Cloned-Restore Fails on OpenShift Clusters with ImagePull Error
Challenge Cloned-restore jobs time-out and eventually fail due to an ImagePullError. Upon further investigation, the root cause is that the default SA in the target namespace does not have ImageStreams for the pod to pull an image from a private registry. Failed to pull image...
XFS Volume Restore Fails to Mount
Challenge Description: XFS kernel module remembers the old UUID and refuses to mount Cloning an XFS volume and mounting the filesystem of the cloned volume on the same node as the source volume will throw a “duplicate UUID” error or similar errors. The cloning of the volume reflects everything fr...
How to Configure an Air-Gapped Veeam Kasten for Kubernetes Deployment Using JFrog Artifactory
Purpose This article provides a step-by-step approach to configuring a JFrog Artifactory server and installing Veeam Kasten for Kubernetes. This allows for creating an air-gapped installation using a private container registry to install Veeam Kasten for Kubernetes. While this can always be done...
Backup Failing With `Too many snapshots` When Using Longhorn as a Storage Provisioner
Challenge Veeam Kasten for Kubernetes backup action for longhorn volumes fails with the error message: too many snapshots created Cause When integrating with CSI-based volumes, Veeam Kasten for Kubernetes employs VolumeSnapshot resources to create snapshots during backup operations. With Longhorn...
How to Clean Up Veeam Kasten for Kubernetes Orphaned Volumes on GKE k8s 1.28.x
Purpose This guide provides instructions on how to clean up Veeam Kasten for Kubernetes provisioned PVs that are not deleted, causing orphaned volumes in the GKE k8s 1.28 release. It has been observed on GKE running k8s 1.28, Veeam Kasten for Kubernetes provisioned PVs using in-tree provisioner...
How to Patch Veeam Kasten for Kubernetes CR used by Red Hat Marketplace Operator
Purpose This article provides an easy way to patch the Veeam Kasten for Kubernetes operand and make changes to the Veeam Kasten for Kubernetes configuration in an operator-based installation. Solution Veeam Kasten for Kubernetes is available for installation in an OCP cluster through either helm...
How to Configure Veeam Kasten for Kubernetes Dashboard to Use a New Certificate
Purpose This article documents how to troubleshoot and replace the certificate used by the Veeam Kasten for Kubernetes Dashboard. Scenario Example The Veeam Kasten for Kubernetes Dashboard becomes inaccessible due to an expired certificate, resulting in an HTTP ERROR 401 , indicating a lack of...
Upgrading Veeam Kasten for Kubernetes Fails With Parse Error
Challenge When upgrading to Veeam Kasten for Kubernetes 6.5.3 or higher the upgrade fails with: parse error at k10/templates/v0services.yaml:128: function “continue” not defined Cause This issue is related to the Helm binary version that is installed. Solution To resolve this issue, download the...
Veeam Kasten for Kubernetes - vSphere Block Mode Exports Failure With Error 14009
Challenge During vSphere block mode export, in certain situations, it might be observed that the folders/objects get created in the storage bucket. However, the export job remains stuck, and the following error message is seen in the debug logs: "Open virtual disk file failed. The error code is...
How to Add a Blueprint Target Using Execution Hooks From the Policy API
Purpose This article explains how to set resources other than namespaces as the hooks for blueprint subjects. By default, hooks sets the namespace as the target for the blueprint unless specified otherwise. Solution Veeam Kasten for Kubernetes supports kanister execution hooks through Policy API...
Alternate Repository for Veeam Kasten for Kubernetes Container Images
Solution This article documents an alternate location where Kasten's images are stored. Kasten's container images are hosted on the Google Container Registry gcr.io/kasten-images; however, this location may be inaccessible due to country-specific or company-imposed firewalls. If allowing access t...
How to Configure Veeam Intelligent Diagnostics Log Location
Purpose This article documents how to change the location where Veeam Intelligent Diagnostics stores the logs it collects—allowing customers to configure both where those logs are stored temporarily on the Veeam Backup Server before being transferred to the Veeam ONE server and where those logs a...
How to Configure AWS Application Loadbalancer (ALB) for Veeam Kasten for Kubernetes
Purpose This article provides an example of configuring AWS Application Loadbalancer ALB for accessing the Veeam Kasten for Kubernetes UI. Note: This documentation assumes that the ALB ingress controller has already been installed and configured. Solution What is an ALB An Application Load Balanc...
Resolving Pending State During Cluster Removal with k10multicluster Tool
Challenge This article helps in finding and deleting long-running finalizers that are waiting on other resources and can cause deletions to appear stuck in the terminating state. Description: When Kubernetes attempts to delete the cluster object that has finalizers “dist.kio.kasten.io/debootstrap...
How to Enable Veeam Kasten for Kubernetes Disaster Recovery using CLI
Purpose This article provides step-by-step instructions on how to enable Veeam Kasten for Kubernetes Disaster Recovery KDR using CLI. Veeam Kasten for Kubernetes Documentation - Disaster Recovery Solution 1. Replace in the following command and run it to create a DR secret with a passphrase: Note...
Veeam Recovery Orchestrator Vulnerability (CVE-2024-29855)
Issue Details CVE-2024-29855 A vulnerability CVE-2024-29855 in Veeam Recovery Orchestrator VRO version 7.0.0.337 allows an attacker to access the VRO web UI with administrative privileges. Note: The attacker must know the exact username and role of an account that has an active VRO UI access toke...
How to Enable Changed Block Tracking for Guest Cluster on vSphere with Tanzu
Purpose Changed Block Tracking is a VMware feature that tracks changes in virtual disks. Veeam Kasten for Kubernetes uses this feature in vSphere with Tanzu Guest Clusters to efficiently back up Persistent Volumes. Enabling Changed Block Tracking in each Supervisor Cluster where Veeam Kasten for...
Debugging Backups with Longhorn CSI
Challenge Veeam Kasten for Kubernetes backup job that doesn't complete while using Longhorn CSI drivers is never-ending even after the proper installation of CSI snapshotter components and controllers. No errors are noticed generally for this issue. The job waits for the volumesnapshot object in...
How to ‘Transform’ Multiple Resources with Regex
Purpose Veeam Kasten makes use of the ‘Transforms’ feature to migrate applications between different clusters, between different storage backends, and many other use cases. Transforms enable modifications to Kubernetes resources on restore. The ability to move an application across clusters is an...
Troubleshooting '401 - Unauthorized' or 'x509' Errors When Accessing the Veeam Kasten for Kubernetes Dashboard
Challenge When attempting to access the Veeam Kasten for Kubernetes dashboard the following error occurs despite using the cacertconfigmap.name helm value: 401 - Unauthorized This issue may also manifest as errors related to x509 in the gateway pod or auth-svc pod logs: x509: certificate signed b...
How to Properly Off-Board a Namespace From Veeam Kasten for Kubernetes Backups
Purpose This article documents the procedure to properly off-board a namespace in Veeam Kasten for Kubernetes. Solution 1. Check all restorepointcontents for the policy using the following command: Replace with the actual policy name. kubectl get restorepointcontents -l k10.kasten.io/policyName=...
What Data Does Veeam Kasten for Kubernetes Include in the Phone Home Reports
Purpose This article documents what data is included in the phone home reports sent by Veeam Kasten for Kubernetes. Solution The data is divided into the following sections Basic Data Infrastructure Data Run Time Data for Billing Purposes Additional Metrics Profiles Artifact Metrics Basic Data...