PVC Export Fails Due to Timeout

Challenge

The export action will fail if Veeam Kasten for Kubernetes takes more than 45 minutes to complete the job.

Checking the export action details, the message below is the clue where Veeam Kasten for Kubernetes hits the timeout for the operation (waitWithBackoffwithRetries) with the duration field showing 45 mins.

errors:
      - cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"context 
        **Deadline exceeded**"},"function":"kasten.io/k10/kio/poll.**waitWithBackoffWithRetries**",
        "linenumber":78,"message":"Context done while polling"},​​​​​
        "fields":[{​​​​​"name":"**duration**","value":"**44m59.989920898s**"}​​​​​

Cause

This may occur when:

• A large amount of data is exported.
• There is a slow network connection to the external repository.
• Manymillions smaller files are exported.

Solution

Veeam Kasten for Kubernetes provides a reliable way to export a snapshot to an external repository during policy execution. Veeam Kasten for Kubernetes uses timeout settings to ensure that a job doesn't run forever and impact subsequent policy executions. These timeouts may need to be adjusted based on the environment, workloads, and the bandwidth available for the data transfers during the export.

When a large amount of data needs to be exported, these default timeouts for export actions may not be suitable.

This document describes how to increase the default timeout for export actions.

Adjusting Timeout Value

The issue can be resolved by increasing the value of the KanisterBackupTimeout parameter.

The command below will gather the current helm values used for installation into a file and run a helm upgrade to the Veeam Kasten for Kubernetes installation, increasing the timeout to a specific number, in this example, 150 minutes. We recommend using a higher value and then fine-tuning it based on the environment.

helm get values k10 --output yaml --namespace=kasten-io > k10_val.yaml && \  
helm upgrade k10 kasten/k10 --namespace=kasten-io -f k10_val.yaml \  
--set kanister.backupTimeout=150 --version=<current-k10-version>

Copy

After the helm upgrade, validate the timeout value on the k10-config configmap.

get cm -n kasten-io k10-config -o yaml | egrep "version|KanisterBackupTimeout"

Copy

To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.