Lucene search

K
veeamVeeam softwareVEEAM:KB4483
HistoryAug 18, 2023 - 12:00 a.m.

Unexpected Increase in Exchange Incremental Data Processed By Veeam Backup for Microsoft 365

2023-08-1800:00:00
Veeam software
www.veeam.com
37
veeam
backup
microsoft 365
exchange
data
processed
incremental

AI Score

6.7

Confidence

High


August 2024 Update

This article has been updated with new information after the issue, initially resolved in August 2023, was observed to have reoccurred in late August 2024.

Issue Summary

First Occurrence – August 7 -> August 21, 2023

Some customers using Veeam Backup for Microsoft 365 to back up Microsoft 365 Exchange data may have experienced an unexpected increase in the volume of data being backed up between August 7 and August 21, 2023. Microsoft confirmedMicrosoft has confirmed this issue in the message EX6700041, which will be displayed in the Microsoft 365 Admin Center of some organizations with affected mailboxes. at that time that this was due to a limited incident, which caused older emails to resurface unexpectedly in some Microsoft Exchange Online environments, resulting in increased email data in some Exchange Online mailboxes.

Second Occurrence – March 8 -> April 23, 2024

Beginning on March 8th, 2024, Veeam Support started to receive reports from customers about an unexpected increase in the amount of Exchange data being backed up. Based on the experience with the first occurrence of this issue, Veeam’s development team promptly engaged with Microsoft Support to begin investigating the root cause. On April 23rd, Microsoft provided the statement below to address this instance.

Third Occurrence – May 28 -> June 12, 2024

__Around May 28th, Veeam Support received reports from customers observing an unexpected increase in the amount of Exchange data being backed up. Veeam’s development team engaged with Microsoft Support, and we believe the core issue has been resolved and that no further impact should occur. Microsoft has released a statement regarding this occurrence.

Fourth Occurrence – Late August 2024 -> (ongoing)

Around Aug 16th, Veeam Support received initial reports from customers observing an unexpected increase in the amount of Exchange data being backed up. Veeam’s development team is engaged with Microsoft Support to learn if this issue is the same as the previous occurrences and is working towards a mitigation strategy.

Around September 3rd, Microsoft published event EX870973, which is expected to be present in the Service Health dashboard for all affected tenants. This direct communication from Microsoft addresses the situation and identifies that the root cause may have been isolated (specific delays with EWS requests in Exchange Online resulted in an increased number of events and items to back up). The statement goes on to explain that a remediation plan is currently being processed to resolve this issue by September 20th.


This article will be updated when further information becomes available. As with every occurrence, the Advised Actions below remain the same; the jobs must be allowed to complete.

Backup Impact

Emails that were changed, causing them to resurface unexpectedly, must be captured by Veeam Backup_ for Microsoft 365_ to ensure backup continuity. Even though these emails may have been previously backed up, their modified state means they must be backed up again. There is no way to reclaim the space utilized by the backup of these resurfaced emails, as it is technically changed data.

This situation is discussed in further detail in Veeam’s RND forums.

Statements from Microsoft

Microsoft provided the following statements to Veeam for us to share with our mutual customers.

Microsoft statement regarding March 2024 incidence:

Due to changes we [Microsoft] are making to optimize search and compatibility functionality within our service, this action led to re-indexing for some of our customers’ data. The re-indexing is not indicative of a change to fields material to our BCDR and storage partners, so we will be working on a change to minimize the re-indexing that is prompting another backup of customer data. We made changes over the weekend [of 2024-04-20] that should pause the unexpected behavior and those changes will be in place until Microsoft and our partners are able to make longer term enhancements to avoid future issues from taking place.
Thank you for your patience.

Microsoft provided the following statement regarding the June 2024 incidence:

Microsoft routinely performs system maintenance on items within the storage system. A recent system maintenance task on items with attachments unexpectedly created EWS SyncFolderItems events to be triggered. The task finished processing about 70% of environments worldwide before it was postponed. The necessary fixes were rolled out to help prevent unexpected sync events for the remaining 30% of environments that need to be processed.

Advised Actions

  • Check the reports and monitoring activity around the dates indicated in the Issue Summary to determine if your setup is affected by this issue.
  • Any impacted Veeam Backup for Microsoft 365 jobs **must**be allowed to be completed.
  • If you have opened a Microsoft Support Case on behalf of your contract and would like more Veeam details regarding the issue to help address it with Microsoft, or should you have any questions, please don't hesitate to contact Veeam Customer Support.

More Information

Veeam Backup for Microsoft 365 uses officially supported methods to back up Exchange Online data, specifically using EWS API’s SyncFolderItems to identify changed items. As such, Veeam Backup for Microsoft 365 is affected by these internal changes made by Microsoft’s maintenance.

The problem described in this article relates to internal changes on the Microsoft side that triggered:

  • An item to be resynced via the EWS SyncFolderItems API, which is the standard way to perform backup operations.
  • An item to have a new LastModifiedDateTime, as this can change for any Exchange item that has its internal properties modified.

Note: Veeam Backup for Microsoft 365 does not use the LastModifiedTime or ChangeKey as a primary indicator of change but can use them as a secondary indicator to avoid duplicate items with the same values when storing data for backup purposes.

Microsoft has said they have made the necessary fixes to prevent unexpected sync events going forward.

To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

AI Score

6.7

Confidence

High