Hi,
(See OUR PROBLEM, later below)
OUR CONFIG:
Our IT team has a Windows General Purpose File Server Cluster "ENGRFILER1" (active/passive) that is sharing a highly used volume (U: drive) which is 2 TB (915 GB currently in use). The volume is NTFS with deduplication enabled at the file server level. The file server cluster nodes are running Windows Server Core.
File server cluster node config (using MSINFO32):
OS Name Microsoft Windows Server 2019 Datacenter
Version 10.0.17763 Build 17763
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name ENGR-FILER1N4
System Manufacturer Microsoft Corporation
System Model Virtual Machine
System Type x64-based PC
System SKU Not Available
Processor Intel(R) Xeon(R) Gold 6144 CPU @ 3.50GHz, 3492 Mhz, 3 Core(s), 6 Logical Processor(s)
BIOS Version/Date Microsoft Corporation Hyper-V UEFI Release v4.0, 3/13/2019
SMBIOS Version 3.1
BIOS Mode Not Available
BaseBoard Manufacturer Microsoft Corporation
BaseBoard Product Virtual Machine
BaseBoard Version Hyper-V UEFI Release v4.0
Platform Role Not Available
Secure Boot State Not Available
PCR7 Configuration Not Available
Windows Directory C:\Windows
System Directory C:\Windows\system32
Boot Device \Device\HarddiskVolume2
Locale United States
Hardware Abstraction Layer Version = "10.0.17763.831"
User Name Not Available
Time Zone Eastern Daylight Time
Installed Physical Memory (RAM) Not Available
Total Physical Memory 24.0 GB
Available Physical Memory 11.4 GB
Total Virtual Memory 27.5 GB
Available Virtual Memory 11.8 GB
Page File Space 3.50 GB
Page File C:\pagefile.sys
Kernel DMA Protection Off
Virtualization-based security Not enabled
Device Encryption Support Not Available
A hypervisor has been detected. Features required for Hyper-V will not be displayed.
( Note: The file server cluster nodes are virtualized, running hyper-converged within a Storage Spaces Direct SOFS cluster. The file server cluster node NTFS U: drive mounted volume share is actually a VHDX file inside a ReFS CSV on the storage
spaces pool. Deduplication is NOT enabled at the CSV S2D level. )
We have built a Microsoft Data Protection Manager 2019 server for backing up our file server volumes.
The DPM version is: 2019 - UR1 (10.19.260.0)
The DPM SQL Sever version info:
SQL Server Management Studio 15.0.18330.0
SQL Server Management Objects (SMO) 16.100.37971.0
Microsoft Analysis Services Client Tools 15.0.19040.0
Microsoft Data Access Components (MDAC) 10.0.17763.1
Microsoft MSXML 3.0 6.0
Microsoft .NET Framework 4.0.30319.42000
Operating System 10.0.17763
The server that DPM is running on is: Windows Server 2019 Datacenter (1809, 17763.1131)
DPM sever config (MSINFO32):
OS Name Microsoft Windows Server 2019 Datacenter
Version 10.0.17763 Build 17763
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name ENGR-BACKUP1
System Manufacturer Dell Inc.
System Model PowerEdge R610
System Type x64-based PC
System SKU Not Available
Processor Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2394 Mhz, 4 Core(s), 8 Logical Processor(s)
Processor Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2394 Mhz, 4 Core(s), 8 Logical Processor(s)
BIOS Version/Date Dell Inc. 6.1.0, 10/18/2011
SMBIOS Version 2.6
BIOS Mode Not Available
BaseBoard Manufacturer Dell Inc.
BaseBoard Product 0F0XJ6
BaseBoard Version A13
Platform Role Not Available
Secure Boot State Not Available
PCR7 Configuration Not Available
Windows Directory C:\Windows
System Directory C:\Windows\system32
Boot Device \Device\HarddiskVolume1
Locale United States
Hardware Abstraction Layer Version = "10.0.17763.1131"
User Name Not Available
Time Zone Eastern Daylight Time
Installed Physical Memory (RAM) Not Available
Total Physical Memory 96.0 GB
Available Physical Memory 90.2 GB
Total Virtual Memory 110 GB
Available Virtual Memory 103 GB
Page File Space 14.0 GB
Page File C:\pagefile.sys
Kernel DMA Protection Off
Virtualization-based security Not enabled
Device Encryption Support Not Available
Hyper-V - VM Monitor Mode Extensions Yes
Hyper-V - Second Level Address Translation Extensions Yes
Hyper-V - Virtualization Enabled in Firmware No
Hyper-V - Data Execution Protection Yes
The DPM server has all patches and is up to date as of: 4/21/2020
The DPM server and file server cluster are members of our AD domain.
The DPM storage pool is:
* 50 TB (25 TB currently free) on a 10 Gbit iSCSI connected SAN disk utilizing a SSD flash front-end tierd disk subsystem.
* Formatted as "Modern Backup Storage" (MBS) ReFS without depduplication enabled.
The connectivity of the DPM server to the file server cluster is 10 Gbit throughout.
The above config is all BOG standard Microsoft DPM and storage config, out-of-the-box, vannilla stuff.
---------------
OUR PROBLEM
When we use DPM to backup the U: drive volume on our file sever cluster we are seeing the exact same problem as outlined in the following thread:
https://social.technet.microsoft.com/Forums/en-US/52a4a412-3e78-447e-968a-3f2b291fc15b/fileserver-protection?forum=dpmfilebackup
The error message we have is:
U:\ on ENGRFILER1FS4.EngrFiler1.domain.x.y
C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\f553adae-8a21-403e-b5d1-9a1a5e0b1a31\5564c443-64e0-45ff-b8bb-1a4adec0bba2\Full\
DPM failed to clean up data of old incremental backups on the replica for Volume U:\ on ENGRFILER1FS4.EngrFiler1.domain.x.y.
Synchronization will fail until the replica cleanup succeeds.
(ID 30134 Details: Cannot create a file when that file already exists (0x800700B7))
This error message occurs every time we backup the volume.
For troubleshooting, we have rebuilt the entire DPM server from scratch and still get the same error.
For troubleshooting we have disabled Windows defender on the DPM server, and we do not run any other antivirus products, nor do we have defender enabled on our file server cluster nodes.
We have not had trouble with DPM when backing up our other servers for our IT environment.
We seem to only have trouble with the volumes exposed by our file server cluster.
Running a "synchronization job with consistency check" seems to fix the problem for a short time, then the problem will re-appear within a few hours, or a day.
I have a guess that this problem is somehow caused by the U: drive volume we are backing up is being deduplicated, but I can't confirm this, as others have stated that turning off dedup didn't solve their problem.
This page says a lot about running deduplication with the DPM storage pool, but it says very little (if anything at all that I can find) about the whether there will be problems when backing up when the volume being backed up is deduplicated:
https://docs.microsoft.com/en-us/system-center/dpm/dpm-support-issues?view=sc-dpm-2019
The page needs more info for consumers if there are issues with backing up deduplicated volumes (on file server clusters or otherwise).
Any help would be greatly appreciated!
Thank you.