Finding Why a Windows Server In-Place Upgrade Rolls Back

A Windows Server in-place upgrade that reaches a late percentage, restarts, and comes back on the old version has rolled back. Setup records why in its own logs, and it keeps those logs after the rollback. This post covers how to read the error code, open the log that matches it, and fix the cause before the next attempt.

A recent r/sysadmin thread had a Windows Server 2016 to 2022 upgrade that stalled at 88% and rolled back. SetupDiag reported 0xC1900101-0x4000D against the x86 copy of ntprint.inf, and SFC and DISM had already been run. The steps below follow a result like that back to the driver or file that stopped the upgrade.

Read the error code

A Windows Setup error has a result code and an extend code. Result code 0xC1900101 is the generic rollback code, and Microsoft's documentation says the cause is usually an incompatible driver. The extend code after the dash says where setup was when it failed. Its first digit is the phase and its last two digits are the operation.

Phase 1 is the down-level phase, while the old operating system is still running. Phase 2 is the Safe OS phase, 3 is the first boot into the new system, and 4 is the boot after that. Operation 0D is data migration, 17 is boot, and 18 is sysprep. So 0x4000D is a failure while setup migrated data after the second restart, and 0x30018 is a failure during the sysprep operation.

Read the SetupDiag result

SetupDiag reads the setup logs and reports the rule that matched the failure. Windows Setup extracts it to C:\$Windows.~BT\Sources and runs it on its own after a failed upgrade. It writes the result to %WinDir%\Logs\SetupDiag\SetupDiagResults.xml and to the registry under HKLM\SYSTEM\Setup\SetupDiag\Results.

Get-Content "$env:windir\Logs\SetupDiag\SetupDiagResults.xml"

When that file isn't there, download the current SetupDiag from Microsoft and run it from an elevated prompt on the server that failed. It needs .NET Framework 4.7.2 or later, and it finds the logs on that server without any arguments.

SetupDiag.exe /Output:C:\SetupDiag\Results.log

The result names the failure and often the file or driver package involved. In the thread above it named the x86 printer driver package. When SetupDiag reports no match, read the logs directly.

Open the log for the phase

Setup writes setupact.log for actions and setuperr.log for errors in a different folder for each phase. Both files are in the same folder.

• C:\$Windows.~BT\Sources\Panther holds the down-level phase and is the starting point for rollback investigations.
• C:\$Windows.~BT\Sources\Rollback holds the rollback itself and is the one Microsoft lists for generic 0xC1900101 rollbacks.
• C:\$Windows.~BT\Sources\Panther\UnattendGC holds the OOBE phase.
• C:\Windows\Panther holds the logs after an upgrade that finished.

Search for the two strings Microsoft documents for a failed apply, and keep the last matches.

Select-String -Path 'C:\$Windows.~BT\Sources\Panther\setupact.log','C:\$Windows.~BT\Sources\Rollback\setupact.log' -Pattern 'Shell application requested abort','Abandoning apply due to error for object' | Select-Object -Last 5 Filename,LineNumber,Line

Open the log at that line number and read the lines above it. The SP, MIG, and CONX entries come from the setup platform, the migration engine, and the compatibility checks, and they name the object setup was working on. Write down the timestamp, then find the same time in setuperr.log. Decode any Win32 error code you find there with certutil.

certutil -error 0x570

Microsoft's own example of this search ends at error 0x570, ERROR_FILE_CORRUPT, on a machine key file under C:\ProgramData\Microsoft\Crypto\RSA\S-1-5-18. Deleting that file resolved the upgrade in their example. The file or driver in your log is the one to fix.

When the rollback followed a crash

A 0x4000D rollback often follows a stop error during the second boot. The rollback folder keeps a minidump as setupmem.dmp, the event logs from the attempt as .evtx files, and the device install log at setupapi\setupapi.dev.log. Open setupmem.dmp in WinDbg, run !analyze -v, and read the MODULE_NAME and IMAGE_NAME lines. Update or remove the driver the dump names. For a 0x30018 rollback, read setupapi.dev.log for the device whose install ends in a failure status.

Fix the cause

For an x86 printer driver like the one in the thread, list the 32-bit drivers and remove each one from the driver store.

Get-PrinterDriver | Where-Object PrinterEnvironment -eq 'Windows NT x86' | Select-Object Name,PrinterEnvironment
Remove-PrinterDriver -Name "<name>" -PrinterEnvironment 'Windows NT x86' -RemoveFromDriverStore

A driver still assigned to a shared printer won't remove. Set that printer to its x64 driver only under Sharing, then Additional Drivers in Print Management, and remove the x86 driver after that.

For other drivers, update the driver or remove the device software that installed it. Microsoft lists antivirus filter and encryption drivers as a cause of 0x40017 rollbacks after the second restart, so uninstall third-party antivirus and do a clean boot before the next attempt. Check that the system drive has at least 20 GB free, the minimum Microsoft lists for a 64-bit upgrade.

Before the next attempt

Microsoft's in-place upgrade guide asks for a full backup with a tested restore, and it rules out two kinds of server. Don't run an in-place upgrade on a domain controller. Build a new one and demote the old one instead. A clustered server takes Cluster-Aware Updating or a cluster operating system rolling upgrade instead.

I'd run the next attempt on a restored copy of the server first. When it finishes there, the same fix is ready for the live server, and the copy gives you the time each phase takes for the maintenance window.

Where ET Ducky shows it

I build ET Ducky, and its Windows agent reads process creation and file events from the kernel through ETW on every managed server. You can open a Live Session shell from the browser without the need for a VPN and run the Select-String and SetupDiag commands above on the server that rolled back.

You'd start a Multi-Agent Session across a server that upgraded and the one that rolled back. It compares the kernel events from both to find the outlier. Only counts-only summaries leave each host, and the raw kernel events stay on it.