WinDbg wasn't breaking because it had set the breakpoint at CRYPT32!RegOpenKeyExW. After I specified ADVAPI32!RegOpenKeyExW and tried the experiment in question, I could see that RegOpenKeyExW returns 2, which seems correct.
I took a look at that code (which is in a third-party library). They are throwing an exception if RegOpenKeyEx returns an error; the (generic) exception handler then calls GetLastError(). For 21.0.40, I just changed that to pass the error returned from RegOpenKeyEx. (Though to be fair to the original developer, the MS docs never say which of the Windows APIs do not support GetLastError().)