Windows Compile Python 3.9 with FIPS enabled OpenSSL 3

发布于 2022-09-21  4.11k 次阅读

Introduction

This article is a step-to-step record of myself trying to compile a FIPS compliance Python 3.9.14 above OpenSSL 3.0.5 with Visual Studio 2017 under Windows Server 2019.

Version Selection

Most of the other guides are using OpenSSL 1.0.2u, which is EOL since 31 December 2019. See here. And OpenSSL 1.1 doesn't support the FIPS module so we have to use OpenSSL 3.0, by this time 3.0.5.

Python 3.6 reached its EOL on 23 December 2021. See here. Hence we have to use Python 3.7/8/9/10.

According to This Github Issue,  Python 3.9/10/11 supported OpenSSL 3.0 on 9 Sep 2021. Therefore we decide to start with Python 3.9.14.

Another thing worth mentioning is that we actually tried OpenSSL 1.0.2u but compiled python leading to the following error when trying to enable FIPS_mode():

FIPS routines:FIPS_check_incore_fingerprint:fingerprint does not match

We debugged into FIPS's code and found it hard to deal with. That's another reason we have to switch to OpenSSL 3.0. Hope anyone who comes across the above error can find our solutions.

Steps

1. Enabled FIPS on Windows Server

We're using Windows Server 2019 Base with Graphic Interface. Version 1809(OS Build 17763.3287).

Following Oracle's guide: https://blogs.oracle.com/cloud-infrastructure/post/windows-server-fips-compliance

Plus, we need to enable '.NET Framework 3.5 Features' and '.NET Framework 4.7 Features' since generating python installer needs them.

2. Install Visual Studio and Base Components:

Git for Windows: https://git-scm.com/download/win

NASM: https://www.nasm.us/

Strawberry Perl: https://strawberryperl.com/

Visual Studio Code: https://code.visualstudio.com/download

After installing NASM, we need to manually add C:\Program Files\NASM to the system PATH:

Visual Studio 2017 community version is enough:

When installing VS, make sure to select 'Python Development' and 'Desktop development with C++'.

Under 'Desktop development with C++', select 'VC++ 2015.3 v14.00 (v140) toolset for desktop':

You don't need to open Visual Studio at all in the whole process. Visual Studio Code is enough.

3. Download and extract source code

MAKE SURE to remove the symbol links inside the archives using tar and zip. Please find a Linux/macOS/Cygwin environment, first extract them:

tar -xzvf openssl-3.0.5.tar
tar -xzvf Python-3.9.14.tar

Then pack with zip:

zip -9 -r openssl-3.0.5.zip openssl-3.0.5/
zip -9 -r Python-3.9.14.zip Python-3.9.14/

Finally, transfer the open-3.0.5.zip and Python-3.9.14.zip to the destination Windows Server.

Let's assume we put them into C:\work

4. Compile OpenSSL with FIPS provider enabled

OpenSSL has a really detailed introduction about the compiling process and FIPS provider. It's strongly recommended to read the following documents:

Compile Process: https://github.com/openssl/openssl/blob/master/INSTALL.md

Install FIPS Module: https://github.com/openssl/openssl/blob/master/README-FIPS.md

Enable FIPS Module: https://www.openssl.org/docs/manmaster/man7/fips_module.html

Let's open 'x64 Native Tools Command Prompt for VS 2017' to perform the following commands.

Config OpenSSL:

cd C:\work\openssl-3.0.5
perl Configure VC-WIN64A no-pinshared enable-fips enable-acvp-tests --prefix=/usr/local/ssl --openssldir=/usr/local/ssl
  • no-pinshared: Don't pin the shared libraries in memory.
  • enable-fips: Build and install FIPS provider. Note we still need to enable it manually later on.
  • enable-acvp-tests: Build support for Automated Cryptographic Validation Protocol (ACVP) tests to pass FIPS validation process.

In case you're a newbie to compiling OpenSSL and encountered any errors, you can try to build with minimum configuration first with  perl Configure VC-WIN64A.

Build and Install OpenSSL:

nmake
nmake install

Take a cup of tea and wait for a while. On a machine with 4 Cores 8259C, 16GB DDR4 2933 it takes around 20 minutes. The compiled files are in C:\usr\local\ssl

If you made any modification to the OpenSSL codebase, please run nmake test before nmake install.

Default enable FIPS Provider:

Open C:\usr\local\ssl\openssl.cnf with Visual Studio Code:

  • Add .include /usr/local/ssl/fipsmodule.cnf
  • Add/Uncomment fips = fips_sect under [provider_sect]
  • Add alg_section = algorithm_sect under [provider_sect]
  • Add section algorithm_sect:
[algorithm_sect]
default_properties = fips=yes

Test if FIPS enabled:

cd C:\usr\local\ssl\bin
openssl.exe sha1 openssl.exe
openssl.exe md5 openssl.exe

The sha1 will succeed, and md5 will fail with:

Error setting digest
DC190000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto\evp\evp_fetch.c:349:Global default library context, Algorithm (MD5 : 102), Properties ()
DC190000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto\evp\digest.c:252:

Bingo, now we have a FIPS-enabled OpenSSL.

Note that do not copy compiled OpenSSL libraries to other machines since the FIPS check will not pass.

5. Compile Python 3.9 with custom OpenSSL

Compile a vanilla python first

cd C:\work\Python-3.9.14\PCbuild
build.bat -v -e -d -p x64

The script will download require components using NuGet into C:\work\Python-3.9.14\externals. Then perform the compilation.

If you encounter any issues, please solve them first before continuing.

Use custom OpenSSL

We need to change a few files in the python source. Open C:\work\Python-3.9.14 with Visual Studio Code.

PCBuild\python.props

Change opensslDir and opensslOutDir to C:\usr\local\ssl\:

PCBuild\openssl.props

Under PropertyGroup, change the first _DLLSuffix from -1-1 to -3, and add another _DLLSuffix like this:

<PropertyGroup>
  <_DLLSuffix>-3</_DLLSuffix>
  <_DLLSuffix Condition="$(Platform) == 'x64'">$(_DLLSuffix)-x64</_DLLSuffix>
  <_DLLSuffix Condition="$(Platform) == 'ARM'">$(_DLLSuffix)-arm</_DLLSuffix>
  <_DLLSuffix Condition="$(Platform) == 'ARM64'">$(_DLLSuffix)-arm64</_DLLSuffix>
</PropertyGroup>

Then add \bin after $(opensslOutDir) like this:

<ItemGroup>
  <_SSLDLL Include="$(opensslOutDir)\bin\libcrypto$(_DLLSuffix).dll" />
  <_SSLDLL Include="$(opensslOutDir)\bin\libcrypto$(_DLLSuffix).pdb" />
  <_SSLDLL Include="$(opensslOutDir)\bin\libssl$(_DLLSuffix).dll" />
  <_SSLDLL Include="$(opensslOutDir)\bin\libssl$(_DLLSuffix).pdb" />
</ItemGroup>

You can check if C:\usr\local\ssl\bin contains the above DLL and PDBs, if not you probably added no-share when configuring OpenSSL in the previous step.

Finally add $(opensslOutDir)lib  before libcrypto.lib and libssl.lib like this:

<Link>
  <AdditionalLibraryDirectories>$(opensslOutDir);%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
  <AdditionalDependencies>ws2_32.lib;$(opensslOutDir)lib\libcrypto.lib;$(opensslOutDir)lib\libssl.lib;%(AdditionalDependencies)</AdditionalDependencies>
</Link>

PCbuild\_ssl.vcxproj

In ClCompile change from $(opensslIncludeDir)\applink.c to $(opensslIncludeDir)\openssl\applink.c:

<ClCompile Include="$(opensslIncludeDir)\openssl\applink.c">
  <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;$(PreprocessorDefinitions)</PreprocessorDefinitions>
</ClCompile>

Re-Compile Python!

Here comes the exciting part, let's re-compile python. I usually delete amd64 and obj folders before re-compiling.

cd C:\work\Python-3.9.14\PCbuild
build.bat -v -e -d -p x64

If you followed the above steps, you should generate a valid python executable by now in C:\work\Python-3.9.14\PCbuild\amd64\python_d.exe. Most of the functions will work but not the APIs related with Win32API. To solve this, we need to generate the real Donald Trump python.exe.

Generate Python.exe and windows installer

Before that, we need to edit some files.

Tools\msi\msi.props

Change ssltag=-1-1 to ssltag=-3-x64:

Afterwards, we need to temporarily turn off the 'System cryptography: Use FIPS compliant algorithms for encrypting, hashing, and signing' in Group Policy -> Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> Security Options. We can turn that switch back on after building python.exe.

Let's start building:

cd C:\work\Python-3.9.14\Tools\msi
buildrelease.bat -x64

Finally, we have our lovely python.exe sitting under C:\work\Python-3.9.14\PCbuild\amd64 and python-3.9.14-amd64.exe under PCbuild\amd64\en-us:

Test if FIPS is enabled in the built python

The SHA1 will execute normally and MD5 will failed with an unsupported error.

import hashlib
print(hashlib.sha1("test_str".encode('utf-8')).hexdigest())
print(hashlib.md5("test_str".encode('utf-8')).hexdigest())

Congratulations! Now you have a working FIPS-Compliant python build. It's recommended to use the installer to install Python along with pip and other components before start doing serious jobs.

At last, very much appreciated for the following articles:

https://www.gyanblog.com/security/how-build-patch-python-3.9.x-fips-enable/#output

https://github.com/openssl/openssl/

https://zhuanlan.zhihu.com/p/387906689

https://blogs.oracle.com/cloud-infrastructure/post/windows-server-fips-compliance