There are different methods or commands a user can use to generate a Private Key and Certificate Signing Request (CSR) through OpenSSL.
If somehow a user generated a Private Key without a password, this will not be accepted by the system when the key is submitted along with the SSL Certificate and Intermediate Certificate under Setup > Site Builder > Domains > Secure Domains tab, as it would ask the user for a password.
To replace this existing key and generate a new Private Key with a password which is compatible with the SSL certificate, run this command in OpenSSL:
rsa -in [Existing Private Key].key -out [New Private Key].key -des3
This command will generate a new Private Key with a password which the user can use with the SSL certificate and Intermediate certificate to setup their custom checkout domain.