This is a continuation of our series of blogs about deployment of MAS. In this case, we are going to describe how to configure Automatic Certificate Management via the JetStack Cert Manager or the IBM Cert Manager Operators. Installing either is not part of this Blog Entry and the IBM Cert Manager should come automatically with the MAS Core as a dependency; however, we prefer to use the JetStack Cert Manager as it supports wider use cases than the IBM Cert Manager (such as split horizon DNS setups). We should be also aware of that IBM has officially supported the IBM Cert Manager and made statements that they will support other Cert Managers (RedHat’s or JetStack). In our experience, the JetStack Cert Manager works well with the MAS Core (and applications) up to the latest version at the time of writing 8.11.10.
There are some considerations to having Certificates automatically issued and maintained (i.e. re-issued before they expire) using Let’s Encrypt with Cert Manager, these are being described below:
- The MAS Core creates wildcard Certificates and, at the time of writing, the only way of generating wildcard certificates with Let’s Encrypt is by using the ACME Issuer and the DNS01 Challenge Provider.
- The public domain or sub-domain (part of the URL) for which these Certificates will be generated must be hosted in one of the supported providers by the above ACME/DNS01 combination. The link describing which providers are supported is included: https://cert-manager.io/docs/configuration/acme/dns01/
- Depending on the DNS provider that is being used, different authentication mechanisms need to be implemented to allow Cert Manager to insert the Challenge (TXT) record into the DNS for it to be verified by Let’s Encrypt to assert domain ownership. Consult the above link for details. In our example, we are going to use CloudFlare as the DNS provider for a test domain and a configured API Token with the right permissions in the domain.
- There is also the possibility of having a split horizon DNS setup where the OpenShift Cluster uses a local/private network resolver that resolves Domain records and that is separate from the Public DNS for the same Domain. In such case, without further configuration, the Challenge (TXT) record cannot be confirmed by Cert Manager as the local/private resolver will not have it. We will discuss how to address and any limitations below.
First, as stated above, we need to ensure that for our Domain, we have it hosted in one of the supported providers and that we generated the proper credentials to use dynamic DNS to insert records automatically (the mechanism is specific to each provider). In our case, we created a CloudFlare API Token (detailed documentation at https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) for a test domain with the following configuration:
Notice that although the official Cert Manager configuration states that under “Zone Resources” we should select “All Zones” instead of “Specific Zones” but we find more secure to choose “Specific Zone” and then the zone in the zone selector to its right in case the same account is being used to manage multiple domains.
Next, we should securely store the API Token as once generated it is not shown again. In case we want to verify the API Token, we could make the following call:
curl "https://api.cloudflare.com/client/v4/user/tokens/verify" -H "Authorization: Bearer <API_TOKEN>"
And the CloudFlare API should respond “This API Token is valid and active". Be also aware that if we are using a CloudFlare Enterprise Account, as per https://developers.cloudflare.com/fundamentals/api/how-to/control-api-access/ then we need to ensure the “Enable API Access” setting is enabled for the user for which the API Token was generated.
Next, we should create a standard OpenShift Secret on the same project/namespace where the Cert Manager Pods are running (in our case: “openshift-operators” for the JetStack Cert Manager or “ibm-common-services” for the IBM Cert Manager). Like the below:
kind: Secret
apiVersion: v1
metadata:
name: cloudflare-api-token-for-sample-domain
namespace: ${CERT_MANAGER_NAMESPACE}
stringData:
api-token: ${CLOUDFLARE_API_TOKEN}
Notice that the Secret syntax depends on the DNS provider and in our case, it is specific to CloudFlare. Ensure to replace the “namespace” and “api-token” entries with the right values. The “name” can also be modified to suit our preferences but be sure to also modify any references to it below.
Next, we are going to create a Cluster Issuer for the Domain (let’s say our domain is “sample.domain”), by submitting it as in the below example:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: sample-domain-dns01-cluster-issuer
spec:
acme:
email: ${ORGANIZATION_SUPPORT_EMAIL}
preferredChain: ISRG Root X1
privateKeySecretRef:
name: sample-domain-dns01-cluster-issuer-privkey-secret
server: 'https://acme-v02.api.letsencrypt.org/directory'
solvers:
- dns01:
cloudflare:
apiTokenSecretRef:
name: cloudflare-api-token-for-sample-domain
key: api-token
selector:
dnsZones:
- sample.domain
Ensure to replace the “email” and “dnsZones” settings with proper values. Also, the “name” and “privateKeySecretRef.name” can be modified to our preferences and any further references to the ClusterIssuer should incorporate the correct name.
The submission of the above YAML should cause the Cert Manager to process it and it should populate its “status” section with a Condition of “type: Ready” reading “The ACME account was registered with the ACME server”.
Next, to configure a specific MAS Core Instance to issue Public Certificates for their URLs, we should locate the “Suite” Custom Resource for the MAS Core Instance and add the following to their “spec” section:
certManagerNamespace: ${CERT_MANAGER_NAMESPACE}
certificateIssuer:
duration: 2160h
name: sample-domain-dns01-cluster-issuer
renewBefore: 360h
domain: dev.ocp01.mas.sample.domain
Notice that the specified “duration” and “renewBefore” have been set to values that corresponds to the current policy of Let’s Encrypt to issue Certificates valid for 3 months that can be renewed up to a month before expiration. Also, ensure to use correct values for “certManagerNamespace” and “name” (as the Cluster Issuer name). Finally, the “domain” entry specifies the “base” URL for the MAS Core instance. All other URLs for the instance (and any installed or to be installed applications) will be derived from that “root” URL (and of course it must belong to the configured domain, in our case “sample.domain”).
The above should be all the configurations we need to automatically issue and refresh valid Let’s Encrypt Certificates for our MAS Instances.
As a recommendation, we could also check that a valid Certificate gets processed and issued before we submit the MAS Core configuration. We could do so by creating a Certificate resource and watching the Cluster Events or going to the Cert Manager Controller Pod and watching its logs for any messages. Below there is an example of such Certificate for testing purposes:
metadata:
name: test-wildcard-cert
namespace: default
spec:
commonName: 'tst.ocp01.mas.sample.domain'
dnsNames:
- tst.ocp01.mas.sample.domain
- '*.tst.ocp01.mas.sample.domain'
issuerRef:
kind: ClusterIssuer
name: sample-domain-dns01-cluster-issuer
secretName: test-wildcard-cert-secret
Notice that we should replace for proper values for the “commonName”, “dnsNames” and the Cluster Issuer “name” under “issuerRef”. Also, the “namespace” the Secret will be generated and the “secretName” that will contain the generated actual Certificate as well.
If everything goes well, after a while, the “status” section of the Certificate should be populated with a Condition of “type: Ready” reading “Certificate is up to date and has not expired”.
Also, we could check the actual Certificate contents by going to the related generated Secret (in our case “test-wildcard-cert-secret” in the “default” namespace) and looking for it “tls.crt” entry (if using the OpenShift Console, press the <Reveal Values> button first), then copying the contents and pasting in any online certificate decoder such as https://www.sslshopper.com/certificate-decoder.html.
Last, to bypass the Local DNS resolution for the Cert Manager Controller (to work around a split horizon DNS setup as mentioned above), we should add the following lines to the installed JetStack Cert Manager Cluster Service Version (notice that this configuration only applies to the JetStack Cert Manager).
spec:
containers:
- args:
....<some non-related arguments here>....
- --dns01-recursive-nameservers="<external-resolver-ip>:53"
- --dns01-recursive-nameservers-only
Where “<external-resolver-ip>” is the IP address of an external resolver such as Google DNS (8.8.8.8) or CloudFlare (1.1.1.1), for example.
And finally, to have the JetStack Cert Manager installed and for the IBM Cert Manager to not to get automatically installed and pushed by dependencies on the system, we should either create or update the “ibm-cpp-config” Config Map on the “ibm-common-services” namespace by adding deployCSCertManagerOperands: "false" to its data section as shown in the below example:
kind: ConfigMap
apiVersion: v1
metadata:
name: ibm-cpp-config
namespace: ibm-common-services
data:
deployCSCertManagerOperands: "false"