Challenge

When attempting to access the Veeam Kasten for Kubernetes dashboard the following error occurs despite using the **cacertconfigmap.name** helm value:

**401 - Unauthorized**

This issue may also manifest as errors related to x509 in the gateway pod orauth-svc pod logs:

**x509: certificate signed by unknown authority**

Solution

Debugging Authentication

You can download the latest version of the debug tool (k10tools) from the KastenHQ GitHub page.

To debug certificate auth issues use the following command:

./k10tools debug auth

Copy

• If the connection to K10 succeeds without the need for a certificate, the following output will be seen:

  Dex:
  

  OIDC Provider URL: https://onkar-1.dev.do.kasten.io
  Release name: k10
  Dex well known URL:https://onkar-1.dev.do.kasten.io/k10/dex/.well-known/openid-configuration
  Trying to connect to Dex without TLS (insecureSkipVerify=false)
  Connection succeeded - OK

• If the connection to K10 does require a certificate, then the following output may be seen:

  Dex:
  

  OIDC Provider URL: https://example.com/k10/dex
  Release name: k10
  Dex well known URL:https://example.com/k10/dex/.well-known/openid-configuration
  Trying to connect to Dex without TLS (insecureSkipVerify=false)
  Connection failed, testing other options
  Trying to connect to Dex with TLS but verification disabled (insecureSkipVerify=true)
  Connection succeeded
  Trying to connect to Dex with TLS verification enabled and using a CA certificate
  Connection failed ({“message”:“HTTP Get for Dex’s well known endpoint failed”,“function”:"kasten.io/k10/kio/tools/k10primer/k10debugger.(*oidcOperate).testDexConnectivity",“linenumber”:29,“fields”:[{“name”:“statusCode”,“value”:null}],“cause”:{“Op”:“Get”,“URL”:"<https://example.com/k10/dex/.well-known/openid-configuration&gt;“,“Err”:{“Cert”:{“Raw”:”) - Error

In this example, the debugging tool was able to connect to Veeam Kasten for Kubernetes with SSL verification disabled. But when it tried to connect with SSL verification enabled, the verification failed. This indicates that the certificate installed with Veeam Kasten for Kubernetes may not be the right one for accessing the Veeam Kasten for Kubernetes dashboard.

Debugging CA Certificate

Use the following command to debug the CA certificate:

./k10tools debug ca-certificate

Copy

Example output:

 CA Certificate Checker:
    Fetching configmap which contains CA Certificate information : custom-ca-bundle-store
    Certificate exists in configmap  -  OK
    Found container : aggregatedapis-svc to extract certificate
    Certificate exists in container at /etc/ssl/certs/custom-ca-bundle.pem
    Certificates matched successfully  -  OK

Review Certificate Chain

To view the certificate chain involved when accessing the Veeam Kasten for Kubernetes Dashboard, use the following openssl command with the -host option set to the domain name of the dashboard.

openssl s_client -host k10-dashboard-example.com -port 443 -prexit -showcerts  

Copy

Within the output, you may see Root CA and Intermediate CA certificates in the chain. Please copy all of the CA certificates into a file named custom-ca-bundle.pem and review: ‘Using Trusted Root Certificate Authority Certificates for TLS’.

Example Command Output

In this output, there are 4 certificates belonging to the following Certificate Authorities:

• The Amazon Root CA 1

• The Server CA 1B

• Starfield Services Root Certificate Authority

• Starfield Class 2 Certification Authority

  openssl s_client -host kasten.io -port 443 -prexit -showcerts

  CONNECTED(00000005)
  depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
  verify return:1
  depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
  verify return:1
  depth=0 CN = kasten.io
  verify return:1

  Certificate chain
  0 s:CN = kasten.io
     i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
  -----BEGIN CERTIFICATE-----
  <cert>
  -----END CERTIFICATE-----
  1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
     i:C = US, O = Amazon, CN = Amazon Root CA 1
  -----BEGIN CERTIFICATE-----
  <cert>
  -----END CERTIFICATE-----
  2 s:C = US, O = Amazon, CN = Amazon Root CA 1
     i:C = US, ST = Arizona, L = Scottsdale, O = “Starfield Technologies, Inc.”, CN = Starfield Services Root Certificate Authority - G2
  -----BEGIN CERTIFICATE-----
  <cert>
  -----END CERTIFICATE-----
  3 s:C = US, ST = Arizona, L = Scottsdale, O = “Starfield Technologies, Inc.”, CN = Starfield Services Root Certificate Authority - G2
     i:C = US, O = “Starfield Technologies, Inc.”, OU = Starfield Class 2 Certification Authority
  -----BEGIN CERTIFICATE-----
  <cert>
  -----END CERTIFICATE-----

  Server certificate
  subject=CN = kasten.io
  issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

  No client certificate CA names sent
  Peer signing digest: SHA256
  Peer signature type: RSA-PSS
  Server Temp Key: X25519, 253 bits

  SSL handshake has read 5369 bytes and written 375 bytes
  Verification: OK

  New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
  Server public key is 2048 bit
  Secure Renegotiation IS NOT supported
  Compression: NONE
  Expansion: NONE
  No ALPN negotiated
  Early data was not sent
  Verify return code: 0 (ok)

To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.