One post tagged with "mTLS"

Imagine a situation where you have multiple different clusters, each running an Istio service mesh and which are federated together to talk to each other. Now also imagine that these clusters are networked together such that each pod IP is uniquely addressable and able to be communicated with from any other cluster.

Context​

I faced a situation like this, where I needed to group endpoints into logical hostnames that represented services backed by those endpoints. Because endpoints could live anywhere, in any cluster, I landed on using a ServiceEntry to register the hostname and WorkloadEntries to represent the endpoints service that hostname.

But a problem came on enabling PeerAuthentication:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod-istio-system # The namespace of our istio installation
spec:
  mtls:
    mode: STRICT

Curl stopped working!

$ curl global-echo
upstream connect error or disconnect/reset before headers. reset reason: connection termination

Through some debugging, I was able to get it to:

$ curl global-echo
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, 
transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

But to find the true root cause and solution took some digging.

Let's get into it