Securing Gateways with HTTPS With a File Mount-Based Approach

The Control Ingress Traffic task describes how to configure an ingress gateway to expose the HTTP endpoint of a service to external traffic. This task shows how to do it but using HTTPS access to the service with either simple or mutual TLS. The private key, server certificate, and the root certificate required by mutual TLS are configured using a file mount based approach.

Before you begin

  1. Perform the steps in the Before you begin and Determining the ingress IP and ports sections of the Control Ingress Traffic task. After performing those steps you should have Istio and the httpbin service deployed, and the environment variables INGRESS_HOST and SECURE_INGRESS_PORT set.

  2. For macOS users, verify that you use curl compiled with the LibreSSL library:

    $ curl --version | grep LibreSSL
    curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
    

    If a version of LibreSSL is printed as in the output above, your curl should work correctly with the instructions in this task. Otherwise, try another installation of curl, for example on a Linux machine.

Generate client and server certificates and keys

For this task you can use your favorite tool to generate certificates and keys. This example uses a script from the https://github.com/nicholasjackson/mtls-go-example repository.

  1. Clone the https://github.com/nicholasjackson/mtls-go-example repository:

    $ git clone https://github.com/nicholasjackson/mtls-go-example
    
  2. Change directory to the cloned repository:

    $ pushd mtls-go-example
    
  3. Generate the certificates for httpbin.example.com. Change password to any value you like in the following command:

    $ ./generate.sh httpbin.example.com password
    

    When prompted, select y for all the questions. The command will generate four directories: 1_root, 2_intermediate, 3_application, and 4_client containing the client and server certificates you use in the procedures below.

  4. Move the certificates into a directory named httpbin.example.com:

    $ mkdir ~+1/httpbin.example.com && mv 1_root 2_intermediate 3_application 4_client ~+1/httpbin.example.com
    
  5. Go back to your previous directory:

    $ popd
    

Configure a TLS ingress gateway with a file mount-based approach

In this section you configure an ingress gateway with port 443 to handle HTTPS traffic. You first create a secret with a certificate and a private key. The secret is mounted to a file on the /etc/istio/ingressgateway-certs path. You can then create a gateway definition that configures a server on port 443.

  1. Create a Kubernetes secret to hold the server’s certificate and private key. Use kubectl to create the secret istio-ingressgateway-certs in namespace istio-system . The Istio gateway will load the secret automatically.

    $ kubectl create -n istio-system secret tls istio-ingressgateway-certs --key httpbin.example.com/3_application/private/httpbin.example.com.key.pem --cert httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem
    secret "istio-ingressgateway-certs" created
    

    Note that by default all the pods in the istio-system namespace can mount this secret and access the private key. You may want to deploy the ingress gateway in a separate namespace and create the secret there, so that only the ingress gateway pod will be able to mount it.

    Verify that tls.crt and tls.key have been mounted in the ingress gateway pod:

    $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-certs
    
  2. Define a Gateway with a server section for port 443.

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      name: httpbin-gateway
    spec:
      selector:
        istio: ingressgateway # use istio default ingress gateway
      servers:
      - port:
          number: 443
          name: https
          protocol: HTTPS
        tls:
          mode: SIMPLE
          serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
          privateKey: /etc/istio/ingressgateway-certs/tls.key
        hosts:
        - "httpbin.example.com"
    EOF
    
  3. Configure routes for traffic entering via the Gateway. Define the same VirtualService as in the Control Ingress Traffic task:

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: httpbin
    spec:
      hosts:
      - "httpbin.example.com"
      gateways:
      - httpbin-gateway
      http:
      - match:
        - uri:
            prefix: /status
        - uri:
            prefix: /delay
        route:
        - destination:
            port:
              number: 8000
            host: httpbin
    EOF
    
  4. Access the httpbin service with HTTPS by sending an https request using curl to SECURE_INGRESS_PORT.

    The --resolve flag instructs curl to supply the SNI value httpbin.example.com when accessing the gateway IP over TLS. The --cacert option instructs curl to use your generated certificate to verify the server.

    By sending the request to the /status/418 URL path, you get a nice visual clue that your httpbin service was indeed accessed. The httpbin service will return the 418 I’m a Teapot code.

    $ curl -v -HHost:httpbin.example.com --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    ...
    Server certificate:
      subject: C=US; ST=Denial; L=Springfield; O=Dis; CN=httpbin.example.com
      start date: Jun 24 18:45:18 2018 GMT
      expire date: Jul  4 18:45:18 2019 GMT
      common name: httpbin.example.com (matched)
      issuer: C=US; ST=Denial; O=Dis; CN=httpbin.example.com
    SSL certificate verify ok.
    ...
    HTTP/2 418
    ...
    -=[ teapot ]=-
    
       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`
    

    Look for the Server certificate section in the curl output and specifically a line with the matched common name: common name: httpbin.example.com (matched). The line SSL certificate verify ok in the output indicates that the server’s certificate was verified successfully. If all went well, you should also see a returned status of 418 along with a nice drawing of a teapot.

Configure a mutual TLS ingress gateway

In this section you extend your gateway’s definition from the previous section to support mutual TLS between external clients and the gateway.

  1. Create a Kubernetes Secret to hold the CA certificate that the server will use to verify its clients. Create the secret istio-ingressgateway-ca-certs in namespace istio-system using kubectl. The Istio gateway will automatically load the secret.

    $ kubectl create -n istio-system secret generic istio-ingressgateway-ca-certs --from-file=httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem
    secret "istio-ingressgateway-ca-certs" created
    
  2. Redefine your previous Gateway to change the tls mode to MUTUAL and to specify caCertificates:

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      name: httpbin-gateway
    spec:
      selector:
        istio: ingressgateway # use istio default ingress gateway
      servers:
      - port:
          number: 443
          name: https
          protocol: HTTPS
        tls:
          mode: MUTUAL
          serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
          privateKey: /etc/istio/ingressgateway-certs/tls.key
          caCertificates: /etc/istio/ingressgateway-ca-certs/ca-chain.cert.pem
        hosts:
        - "httpbin.example.com"
    EOF
    
  3. Access the httpbin service by HTTPS as in the previous section:

    $ curl -HHost:httpbin.example.com --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    curl: (35) error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
    

    This time you will get an error since the server refuses to accept unauthenticated requests. You need to pass curl a client certificate and your private key for signing the request.

  4. Resend the previous request by curl, this time passing as parameters your client certificate (additional --cert option) and your private key (the --key option):

    $ curl -HHost:httpbin.example.com --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem --cert httpbin.example.com/4_client/certs/httpbin.example.com.cert.pem --key httpbin.example.com/4_client/private/httpbin.example.com.key.pem https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    
    -=[ teapot ]=-
    
       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`
    

    This time the server performed client authentication successfully and you received the pretty teapot drawing again.

Configure a TLS ingress gateway for multiple hosts

In this section you will configure an ingress gateway for multiple hosts, httpbin.example.com and bookinfo.com. The ingress gateway will present to clients a unique certificate corresponding to each requested server.

Unlike the previous sections, the Istio default ingress gateway will not work out of the box because it is only preconfigured to support one secure host. You’ll need to first configure and redeploy the ingress gateway server with another secret, before you can use it to handle a second host.

Generate client and server certificates and keys for bookinfo.com

Perform the same steps as in Generate client and server certificates and keys, only this time for host bookinfo.com instead of httpbin.example.com.

  1. Change directory to the cloned repository:

    $ pushd mtls-go-example
    
  2. Generate the certificates for bookinfo.com. Change password to any value you like in the following command:

    $ ./generate.sh bookinfo.com password
    

    When prompted, select y for all the questions.

  3. Move the certificates into a directory named bookinfo.com:

    $ mkdir ~+1/bookinfo.com && mv 1_root 2_intermediate 3_application 4_client ~+1/bookinfo.com
    
  4. Go back to your previous directory:

    $ popd
    

Redeploy istio-ingressgateway with the new certificates

  1. Create a new secret to hold the certificate for bookinfo.com:

    $ kubectl create -n istio-system secret tls istio-ingressgateway-bookinfo-certs --key bookinfo.com/3_application/private/bookinfo.com.key.pem --cert bookinfo.com/3_application/certs/bookinfo.com.cert.pem
    secret "istio-ingressgateway-bookinfo-certs" created
    
  2. Generate the istio-ingressgateway deployment with a volume to be mounted from the new secret. Use the same options you used for generating your istio.yaml:

    $ helm template $HOME/istio-fetch/istio --name istio --namespace istio-system -x charts/gateways/templates/deployment.yaml --set gateways.istio-egressgateway.enabled=false \
    --set 'gateways.istio-ingressgateway.secretVolumes[0].name'=ingressgateway-certs \
    --set 'gateways.istio-ingressgateway.secretVolumes[0].secretName'=istio-ingressgateway-certs \
    --set 'gateways.istio-ingressgateway.secretVolumes[0].mountPath'=/etc/istio/ingressgateway-certs \
    --set 'gateways.istio-ingressgateway.secretVolumes[1].name'=ingressgateway-ca-certs \
    --set 'gateways.istio-ingressgateway.secretVolumes[1].secretName'=istio-ingressgateway-ca-certs \
    --set 'gateways.istio-ingressgateway.secretVolumes[1].mountPath'='/etc/istio/ingressgateway-ca-certs \
    --set 'gateways.istio-ingressgateway.secretVolumes[2].name'=ingressgateway-bookinfo-certs \
    --set 'gateways.istio-ingressgateway.secretVolumes[2].secretName'=istio-ingressgateway-bookinfo-certs \
    --set 'gateways.istio-ingressgateway.secretVolumes[2].mountPath'=/etc/istio/ingressgateway-bookinfo-certs > \
    $HOME/istio-ingressgateway.yaml
    
  3. Redeploy istio-ingressgateway:

    $ kubectl apply -f $HOME/istio-ingressgateway.yaml
    deployment "istio-ingressgateway" configured
    
  4. Verify that the key and certificate have been successfully loaded in the istio-ingressgateway pod:

    $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-bookinfo-certs
    

    tls.crt and tls.key should appear in the directory contents.

Configure traffic for the bookinfo.com host

  1. Deploy the Bookinfo sample application, without a gateway:

    Zip
    $ kubectl apply -f @samples/bookinfo/platform/kube/bookinfo.yaml@
    
  2. Define a gateway for bookinfo.com:

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      name: bookinfo-gateway
    spec:
      selector:
        istio: ingressgateway # use istio default ingress gateway
      servers:
      - port:
          number: 443
          name: https-bookinfo
          protocol: HTTPS
        tls:
          mode: SIMPLE
          serverCertificate: /etc/istio/ingressgateway-bookinfo-certs/tls.crt
          privateKey: /etc/istio/ingressgateway-bookinfo-certs/tls.key
        hosts:
        - "bookinfo.com"
    EOF
    
  3. Configure the routes for bookinfo.com. Define a VirtualService like the one in samples/bookinfo/networking/bookinfo-gateway.yaml:

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: bookinfo
    spec:
      hosts:
      - "bookinfo.com"
      gateways:
      - httpbin-gateway
      http:
      - match:
        - uri:
            exact: /productpage
        - uri:
            exact: /login
        - uri:
            exact: /logout
        - uri:
            prefix: /api/v1/products
        route:
        - destination:
            host: productpage
            port:
              number: 9080
    EOF
    
  4. Send a request to the Bookinfo productpage:

    $ curl -o /dev/null -s -v -w "%{http_code}\n" -HHost:bookinfo.com --resolve bookinfo.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert bookinfo.com/2_intermediate/certs/ca-chain.cert.pem -HHost:bookinfo.com https://bookinfo.com:$SECURE_INGRESS_PORT/productpage
    ...
    Server certificate:
      subject: C=US; ST=Denial; L=Springfield; O=Dis; CN=bookinfo.com
      start date: Aug 12 13:50:05 2018 GMT
      expire date: Aug 22 13:50:05 2019 GMT
      common name: bookinfo.com (matched)
      issuer: C=US; ST=Denial; O=Dis; CN=bookinfo.com
    SSL certificate verify ok.
    ...
    200
    
  5. Verify that httbin.example.com is accessible as previously. Send a request to it and see again the teapot you should already love:

    $ curl -v -HHost:httpbin.example.com --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    ...
    -=[ teapot ]=-
    
       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`
    

Troubleshooting

  • Inspect the values of the INGRESS_HOST and SECURE_INGRESS_PORT environment variables. Make sure they have valid values, according to the output of the following commands:

    $ kubectl get svc -n istio-system
    $ echo INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT
    
  • Verify that the key and the certificate are successfully loaded in the istio-ingressgateway pod:

    $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-certs
    

    tls.crt and tls.key should exist in the directory contents.

  • If you created the istio-ingressgateway-certs secret, but the key and the certificate are not loaded, delete the ingress gateway pod and force the ingress gateway pod to restart and reload key and certificate.

    $ kubectl delete pod -n istio-system -l istio=ingressgateway
    
  • Verify that the Subject is correct in the certificate of the ingress gateway:

    $ kubectl exec -i -n istio-system $(kubectl get pod -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')  -- cat /etc/istio/ingressgateway-certs/tls.crt | openssl x509 -text -noout | grep 'Subject:'
        Subject: C=US, ST=Denial, L=Springfield, O=Dis, CN=httpbin.example.com
    
  • Verify that the proxy of the ingress gateway is aware of the certificates:

    $ kubectl exec -ti $(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system -- curl  127.0.0.1:15000/certs
    {
      "ca_cert": "",
      "cert_chain": "Certificate Path: /etc/istio/ingressgateway-certs/tls.crt, Serial Number: 100212, Days until Expiration: 370"
    }
    
  • Check the log of istio-ingressgateway for error messages:

    $ kubectl logs -n istio-system -l istio=ingressgateway
    
  • For macOS users, verify that you use curl compiled with the LibreSSL library, as described in the Before you begin section.

Troubleshooting for mutual TLS

In addition to the steps in the previous section, perform the following:

  • Verify that the CA certificate is loaded in the istio-ingressgateway pod:

    $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-ca-certs
    

    ca-chain.cert.pem should exist in the directory contents.

  • If you created the istio-ingressgateway-ca-certs secret, but the CA certificate is not loaded, delete the ingress gateway pod and force it to reload the certificate:

    $ kubectl delete pod -n istio-system -l istio=ingressgateway
    
  • Verify that the Subject is correct in the CA certificate of the ingress gateway:

    $ kubectl exec -i -n istio-system $(kubectl get pod -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')  -- cat /etc/istio/ingressgateway-ca-certs/ca-chain.cert.pem | openssl x509 -text -noout | grep 'Subject:'
    Subject: C=US, ST=Denial, L=Springfield, O=Dis, CN=httpbin.example.com
    

Cleanup

  1. Delete the Gateway configuration, the VirtualService, and the secrets:

    $ kubectl delete gateway --ignore-not-found=true httpbin-gateway bookinfo-gateway
    $ kubectl delete virtualservice httpbin
    $ kubectl delete --ignore-not-found=true -n istio-system secret istio-ingressgateway-certs istio-ingressgateway-ca-certs
    $ kubectl delete --ignore-not-found=true virtualservice bookinfo
    
  2. Delete the directories of the certificates and the repository used to generate them:

    $ rm -rf httpbin.example.com bookinfo.com mtls-go-example
    
  3. Remove the file you used for redeployment of istio-ingressgateway:

    $ rm -f $HOME/istio-ingressgateway.yaml
    
  4. Shutdown the httpbin service:

    Zip
    $ kubectl delete --ignore-not-found=true -f @samples/httpbin/httpbin.yaml@