Remotely Accessing Telemetry Addons

This task shows how to configure Istio to expose and access the telemetry addons outside of a cluster.

Configuring remote access

Remote access to the telemetry addons can be configured in a number of different ways. This task covers two basic access methods: secure (via HTTPS) and insecure (via HTTP). The secure method is strongly recommended for any production or sensitive environment. Insecure access is simpler to set up, but will not protect any credentials or data transmitted outside of your cluster.

Option 1: Secure access (HTTPS)

A server certificate is required for secure access. Follow these steps to install and configure server certificates for a domain that you control.

You may use self-signed certificates instead. Visit our Securing Gateways with HTTPS Using Secret Discovery Service task for general information on using self-signed certificates to access in-cluster services.

  1. Install Istio in your cluster and enable the cert-manager flag and configure istio-ingressgateway to use the Secret Discovery Service.

    To install Istio accordingly, use the following Helm installation options:

    • --set gateways.enabled=true
    • --set gateways.istio-ingressgateway.enabled=true
    • --set gateways.istio-ingressgateway.sds.enabled=true
    • --set certmanager.enabled=true
    • --set certmanager.email=mailbox@donotuseexample.com

    To additionally install the telemetry addons, use the following Helm installation options:

    • Grafana: --set grafana.enabled=true
    • Kiali: --set kiali.enabled=true
    • Prometheus: --set prometheus.enabled=true
    • Tracing: --set tracing.enabled=true
  2. Configure the DNS records for your domain.

    1. Get the external IP address of the istio-ingressgateway.

      $ kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
      <IP ADDRESS OF CLUSTER INGRESS>
      
    2. Set an environment variable to hold your target domain.

      $ TELEMETRY_DOMAIN=<your.desired.domain>
      
    3. Point your desired domain at that external IP address via your domain provider.

      The mechanism for achieving this step varies by provider. Here are a few example documentation links:

    4. Verify that the DNS records are correct.

      $ dig +short $TELEMETRY_DOMAIN
      <IP ADDRESS OF CLUSTER INGRESS>
      
  3. Generate a server certificate

    $ cat <<EOF | kubectl apply -f -
    apiVersion: certmanager.k8s.io/v1alpha1
    kind: Certificate
    metadata:
      name: telemetry-gw-cert
      namespace: istio-system
    spec:
      secretName: telemetry-gw-cert
      issuerRef:
        name: letsencrypt
        kind: ClusterIssuer
      commonName: $TELEMETRY_DOMAIN
      dnsNames:
      - $TELEMETRY_DOMAIN
      acme:
        config:
        - http01:
            ingressClass: istio
          domains:
          - $TELEMETRY_DOMAIN
    ---
    EOF
    certificate.certmanager.k8s.io "telemetry-gw-cert" created
    
  4. Wait until the server certificate is ready.

    $ JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status}{end}{end}' && kubectl -n istio-system get certificates -o jsonpath="$JSONPATH"
    telemetry-gw-cert:Ready=True
    
  5. Apply networking configuration for the telemetry addons.

    1. Apply the following configuration to expose Grafana:

      $ cat <<EOF | kubectl apply -f -
      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: grafana-gateway
        namespace: istio-system
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 15031
            name: https-grafana
            protocol: HTTPS
          tls:
            mode: SIMPLE
            serverCertificate: sds
            privateKey: sds
            credentialName: telemetry-gw-cert
          hosts:
          - "$TELEMETRY_DOMAIN"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: grafana-vs
        namespace: istio-system
      spec:
        hosts:
        - "$TELEMETRY_DOMAIN"
        gateways:
        - grafana-gateway
        http:
        - match:
          - port: 15031
          route:
          - destination:
              host: grafana
              port:
                number: 3000
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: grafana
        namespace: istio-system
      spec:
        host: grafana
        trafficPolicy:
          tls:
            mode: DISABLE
      ---
      EOF
      gateway.networking.istio.io "grafana-gateway" configured
      virtualservice.networking.istio.io "grafana-vs" configured
      destinationrule.networking.istio.io "grafana" configured
      
    2. Apply the following configuration to expose Kiali:

      $ cat <<EOF | kubectl apply -f -
      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: kiali-gateway
        namespace: istio-system
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 15029
            name: https-kiali
            protocol: HTTPS
          tls:
            mode: SIMPLE
            serverCertificate: sds
            privateKey: sds
            credentialName: telemetry-gw-cert
          hosts:
          - "$TELEMETRY_DOMAIN"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: kiali-vs
        namespace: istio-system
      spec:
        hosts:
        - "$TELEMETRY_DOMAIN"
        gateways:
        - kiali-gateway
        http:
        - match:
          - port: 15029
          route:
          - destination:
              host: kiali
              port:
                number: 20001
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: kiali
        namespace: istio-system
      spec:
        host: kiali
        trafficPolicy:
          tls:
            mode: DISABLE
      ---
      EOF
      gateway.networking.istio.io "kiali-gateway" configured
      virtualservice.networking.istio.io "kiali-vs" configured
      destinationrule.networking.istio.io "kiali" configured
      
    3. Apply the following configuration to expose Prometheus:

      $ cat <<EOF | kubectl apply -f -
      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: prometheus-gateway
        namespace: istio-system
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 15030
            name: https-prom
            protocol: HTTPS
          tls:
            mode: SIMPLE
            serverCertificate: sds
            privateKey: sds
            credentialName: telemetry-gw-cert
          hosts:
          - "$TELEMETRY_DOMAIN"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: prometheus-vs
        namespace: istio-system
      spec:
        hosts:
        - "$TELEMETRY_DOMAIN"
        gateways:
        - prometheus-gateway
        http:
        - match:
          - port: 15030
          route:
          - destination:
              host: prometheus
              port:
                number: 9090
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: prometheus
        namespace: istio-system
      spec:
        host: prometheus
        trafficPolicy:
          tls:
            mode: DISABLE
      ---
      EOF
      gateway.networking.istio.io "prometheus-gateway" configured
      virtualservice.networking.istio.io "prometheus-vs" configured
      destinationrule.networking.istio.io "prometheus" configured
      
    4. Apply the following configuration to expose the tracing service:

      $ cat <<EOF | kubectl apply -f -
      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: tracing-gateway
        namespace: istio-system
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 15032
            name: https-tracing
            protocol: HTTPS
          tls:
            mode: SIMPLE
            serverCertificate: sds
            privateKey: sds
            credentialName: telemetry-gw-cert
          hosts:
          - "$TELEMETRY_DOMAIN"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: tracing-vs
        namespace: istio-system
      spec:
        hosts:
        - "$TELEMETRY_DOMAIN"
        gateways:
        - tracing-gateway
        http:
        - match:
          - port: 15032
          route:
          - destination:
              host: tracing
              port:
                number: 80
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: tracing
        namespace: istio-system
      spec:
        host: tracing
        trafficPolicy:
          tls:
            mode: DISABLE
      ---
      EOF
      gateway.networking.istio.io "tracing-gateway" configured
      virtualservice.networking.istio.io "tracing-vs" configured
      destinationrule.networking.istio.io "tracing" configured
      
  6. Visit the telemetry addons via your browser.

    • Kiali: https://$TELEMETRY_DOMAIN:15029/
    • Prometheus: https://$TELEMETRY_DOMAIN:15030/
    • Grafana: https://$TELEMETRY_DOMAIN:15031/
    • Tracing: https://$TELEMETRY_DOMAIN:15032/

Option 2: Insecure access (HTTP)

  1. Install Istio in your cluster with your desired telemetry addons.

    To additionally install the telemetry addons, use the following Helm installation options:

    • Grafana: --set grafana.enabled=true
    • Kiali: --set kiali.enabled=true
    • Prometheus: --set prometheus.enabled=true
    • Tracing: --set tracing.enabled=true
  2. Apply networking configuration for the telemetry addons.

    1. Apply the following configuration to expose Grafana:

      $ cat <<EOF | kubectl apply -f -
      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: grafana-gateway
        namespace: istio-system
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 15031
            name: http-grafana
            protocol: HTTP
          hosts:
          - "*"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: grafana-vs
        namespace: istio-system
      spec:
        hosts:
        - "*"
        gateways:
        - grafana-gateway
        http:
        - match:
          - port: 15031
          route:
          - destination:
              host: grafana
              port:
                number: 3000
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: grafana
        namespace: istio-system
      spec:
        host: grafana
        trafficPolicy:
          tls:
            mode: DISABLE
      ---
      EOF
      gateway.networking.istio.io "grafana-gateway" configured
      virtualservice.networking.istio.io "grafana-vs" configured
      destinationrule.networking.istio.io "grafana" configured
      
    2. Apply the following configuration to expose Kiali:

      $ cat <<EOF | kubectl apply -f -
      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: kiali-gateway
        namespace: istio-system
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 15029
            name: http-kiali
            protocol: HTTP
          hosts:
          - "*"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: kiali-vs
        namespace: istio-system
      spec:
        hosts:
        - "*"
        gateways:
        - kiali-gateway
        http:
        - match:
          - port: 15029
          route:
          - destination:
              host: kiali
              port:
                number: 20001
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: kiali
        namespace: istio-system
      spec:
        host: kiali
        trafficPolicy:
          tls:
            mode: DISABLE
      ---
      EOF
      gateway.networking.istio.io "kiali-gateway" configured
      virtualservice.networking.istio.io "kiali-vs" configured
      destinationrule.networking.istio.io "kiali" configured
      
    3. Apply the following configuration to expose Prometheus:

      $ cat <<EOF | kubectl apply -f -
      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: prometheus-gateway
        namespace: istio-system
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 15030
            name: http-prom
            protocol: HTTP
          hosts:
          - "*"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: prometheus-vs
        namespace: istio-system
      spec:
        hosts:
        - "*"
        gateways:
        - prometheus-gateway
        http:
        - match:
          - port: 15030
          route:
          - destination:
              host: prometheus
              port:
                number: 9090
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: prometheus
        namespace: istio-system
      spec:
        host: prometheus
        trafficPolicy:
          tls:
            mode: DISABLE
      ---
      EOF
      gateway.networking.istio.io "prometheus-gateway" configured
      virtualservice.networking.istio.io "prometheus-vs" configured
      destinationrule.networking.istio.io "prometheus" configured
      
    4. Apply the following configuration to expose the tracing service:

      $ cat <<EOF | kubectl apply -f -
      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: tracing-gateway
        namespace: istio-system
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 15032
            name: http-tracing
            protocol: HTTP
          hosts:
          - "*"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: tracing-vs
        namespace: istio-system
      spec:
        hosts:
        - "*"
        gateways:
        - tracing-gateway
        http:
        - match:
          - port: 15032
          route:
          - destination:
              host: tracing
              port:
                number: 80
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: tracing
        namespace: istio-system
      spec:
        host: tracing
        trafficPolicy:
          tls:
            mode: DISABLE
      ---
      EOF
      gateway.networking.istio.io "tracing-gateway" configured
      virtualservice.networking.istio.io "tracing-vs" configured
      destinationrule.networking.istio.io "tracing" configured
      
  3. Visit the telemetry addons via your browser.

    • Kiali: http://<IP ADDRESS OF CLUSTER INGRESS>:15029/
    • Prometheus: http://<IP ADDRESS OF CLUSTER INGRESS>:15030/
    • Grafana: http://<IP ADDRESS OF CLUSTER INGRESS>:15031/
    • Tracing: http://<IP ADDRESS OF CLUSTER INGRESS>:15032/

Cleanup

  • Remove all related Gateways:

    $ kubectl -n istio-system delete gateway grafana-gateway kiali-gateway prometheus-gateway tracing-gateway
    gateway.networking.istio.io "grafana-gateway" deleted
    gateway.networking.istio.io "kiali-gateway" deleted
    gateway.networking.istio.io "prometheus-gateway" deleted
    gateway.networking.istio.io "tracing-gateway" deleted
    
  • Remove all related Virtual Services:

    $ kubectl -n istio-system delete virtualservice grafana-vs kiali-vs prometheus-vs tracing-vs
    virtualservice.networking.istio.io "grafana-vs" deleted
    virtualservice.networking.istio.io "kiali-vs" deleted
    virtualservice.networking.istio.io "prometheus-vs" deleted
    virtualservice.networking.istio.io "tracing-vs" deleted
    
  • If installed, remove the gateway certificate:

    $ kubectl -n istio-system delete certificate telemetry-gw-cert
    certificate.certmanager.k8s.io "telemetry-gw-cert" deleted