[FIXED] How to configure ingress-nginx-controller-metrics in Prometheus using Helm chart with values file?

Issue

I want to get my nginx-ingress metrics to be added in Prometheus so I can see my app logs to be shown in loki. However, I have not had much success. I have been following this guide here but I want to do this for nginx-ingress instead, the guide is for ambassador ingress.

Nginx-Ingress installed using the following command:

NGINX_CHART_VERSION="4.0.6";
helm upgrade ingress-nginx ingress-nginx/ingress-nginx --version "$NGINX_CHART_VERSION" --namespace ingress-nginx -f "03-setup-ingress-controller/assets/manifests/nginx-values-v${NGINX_CHART_VERSION}.yaml"  --set controller.metrics.enabled=true  --set-string controller.podAnnotations."prometheus\.io/scrape"="true"  --set-string controller.podAnnotations."prometheus\.io/port"="10254"

So I have my nginx-ingress metrics exposed in my k8 cluster as you can see:

kubectl get svc -n ingress-nginx                                    
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.245.57.3      <REDACTED>   80:31150/TCP,443:31740/TCP   9d
ingress-nginx-controller-admission   ClusterIP      10.245.186.61    <none>            443/TCP                      9d
ingress-nginx-controller-metrics     ClusterIP      10.245.240.243   <none>            10254/TCP                    20h

I did a Helm upgrade with a values file that contains my ingress-nginx-controller-metrics :

 HELM_CHART_VERSION="27.2.1";
helm upgrade kube-prom-stack prometheus-community/kube-prometheus-stack \
--version "${HELM_CHART_VERSION}"  --namespace monitoring  \ 
-f "04-setup-prometheus-stack/assets/manifests/prom-stack-values-v${HELM_CHART_VERSION}.yaml"

and I updated it with the values file to contains, customized from git repo config) as:

## Starter Kit components service monito
  #
additionalServiceMonitors:
  - name: "ingress-nginx-monitor"
    selector:
      matchLabels:
        service: "ingress-nginx-controller
    namespaceSelector:
      matchNames:
        - ingress-nginx
    endpoints:
      - port: "ingress-nginx-controller-me
        path: /metrics
        scheme: http

However, when I check the Prometheus Targets I don't see the nginx ingress there.

Solution

additionalServiceMonitors for the kube-prom-stack should be :

    additionalServiceMonitors:
        - name: "ingress-nginx-monitor"
          selector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
          namespaceSelector:
            matchNames:
              - ingress-nginx
          endpoints:
            - port: "metrics"

Answered By - leachim742

Answer Checked By - Katrina (PHPFixing Volunteer)