Running kubernetes 1.15 in azure.
I need a basic alert (e-mail/slack notification) when one or more of my applications/pods are down in kubernetes.
As an example I have https://cert-manager.io/docs/ running in multiple clusters (hosted in azure) and I would like to get an alert (e-mail/slack notification) if it stops running.
Based on this post:
How do I set up a hook to send an email on Kubernetes pod restart?
it seems to get an e-mail alert I need to install Prometheus + Grafana access the web-ui and configure alerts so based on:
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack
I have tried:
helm versionversion.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}helm repo add prometheus-community https://prometheus-community.github.io/helm-chartshelm repo add stable https://kubernetes-charts.storage.googleapis.com/helm repo updatehelm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoringBut that gives:
Error: failed to install CRD crds/crd-alertmanager.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1"Here there is some guide on how to create the crds manually:
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#helm-fails-to-create-crds
but that should only be necessary if running helm 2.x which I am not I am running 3.1.2.
Also if I try to install them manually I get:
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yamlerror: unable to recognize "https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1"$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yamlerror: unable to recognize "https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1"$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yamlerror: unable to recognize "https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1"...Also I found this kube-prometheus stack compatibility matrix:
https://github.com/prometheus-operator/kube-prometheus#compatibility
but versions in that matris does not match the ones I get:
$ helm search repo prometheus-community/kube-prometheus-stack --versionsNAME CHART VERSION APP VERSION DESCRIPTION prometheus-community/kube-prometheus-stack 10.1.2 0.42.1 kube-prometheus-stack collects Kubernetes manif...prometheus-community/kube-prometheus-stack 10.1.1 0.42.1 kube-prometheus-stack collects Kubernetes manif...prometheus-community/kube-prometheus-stack 10.1.0 0.42.1 kube-prometheus-stack collects Kubernetes manif...prometheus-community/kube-prometheus-stack 10.0.2 0.42.1 kube-prometheus-stack collects Kubernetes manif...prometheus-community/kube-prometheus-stack 10.0.1 0.42.1 kube-prometheus-stack collects Kubernetes manif...So seems that might be a 3rd way to install Prometheus
Any input appreciated.
UPDATE:
Randomly selecting the previous major version (9.4.10) seems to work:
$ helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoring --version 9.4.10manifest_sorter.go:192: info: skipping unknown hook: "crd-install"manifest_sorter.go:192: info: skipping unknown hook: "crd-install"manifest_sorter.go:192: info: skipping unknown hook: "crd-install"manifest_sorter.go:192: info: skipping unknown hook: "crd-install"manifest_sorter.go:192: info: skipping unknown hook: "crd-install"manifest_sorter.go:192: info: skipping unknown hook: "crd-install"NAME: kube-prometheus-stackLAST DEPLOYED: Fri Oct 23 15:15:03 2020NAMESPACE: monitoringSTATUS: deployedREVISION: 1NOTES:kube-prometheus-stack has been installed. Check its status by running:kubectl --namespace monitoring get pods -l "release=kube-prometheus-stack"Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.Guess trial and error is the way to go when installing stuff on older k8s version, could be great with compatibility matrices though.
Best Answer
Based on the kube-prometheus-stack repo, this helm chart is restricted for K8s versions 1.16.0 or above;
kubeVersion: ">=1.16.0-0"
Even though the github README says the prerequisites as Kubernetes 1.10+ with Beta APIs, internally the helm chart checks for the kube version to be 1.16.0 or above.
So I believe, you will need to try this on an upgrade K8s cluster.
If upgrading the cluster is not an option, maybe you could try the deprecated old version of this.
https://github.com/helm/charts/tree/master/stable/prometheus