Fires when jaas_webhook_cert_renewal_failures_total has increased above the configured per-hour threshold. The Renewer background goroutine rotates the self-signed TLS material every Validity / 3 (typically every few months for a year-long cert). When it can’t, the existing cert keeps working until its natural expiry — at which point the apiserver stops trusting the chain and every JsonnetSnippet admission fails cluster-wide with x509 errors.

Symptom

• JaaSWebhookCertRenewalFailing alert is firing (severity: critical).
• Operator pod logs carry repeated Self-signed webhook cert renewal failed warnings at the Renewer.Interval cadence.
• kubectl describe validatingwebhookconfiguration <name> shows a caBundle that hasn’t rotated since the failures started.
• The pod stays Ready=True — the renewer’s failures don’t gate the readiness probe.

Diagnosis

The most common causes, in order of frequency:

Cause A — RBAC drift on the named VWC

The operator’s ClusterRole pins resourceNames: [<VWCName>] on the validatingwebhookconfigurations patch verb. A chart upgrade that changes operator.webhook.vwcName (or a manual chart edit) leaves the running pod patching a name it no longer has permission for.

kubectl auth can-i patch validatingwebhookconfiguration/<vwc-name> \
  --as=system:serviceaccount:<namespace>:<operator-sa>

If the answer is “no”, the chart’s operator-cluster ClusterRole needs the current VWC name added to resourceNames (or the running pod restarted to pick up the new name).

Cause B — VWC renamed out from under the operator

A separate controller (admission policy automation, GitOps drift correction) renamed the VWC. The operator is patching a stale name.

kubectl get validatingwebhookconfigurations \
  --selector 'app.kubernetes.io/instance=<release-name>'

If the live name differs from the operator’s --webhook-validating-config-name flag, redeploy the operator with the correct flag or rename the VWC back.

Cause C — CertDir gone read-only

The chart mounts CertDir as an emptyDir by default. A kubectl apply that adds a readOnlyRootFilesystem: true security context or a sidecar that re-mounts the volume can break writes.

kubectl --namespace <ns> exec <operator-pod> -- ls -l /tmp/k8s-webhook-server/serving-certs/
kubectl --namespace <ns> exec <operator-pod> -- touch /tmp/k8s-webhook-server/serving-certs/.write-probe

If the touch fails, the security-context or volume mount needs fixing.

Remediation

1. Fix the root cause (RBAC, name, or mount).

2. Roll the operator pod to force a fresh bootstrap of the cert and a re-patch of the VWC:

   kubectl --namespace <ns> rollout restart deployment <operator-deployment>
   

   The new pod’s bootstrap path merges its CA into the VWC’s existing CA set rather than replacing it, so existing replicas stay trusted across the rotation.

3. Verify renewal is healthy after the bounce — the jaas_webhook_cert_renewal_failures_total counter should stop increasing, and the alert clears once the for: window passes.

When to consider switching to cert-manager

If the self-signed renewer keeps tripping over your environment’s RBAC story or pod-security policies, the chart supports operator.webhook.certMode: cert-manager — cert-manager handles the rotation and the operator mounts the resulting secret. Trade-off: requires cert-manager installed and an Issuer configured.