Jsonnet-as-a-Service

Admission webhook

Mon, 01 Jan 0001 00:00:00 +0000

JaaS ships an optional validating admission webhook for JsonnetSnippet. It is independent of, but layered on top of, operator mode .

Enabling it

Set --enable-webhook to boot the webhook server. It requires --enable-flux-integration (the webhook is wired only inside the operator boot path; enabling it alone is rejected as a flag error) and TLS material — tls.crt and tls.key — under --webhook-cert-dir (default /tmp/k8s-webhook-server/serving-certs). The server binds --webhook-port (default 9443).

What it validates

The webhook rejects a JsonnetSnippet whose spec.externalVariables declares a key that collides with an operator-level --ext-var. An operator-supplied external variable always wins, so a snippet that tries to redeclare one would render against a value it does not control; the webhook refuses the snippet at admission time instead.

Alerting

Mon, 01 Jan 0001 00:00:00 +0000

JaaS turns a sustained problem into a notification two ways: a Prometheus PrometheusRule that pages on its metrics , and Kubernetes Events that Flux’s notification-controller can route to chat or e-mail.

The binary

The operator emits a standard Kubernetes Event on every Ready-condition transition — Normal for Synced, Warning for every other reason. The reason string fills both the event reason and action. These Events need no flag to enable; they are written whenever the operator reconciles.

ArtifactTooLarge

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=ArtifactTooLarge. The Message states the rendered byte count and the configured cap.

Cause

The snippet’s rendered output exceeds the operator’s --max-artifact-bytes (Helm: operator.storage.maxArtifactBytes). The cap is a defense-in-depth control — one runaway snippet shouldn’t fill a shared storage volume.

Common triggers:

a snippet generating massive arrays via std.range(n) with a much larger n than intended
accidentally inlining a large data fixture via importstr
forgetting to project / filter when fanning out per-tenant configs

Diagnosis

Check the rendered size locally:

Building and Testing

Mon, 01 Jan 0001 00:00:00 +0000

The host needs no Go toolchain. All build and test commands run inside a containerized dev shell defined by dev/Containerfile and driven by ilo. The .ilo.rc at the repo root supplies the shell arguments, so the short form is always available:

ilo bash -c '<command>'

The dev shell pre-installs the Go toolchain, all static-analysis tools, controller-gen, and the envtest asset bundle. The Go module cache is mounted from ${XDG_CACHE_HOME}/go so repeated builds do not re-download dependencies.

CI and Releases

Mon, 01 Jan 0001 00:00:00 +0000

Static analysis

golangci-lint is not used. The tools below run directly, both in CI and locally inside the dev shell (see Building and Testing ). Every tool is a separate, auditable binary with its own config file.

Tool	Scope	Config
`go vet` (all analyzers)	Go correctness	—
staticcheck	Bugs, simplifications, style	`staticcheck.conf` (`checks = ["all"]`)
gosec	Security patterns	inline `#nosec` justifications
govulncheck	Known vulnerabilities in the dependency graph	—
arch-go	Architecture rules	`arch-go.yml`
gofumpt	Strict formatting	—
REUSE	License / copyright metadata on every file	`REUSE.toml`
yamllint	YAML	`.yamllint.yaml`
actionlint	GitHub Actions workflows	—
markdownlint	Markdown	`.markdownlint.yaml`
typos	Spelling	`.typos.toml`
Trivy	Container image CVEs	—

Architecture rules

arch-go.yml pins two invariants enforced with 100% compliance:

Configuration reference

Mon, 01 Jan 0001 00:00:00 +0000

Every JaaS flag is listed here with its default and a one-line description. Run jaas --help to see the same list at runtime. The tables on this page are generated from the binary’s own flag definitions, so they never drift from the runtime contract.

The Helm chart exposes most flags under arguments.*; operator-specific flags are under operator.*. The full set of chart values is in the Helm chart values reference.

Jsonnet server

The Jsonnet server evaluates snippets and returns JSON. It binds on --listen-address:--port by default.

CRD watch engagement failing

Mon, 01 Jan 0001 00:00:00 +0000

Fires when jaas_crd_watch_engagement_failures_total{gvk=...} has increased above the per-hour threshold for the alert window. JaaS lazy-watches Flux source CRDs: at boot, only the CRDs already installed get a watch; when a previously-missing CRD becomes Established=True (operator installed source-controller post hoc, say), the crdWatcher engages a runtime watch on it via Controller.Watch. When that engagement fails, the apiextensions informer fires no further events on a stable CRD — meaning the watch stays un-engaged forever until either the CRD object’s metadata/status is changed by something else, or the operator restarts.

Creating source artifacts

Mon, 01 Jan 0001 00:00:00 +0000

A JsonnetSnippet’s spec.sourceRef consumes a Flux source. The operator reads the referenced source CR’s status.artifact.url, downloads the tarball Flux’s source-controller serves there, verifies its status.artifact.digest, and extracts it into the snippet’s file tree. Every supported kind — GitRepository, OCIRepository, and Bucket — reaches the operator through that same status.artifact contract, so the operator never talks to a git remote, an OCI registry, or an object store directly. Flux owns that fetch; the operator consumes the artifact Flux already produced.

CrossNamespaceRefRejected

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=CrossNamespaceRefRejected. The Message names the offending reference (a library or a sourceRef).

Cause

The operator is running with --no-cross-namespace-refs=true (the chart default) and the snippet references a library or Flux source in a different namespace.

This is a deliberate isolation control — it mirrors Flux’s --no-cross-namespace-refs and stops a tenant in namespace A from reaching libraries / sources in namespace B without an explicit relationship.

Diagnosis

Inspect the spec and identify which reference points outside the snippet’s namespace:

DependencyCycle

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=DependencyCycle. The Message names the snippet that closes the cycle.

Cause

The snippet’s spec.sourceRef chain transitively points back at the snippet itself. The reconciler detects this and refuses to publish so chained snippets don’t loop forever (each republish would trigger every downstream snippet to re-render, which would re-trigger the upstream, and so on).

Two cycle shapes:

Direct sourceRef cycle: A.spec.sourceRef → ExternalArtifact/A. A snippet sourcing from its own published artifact.
Library-mediated cycle: A.spec.libraries → JsonnetLibrary/L, where L.spec.sourceRef → ExternalArtifact/A (or a longer chain back to A).

The validating webhook (--enable-webhook) rejects new CRs that introduce a cycle at admission; the reconciler check is a fallback for when admission is bypassed or the cycle is introduced retroactively (e.g., adding a new library that closes a loop with existing snippets).

Deploying manifests with StageSet

Mon, 01 Jan 0001 00:00:00 +0000

JaaS pairs with stageset-controller to deploy Kubernetes manifests as code: you author the manifests in Jsonnet, the JaaS operator renders and publishes them as a Flux ExternalArtifact, and stageset-controller rolls that artifact out across ordered, gated stages.

This tutorial covers the JaaS side — authoring the manifests with top-level arguments and external variables, and publishing the rendered JSON. The rollout side (the StageSet resource, its stages, gates, and actions) lives on the stageset-controller site and is linked at the end.

Eval-concurrency saturation

Mon, 01 Jan 0001 00:00:00 +0000

Not tied to a single Reason — this page covers what to do when the global concurrent-eval cap (--max-concurrent-evals) is full and JaaS is shedding new evaluations. The cap exists because the synchronous go-jsonnet API has no context-aware cancellation: once an eval starts it runs to natural completion, so an unbounded queue lets a runaway snippet pile up goroutines that outlive every caller’s deadline.

Symptom

One or more of:

JaaSEvalSaturation is firing — jaas_eval_in_flight / jaas_eval_max_concurrent has been above the threshold (default 0.9) for the alert window.
JaaSEvalRejected is firing — rate(jaas_eval_unavailable_total[5m]) has been above the threshold.
HTTP clients see 503 Service Unavailable with body {"error": "evaluation_unavailable", "message": "concurrent-eval cap is full; retry after backoff"}.
kubectl describe jsonnetsnippet shows recurring Warning EvalUnavailable events with message reconcile deferred for 1s by --max-concurrent-evals. Ready condition stays untouched (backpressure is not failure).
jaas_eval_outstanding_timed_out is also elevated — confirms the runaway-snippet diagnosis: orphaned evals are pinning slots while their parents have already given up.

Diagnosis: why is the cap full?

The cap fills for two distinct reasons. The right remediation depends on which.

Evaluation and security

Mon, 01 Jan 0001 00:00:00 +0000

JaaS runs Jsonnet on the server and returns the result over HTTP. Three caps bound each evaluation, and a small security model governs what a snippet and its callers can reach. Review and tune both sections before exposing the service to a wider audience.

Evaluation caps

Flag	Default	Effect
`--evaluation-timeout`	`5s`	Wall-clock budget per evaluation. Exceeding it returns `504 evaluation_timeout`. `0` disables the timeout.
`--max-stack`	`500`	Maximum Jsonnet call-stack depth. `0` uses go-jsonnet’s own default.
`--max-concurrent-evals`	`max(GOMAXPROCS*4, 16)`	In-flight evaluations allowed at once. Excess requests return `503 evaluation_unavailable`. `0` disables the cap.

./jaas \
 --snippet-directory examples/snippets/dashboards \
 --evaluation-timeout 2s \
 --max-stack 1000 \
 --max-concurrent-evals 32

The default for --max-concurrent-evals bounds worst-case goroutine pile-up under a runaway snippet. Each in-flight evaluation pins roughly one CPU for its working set, so raising the cap far above the available parallelism queues work without adding throughput.

EvaluationFailed

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=EvaluationFailed. The Message contains the raw go-jsonnet diagnostic — file name, line, column, and the underlying error.

Cause

The snippet failed to evaluate. Three broad categories:

Syntax error — unclosed brace, missing comma, bad indent.
Runtime error — std.extVar('missing') for an unset variable, division by zero, error '...' thrown explicitly.
Import error — import 'missing.libsonnet' resolves to nothing in the snippet’s file map or library imports.

Diagnosis

Read the Message — it names the file and line. Reproduce locally:

EvaluationTimeout

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=EvaluationTimeout. The snippet’s eval ran longer than the operator’s --evaluation-timeout.

Cause

Snippets are evaluated synchronously per reconcile. The deadline is wall-clock, not CPU — but go-jsonnet has no mid-evaluation cancellation, so a snippet that runs over the deadline still keeps consuming CPU on the operator pod until it returns naturally.

Common triggers:

a snippet recursing deeper than necessary (try lowering --max-stack to surface this as a stack-limit error instead, then optimize)
a snippet that loads a huge sourceRef tarball and walks it
a snippet that calls std.set / std.uniq over a very large array

Diagnosis

Time it locally:

External variables and TLAs

Mon, 01 Jan 0001 00:00:00 +0000

JaaS feeds two kinds of input into an evaluation: external variables, set by the process owner at startup, and top-level arguments, supplied per request through the URL query string.

External variables

External variables come from two sources. The environment mechanism reads every variable prefixed with JAAS_EXT_VAR_ — the suffix is the variable name:

JAAS_EXT_VAR_name=Alice \
JAAS_EXT_VAR_key=secret \
 ./jaas --snippet-directory examples/snippets/dashboards

The --ext-var KEY=VALUE flag does the same and is repeatable. On a key conflict, the flag takes precedence over the environment value:

ExternalArtifact output contract

Mon, 01 Jan 0001 00:00:00 +0000

For every successfully evaluated JsonnetSnippet, the JaaS operator upserts a Flux ExternalArtifact CR (source.toolkit.fluxcd.io/v1) in the same namespace as the snippet. JaaS does not own the ExternalArtifact CRD — it is defined and installed by Flux’s source-controller. The full CRD schema is in the Flux ExternalArtifact reference ; below is the subset JaaS writes and the invariants downstream consumers can rely on.

For task-oriented context, see Operator mode .

What JaaS writes

`spec.sourceRef`

JaaS stamps a back-pointer to the originating snippet on spec.sourceRef:

ExternalVariableConflict

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=ExternalVariableConflict. The Message names the conflicting key.

Cause

The snippet’s spec.externalVariables declares a key that the operator already owns via --ext-var (cluster operator-level). Operator keys win by design — they’re how the cluster admin pins cluster-scoped values like cluster, region, environment so a tenant snippet can’t override them.

Diagnosis

# Which keys does the operator own?
kubectl --namespace <jaas-ns> get pod --selector app.kubernetes.io/name=jaas --output yaml | grep -A1 "\--ext-var="

Cross-reference with the snippet’s spec.externalVariables.

Grafana dashboards

Mon, 01 Jan 0001 00:00:00 +0000

JaaS pairs with the grafana-operator to manage Grafana dashboards as code: you author the dashboard in Jsonnet, the JaaS operator renders it and publishes the dashboard JSON as a Flux ExternalArtifact, and the grafana-operator reconciles that artifact into a live Grafana instance.

This tutorial covers the JaaS side — authoring the dashboard, importing grafonnet as a JsonnetLibrary, and publishing the rendered JSON. The grafana-operator side (the GrafanaDashboard CR, datasources, folders) lives on their site and is linked at the end.

Helm chart values

Mon, 01 Jan 0001 00:00:00 +0000

The jaas Helm chart lives in the metio/helm-charts monorepo and is published at oci://ghcr.io/metio/helm-charts/jaas. The tables below are generated from each chart’s values.yaml, so they track the chart’s current values rather than a hand-maintained copy.

For how the values map onto the binary’s runtime behaviour, see the Configuration reference — every arguments.* value drives the corresponding --flag.

jaas chart

Value	Type	Default	Description
`additionalLibraries`	`object\|null`	(empty)	Additional library OCI volumes to mount. The key is the name of the directory and the value is an OCI volume that will be mounted beneath .Values.paths.libraries, e.g., the commented example below would result in all files from 'ghcr.io/metio/something:latest' to be mounted at '/srv/libraries/something/' (using the default configuration) OCI mounts suit large, independently-released library bundles pinned by digest. For shared libraries you want to manage as ordinary manifests — published and updated via kubectl/GitOps without an operator restart — apply a namespaced JsonnetLibrary instead and reference it from a snippet's spec.libraries.
`arguments`	`object`	(empty)	CLI flags for the jaas binary. These arguments map directly to their corresponding CLI flag.
`arguments.evaluationTimeout`	`string`	`5s`	Per-request go-jsonnet evaluation timeout. 0 disables.
`arguments.jsonnetEndpointPath`	`string`	`jsonnet`	URL path segment the jsonnet handler is mounted under (GET /<path>/...).
`arguments.listenAddress`	`string`	`::`	Bind address for the jsonnet HTTP and management servers. `::` listens on every IPv6 interface and — because Go opens IPV6_V6ONLY=0 by default on Linux — also accepts IPv4 connections via v4-mapped addresses. Use `0.0.0.0` to constrain to IPv4 on clusters that don't support dual-stack pods.
`arguments.logFormat`	`string`	`json`	Log output format for the jaas binary (json, text).
`arguments.logLevel`	`string`	`info`	Log verbosity for the jaas binary (debug, info, warn, error).
`arguments.managementListenAddress`	`string`	`::`	Bind address for the management (probes) HTTP server. Same dual-stack semantics as listenAddress.
`arguments.managementReadTimeout`	`string`	`10s`	Read timeout on the management (probes) HTTP server.
`arguments.managementWriteTimeout`	`string`	`10s`	Write timeout on the management (probes) HTTP server.
`arguments.maxConcurrentEvals`	`integer`	`32`	Maximum in-flight Jsonnet evaluations across both HTTP and operator paths. Excess requests return 503 (HTTP) or RequeueAfter (operator) with EvalUnavailable backpressure events. Sized to bound worst-case goroutine pile-up when a runaway snippet's synchronous eval outlives every caller's ctx — the cap turns "unbounded leak" into a tunable backpressure boundary. 0 disables the gate; the operator default (process-side) is max(GOMAXPROCS*4, 16).
`arguments.maxStack`	`integer`	`500`	go-jsonnet stack-depth ceiling. Guards against unbounded recursion.
`arguments.readTimeout`	`string`	`10s`	Read timeout on the jsonnet HTTP server.
`arguments.shutdownDelay`	`string`	`5s`	Grace period readiness stays false before in-flight requests are aborted on SIGTERM, so endpoint controllers can drain the pod first.
`arguments.writeTimeout`	`string`	`10s`	Write timeout on the jsonnet HTTP server.
`crds`	`object`	(empty)	CRD lifecycle. The chart ships the controller's CRDs under templates/ so a `helm upgrade` applies schema changes automatically. Set create=false to manage them out-of-band — e.g. pre-installed cluster-wide, or in CI where the same chart is installed once per values file and a kept cluster-scoped CRD owned by the first release can't be re-adopted by the next.
`crds.create`	`boolean`	`true`
`deployment`	`object`	(empty)	Fine tune the Deployment resource.
`deployment.additionalLabels`	`object`	(empty)	Extra labels merged onto the Deployment metadata.
`deployment.revisionHistoryLimit`	`integer`	`1`	spec.revisionHistoryLimit on the Deployment.
`externalVariables`	`object\|null`	(empty)	External variables (std.extVar). They will be added as JAAS_EXT_VAR_your_name and are available as your_name in your Jsonnet snippets.
`externalVariablesFrom`	`object`	(empty)	External variables loaded from ConfigMaps or Secrets. Each array expects the name of either ConfigMaps or Secrets. Only those values in ConfigMaps and Secrets that start with JAAS_EXT_VAR will be available as external variable in your Jsonnet snippets.
`externalVariablesFrom.configMaps`	`array`	(empty)	ConfigMap names to source JAAS_EXT_VAR_* env vars from.
`externalVariablesFrom.secrets`	`array`	(empty)	Secret names to source JAAS_EXT_VAR_* env vars from.
`global`	`object`	(empty)	Global values are values that can be accessed from any chart or subchart by exactly the same name.
`image`	`object`	(empty)	Container image for the jaas workload. Overwrite these in case you want to use an internal registry mirror.
`image.pullPolicy`	`string`	`IfNotPresent`	Pull policy for the jaas container. IfNotPresent is the default — safe for tagged release images. Switch to Always when running off a mutable tag (latest, edge, dev) so each restart re-checks the registry. Never is useful when the image is preloaded into the node (kind, k3d).
`image.registry`	`string`	`ghcr.io`
`image.repository`	`string`	`metio/jaas`
`image.tag`	`string`	(empty)	Optional explicit tag override. Defaults to .Chart.AppVersion (the chart's released version). Useful when running a custom build: --set-string image.tag=dev-abc123
`imagePullSecrets`	`array`	(empty)	Optional pull secrets in case you use a private mirror/image
`libraries`	`object`	(empty)	Well-known jsonnet libraries. All OFF by default — a default install mounts no OCI volumes. Enable per renderer-mode need (the standalone HTTP renderer imports these from pre-baked image bundles). Each enabled library renders a `-library-path` arg, a read-only volumeMount, and an OCI `image:` volume. Static OCI mounts are mutually exclusive with operator.enabled (Flux mode): in Flux mode libraries come from JsonnetLibrary CRs, not these image bundles.
`libraries.docsonnet`	`object`	(empty)	docsonnet library. Mounts a JOI OCI bundle when enabled.
`libraries.docsonnet.enabled`	`boolean`	`false`
`libraries.docsonnet.registry`	`string`	`ghcr.io`
`libraries.docsonnet.repository`	`string`	`metio/joi-jsonnet-libs-docsonnet`
`libraries.docsonnet.version`	`string`	`latest`
`libraries.grafonnet`	`object`	(empty)	grafonnet library (Grafana dashboards). Mounts a JOI OCI bundle when enabled.
`libraries.grafonnet.enabled`	`boolean`	`false`
`libraries.grafonnet.registry`	`string`	`ghcr.io`
`libraries.grafonnet.repository`	`string`	`metio/joi-grafana-grafonnet`
`libraries.grafonnet.version`	`string`	`latest`	The JOI images are calendar-tagged (plus a moving `latest`), not tagged by the upstream grafonnet version — there is no `v11.4.0` tag. Match the other libraries and track `latest`; pin to a dated tag (e.g. 2026.6.14) to freeze.
`libraries.xtd`	`object`	(empty)	xtd library. Mounts a JOI OCI bundle when enabled.
`libraries.xtd.enabled`	`boolean`	`false`
`libraries.xtd.registry`	`string`	`ghcr.io`
`libraries.xtd.repository`	`string`	`metio/joi-jsonnet-libs-xtd`
`libraries.xtd.version`	`string`	`latest`
`namespace`	`object`	(empty)	Opt-in install-namespace render with Pod Security Standards labels. Default-off so the chart doesn't claim ownership of a pre-existing namespace operators created via kubectl / Argo. Enable for fresh installs that want the namespace's PSS posture baked into the chart.
`namespace.additionalLabels`	`object`	(empty)	Extra labels on the rendered Namespace.
`namespace.annotations`	`object`	(empty)	Extra annotations on the rendered Namespace.
`namespace.create`	`boolean`	`false`	Whether the chart renders (and owns) its install Namespace.
`namespace.podSecurity`	`object`	(empty)	Pod Security Standards admission labels stamped on the namespace.
`namespace.podSecurity.audit`	`string`	`restricted`	PSS audit level.
`namespace.podSecurity.auditVersion`	`string`	`latest`
`namespace.podSecurity.enforce`	`string`	`restricted`	`restricted` is the strictest PSS level. Image volumes (used by the chart for snippet/library mounts) are restricted-compliant from Kubernetes 1.33 onward; clusters on 1.32 or older must drop this to `baseline`.
`namespace.podSecurity.enforceVersion`	`string`	`latest`
`namespace.podSecurity.warn`	`string`	`restricted`	PSS warn level.
`namespace.podSecurity.warnVersion`	`string`	`latest`
`networkPolicy`	`object`	(empty)	Opt-in NetworkPolicy for the jaas pod (storage port lockdown + per-port ingress controls).
`networkPolicy.additionalIngress`	`array`	(empty)	Free-form additional ingress rules merged into the policy verbatim.
`networkPolicy.calico`	`object`	(empty)	Calico engine (engine=calico) raw passthrough. Entries are merged verbatim into the projectcalico.org NetworkPolicy's spec.ingress / spec.egress.
`networkPolicy.calico.egress`	`array`	(empty)	Extra spec.egress rules merged verbatim (Calico NetworkPolicy schema).
`networkPolicy.calico.ingress`	`array`	(empty)	Extra spec.ingress rules merged verbatim (Calico NetworkPolicy schema).
`networkPolicy.cilium`	`object`	(empty)	Cilium engine (engine=cilium) raw passthrough. Entries are merged verbatim into the CiliumNetworkPolicy's spec.ingress / spec.egress. This is how you tighten the cilium policy — e.g. add fromEndpoints to the ingress rules, or additional toEndpoints / toFQDNs egress.
`networkPolicy.cilium.egress`	`array`	(empty)	Extra spec.egress rules merged verbatim (CiliumNetworkPolicy schema).
`networkPolicy.cilium.ingress`	`array`	(empty)	Extra spec.ingress rules merged verbatim (CiliumNetworkPolicy schema).
`networkPolicy.clusterNetworkPolicy`	`object`	(empty)	ClusterNetworkPolicy engine (engine=clusterNetworkPolicy) raw passthrough. Entries are merged verbatim into the policy.networking.k8s.io ClusterNetworkPolicy's spec.ingress / spec.egress.
`networkPolicy.clusterNetworkPolicy.egress`	`array`	(empty)	Extra spec.egress rules merged verbatim (ClusterNetworkPolicy schema).
`networkPolicy.clusterNetworkPolicy.ingress`	`array`	(empty)	Extra spec.ingress rules merged verbatim (ClusterNetworkPolicy schema).
`networkPolicy.clusterNetworkPolicy.priority`	`integer`	`1000`	spec.priority on the ClusterNetworkPolicy (lower wins within the tier).
`networkPolicy.defaultDeny`	`object`	(empty)	Render a namespace-wide default-deny so EVERY pod in the namespace is denied by default and the per-workload allowlists are the only exceptions (zero-trust namespace). OFF by default because it also denies co-located workloads — enable only when JaaS has its own namespace. Off keeps a pod-scoped setup: only JaaS's pods are locked down, neighbours untouched.
`networkPolicy.defaultDeny.enabled`	`boolean`	`false`
`networkPolicy.defaultDeny.order`	`integer`	`2000`	Calico/ClusterNetworkPolicy sort key for the deny-all. It must rank the deny-all AFTER the per-workload allowlists so the allowlists win: Calico evaluates lower order first (the allowlist policies carry no order = lowest precedence, so this defaults high), and ClusterNetworkPolicy gives lower priority numbers higher precedence (the allowlist priority defaults to networkPolicy.clusterNetworkPolicy.priority, so this defaults higher). The kubernetes and cilium engines have no precedence knob — deny + allow simply combine additively, allow wins.
`networkPolicy.egress`	`object`	(empty)
`networkPolicy.egress.dns`	`boolean`	`true`	When egress is enabled, allow DNS (UDP+TCP 53) to the cluster DNS namespace.
`networkPolicy.egress.dnsNamespace`	`string`	`kube-system`	Namespace of the cluster DNS service (matched by kubernetes.io/metadata.name).
`networkPolicy.egress.enabled`	`boolean`	`false`	Render egress rules. Adds Egress to policyTypes, so everything not explicitly allowed is denied. Off by default: enable only after allowing everything the pod needs (DNS, the kube-apiserver, source-controller, S3, the OTLP collector) via egress.to — otherwise the operator loses cluster access.
`networkPolicy.egress.to`	`array`	(empty)	Additional egress peers merged verbatim into spec.egress. This is where you allow the kube-apiserver (an ipBlock CIDR), the flux-system namespace (source-controller), and external endpoints (S3, OTLP). NetworkPolicy cannot select the apiserver by label, so it must be an ipBlock here.
`networkPolicy.enabled`	`boolean`	`false`	Opt-in. When enabled, a NetworkPolicy targets the jaas pod and locks the storage HTTP port down to the configured consumer selector (defaults to any pod in flux-system — kustomize-controller / helm-controller). Other ports stay open so kubelet probes, the jsonnet HTTP path, and the webhook traffic from kube-apiserver continue to work. Leave OFF on initial install if the cluster has no NetworkPolicy controller (Calico/Cilium/etc.) — without one, the policy is silently inert; with one, default-deny semantics apply to everything not listed here.
`networkPolicy.engine`	`string`	`kubernetes`	Policy engine to render for. kubernetes (default) emits a vanilla networking.k8s.io NetworkPolicy. cilium / calico / clusterNetworkPolicy emit the engine-native equivalent instead (CiliumNetworkPolicy, projectcalico.org NetworkPolicy, or policy.networking.k8s.io ClusterNetworkPolicy). The per-port .from knobs below apply to the kubernetes engine only; alternative engines are pod-scoped allow-all on the required ports and tighten via their native passthrough lists (cilium/calico/clusterNetworkPolicy .ingress / .egress).
`networkPolicy.http`	`object`	(empty)	Restrict sources allowed to hit the jsonnet HTTP endpoint. Empty list allows everything (typical when an Ingress fronts the service).
`networkPolicy.http.from`	`array`	(empty)
`networkPolicy.management`	`object`	(empty)	Restrict sources allowed to hit the management probes. Empty list allows everything — kubelet probes source from the node IP.
`networkPolicy.management.from`	`array`	(empty)
`networkPolicy.metrics`	`object`	(empty)	Sources allowed to scrape the operator metrics port. Empty = all (typically your Prometheus). Only rendered in operator mode with metrics enabled.
`networkPolicy.metrics.from`	`array`	(empty)
`networkPolicy.storage`	`object`	(empty)	Sources allowed to read published ExternalArtifact tarballs from the storage HTTP port. The consumers are the Flux controllers that dereference ExternalArtifact.status.artifact.url — kustomize-controller and helm-controller (both in flux-system) — NOT source-controller, which only produces its own typed artifacts. Defaults to any pod in flux-system so the stock Flux consumers work out of the box. Add an entry per extra consumer namespace (e.g. stageset-controller in stageset-system): - namespaceSelector: matchLabels: kubernetes.io/metadata.name: stageset-system
`networkPolicy.storage.from`	`array`	(empty)
`networkPolicy.webhook`	`object`	(empty)	Restrict sources allowed to dial the webhook. Empty list allows everything — kube-apiserver cannot be expressed as a podSelector.
`networkPolicy.webhook.from`	`array`	(empty)
`operator`	`object`	(empty)	Operator mode — opt-in. When enabled, jaas runs alongside its HTTP path as a Kubernetes operator that watches JsonnetSnippet / JsonnetLibrary CRs and publishes evaluated results as Flux ExternalArtifact resources.
`operator.cleanupOnDelete`	`object`	(empty)	On `helm uninstall`, run a pre-delete Job that bulk-deletes every JsonnetSnippet so the operator's finalizer drops the published ExternalArtifact + tarball BEFORE the operator pod itself is removed. Without this, the operator goes away first and downstream Flux consumers are left referencing orphaned ExternalArtifacts. Disable when uninstall is driven by something other than Helm (e.g., ArgoCD with hooks off) — the manual cleanup pattern is: kubectl delete jsonnetsnippet --all -A --wait=true --timeout=2m helm uninstall jaas -n <ns>
`operator.cleanupOnDelete.activeDeadlineSeconds`	`integer`	`600`	Job-level safety net. activeDeadlineSeconds caps the Job's total runtime; if the operator is wedged, Helm won't be blocked forever.
`operator.cleanupOnDelete.backoffLimit`	`integer`	`2`	backoffLimit on the pre-delete Job.
`operator.cleanupOnDelete.enabled`	`boolean`	`true`
`operator.cleanupOnDelete.image`	`object`	(empty)	Image for the pre-delete cleanup Job (a kubectl image).
`operator.cleanupOnDelete.image.pullPolicy`	`string`	`IfNotPresent`
`operator.cleanupOnDelete.image.registry`	`string`	`registry.k8s.io`
`operator.cleanupOnDelete.image.repository`	`string`	`kubectl`
`operator.cleanupOnDelete.image.tag`	`string`	`v1.34.0`
`operator.cleanupOnDelete.kubectlTimeout`	`string`	`2m`	Maximum wall-clock time the pre-delete Job waits for snippet finalizers to run before it strips them forcibly and proceeds. Bigger when snippets publish to slow S3 endpoints.
`operator.cleanupOnDelete.resources`	`object`	(empty)	Resource requests/limits for the cleanup Job container.
`operator.cleanupOnDelete.resources.cpu`	`string`	`100m`
`operator.cleanupOnDelete.resources.ephemeralStorage`	`string`	`16Mi`	Ephemeral-storage requests/limits. The cleanup Job runs a few `kubectl` calls then exits — disk usage is negligible. Sized small to satisfy kube-score's resource-completeness check without crowding cluster ephemeral-storage budgets.
`operator.cleanupOnDelete.resources.memory`	`string`	`64Mi`
`operator.defaultServiceAccount`	`string`	(empty)	The default ServiceAccount the operator impersonates when a snippet does not specify spec.serviceAccountName. Snippets without an effective SA are rejected at reconcile time.
`operator.enabled`	`boolean`	`false`
`operator.extVars`	`object`	(empty)	Operator-level external variables. Keys here are non-overridable: a JsonnetSnippet whose spec.externalVariables names any of these keys is rejected at admission (and as a fallback at reconcile time).
`operator.labelSelector`	`string`	(empty)	Optional label selector that narrows which CRs the operator watches. Leave empty to watch every CR in the cluster.
`operator.leaderElection`	`object`	(empty)	Leader election guards against two operator replicas double-reconciling the same JsonnetSnippet. With replicas.max: 1 (the chart default) only one replica ever runs, so the lock is just defensive — but operators who scale the Deployment up (rolling restarts, blue-green) MUST keep this on. Off is supported (single replica only) but unsafe to scale.
`operator.leaderElection.enabled`	`boolean`	`true`
`operator.leaderElection.id`	`string`	(empty)	Lease object name. Defaults to "<release name>-operator" so multiple JaaS installs in the same namespace don't fight over a shared lease.
`operator.maxWithdrawWait`	`string`	`1h`	Bound the time a deleted JsonnetSnippet's finalizer can hold while Publisher.Withdraw keeps failing. Past this, the operator emits a Warning WithdrawForced event, drops the finalizer, and lets the snippet be garbage-collected — possibly leaving an orphan tarball in storage (`<namespace>/<name>/<rev>.tar.gz`). Required so a permanently-broken backend (S3 perma-down, deleted bucket, revoked RBAC) doesn't block namespace teardown. 1h rides out transient apiserver/S3 incidents while bounding the wait.
`operator.metrics`	`object`	(empty)	Controller-runtime exposes a Prometheus metrics endpoint covering workqueue depth, reconcile latencies, the manager's REST client, and whatever custom collectors the reconciler registers. The chart wires the endpoint to a dedicated Service; ServiceMonitor rendering is opt-in for clusters running the Prometheus Operator.
`operator.metrics.enabled`	`boolean`	`true`
`operator.metrics.prometheusRule`	`object`	(empty)	Opt-in PrometheusRule shipping a starter set of alerts on the operator's custom metrics plus a handful of standard controller-runtime signals. Requires the Prometheus Operator's `monitoring.coreos.com/v1` API to be installed in the cluster. Thresholds default to conservative values — page on sustained failure patterns, not single bad reconciles. Tune per cluster.
`operator.metrics.prometheusRule.annotations`	`object`	(empty)	Extra annotations on the PrometheusRule.
`operator.metrics.prometheusRule.enabled`	`boolean`	`false`
`operator.metrics.prometheusRule.extraAlertLabels`	`object`	(empty)	Labels merged onto EVERY alert this PrometheusRule renders. Useful for routing all jaas alerts through one Alertmanager receiver (e.g., `team: platform`).
`operator.metrics.prometheusRule.extraRules`	`array`	(empty)	Extra alert rules appended to the rendered PrometheusRule under a separate "jaas-operator-extras" group. Lets operators ship cluster-specific alerts (e.g. custom thresholds on jaas metrics, alerts joined against deployment labels) without forking the chart. The list is rendered verbatim — each entry must be a valid PrometheusRule alert (see Prometheus documentation for the schema). To silence a built-in alert: raise its threshold under the `thresholds` block above to an impossibly high value. There is no per-built-in disable toggle — the threshold pattern is the recommended path because it surfaces "this alert is intentionally inert" in the chart values rather than hiding the configuration behind a flag list. Example: extraRules: - alert: JaaSManySnippetsDown expr: count(jaas_snippet_reconcile_total{status="False"}) > 10 for: 5m labels: { severity: critical, team: platform } annotations: summary: '>10 snippets failing across all namespaces'
`operator.metrics.prometheusRule.interval`	`string`	`30s`	Evaluation interval for every group in the rendered rule.
`operator.metrics.prometheusRule.labels`	`object`	(empty)	Extra labels copied onto the PrometheusRule object itself (typically the labels your Prometheus instance selects on).
`operator.metrics.prometheusRule.runbookAnnotationKey`	`string`	`runbook_url`	Annotation key the runbook URL lands under on every alert. `runbook_url` is the Prometheus-operator convention (kube-prom, Alertmanager templates, most third-party tooling key off it); `runbook` is common in older internal stacks. Anything goes — the value is just a YAML map key.
`operator.metrics.prometheusRule.thresholds`	`object`	(empty)	All-numeric/duration thresholds in one place so operators can adjust the noise floor without copy-pasting the rule body.
`operator.metrics.prometheusRule.thresholds.artifactSizeBytes`	`integer`	`16777216`	JaaSSnippetArtifactGrowing: p99 rendered bytes threshold. Defaults to 16 MiB — well below the 64 MiB MaxArchiveBytes default but enough to catch a runaway snippet early.
`operator.metrics.prometheusRule.thresholds.artifactSizeDuration`	`string`	`30m`
`operator.metrics.prometheusRule.thresholds.crdWatchEngagementFailuresDuration`	`string`	`30m`
`operator.metrics.prometheusRule.thresholds.crdWatchEngagementFailuresPerHour`	`integer`	`1`	JaaSCRDWatchEngagementFailing: hourly increase on the CRD watch engagement failure counter. The retry pipeline will eventually re-engage, but a sustained inability to engage means dependent snippets aren't re-rendering on upstream source events.
`operator.metrics.prometheusRule.thresholds.evalLeakedDuration`	`string`	`5m`
`operator.metrics.prometheusRule.thresholds.evalLeakedFloor`	`integer`	`0`	JaaSEvalLeakedGoroutines: floor on the leak gauge before alerting. The gauge counts eval goroutines whose parent ctx already fired — orphans still consuming CPU until their synchronous go-jsonnet call returns naturally. CLAUDE.md's documented design is "> 0 for 5m" — any sustained leak is textbook runaway-snippet signal. 0 is intentional; the accompanying duration window catches transient spikes without paging.
`operator.metrics.prometheusRule.thresholds.evalRejectedDuration`	`string`	`10m`
`operator.metrics.prometheusRule.thresholds.evalRejectedRate`	`number`	`0.05`	JaaSEvalRejected: per-second rate of evaluations the semaphore turned away. 0.05/s ≈ one rejection every 20s, a rate that indicates sustained overload rather than a transient spike a single retry would clear.
`operator.metrics.prometheusRule.thresholds.evalSaturationDuration`	`string`	`10m`
`operator.metrics.prometheusRule.thresholds.evalSaturationRatio`	`number`	`0.9`	JaaSEvalSaturation: ratio of in-flight evaluations to the configured cap before alerting. 0.9 means "running at 90 %+ of the eval semaphore for evalSaturationDuration" — close enough to the cap that any new burst will start landing rejections. The alert guards on jaas_eval_max_concurrent > 0 so a disabled gate (--max-concurrent-evals=0) never fires.
`operator.metrics.prometheusRule.thresholds.forceDropsDuration`	`string`	`5m`
`operator.metrics.prometheusRule.thresholds.forceDropsPerHour`	`integer`	`0`	JaaSForceDropsAccumulating: hourly increase on the force-drop counter. Even a single force-drop means an orphan tarball was left behind; sustained drops mean a permanently-broken pipeline (revoked RBAC on the EA Delete, deleted S3 bucket, missing CRD). The remediation runbook is storage-recovery.md.
`operator.metrics.prometheusRule.thresholds.podDownDuration`	`string`	`5m`	JaaSOperatorPodDown: how long an operator pod can stay NotReady before paging. Below 5m the alert tends to fire during normal pod rolls.
`operator.metrics.prometheusRule.thresholds.reconcileErrorDuration`	`string`	`10m`
`operator.metrics.prometheusRule.thresholds.reconcileErrorRate`	`number`	`0.1`	JaaSSnippetReconcileErrorsHigh: per-snippet Ready=False rate (reconciles per second, 5m window). 0.1/s ≈ one failure every 10s — a clearly stuck snippet, not transient flapping.
`operator.metrics.prometheusRule.thresholds.reconcileLatencyDuration`	`string`	`15m`
`operator.metrics.prometheusRule.thresholds.reconcileLatencySeconds`	`integer`	`30`	JaaSReconcileLatencyHigh: reconcile-time p99 ceiling, in seconds. 30s is generous — long enough for an OCI fetch plus eval of a non-trivial snippet, short enough to catch a genuinely stuck reconciler.
`operator.metrics.prometheusRule.thresholds.sweepFailuresDuration`	`string`	`30m`
`operator.metrics.prometheusRule.thresholds.sweepFailuresPerHour`	`integer`	`3`	JaaSStorageSweepFailures: per-hour count of failing background sweep passes before alerting. Default 3 absorbs the occasional flake without paging while catching a genuinely degraded backend (chronic disk full, revoked permissions). The sweep runs on operator.storage.sweep.interval (default 10m), so 3/hour ≈ half the passes failing.
`operator.metrics.prometheusRule.thresholds.tenantTokenMintFailureDuration`	`string`	`10m`
`operator.metrics.prometheusRule.thresholds.tenantTokenMintFailureRate`	`number`	`0.01`	JaaSTenantTokenMintFailing: per-second mint failure rate across the cluster. Even one mint failure on a (namespace, SA) pair flags revoked `serviceaccounts/token: create` or a deleted SA — actionable immediately.
`operator.metrics.prometheusRule.thresholds.webhookCertRenewalFailuresDuration`	`string`	`30m`
`operator.metrics.prometheusRule.thresholds.webhookCertRenewalFailuresPerHour`	`integer`	`1`	JaaSWebhookCertRenewalFailing: hourly increase on the cert renewal failure counter. The Renewer ticks every Validity/3 (typically every few months), so a single failure shouldn't page; sustained failures across multiple ticks (RBAC drift, CertDir write-perm loss) mean the rotation pipeline is broken and the existing cert's natural expiry is the deadline.
`operator.metrics.prometheusRule.thresholds.workqueueDepth`	`integer`	`50`	JaaSControllerWorkqueueDepthHigh: items the workqueue holds before alerting. Default 50 is high enough to absorb a normal roll-out, low enough that genuine apiserver stalls page.
`operator.metrics.prometheusRule.thresholds.workqueueDuration`	`string`	`15m`
`operator.metrics.serviceMonitor`	`object`	(empty)	Opt-in ServiceMonitor (Prometheus Operator) for the metrics endpoint.
`operator.metrics.serviceMonitor.annotations`	`object`	(empty)	Extra annotations on the ServiceMonitor.
`operator.metrics.serviceMonitor.enabled`	`boolean`	`false`
`operator.metrics.serviceMonitor.interval`	`string`	`30s`	Scrape interval the Prometheus instance honors. Match this to your shared scrape config.
`operator.metrics.serviceMonitor.labels`	`object`	(empty)	Extra labels the ServiceMonitor carries (typically the labels your Prometheus instance selects on, e.g., release: kube-prom).
`operator.metrics.serviceMonitor.scrapeTimeout`	`string`	`10s`	Per-scrape timeout. Must stay strictly less than interval.
`operator.noCrossNamespaceRefs`	`boolean`	`true`	When true, the operator rejects JsonnetSnippet / library refs that point at a SourceRef in a different namespace.
`operator.rbac`	`object`	(empty)	Opt out of chart-rendered RBAC when external governance owns ClusterRole/RoleBinding shape (ArgoCD app-of-apps, Crossplane, GitOps controller that manages RBAC separately). The operator still expects the named SA to have the verbs documented in README's "Operator Mode" section; with rbac.create=false you're responsible for granting them.
`operator.rbac.create`	`boolean`	`true`
`operator.rerenderBurst`	`integer`	`120`	Per-snippet token-bucket depth.
`operator.rerenderRate`	`string`	`60/min`	Per-snippet steady-state re-render budget. N/period where period is one of sec, min, hour. Token-bucket combined with rerenderBurst.
`operator.serviceAccount`	`object`	(empty)	ServiceAccount the operator runs as. When create is true the chart provisions one; otherwise the operator binds against the named SA.
`operator.serviceAccount.annotations`	`object`	(empty)	Annotations to attach to the operator ServiceAccount. Common uses: AWS IRSA (eks.amazonaws.com/role-arn), GCP Workload Identity (iam.gke.io/gcp-service-account), Azure Workload Identity (azure.workload.identity/client-id). These bind cloud-IAM identities to the SA so the operator can access cloud resources without static credentials.
`operator.serviceAccount.create`	`boolean`	`true`
`operator.serviceAccount.name`	`string`	`jaas`
`operator.storage`	`object`	(empty)	Artifact storage for published ExternalArtifact tarballs (operator mode).
`operator.storage.backend`	`string`	`local`	Artifact backend used to persist ExternalArtifact tarballs. local — emptyDir or PVC mounted into the pod. Simple, single-pod (RWO PVC) unless a ReadWriteMany storage class is configured. s3 — any S3-compatible bucket (AWS S3, MinIO, Ceph RGW, etc.). The chart-default story for multi-replica HA: every replica reads from the same bucket; leader election still gates writes. PVC settings below are ignored when backend=s3. Artifact backend used to persist ExternalArtifact tarballs (local \| s3).
`operator.storage.baseURL`	`string`	(empty)	Public URL prefix downstream Flux consumers dereference. Defaults to the in-cluster Service DNS name when empty — operators wiring this up via Ingress should set it explicitly.
`operator.storage.gcGrace`	`string`	`5m`	Minimum time a superseded artifact revision stays fetchable after being evicted from the keep-set. Closes the pin→fetch race in which a Flux consumer reads ExternalArtifact.status.artifact a moment before the operator garbage-collects the superseded revision and then 404s on the URL. Supersession time is derived from on-disk storage metadata so the window survives operator restarts. Zero restores eager pruning; the snippet teardown path (finalizer Withdraw) is unaffected. See docs/consumers.md for when to leave the default vs raise spec.history for deliberate rollback retention.
`operator.storage.headless`	`boolean`	`false`	When true, the storage Service is rendered as headless (clusterIP: None) so DNS resolves each pod IP directly. Useful for external sidecars that want stable per-pod targets, or for traffic that should skip kube-proxy. Default ClusterIP otherwise.
`operator.storage.listenAddress`	`string`	`::`	Bind address for the storage HTTP server. Same dual-stack semantics as arguments.listenAddress: `::` accepts both IPv6 and IPv4-mapped traffic on Linux.
`operator.storage.maxArtifactBytes`	`integer`	`0`	Cap the rendered artifact size in bytes. Snippets whose rendered output exceeds this fail with ReasonArtifactTooLarge before any disk/S3 write — stops one runaway snippet from filling a shared storage volume. Zero disables.
`operator.storage.path`	`string`	`/var/lib/jaas/artifacts`	Directory inside the container the operator writes tarballs to. Only consulted when backend=local.
`operator.storage.persistence`	`object`	(empty)	Persist tarballs across pod restarts via a PVC. With persistence OFF (the default) the chart uses an emptyDir, which is lost on pod restart — downstream Flux consumers then see brief 404s while every snippet re-renders. With persistence ON, the PVC survives restarts and downstream consumers see no gap. PVC access mode is RWO by default, which constrains the chart to a single replica (RWO can only attach to one pod). For multi-replica HA you need an RWX storage class — uncommon in cloud providers.
`operator.storage.persistence.accessModes`	`array`	(empty)	accessModes the PVC declares. ReadWriteOnce is the safe default; ReadWriteMany unlocks multi-replica HA when the StorageClass supports it.
`operator.storage.persistence.annotations`	`object`	(empty)	Extra annotations on the PVC (e.g., CSI volume tags).
`operator.storage.persistence.enabled`	`boolean`	`false`
`operator.storage.persistence.labels`	`object`	(empty)	Extra labels on the PVC.
`operator.storage.persistence.selector`	`object`	(empty)	Optional PVC selector for matching pre-provisioned volumes.
`operator.storage.persistence.size`	`string`	`10Gi`	Requested volume size. 10 GiB holds many thousand small tarballs.
`operator.storage.persistence.storageClassName`	`string`	(empty)	storageClassName picks the StorageClass. Empty means "cluster default" — usually right.
`operator.storage.readTimeout`	`string`	`30s`	Read timeout on the storage HTTP server.
`operator.storage.s3`	`object`	(empty)	S3 backend configuration. Only consulted when backend=s3.
`operator.storage.s3.anonymous`	`boolean`	`false`	Skip request signing entirely. Public-bucket test mode only.
`operator.storage.s3.bucket`	`string`	(empty)	Bucket the artifacts live in. Must already exist; jaas does not create it. Required.
`operator.storage.s3.credentialsSecret`	`object`	(empty)	Credentials are supplied one of two ways — never inline. Inline values would land on the Pod's command line (visible in `ps`, in the PodSpec, and in the stored Helm release), so the chart deliberately offers no accessKey/secretKey/sessionToken field: credentialsSecret.name — an existing Secret carrying AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN. It is wired into the pod via envFrom.secretRef; minio-go's env chain picks the keys up automatically since no -s3-* credential flag is passed. leave credentialsSecret empty — minio-go's IAM/IRSA discovery chain takes over (AWS_* env vars, EKS web-identity, EC2 metadata). Bind a cloud identity to the operator's ServiceAccount via operator.serviceAccount.annotations.
`operator.storage.s3.credentialsSecret.name`	`string`	(empty)
`operator.storage.s3.endpoint`	`string`	(empty)	Endpoint host:port (e.g. s3.amazonaws.com, minio.minio.svc:9000). Required.
`operator.storage.s3.prefix`	`string`	(empty)	Optional object-key prefix so jaas can coexist with other tenants in one bucket: <prefix>/<namespace>/<snippet>/<rev>.tar.gz
`operator.storage.s3.region`	`string`	(empty)	Region the bucket lives in. Required for AWS multi-region; ignored by most S3-compatible servers.
`operator.storage.s3.useSSL`	`boolean`	`true`	Talk HTTPS. Disable only for local MinIO over HTTP.
`operator.storage.sizeLimit`	`string`	`256Mi`	emptyDir size limit. Only consulted when persistence.enabled is false. Tarballs typically are well under 1 MiB; the sizing here is generous so transient bursts don't OOM the pod.
`operator.storage.sweep`	`object`	(empty)	Periodic GC: sweep orphaned <rev>.tar.gz.tmp residue left by Puts whose process died mid-rename. Local backend only — the S3 backend's Put is atomic so no .tmp files exist. Set interval to "0s" to disable.
`operator.storage.sweep.interval`	`string`	`10m`
`operator.storage.sweep.maxTmpAge`	`string`	`30m`
`operator.storage.writeTimeout`	`string`	`5m`	Write timeout on the storage HTTP server (--storage-write-timeout): how long a single response is allowed to take before the connection is closed. To cap artifact size, use operator.storage.maxArtifactBytes.
`operator.tracing`	`object`	(empty)	OpenTelemetry tracing for the operator. When endpoint is empty (the default), no spans are emitted and the OTel SDK is in no-op mode. Set the endpoint to ship spans to an OTLP gRPC collector: tracing: endpoint: otel-collector.observability.svc:4317 insecure: true
`operator.tracing.endpoint`	`string`	(empty)
`operator.tracing.insecure`	`boolean`	`false`
`operator.tracing.sampleRatio`	`number`	`1`
`operator.watchNamespaces`	`array`	(empty)	Namespaces this operator instance scopes its cache to. Empty (the default) means cluster-wide. When set, the manager's informer cache only observes CRs in these namespaces, AND the rendered RBAC pivots from a single ClusterRoleBinding to one RoleBinding per namespace — multi-tenant operator-instances pattern.
`operator.webhook`	`object`	(empty)	Validating admission webhook (operator mode). Rejects JsonnetSnippets whose spec.externalVariables collide with the operator's extVars.
`operator.webhook.certDir`	`string`	`/tmp/k8s-webhook-server/serving-certs`	Directory inside the container where the TLS cert and key land. With certMode=cert-manager this is mounted read-only from the secretName below. With certMode=self-signed it's a writable emptyDir the operator writes into on startup.
`operator.webhook.certManager`	`object`	(empty)	cert-manager-mode knobs. Only consulted when certMode is "cert-manager". When self-signed, this block is ignored and no cert-manager API objects are rendered.
`operator.webhook.certManager.enabled`	`boolean`	`true`
`operator.webhook.certManager.issuerRef`	`object`	(empty)
`operator.webhook.certManager.issuerRef.kind`	`string`	`Issuer`	Issuer kind: Issuer or ClusterIssuer
`operator.webhook.certManager.issuerRef.name`	`string`	(empty)	Issuer name in the chart's namespace (Issuer) or cluster (ClusterIssuer). Required when certMode=cert-manager.
`operator.webhook.certMode`	`string`	`self-signed`	How TLS material for the webhook is provisioned: cert-manager — chart renders a cert-manager Certificate; the Secret is mounted into the pod read-only. self-signed — operator generates a CA + serving cert at startup and stamps its own ValidatingWebhookConfiguration caBundle. Requires the operator to have get/update on the named VWC (added automatically). Removes the cert-manager dependency entirely. The default is self-signed because it needs no prerequisites and always works; cert-manager mode only works when cert-manager is installed in the cluster.
`operator.webhook.enabled`	`boolean`	`false`
`operator.webhook.failurePolicy`	`string`	`Fail`	Failure policy for the ValidatingWebhookConfiguration. Fail is the safer default — a webhook outage blocks JsonnetSnippet writes cluster-wide until the operator is back. Switch to Ignore only when you're willing to accept invalid snippets reaching the reconciler. Either way, the reconciler enforces the same ext-var-conflict invariant as a fallback so admission bypass never silently breaks the contract. During an operator restart (rolling update, crash-loop, eviction) the webhook is briefly unreachable and Fail blocks every JsonnetSnippet create/update for that window — typically <5s with `LeaderElectionReleaseOnCancel: true`. If your CI/GitOps tooling can't tolerate that, scope the webhook with objectSelector / namespaceSelector to limit the blast radius, or set failurePolicy: Ignore.
`operator.webhook.matchConditions`	`array`	(empty)	matchConditions are CEL expressions evaluated server-side before the webhook is dialed. Lets operators short-circuit admission checks for specific snippet shapes without a round trip. Requires Kubernetes 1.30+.
`operator.webhook.namespaceSelector`	`object`	(empty)	namespaceSelector restricts the webhook to Namespaces carrying the listed labels. A common pattern: opt-in per namespace via a label so the apiserver doesn't dial the webhook for unrelated namespaces during operator outages. Example: namespaceSelector: matchLabels: jaas.metio.wtf/admission: enforce
`operator.webhook.objectSelector`	`object`	(empty)	objectSelector restricts the webhook to JsonnetSnippets carrying the listed labels. Useful when only some snippets are operator-managed (the rest live in Argo or kustomize). Example: objectSelector: matchLabels: jaas.metio.wtf/managed: "true"
`operator.webhook.secretName`	`string`	`jaas-webhook-cert`	The Secret the cert lives in. cert-manager writes it; the Deployment mounts it read-only at certDir.
`operator.webhook.selfSignedValidity`	`string`	`8760h`	Validity of the self-signed serving cert. Operators that want short-lived rotation should use cert-manager instead — the self-signed mode renews on every pod restart, no in-flight rotation. Only consulted when certMode=self-signed.
`operator.webhook.sideEffects`	`string`	`None`	Side-effect mode for the ValidatingWebhookConfiguration.
`paths`	`object`	(empty)	Filesystem paths inside the container. Modify the paths used, use this in case you run a custom image.
`paths.libraries`	`string`	`/srv/libraries`	Directory library OCI volumes mount beneath.
`paths.snippets`	`string`	`/srv/snippets`	Directory snippet OCI volumes mount beneath.
`pdb`	`object`	(empty)	Fine tune the PodDisruptionBudget. Rendered only when replicas.max > replicas.min.
`pdb.maxUnavailable`	`integer\|string\|null`	`1`	PDB maxUnavailable. Default 1 caps voluntary disruptions to one pod at a time. Set to null when using minAvailable instead.
`pdb.minAvailable`	`integer\|string\|null`	`null`	Disruption budget shape. Exactly one of minAvailable / maxUnavailable is rendered into the PDB spec; setting both is a config error K8s would reject. Default `maxUnavailable: 1` caps voluntary disruptions to one pod at a time regardless of replica count — the right knob for a rolling restart / node drain story. `minAvailable: N` is the alternative when you want "always keep at least N pods up" instead — useful only when the workload's headroom is a fixed floor, not the more common "one out at a time". Override examples: pdb: { maxUnavailable: 2 } # allow draining two nodes in parallel pdb: { minAvailable: 1, maxUnavailable: null } # floor of 1 instead pdb: { minAvailable: "50%", maxUnavailable: null } # percentage form PDB minAvailable. Mutually exclusive with maxUnavailable; null leaves it unset.
`pdb.unhealthyPodEvictionPolicy`	`string`	(empty)	spec.unhealthyPodEvictionPolicy on the PDB. Empty defers to the cluster default.
`pod`	`object`	(empty)	Fine tune the managed Pod resources.
`pod.additionalLabels`	`object`	(empty)	Extra labels merged onto the pod template.
`ports`	`object`	(empty)	Container port assignments for the jaas pod.
`ports.http`	`integer`	`8080`	Used by the /jsonnet endpoint
`ports.management`	`integer`	`8081`	Used by startup, readiness, and liveness probes
`ports.metrics`	`integer`	`8083`	Used by controller-runtime's Prometheus metrics endpoint (operator.enabled only). Defaults to 8083 to avoid the collision with controller-runtime's own :8080 default and the jsonnet HTTP port. Set to 0 in operator.metrics.enabled to disable entirely.
`ports.storage`	`integer`	`8082`	Used by the operator's storage HTTP server (operator.enabled only)
`ports.webhook`	`integer`	`9443`	Used by the validating admission webhook (operator.webhook.enabled only)
`replicas`	`object`	(empty)	Replica bounds for the Deployment. It is safe to increase the number of replicas.
`replicas.max`	`integer`	`1`
`replicas.min`	`integer`	`1`
`resources`	`object`	(empty)	Per-container resource requests/limits. Smallest possible values here, increase if you have more resources to spare.
`resources.cpu`	`string`	`32m`
`resources.ephemeralStorage`	`string`	`10Mi`
`resources.memory`	`string`	`64Mi`
`service`	`object`	(empty)	Fine tune the Service resource.
`service.ipFamilies`	`array`	(empty)	Explicit list of IP families for the Service. Empty defers to the cluster's default. Example for explicit IPv4+IPv6: ipFamilies: [IPv4, IPv6] Order matters: the first family becomes the primary ClusterIP.
`service.ipFamilyPolicy`	`string`	(empty)	K8s Service ipFamilyPolicy. Empty leaves the cluster's default (usually SingleStack). Set to PreferDualStack on dual-stack clusters that may still have single-stack nodes, or RequireDualStack to refuse single-stack clusters at admission.
`serviceMesh`	`object`	(empty)	Opt-in service-mesh L7 authorization + mTLS for the jaas pod. Complementary to networkPolicy, NOT a replacement: networkPolicy is L3/L4 (which pods/IPs may open a connection), serviceMesh is L7/identity (which mesh identities may call which port). Both can be enabled at once and stack additively. Authorizes only the mesh-reachable ports — http (jsonnet), storage (artifacts), and metrics (operator). The webhook (kube-apiserver) and management/probe (kubelet) ports are deliberately left out: those callers speak plain TLS / no mesh identity, and locking them to mesh principals would break admission and health probing.
`serviceMesh.defaultDeny`	`object`	(empty)	Render a namespace-wide default-deny so every pod in the install namespace rejects unauthorized mesh traffic and the per-workload allows above are the only exceptions (zero-trust namespace). Istio renders an empty-spec AuthorizationPolicy (deny-all) scoped to the whole namespace, which sits at lower precedence than the workload ALLOW. Linkerd has no per-object deny-all; the namespace default is set via the config.linkerd.io/default-inbound-policy annotation, so this is stamped onto the chart-managed Namespace (requires namespace.create=true) — otherwise annotate the namespace out-of-band. Enable only when jaas owns its namespace; it also denies co-located workloads.
`serviceMesh.defaultDeny.enabled`	`boolean`	`false`
`serviceMesh.enabled`	`boolean`	`false`	Opt-in. When enabled, the chart renders the selected engine's authorization + mTLS objects for the jaas pod. Inert unless that mesh (Istio or Linkerd) is actually installed and the pod is injected.
`serviceMesh.engine`	`string`	`istio`	Mesh engine to render for. istio emits security.istio.io AuthorizationPolicy (+ optional PeerAuthentication). linkerd emits policy.linkerd.io Server / AuthorizationPolicy / MeshTLSAuthentication objects.
`serviceMesh.http`	`object`	(empty)	Per-port allowed mesh identities. Each `from` entry is a source matcher: - principals: SPIFFE/mesh identities (Istio source.principals, e.g. cluster.local/ns/<ns>/sa/<sa>). For Linkerd these map to MeshTLSAuthentication identities (the proxy identity string, e.g. <sa>.<ns>.serviceaccount.identity.linkerd.cluster.local, or ""). - namespaces: source namespaces (Istio source.namespaces). ISTIO-ONLY — Linkerd authenticates by workload identity, not by namespace, so this field is ignored on the linkerd engine. An EMPTY `from` list on a port means OPEN (allow any caller), mirroring networkPolicy's empty-`from`=open semantics. On Istio that is a port rule with no `from`; on Linkerd it is a MeshTLSAuthentication of identities: [""] (any authenticated meshed client). Mesh identities allowed to hit the jsonnet HTTP endpoint (ports.http). Empty = open.
`serviceMesh.http.from`	`array`	(empty)
`serviceMesh.istio`	`object`	(empty)	Istio engine raw passthrough. Entries are merged verbatim into the AuthorizationPolicy's spec.rules (security.istio.io/v1 rule schema). Use this to add rules the per-port `from` knobs above can't express — path/method matchers, `when` JWT-claim conditions, ipBlocks, etc.
`serviceMesh.istio.rules`	`array`	(empty)
`serviceMesh.linkerd`	`object`	(empty)	Linkerd engine raw passthrough. Entries are appended verbatim as additional documents after the rendered Server / AuthorizationPolicy / MeshTLSAuthentication set — each entry must be a complete object (policy.linkerd.io AuthorizationPolicy, HTTPRoute, etc.).
`serviceMesh.linkerd.authorizations`	`array`	(empty)
`serviceMesh.metrics`	`object`	(empty)	Mesh identities allowed to scrape the operator metrics port (ports.metrics). Operator mode + metrics only. Empty = open.
`serviceMesh.metrics.from`	`array`	(empty)
`serviceMesh.mtls`	`string`	(empty)	mTLS posture (Istio engine only). Empty defers to the mesh's own default (mesh-wide PeerAuthentication / MeshConfig). permissive renders a PeerAuthentication accepting both mTLS and plaintext. strict requires mTLS on the workload's ports EXCEPT the webhook + management ports, which get a port-level PERMISSIVE carve-out so the non-mesh kube-apiserver and kubelet still connect. Linkerd negotiates mTLS automatically between meshed pods, so this knob does not apply to the linkerd engine.
`serviceMesh.storage`	`object`	(empty)	Mesh identities allowed to read published artifacts from the storage port (ports.storage). Operator mode only. Empty = open.
`serviceMesh.storage.from`	`array`	(empty)
`snippets`	`object\|null`	(empty)	Snippet OCI volumes to mount. The key is the name of the directory and the value is an OCI volume that will be mounted beneath .Values.paths.snippets, e.g., the commented example below would result in all files from 'ghcr.io/metio/something:latest' to be mounted at '/srv/snippets/something/' (using the default configuration)

joi library chart

The joi chart publishes Jsonnet OCI Images as JsonnetLibrary + OCIRepository pairs, so snippets can import vendored libraries (grafonnet, k8s-libsonnet, …) without bundling them. Deploy it alongside jaas when snippets reference shared libraries.

High reconcile latency

Mon, 01 Jan 0001 00:00:00 +0000

Linked from the JaaSReconcileLatencyHigh alert. Fires when the controller-runtime controller_runtime_reconcile_time_seconds histogram p99 exceeds the configured threshold (default 30s) for the alert window.

Symptom

ALERTS{alertname="JaaSReconcileLatencyHigh", controller="jsonnetsnippet"}

kubectl get jsonnetsnippet shows status updates trickling in well after spec changes.
Operator pod CPU is moderate-to-high but the queue is draining (distinguishes this from workqueue-saturation.md , where the queue itself is growing).

Cause

Reconcile latency is the wall-clock cost of one Reconcile() call. Inside the call, JaaS does (in order):

InvalidSpec

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=InvalidSpec. The condition Message names which field is at fault.

Cause

Spec-level validation that admission should have caught but the reconciler is enforcing as a fallback:

spec.entryFile is empty
both spec.files and spec.sourceRef are set (mutually exclusive)
neither spec.files nor spec.sourceRef is set
spec.entryFile does not match any key in the resolved file map

Diagnosis

kubectl describe jsonnetsnippet <name>

Read the Message — it names the field.

Remediation

Fix the spec and reapply. If the validating webhook is enabled (--enable-webhook), kubectl apply rejects the invalid spec at admission instead of letting it land and fail later.

JaaS and grafana-operator

Mon, 01 Jan 0001 00:00:00 +0000

JaaS and the grafana-operator are not alternatives — they do different jobs and are commonly used together. JaaS produces dashboard JSON from Jsonnet; grafana-operator consumes dashboard JSON and reconciles it into a Grafana instance.

Division of labour

grafana-operator manages Grafana itself. It reconciles GrafanaDashboard, GrafanaDatasource, GrafanaFolder, and related resources into one or more Grafana instances, handling authentication, folder placement, datasource wiring, and drift correction inside Grafana. A GrafanaDashboard can take its dashboard model from inline JSON, a URL, a ConfigMap, a grafana.com dashboard ID, or a remote source.

JaaS vs jsonnet-controller

Mon, 01 Jan 0001 00:00:00 +0000

pelotech/jsonnet-controller is a Flux-style controller that builds Jsonnet inside the controller and applies the result to the cluster, with a configuration model compatible with kubecfg . JaaS and jsonnet-controller both turn Jsonnet into Kubernetes objects under Flux, but they draw the boundary between rendering and deployment in different places.

Coupled build-and-apply vs a rendering service

jsonnet-controller couples the two halves: one controller reads a Jsonnet source, evaluates it, and applies the resulting objects to the cluster. The rendered output lives inside the controller’s reconcile loop; the unit of configuration is “build this Jsonnet and apply it here.”

JaaS vs Tanka

Mon, 01 Jan 0001 00:00:00 +0000

Grafana Tanka and JaaS both render Kubernetes manifests from Jsonnet, and both build on the same jsonnet-bundler vendoring conventions. The difference is where the rendering runs and how the result reaches the cluster.

The two models

Tanka renders and applies from a developer workstation or a CI runner. You organise your code as environments (environments/<env>/main.jsonnet plus a spec.json), run tk show or tk export to inspect the rendered objects, and tk apply to push them to the cluster the environment names in its apiServer field. The workstation or CI runner needs the Jsonnet toolchain, the vendored library tree, and credentials for the target cluster.

JaaS vs the jsonnet CLI

Mon, 01 Jan 0001 00:00:00 +0000

The jsonnet command-line tool — usually paired with jsonnet-bundler (jb) for vendoring libraries — evaluates Jsonnet to JSON on your machine. JaaS runs the same go-jsonnet core as a service. This is not a question of which implementation is correct; it is a question of where the evaluation runs and what surrounds it.

What the service adds

Over a local binary invocation, JaaS adds:

An HTTP endpoint other systems can call. GET /jsonnet/<snippet> returns the evaluated JSON, with Top Level Arguments supplied as query parameters and external variables configured on the service. Anything that speaks HTTP can request a render without installing the toolchain or the vendor tree. See the rendering endpoint usage page.
An operator that turns a snippet into a revisioned Flux artifact. With --enable-flux-integration, a JsonnetSnippet is evaluated continuously and published as a content-addressed ExternalArtifact that Flux consumers apply in-cluster — re-rendered automatically when its source changes. See operator mode .
Import resolution that matches jsonnet -J vendor. JaaS resolves imports with the same semantics as the CLI’s JPATH/vendor search, so the JSON a snippet produces under the service matches what the CLI produces locally.
Evaluation caps. --evaluation-timeout bounds wall-clock time per render, --max-stack bounds call-stack depth, and --max-concurrent-evals bounds how many evaluations run at once — so one expensive snippet cannot exhaust a shared server. The CLI imposes none of these on its own.
Read-scope sandboxing. Snippet-name resolution goes through Go’s os.Root, which rejects .. traversal and symlinks that escape the configured snippet directory, so a crafted request cannot read arbitrary files. The evaluation and security page details the caps and the boundaries.

When the plain CLI is the right tool

The CLI is the better choice for work that is local and one-off:

JOI images

Mon, 01 Jan 0001 00:00:00 +0000

Jsonnet OCI Images (JOI) package popular Jsonnet libraries as single-layer OCI images, one per upstream library, published at ghcr.io/metio/joi-<org>-<repo>. Because each image is a single layer, the same artifact serves two roles: a container image volume mounted into jaas, and a Flux OCIRepository source the operator fetches — so a snippet imports a vendored library without bundling it.

Deploy them with the joi Helm chart , which renders a JsonnetLibrary + OCIRepository pair for each enabled library. A snippet then imports a library by its alias, choosing the version in the import path:

Jsonnet libraries

Mon, 01 Jan 0001 00:00:00 +0000

Snippets import reusable Jsonnet from two places: namespaced JsonnetLibrary custom resources and OCI-mounted shared libraries the operator carries on disk. Both feed the same import-alias namespace, so a snippet’s import statements look identical regardless of where the library comes from.

The JsonnetLibrary CRD

A JsonnetLibrary is a namespaced bundle of .libsonnet files. Like a snippet, it declares exactly one source — inline spec.files or a spec.sourceRef to a Flux source (GitRepository, OCIRepository, Bucket, ExternalArtifact). The library carries no registration name of its own; the import alias is chosen on the snippet side.

JsonnetLibrary

Mon, 01 Jan 0001 00:00:00 +0000

JsonnetLibrary (jlib) is a namespaced bundle of .libsonnet files that JsonnetSnippet CRs in the same namespace can import. The library carries no artifact of its own and has no controller reconciling it today — it exists purely as a supply-side source for snippets. The import alias is set on the snippet side via LibraryRef.importPath (defaulting to the library’s metadata.name); the library itself carries no registration name. Task-oriented guidance lives in Jsonnet libraries .

JsonnetSnippet

Mon, 01 Jan 0001 00:00:00 +0000

JsonnetSnippet (jsnip) is the published unit of Jsonnet evaluation. The JaaS operator watches these namespaced CRs, evaluates the Jsonnet they describe, and upserts a Flux ExternalArtifact whose status.artifact.url points at the rendered result. Task-oriented guidance lives in Operator mode and Snippet sources .

Example

apiVersion: jaas.metio.wtf/v1
kind: JsonnetSnippet
metadata:
 name: hello-world
 namespace: default
spec:
 serviceAccountName: hello-world-tenant
 entryFile: main.jsonnet
 output: rendered
 history: 3
 interval: 10m
 suspend: false
 files:
 main.jsonnet: |
 local lib = import 'mylib/main.libsonnet';
 lib.dashboard(std.extVar('env'), std.extVar('cluster'))
 libraries:
 - kind: JsonnetLibrary
 name: mylib
 importPath: mylib
 externalVariables:
 env: production
 cluster: eu-west-1
 tlas:
 title:
 - My Dashboard

Exactly one of spec.files or spec.sourceRef must be set. Admission rejects CRs that set neither or both.

Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

JaaS ships as a container image at ghcr.io/metio/jaas:latest and as a Helm chart at oci://ghcr.io/metio/helm-charts/jaas. Pre-built binaries for Linux, macOS, and Windows are attached to each GitHub release for operators who prefer to run the binary directly.

Prerequisites

A Kubernetes cluster, v1.28 or later, with kubectl configured against it.
Helm v3.14 or later — OCI chart support is required to pull the chart from ghcr.io.

The Flux CR-based mode (below) additionally needs:

LibraryNotFound

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=LibraryNotFound. The Message names the missing library.

Cause

A spec.libraries[*] entry references a JsonnetLibrary CR that the operator cannot Get. Common reasons:

the library CR doesn’t exist (typo, wrong namespace, not yet applied)
the tenant ServiceAccount doesn’t have get on the library kind in the library’s namespace
the library is in a different namespace and --no-cross-namespace-refs=true

Diagnosis

# Confirm the library exists.
kubectl --namespace <ns> get jsonnetlibrary <name>

# Test the tenant's RBAC.
kubectl auth can-i get jsonnetlibrary <name> \
 --as=system:serviceaccount:<ns>:<tenant-sa> -n <library-ns>

If can-i returns no, RBAC is the gap.

Local rendering

Mon, 01 Jan 0001 00:00:00 +0000

JaaS runs as a cluster-free Jsonnet renderer: point it at a directory of snippets and a directory of libraries, then GET a snippet name to receive the evaluated JSON. No Kubernetes, no operator mode, no Flux. The evaluation core is the same one the operator uses, so a snippet that renders correctly here renders identically in-cluster.

This tutorial runs against this repository’s examples/ layout, so clone the repo first:

git clone https://github.com/metio/jaas
cd jaas

Step 1 — Get the binary or container image

Pre-built binaries are attached to each GitHub release . Download the archive for your platform, unpack it, and the jaas binary is inside.

Logging

Mon, 01 Jan 0001 00:00:00 +0000

JaaS logs through Go’s log/slog. Every request, reconcile, and lifecycle event is a structured record you can filter and parse rather than scrape with a regex. In operator mode, controller-runtime’s own output — leader election, cache sync, manager startup — flows through the same slog handler via the logr bridge (ctrl.SetLogger(logr.FromSlogHandler(...))), so the manager’s logs share the configured level and format instead of emitting controller-runtime’s default zap output.

The binary

Two flags control logging. They apply in every mode JaaS runs in:

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The JaaS operator exposes a Prometheus metrics endpoint covering controller-runtime’s standard families plus a custom jaas_* family the reconciler registers. Scrape it for dashboards and feed it into the shipped alerts .

The binary

controller-runtime’s Prometheus endpoint binds --metrics-bind-address (default :8083), serving the standard text exposition format at /metrics. Setting it to 0 disables the endpoint. The default deliberately avoids controller-runtime’s built-in :8080, which would collide with the Jsonnet HTTP port.

The full flag list with defaults is on the configuration page .

Network policy

Mon, 01 Jan 0001 00:00:00 +0000

The Helm chart ships an opt-in NetworkPolicy for the JaaS pod. It is off by default and renders only when networkPolicy.enabled is true. Two independent layers are on offer: pod-scoped allowlists that lock down only JaaS’s own pods (the safe default), and an additional namespace-wide default-deny for a zero-trust namespace. The ingress and egress tables below describe exactly what traffic JaaS depends on — in both renderer mode and operator mode — so everything else can be denied.

Observability

Mon, 01 Jan 0001 00:00:00 +0000

JaaS gives you four ways to see what it is doing in a cluster. Structured logs tell you what happened on a single request or reconcile; traces follow one operation across its spans; metrics aggregate behaviour into time series for dashboards and alerts; and alerts plus Kubernetes Events turn a sustained problem into a page or a notification.

Each pillar has its own page covering both the binary’s flags and the Helm chart keys that drive them:

Operations

Mon, 01 Jan 0001 00:00:00 +0000

Day-two operations for a running JaaS install. Initial install and hardening decisions are in Kubernetes and Production .

Graceful shutdown and drain

When Kubernetes sends SIGTERM, JaaS executes a two-phase shutdown to avoid dropping in-flight requests:

The readiness probe flips to false (503 on /ready). Kubernetes endpoint controllers begin deregistering the pod from Services.
JaaS waits for --shutdown-delay (default 5s) before closing its listeners. This window lets the endpoint propagation complete so no new traffic arrives after the server closes.
After the delay, the servers shut down gracefully with a 30-second context.WithTimeout. The operator goroutine is also cancelled and awaited within the same 30-second window.

The distroless runtime image has no sleep binary, so the drain delay is implemented in the binary rather than via a preStop hook. A second SIGTERM (or SIGINT) during the drain cuts the wait short.

Operator mode

Mon, 01 Jan 0001 00:00:00 +0000

JaaS runs as a Kubernetes operator alongside its HTTP renderer. In this mode it watches custom resources, evaluates the Jsonnet they describe, and publishes the result as a Flux ExternalArtifact that downstream controllers consume. The HTTP renderer keeps running; the operator is an additional set of goroutines, not a separate binary.

Enabling the operator

Set --enable-flux-integration on the binary:

jaas --enable-flux-integration \
 --storage-path=/var/lib/jaas/artifacts \
 --storage-base-url=http://jaas-storage.jaas.svc:8082

--storage-path and --storage-base-url are required in operator mode — they tell the operator where to write artifact tarballs and the public URL prefix downstream consumers fetch them from.

Operator pod not ready

Mon, 01 Jan 0001 00:00:00 +0000

Linked from the JaaSOperatorPodDown alert. Fires when at least one jaas pod has been Ready=False for the alert window (default 5m).

Symptom

ALERTS{alertname="JaaSOperatorPodDown", namespace="<jaas-ns>"}

New JsonnetSnippets stay Ready=Unknown indefinitely (no controller reconciling them).
Existing snippets keep serving stale ExternalArtifact content (the storage HTTP server may still respond; reconciliation is the part that stopped).

Cause

One of the chart’s two probes is failing:

Liveness (/live) is unconditional 200 — a failure here means the pod’s HTTP server itself isn’t responding (deadlock, OOM, panic).
Readiness (/ready) consults HealthState, which goes false during drainBeforeShutdown or before the listeners bind.
Startup (/start) returns 503 until MarkStarted() is called — bind failures (port already in use, permission denied) keep the pod stuck here forever.

Frequent causes:

Pending

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

kubectl get jsonnetsnippet shows READY=Unknown (or False with REASON=Pending) immediately after the snippet was created or its spec was updated.

Cause

The operator has observed the CR but hasn’t completed its first reconcile pass yet. Transient by design.

Diagnosis

kubectl describe jsonnetsnippet <name>

If the timestamp on the Pending condition is older than ~30 seconds, the operator is either:

not running (kubectl --namespace <jaas-namespace> get pods)
backed up on its work queue (check kubectl logs deploy/jaas and the workqueue_depth metric)
not the leader (multi-replica install, kubectl --namespace <jaas-namespace> get lease shows the holder)

Remediation

If transient, wait. If persistent:

Production

Mon, 01 Jan 0001 00:00:00 +0000

The chart’s defaults are safe for an initial install but not optimised for sustained production workloads. Work through these decisions before exposing JaaS to real traffic. Each links to the detailed guide.

1. Pick a storage backend

The single largest decision. Artifacts must survive pod restarts and, for HA, be readable by every replica simultaneously.

Backend	Persistence	Multi-replica HA
`local` + emptyDir (chart default)	No	No
`local` + RWO PVC	Yes	No — single replica only
`local` + RWX PVC	Yes	Yes — requires RWX storage class
`s3`	Yes	Yes — leader writes, all replicas read

For cloud installs, s3 (AWS S3, MinIO, Ceph RGW, GCS S3-compat API) is the recommended backend. Pair it with leader election (on by default) so only the lease-holder writes. For on-prem, a PVC with the access mode your storage class supports is the practical path.

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial takes you from an empty cluster to one published Flux ExternalArtifact carrying rendered JSON. The path is operator mode with no optional knobs — no webhook, no S3, no Flux source CRs — and a single JsonnetSnippet whose source is inline spec.files.

Prerequisites

A Kubernetes cluster. kind, minikube, or a managed cluster all work.
kubectl configured to talk to it.
helm 3.x.
Flux installed, at v2.7.0 or newer. A JsonnetSnippet publishes its result as a Flux ExternalArtifact, and the ExternalArtifact CRD lands in source-controller v1.7.0 (Flux v2.7.0) — earlier bundles have no such CRD and the publish path fails. Install all of Flux:

RBACDenied

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

kubectl describe jsonnetsnippet <name>
...
Status:
 Conditions:
 Reason: RBACDenied
 Status: False
 Type: Ready
 Message: RBAC denied reading the source CR — grant the tenant ServiceAccount get on the source kind ...

Or for a missing CRD:

 Message: source CR's kind is not registered with the apiserver — install the corresponding CRD ...

The reconciler logs at warn level and stops engaging backoff for this snippet. The next reconcile happens only when the snippet’s spec changes, a referenced library / source CR’s status flips, or spec.interval ticks — so the workqueue isn’t burning cycles on a permanently-failing call.

Rendering endpoint

Mon, 01 Jan 0001 00:00:00 +0000

Send a GET to the rendering endpoint with a snippet name and JaaS returns the evaluated Jsonnet as JSON:

curl http://127.0.0.1:8080/jsonnet/example1

The Jsonnet server binds 127.0.0.1:8080 by default (--listen-address, --port). The URL shape is GET /<jsonnet-endpoint-path>/{snippet...}, where {snippet...} is a trailing path segment that may contain slashes. A successful response carries Content-Type: application/json and the rendered document.

The endpoint path

The leading path segment defaults to jsonnet and is set with --jsonnet-endpoint-path. Running with --jsonnet-endpoint-path render moves the endpoint to GET /render/{snippet...}:

Self-signed webhook cert renewal failing

Mon, 01 Jan 0001 00:00:00 +0000

Fires when jaas_webhook_cert_renewal_failures_total has increased above the configured per-hour threshold. The Renewer background goroutine rotates the self-signed TLS material every Validity / 3 (typically every few months for a year-long cert). When it can’t, the existing cert keeps working until its natural expiry — at which point the apiserver stops trusting the chain and every JsonnetSnippet admission fails cluster-wide with x509 errors.

Symptom

JaaSWebhookCertRenewalFailing alert is firing (severity: critical).
Operator pod logs carry repeated Self-signed webhook cert renewal failed warnings at the Renewer.Interval cadence.
kubectl describe validatingwebhookconfiguration <name> shows a caBundle that hasn’t rotated since the failures started.
The pod stays Ready=True — the renewer’s failures don’t gate the readiness probe.

Diagnosis

The most common causes, in order of frequency:

Service mesh

Mon, 01 Jan 0001 00:00:00 +0000

The Helm chart ships an opt-in service-mesh authorization layer for the JaaS pod. It is off by default and renders only when serviceMesh.enabled is true. Where the network policy operates at L3/L4 — which pods and IP ranges may reach which ports — the service mesh operates at L7 with identity-based authorization and mTLS: it asks which mesh identity is calling, proven by a cryptographic workload certificate, not merely which IP the packet came from.

ServiceAccountMissing

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=ServiceAccountMissing.

Cause

The snippet omitted spec.serviceAccountName AND the operator was started without --default-service-account. The operator refuses to reconcile a snippet with no effective ServiceAccount because every reconcile mints a tenant token from that SA — without one, there’s nothing to impersonate.

Diagnosis

kubectl get jsonnetsnippet <name> --output jsonpath='{.spec.serviceAccountName}'

Empty? Either the snippet must set it, or the cluster operator must configure a default.

Remediation

Pick one:

Snippet-side (preferred for multi-tenant setups): set spec.serviceAccountName: <existing-sa> on every snippet. Each tenant uses its own SA → least-privilege impersonation.
Cluster-side (single-tenant clusters): start the operator with --default-service-account=<sa-name>. Every snippet without an explicit SA impersonates this one. The default SA must exist in every snippet’s namespace — the operator looks it up per-reconcile.

Snippet sources

Mon, 01 Jan 0001 00:00:00 +0000

A JsonnetSnippet declares exactly one source for its Jsonnet bytes: either inline spec.files or a spec.sourceRef pointing at a Flux source. Admission rejects a snippet that sets both or neither. The operator resolves the source into an in-memory file tree, evaluates spec.entryFile within it, and publishes the result.

Inline files

spec.files is a map of filename to Jsonnet source. The operator evaluates the entry file (spec.entryFile, default main.jsonnet) against the rest of the map. This is the simplest source — the snippet is self-contained, with no external dependency to fetch:

Snippets and libraries

Mon, 01 Jan 0001 00:00:00 +0000

In renderer mode you declare snippets and libraries on disk through command-line flags. A snippet becomes reachable at the rendering endpoint ; a library is importable by any snippet.

Directory snippets

Point --snippet-directory at a directory whose subdirectories each hold a main.jsonnet. Each subdirectory name becomes a snippet name:

./jaas --snippet-directory examples/snippets/dashboards
curl http://127.0.0.1:8080/jsonnet/example1

Given this layout, example1 resolves to examples/snippets/dashboards/example1/main.jsonnet:

examples/snippets/dashboards
├── example1
│ └── main.jsonnet
├── tla-example
│ └── main.jsonnet
└── multi-tla
 └── main.jsonnet

File snippets

Point --snippet at an individual Jsonnet file. The file path becomes the snippet name:

SourceFetchFailed

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=SourceFetchFailed. The Message describes what went wrong (HTTP error, digest mismatch, tarball too large, etc.).

Cause

The Fetcher resolved the source CR and started downloading the artifact, but the download itself failed. Three subcategories:

HTTP failure — connection refused, 5xx from the source-controller endpoint, TLS handshake error
Digest mismatch — the bytes don’t hash to status.artifact.digest. Possible truncation or in-flight tampering
Tarball oversized — extracted bytes exceed MaxArchiveBytes (default 64 MiB)

Diagnosis

Check the source CR’s status.artifact.url is reachable from the operator pod:

SourceNotReady

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=SourceNotReady. The Message names the source CR (GitRepository/foo, ExternalArtifact/bar, etc.).

Cause

The Flux source CR the snippet references exists but its own status.conditions[Ready] is not yet True (or status.artifact is empty). The operator refuses to fetch from a source it can’t trust as ready.

For chained snippets specifically: the upstream snippet may have failed reconciliation, so its ExternalArtifact is stale or unpopulated.

Diagnosis

kubectl describe <kind> <source-name>
# Look for the Ready condition and any error messages.

For Flux sources, also check the source-controller logs:

SourceRefNotYetSupported

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=SourceRefNotYetSupported.

Cause

The snippet sets spec.sourceRef but the operator was built without a Fetcher wired in. This is a mis-deployment in practice — production binary always wires sources.New(). Seeing this in a real cluster means you’re running:

a test/dev binary
a custom build where defaultBuilder was modified
a future code path that hasn’t enabled sourceRef yet

Diagnosis

kubectl logs deploy/jaas | grep -i "fetcher"

If the operator logs no Fetcher initialization, the binary is incomplete.

Storage and high availability

Mon, 01 Jan 0001 00:00:00 +0000

In operator mode JaaS renders each JsonnetSnippet into a tar.gz artifact, stores it, and publishes an ExternalArtifact CR that points a Flux consumer at the tarball over HTTP. JaaS publishes artifacts through one of two storage backends — local filesystem or S3-compatible object storage — with optional leader election for multi-replica high availability and configurable revision retention.

Serving the tarballs

Regardless of backend, the operator runs an HTTP server that Flux consumers fetch artifacts from. Three flags govern it, and --storage-base-url and --storage-path are required whenever --enable-flux-integration is set:

Storage backend recovery

Mon, 01 Jan 0001 00:00:00 +0000

Not tied to a single Reason — this page covers what to do when the artifact store itself is degraded (PVC lost, S3 endpoint unavailable, the storage HTTP server is down). Downstream Flux consumers (kustomize-controller, helm-controller, grafana-operator) dereference ExternalArtifact.status.artifact.url to fetch tarballs; when that URL stops returning bytes, dependent resources stall.

Symptom

One or more of:

Downstream Flux consumers report 404 Not Found or connection refused against the JaaS storage URL.
kubectl get externalartifact --all-namespaces shows resources whose URL is unreachable from the consumer pods.
The operator pod is healthy (Ready=True on snippets), but the storage Service is unresponsive.
helm upgrade of the chart from persistence.enabled: false to true — or vice versa — caused a gap.

Triage: which backend are you running?

kubectl --namespace <jaas-ns> get deploy jaas \
 --output jsonpath='{.spec.template.spec.containers[0].args}' \
 | tr ',' '\n' | grep -E 'storage-backend|storage-path|s3-endpoint'

--storage-backend=local → filesystem behind --storage-path. Either an emptyDir (chart default) or a PVC.
--storage-backend=s3 → an external S3-compatible bucket; the storage HTTP server in-pod is a thin streaming proxy over minio-go.

Filesystem backend

PVC lost or replaced

Symptom: every ExternalArtifact URL returns 404 even though the snippet’s Ready=True. The Publisher writes idempotently on every reconcile, so making the operator re-render every snippet is the fix:

Suspended

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=Suspended. The snippet’s spec.suspend is set to true.

Cause

An operator (or automation) paused reconciliation for this snippet, typically to investigate a downstream issue without the artifact being rewritten underneath them. The previously-published ExternalArtifact and the on-disk tarball are left intact — downstream Flux consumers continue serving the last successful render.

This is a normal, intentional state. It is not a failure.

Diagnosis

kubectl get jsonnetsnippet <name> --output jsonpath='{.spec.suspend}'

If the value is true, the suspension is set on the spec. Check kubectl describe for the last condition transition timestamp to see when it happened.

Synced

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

kubectl get jsonnetsnippet shows READY=True with REASON=Synced. This is the healthy state — no action required.

Cause

The most recent reconcile pass completed end-to-end: source resolved, libraries resolved, eval succeeded, tarball published, ExternalArtifact upserted.

Diagnosis

To inspect the published artifact:

kubectl get externalartifact <snippet-name> --output yaml

The status.artifact.url points at the operator’s storage HTTP server. Curl it from a pod in the cluster to confirm the bytes match:

kubectl run --rm --stdin --tty --restart=Never tmp --image=docker.io/library/curlimages/curl -- \
 curl -sL <status.artifact.url> | tar tzv

Remediation

None — this is the healthy state.

Tenancy and RBAC

Mon, 01 Jan 0001 00:00:00 +0000

In operator mode the JaaS operator never acts with its own broad privileges when touching tenant resources. Every reconcile of a JsonnetSnippet runs against the RBAC of a tenant ServiceAccount, so a snippet can only reach what its own ServiceAccount is allowed to reach.

Per-snippet impersonation

Each JsonnetSnippet carries a spec.serviceAccountName. On every reconcile the operator mints a short-lived Bearer token for that ServiceAccount through the Kubernetes TokenRequest API (serviceaccounts/token: create) and performs all tenant-side API calls — reading JsonnetLibrary objects, fetching Flux source artifacts, and writing the published ExternalArtifact — as that ServiceAccount. The operator does not use the impersonate verb; it uses a real token, so the apiserver evaluates the tenant’s own RBAC.

Tracing

Mon, 01 Jan 0001 00:00:00 +0000

The JaaS operator exports OpenTelemetry traces over OTLP gRPC. With an endpoint configured, each reconcile and the work it fans out into — source fetch, library resolution, evaluation, publish — becomes a span you can follow in a tracing backend. When no endpoint is set, the OpenTelemetry SDK runs in no-op mode and emits nothing, so tracing carries no cost until you opt in.

The binary

Three flags configure the exporter:

Watch-layer silent failure

Mon, 01 Jan 0001 00:00:00 +0000

Not tied to a per-snippet Reason. This page covers the one RBAC-denial path the reconciler cannot surface itself: when the operator’s own ClusterRole is missing a verb on a watched resource kind, controller-runtime’s informer fails to start its watch, logs warnings, and retries silently. The reconciler never sees the failure — and no snippet’s status condition will tell you about it.

If a per-snippet runbook (rbacdenied.md, sourcefetchfailed.md, sourcenotready.md) doesn’t match the symptoms, this is where to look next.

Workqueue saturation

Mon, 01 Jan 0001 00:00:00 +0000

Linked from the JaaSControllerWorkqueueDepthHigh alert. Fires when the reconciler’s workqueue holds more items than the configured threshold (default 50) for the alert window. Not tied to a Reason constant — workqueue depth is a controller-runtime signal, not a per-snippet status.

Symptom

ALERTS{alertname="JaaSControllerWorkqueueDepthHigh", controller="jsonnetsnippet"}

New snippet writes settle slowly (status takes minutes to flip, not seconds).
Existing snippets re-render later than spec.interval would suggest.
kubectl describe jsonnetsnippet shows a stale ObservedGeneration.

Cause

The operator is dequeuing reconciles slower than the API server enqueues them. Common causes, in observed-frequency order: