JaaS ships an optional validating admission webhook for JsonnetSnippet. It is independent of, but layered on top of, operator mode .

Enabling it

Set --enable-webhook to boot the webhook server. It requires --enable-flux-integration (the webhook is wired only inside the operator boot path; enabling it alone is rejected as a flag error) and TLS material — tls.crt and tls.key — under --webhook-cert-dir (default /tmp/k8s-webhook-server/serving-certs). The server binds --webhook-port (default 9443).

What it validates

The webhook rejects a JsonnetSnippet whose spec.externalVariables declares a key that collides with an operator-level --ext-var. An operator-supplied external variable always wins, so a snippet that tries to redeclare one would render against a value it does not control; the webhook refuses the snippet at admission time instead.

The reconciler enforces the same invariant as a fallback, so a snippet that bypasses admission — for example when the webhook is unreachable under a failurePolicy: Ignore — still fails at reconcile rather than rendering with the wrong value.

Failure policy trade-off

The Helm chart defaults operator.webhook.failurePolicy: Fail. With Fail, a webhook outage blocks every JsonnetSnippet create and update cluster-wide until the operator is back. During a rolling update that window is typically under five seconds, because leader election releases the lease on context-cancel and the next replica takes over immediately.

If your CI or GitOps tooling cannot tolerate even that window, narrow or relax the webhook:

• operator.webhook.objectSelector — match only snippets carrying a label, e.g. require jaas.metio.wtf/managed: "true".
• operator.webhook.namespaceSelector — opt in per namespace.
• failurePolicy: Ignore — let create/update through when the webhook is unreachable, relying on the reconciler-side fallback to catch the colliding-key case.

TLS provisioning

--webhook-cert-mode selects how the serving certificate is provisioned.

cert-manager (default)

--webhook-cert-mode=cert-manager expects external tooling to provision tls.crt/tls.key. The Helm chart renders a cert-manager.io/v1 Certificate and mounts the issued Secret into the pod at --webhook-cert-dir. cert-manager handles renewal; the webhook server hot-reloads TLS when the mounted files change.

self-signed

--webhook-cert-mode=self-signed makes the operator generate a CA and serving certificate in-pod, write them to --webhook-cert-dir, and patch the named ValidatingWebhookConfiguration’s caBundle so the apiserver trusts the chain. A background renewer regenerates and re-writes the files before expiry, and the webhook server hot-reloads without a restart. The relevant flags:

In self-signed mode the operator needs get/update on the named ValidatingWebhookConfiguration. Multiple replicas bootstrapping at once during a rolling update converge on a combined caBundle rather than clobbering each other, so each replica’s CA stays trusted across the rollout.

The full flag list with defaults is on the configuration page .

JaaS turns a sustained problem into a notification two ways: a Prometheus PrometheusRule that pages on its metrics , and Kubernetes Events that Flux’s notification-controller can route to chat or e-mail.

The binary

The operator emits a standard Kubernetes Event on every Ready-condition transition — Normal for Synced, Warning for every other reason. The reason string fills both the event reason and action. These Events need no flag to enable; they are written whenever the operator reconciles.

The operator also threads runbook links into its own status automatically: every actionable Ready-condition Message gains a (runbook: https://jaas.projects.metio.wtf/runbooks/<reason>/) suffix, so kubectl describe jsonnetsnippet points straight at the matching page. Healthy or intentional states (Synced, Suspended, Pending) get no suffix.

Routing Events through Flux

Routing the Events is Flux’s notification-controller: target an Alert CR at kind: JsonnetSnippet and JaaS needs no Provider/Alert plumbing of its own.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: jaas-snippets
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: warn   # 'info' to include success events
  eventSources:
    - kind: JsonnetSnippet
      name: '*'

Wire whatever Provider you already use for Flux source CRs; see the Flux notification-controller documentation for provider configuration.

The alert catalog

The chart ships a starter alert set on the custom metrics plus a handful of controller-runtime signals. Each alert carries its remediation page as a runbook_url annotation so Alertmanager renders a direct link:

JaaSSnippetReconcileErrorsHigh templates its runbook URL on the failing reason, so it lands on the matching per-reason page under /runbooks/ . Each Ready-condition reason and each alert maps to a remediation page there.

The Helm chart

The PrometheusRule is opt-in under operator.metrics.prometheusRule and needs the Prometheus Operator’s monitoring.coreos.com/v1 API in the cluster:

operator:
  enabled: true
  metrics:
    enabled: true
    prometheusRule:
      enabled: true
      interval: 30s
      # Labels your Prometheus instance selects PrometheusRules on.
      labels:
        release: kube-prometheus
      # Merged onto every rendered alert — route all jaas alerts
      # through one Alertmanager receiver.
      extraAlertLabels:
        team: platform
      # Annotation key the runbook URL lands under (Prometheus-operator
      # convention is runbook_url).
      runbookAnnotationKey: runbook_url

Every threshold is a knob under operator.metrics.prometheusRule.thresholds, so the noise floor is tunable without copy-pasting rule bodies. To silence a built-in alert, raise its threshold to an impossibly high value — there is no per-alert disable toggle, and the threshold pattern keeps “this alert is intentionally inert” visible in the chart values. Cluster-specific rules append under a separate group via operator.metrics.prometheusRule.extraRules.

Symptom

READY=False, REASON=ArtifactTooLarge. The Message states the rendered byte count and the configured cap.

Cause

The snippet’s rendered output exceeds the operator’s --max-artifact-bytes (Helm: operator.storage.maxArtifactBytes). The cap is a defense-in-depth control — one runaway snippet shouldn’t fill a shared storage volume.

Common triggers:

• a snippet generating massive arrays via std.range(n) with a much larger n than intended
• accidentally inlining a large data fixture via importstr
• forgetting to project / filter when fanning out per-tenant configs

Diagnosis

Check the rendered size locally:

jsonnet /tmp/snippet/<entry-file> | wc -c

Remediation

Two paths:

1. Shrink the output. Inspect the snippet for unintended fan-out; project only the fields downstream consumers actually need.
2. Raise the cap. --max-artifact-bytes=10485760 (10 MiB) gives more headroom. Pair with PVC sizing in the chart so the volume can hold N rev’s worth of the new max.

If many snippets are bumping against the cap, the cap itself may be too low for the workload — review the cluster-wide ratio of total storage to per-snippet rev count.

The host needs no Go toolchain. All build and test commands run inside a containerized dev shell defined by dev/Containerfile and driven by ilo. The .ilo.rc at the repo root supplies the shell arguments, so the short form is always available:

ilo bash -c '<command>'

The dev shell pre-installs the Go toolchain, all static-analysis tools, controller-gen, and the envtest asset bundle. The Go module cache is mounted from ${XDG_CACHE_HOME}/go so repeated builds do not re-download dependencies.

Building

ilo bash -c 'go build -o jaas .'

The Dockerfile builds the production image. It accepts VERSION and COMMIT build args:

docker build -t ghcr.io/metio/jaas:dev .
docker build --build-arg VERSION=v1 --build-arg COMMIT=abc123 -t ghcr.io/metio/jaas:dev .

Regenerating generated code

CRD manifests and the zz_generated.deepcopy.go file are generated by controller-gen. Run this after touching api/v1/ types:

# Regenerate deep-copy functions
ilo bash -c 'controller-gen object paths=./api/v1/...'

# Regenerate CRD manifests under config/crd/bases/
ilo bash -c 'controller-gen crd paths=./api/v1/... output:crd:dir=./config/crd/bases'

Static analysis

golangci-lint is not used. The standalone tools below run directly, both in CI and locally inside the dev shell:

ilo bash -c 'go vet ./...'
ilo bash -c 'staticcheck ./...'       # config: staticcheck.conf, checks = ["all"]
ilo bash -c 'gofumpt -l .'           # empty output means clean; any output is a failure
ilo bash -c 'gosec ./...'            # inline #nosec justifications silence false positives
ilo bash -c 'govulncheck ./...'      # reachable-from-code advisories only
ilo bash -c 'arch-go'                # architecture rules; config: arch-go.yml

Test layers

Pure unit tests

Table-driven tests with no external state. They live next to the code they cover across internal/... and api/v1/. Several act as drift gates: conditions_test.go verifies that every Reason* constant has a matching docs/runbooks/<reason>.md, and TestErrorResponse_StableCodeValues pins the wire-level ErrCode* strings against accidental rename.

ilo bash -c 'go test -count=1 -race -cover ./...'

To run a single test by name:

ilo bash -c 'go test -count=1 -v -run TestName ./internal/handler/'

Envtest-backed operator tests

Files named envtest_*_test.go (in internal/operator/, internal/webhook/selfsigned/, and main_envtest_test.go at the repo root) boot a real kube-apiserver and etcd via controller-runtime’s envtest package and run the reconciler, webhook, and full run(...) function against them.

The tests share one apiserver instance per test binary, guarded by a sync.Once, so the startup cost is paid once. Each test t.Skips when KUBEBUILDER_ASSETS is unset — there is no build tag. The dev shell pre-stages the envtest asset bundle via setup-envtest (pinned ENVTEST_K8S_VERSION) and exports KUBEBUILDER_ASSETS, so these tests run by default inside ilo bash. On a host without the bundle they silently skip.

The envtest harness sets Config.SkipImpersonation (the only place that setting is allowed) and defaults MetricsBindAddress to "0" so parallel test cases do not fight over the metrics port.

ilo bash -c 'go test -count=1 -race -cover ./...'

Golden / example end-to-end tests

examples_test.go boots the full binary via runInBackground and asserts HTTP responses against golden files under testdata/golden/. Comparison is semantic — both sides are parsed as JSON and compared on the parsed values, so whitespace and key ordering are irrelevant.

After changing an example or adding a new one, regenerate the golden files:

ilo bash -c 'go test -update ./...'

Inspect and commit the diff in testdata/golden/.

Fuzz tests

Fuzz targets in internal/handler/, internal/sources/, and internal/urlguard/ harden the request path, the tar/gzip artifact unpacker, and the SSRF URL/IP parser against adversarial input. CI exercises their seed corpus as ordinary unit tests. To fuzz interactively:

ilo bash -c 'go test -fuzz=FuzzName -fuzztime=30s ./internal/urlguard/'

Benchmarks

Throughput benchmarks in internal/eval/, internal/storage/, and internal/operator/ cover reconcile throughput, watch mapping, and the tenant-client cache. They are baselines, not merge gates. The reconcile benchmark is envtest-backed and skips without KUBEBUILDER_ASSETS.

ilo bash -c 'go test -bench=. -benchmem -run=^$ ./internal/operator/'

Kind operator smoke tests

The cluster-level layer runs outside go test. Pure-kubectl bash scenarios in hack/smoke/ run against a real kind cluster via .github/workflows/kind-smoke.yml. To run a scenario locally against any reachable cluster, deploy JaaS and invoke the scenario scripts directly:

hack/smoke/scenario-inline-files.sh

See CI and releases for how the smoke layer fits into the two-angle end-to-end strategy.

Static analysis

golangci-lint is not used. The tools below run directly, both in CI and locally inside the dev shell (see Building and Testing ). Every tool is a separate, auditable binary with its own config file.

Architecture rules

arch-go.yml pins two invariants enforced with 100% compliance:

• api/v1 depends on neither the operator internals nor sigs.k8s.io/controller-runtime. The CRD types stay importable by external consumers without dragging the manager in. Scheme registration uses apimachinery’s runtime.NewSchemeBuilder for exactly this reason.
• internal/urlguard — the SSRF-defence layer — depends on the standard library only, with no internal and no external imports. This keeps the IP/URL validation logic self-contained and straightforward to fuzz in isolation.

The verify.yml PR gate

.github/workflows/verify.yml fans out into one job per concern. A failure points straight at the offending gate. CI installs each tool fresh via go run <tool>@latest; the dev shell pre-installs the same tools at the same versions, so local and CI runs agree.

All-green aggregate

The workflow ends with a single all-green job:

• needs every other job
• runs if: always()
• fails unless each dependency result is success or skipped

That one job is the only check marked required in branch protection. Adding a new job to the needs list of all-green covers it automatically; no new required check needs to be registered.

The govulncheck gate is a hard blocker. A reachable-from-code advisory that cannot be fixed by bumping a dependency blocks the PR until resolved. Resolution is usually a toolchain bump in go.mod (for stdlib advisories) or go get -u (for module advisories).

The release pipeline

Releases are calendar-based and automated. .github/workflows/release.yml runs on a Monday cron (47 7 * * MON) plus manual workflow_dispatch. The version is computed from the run date:

date +'%Y.%-m.%-d'

For a Monday run on 2026-06-22 that produces 2026.6.22.

goreleaser is not used. GPG is not used. The pipeline is hand-rolled across three jobs.

prepare

Counts commits since the last release touching the build-relevant paths (go.mod main.go internal api config Dockerfile). Every downstream job gates on that count being non-zero (or there being no prior release at all), so an empty week publishes nothing.

build

A cross-compile matrix over ten platform/arch combinations:

• linux/amd64, linux/arm (v7), linux/arm64, linux/ppc64le, linux/riscv64, linux/s390x
• windows/amd64, windows/arm64
• darwin/amd64, darwin/arm64

Each platform compiles with:

CGO_ENABLED=0 go build -trimpath \
  -ldflags="-s -w -X main.version=<ver> -X main.commit=<sha>" .

Archives are tar.gz on linux/darwin and zip on windows (with a .exe binary), each bundling LICENSE and README.md.

container

A single docker buildx multi-arch push to ghcr.io/metio/jaas:{latest,<version>} over the six linux arches. The Dockerfile builder is pinned to $BUILDPLATFORM and cross-compiles via Go’s GOARCH, so the multi-arch build needs no QEMU.

SBOM and provenance are attached. The image is signed with cosign keyless immediately after push:

cosign sign \
  --yes \
  --annotations "repo=metio/jaas" \
  --annotations "workflow=Automated Release" \
  ghcr.io/metio/jaas@<digest>

Identity is proven by the workflow’s OIDC certificate issued by Fulcio; there is no key to distribute.

github

Gates on both build and container succeeding. Downloads all platform archives, computes a single SHA256SUMS over them, signs it with cosign keyless (Sigstore bundle format), and publishes the GitHub Release with all archives, the checksum file, and the bundle attached.

To verify a release download:

cosign verify-blob jaas_<version>_SHA256SUMS \
  --bundle jaas_<version>_SHA256SUMS.bundle \
  --certificate-identity-regexp '^https://github.com/metio/jaas/\.github/workflows/release\.yml@refs/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
sha256sum -c jaas_<version>_SHA256SUMS

To verify the container image:

cosign verify ghcr.io/metio/jaas:<version> \
  --certificate-identity-regexp '^https://github.com/metio/jaas/\.github/workflows/release\.yml@refs/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Every JaaS flag is listed here with its default and a one-line description. Run jaas --help to see the same list at runtime. The tables on this page are generated from the binary’s own flag definitions, so they never drift from the runtime contract.

The Helm chart exposes most flags under arguments.*; operator-specific flags are under operator.*. The full set of chart values is in the Helm chart values reference.

Jsonnet server

The Jsonnet server evaluates snippets and returns JSON. It binds on --listen-address:--port by default.

Management server

The management server exposes the three Kubernetes probe endpoints. It binds on --management-listen-address:--management-port.

Endpoints: GET /start (startup probe), GET /ready (readiness probe), GET /live (liveness probe). Startup and readiness return 503 with a {"status":"…"} JSON body when the server is not yet ready. Liveness is an unconditional 200.

Snippets and libraries

Flags for declaring the Jsonnet files the server serves.

Snippet name resolution uses Go’s os.OpenRoot, which rejects .. traversal and symlinks that escape the configured directory. This is security-critical; see Evaluation and security .

External variables

Environment variable alternative: set JAAS_EXT_VAR_<NAME>=<VALUE> to expose <NAME> as an external variable. The --ext-var flag overrides the env mechanism on key conflict. See External variables and TLAs for usage examples.

Evaluation limits

--evaluation-timeout fires the HTTP response but does not terminate the underlying go-jsonnet call — the evaluation continues consuming CPU until it finishes naturally. Size container resources accordingly and use --max-concurrent-evals to bound worst-case goroutine pile-up. See Evaluation and security for the full discussion.

Lifecycle

Operator (Flux integration)

The following flags are only active when --enable-flux-integration is set.

Environment variable: JAAS_WATCH_NAMESPACES — comma-separated namespace list. Superseded by --watch-namespaces when both are set.

Storage server (local and S3)

The storage server is the HTTP file server that downstream Flux consumers fetch artifacts from. It is started only when --enable-flux-integration is set.

S3 flags

Active only when --storage-backend=s3.

Webhook (TLS provisioning)

Active only when --enable-webhook is set (which also requires --enable-flux-integration).

See Admission webhook for the full failurePolicy trade-off and cert rotation details.

Leader election

Observability

Metrics

Tracing

Logging and lifecycle

Fires when jaas_crd_watch_engagement_failures_total{gvk=...} has increased above the per-hour threshold for the alert window. JaaS lazy-watches Flux source CRDs: at boot, only the CRDs already installed get a watch; when a previously-missing CRD becomes Established=True (operator installed source-controller post hoc, say), the crdWatcher engages a runtime watch on it via Controller.Watch. When that engagement fails, the apiextensions informer fires no further events on a stable CRD — meaning the watch stays un-engaged forever until either the CRD object’s metadata/status is changed by something else, or the operator restarts.

The visible symptom is that snippets with spec.sourceRef.Kind=<the affected kind> stop re-rendering on upstream source updates. There is no per-snippet status signal — they sit at their last-rendered revision, drifting from upstream.

Symptom

• JaaSCRDWatchEngagementFailing alert is firing with gvk labelling the affected kind.
• kubectl describe jsonnetsnippet on snippets referencing that GVK shows a Ready condition that hasn’t moved in hours/days.
• Upstream Flux source CRs (GitRepository, OCIRepository, Bucket, ExternalArtifact) show recent status.artifact.revision changes that the jaas snippets aren’t picking up.

Diagnosis

Step 1 — confirm the CRD is actually installed and Established

kubectl get crd <plural>.source.toolkit.fluxcd.io \
  --output jsonpath='{.status.conditions[?(@.type=="Established")].status}{"\n"}'

Expect True. If the CRD is not installed or not yet Established, the watcher is correct to skip; install / wait.

Step 2 — check the operator’s RBAC on the source kind

kubectl auth can-i list <plural>.source.toolkit.fluxcd.io \
  --as=system:serviceaccount:<ns>:<operator-sa>
kubectl auth can-i watch <plural>.source.toolkit.fluxcd.io \
  --as=system:serviceaccount:<ns>:<operator-sa>

If either is “no”, the chart’s operator-tenants ClusterRole (or per-namespace RoleBinding when watchNamespaces is set) is missing the get/list/watch verbs on this kind. Update the chart’s FluxSourceKinds mapping or add the verb manually.

Step 3 — check controller-runtime cache state

kubectl --namespace <ns> logs <operator-pod> | grep -E 'engage|Failed to watch|cache' | tail -20

Look for cache reconnect, informer failed, or Watch failed: forbidden. A transient cache reconnect during a heavy load period can trip engagement once; the DD7 bounded-retry mechanism re-engages automatically. Sustained failures point at RBAC or a misconfigured MetricsBindAddress.

Remediation

1. Fix the verb / kind / RBAC issue identified above.

2. Roll the operator pod to force a fresh SetupWithManager pass, which re-detects every Flux CRD and re-engages watches that succeed on first try:

   kubectl --namespace <ns> rollout restart deployment <operator-deployment>
   

3. Verify the counter stops increasing and the alert clears.

When the alert is noisy

If jaas_crd_watch_engagement_failures_total ticks once at boot but never again, that’s the expected DD7 bounded-retry behavior: the first attempt failed (transient race during cache start), the retry succeeded. Raise crdWatchEngagementFailuresPerHour if the boot-time blip is noisy enough to page.

A JsonnetSnippet’s spec.sourceRef consumes a Flux source. The operator reads the referenced source CR’s status.artifact.url, downloads the tarball Flux’s source-controller serves there, verifies its status.artifact.digest, and extracts it into the snippet’s file tree. Every supported kind — GitRepository, OCIRepository, and Bucket — reaches the operator through that same status.artifact contract, so the operator never talks to a git remote, an OCI registry, or an object store directly. Flux owns that fetch; the operator consumes the artifact Flux already produced.

The recipes below show how to produce each source kind so a snippet can reference it. Snippet sources covers wiring the finished source into a JsonnetSnippet. For the source CRDs themselves and their full field reference, see the Flux documentation — only what a JaaS source needs is covered here.

A JsonnetSnippet references the source you create with a spec.sourceRef:

apiVersion: jaas.metio.wtf/v1
kind: JsonnetSnippet
metadata:
  name: dashboards
  namespace: default
spec:
  serviceAccountName: dashboards-tenant
  entryFile: dashboards/api-latency.jsonnet
  sourceRef:
    kind: GitRepository      # or OCIRepository, or Bucket
    name: dashboards-source
    path: dashboards/        # optional: narrow extraction to a subtree

The tenant ServiceAccount needs get on the referenced source kind. See Tenancy and RBAC for the exact verbs.

GitRepository

A GitRepository source tracks a branch, tag, or commit of a git repository. source-controller clones the ref and packs the tree into the tarball the operator fetches. There is no packaging or layer constraint — the operator extracts whatever files the commit contains.

1. Lay out your Jsonnet files in a directory. File names and the directory structure carry over verbatim into the snippet’s file tree, so place the entry file where spec.entryFile expects it:

   dashboards/
   ├── api-latency.jsonnet
   ├── error-budget.jsonnet
   └── lib/
       └── panels.libsonnet
   

2. Commit the files and push them to a git repository:

   git add dashboards/
   git commit -m "Add Grafana dashboards"
   git push origin main
   

3. Create the GitRepository source. With the Flux CLI:

   flux create source git dashboards-source \
     --url=https://github.com/example-org/grafana-dashboards \
     --branch=main \
     --interval=5m \
     --namespace=default \
     --export
   

   The equivalent CR YAML, which is authoritative:

   apiVersion: source.toolkit.fluxcd.io/v1
   kind: GitRepository
   metadata:
     name: dashboards-source
     namespace: default
   spec:
     interval: 5m
     url: https://github.com/example-org/grafana-dashboards
     ref:
       branch: main
   

4. Point a snippet’s spec.sourceRef at the source. Set kind: GitRepository, name: dashboards-source, and optionally path: to extract only a subtree:

   apiVersion: jaas.metio.wtf/v1
   kind: JsonnetSnippet
   metadata:
     name: api-latency-dashboard
     namespace: default
   spec:
     serviceAccountName: dashboards-tenant
     entryFile: dashboards/api-latency.jsonnet
     sourceRef:
       kind: GitRepository
       name: dashboards-source
       path: dashboards/
   

When a new commit lands on the tracked branch, source-controller republishes the artifact and the operator’s watch re-renders the snippet.

OCIRepository

An OCIRepository source pulls an OCI artifact from a registry. source-controller unpacks the artifact’s single gzipped-tar layer into the tarball the operator fetches. Producing the artifact with flux push artifact packs a directory into exactly that shape.

1. Lay out your Jsonnet files in a directory, the same as for a git source:

   ./
   ├── main.jsonnet
   └── lib/
       └── panels.libsonnet
   

2. Push the directory as an OCI artifact with the Flux CLI. flux push artifact packs the directory into one gzipped-tar layer and pushes it to the registry:

   flux push artifact oci://ghcr.io/example-org/dashboards:v1 \
     --path=. \
     --source="$(git config --get remote.origin.url)" \
     --revision="$(git rev-parse HEAD)"
   

   --source and --revision stamp provenance metadata onto the artifact; set them to a URL and a version identifier of your choosing.

3. Create the OCIRepository source. With the Flux CLI:

   flux create source oci dashboards-source \
     --url=oci://ghcr.io/example-org/dashboards \
     --tag=v1 \
     --interval=5m \
     --namespace=default \
     --export
   

   The equivalent CR YAML, which is authoritative:

   apiVersion: source.toolkit.fluxcd.io/v1
   kind: OCIRepository
   metadata:
     name: dashboards-source
     namespace: default
   spec:
     interval: 5m
     url: oci://ghcr.io/example-org/dashboards
     ref:
       tag: v1
   

4. Point a snippet’s spec.sourceRef at the source with kind: OCIRepository:

   apiVersion: jaas.metio.wtf/v1
   kind: JsonnetSnippet
   metadata:
     name: api-latency-dashboard
     namespace: default
   spec:
     serviceAccountName: dashboards-tenant
     entryFile: main.jsonnet
     sourceRef:
       kind: OCIRepository
       name: dashboards-source
   

Single layer is mandatory. flux push artifact produces an OCI artifact with exactly one gzipped-tar layer, which is what source-controller expects and the only shape it unpacks. An artifact built any other way — a hand-rolled oras push with one file per layer, a Dockerfile/container-image build, or any tool that splits content across multiple layers — is not consumed correctly. source-controller cannot reconstruct the file tree, the snippet’s source never resolves, and the snippet reports Ready=False. Always build OCI sources with flux push artifact.

Verify the layer count before relying on an artifact. Fetch the manifest and confirm the layers array has length 1:

oras manifest fetch oci://ghcr.io/example-org/dashboards:v1 | \
  jq '.layers | length'

A result of 1 is required. Any other number means the artifact was not built with flux push artifact and will not resolve.

Private registries and Amazon ECR

source-controller performs the pull, so registry credentials belong on the OCIRepository (or on source-controller itself) — never on the JaaS operator or a snippet’s ServiceAccount. The same applies to a JsonnetLibrary whose sourceRef points at an OCIRepository.

For a generic private registry, add a spec.secretRef to a docker-registry Secret. For Amazon ECR you need no pull Secret at all: set spec.provider: aws and source-controller authenticates with its own ambient AWS identity. On EKS that is an IRSA role bound to source-controller’s ServiceAccount with ECR read permissions:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: dashboards-source
  namespace: default
spec:
  interval: 5m
  provider: aws
  url: oci://111122223333.dkr.ecr.eu-west-1.amazonaws.com/dashboards
  ref:
    tag: v1

The IRSA role needs ecr:GetAuthorizationToken (resource *) plus ecr:BatchGetImage and ecr:GetDownloadUrlForLayer on the repository. Because the credential is source-controller’s, one role covers every OCIRepository it pulls, and the JaaS operator stays out of the registry path entirely.

This IRSA role is source-controller’s, not the JaaS operator’s. The JaaS operator uses IRSA only for its own S3 storage backend — a separate concern from pulling sources.

There is a third way to load OCI content that does not go through a sourceRef at all: the chart can mount snippets and libraries from OCI image volumes (snippets / additionalLibraries), read straight from a registry into the pod. Those volumes are pulled by the kubelet, exactly like a container image — so they authenticate the way images do, not through IRSA. On EKS that means the node’s IAM role with ECR read (the AmazonEC2ContainerRegistryReadOnly managed policy the default node role already carries), or an imagePullSecret on the pod. Pod-level IRSA grants the pod’s ServiceAccount AWS API access, which the kubelet does not use when pulling images, so it is not the mechanism for this path. With a node role that can read ECR, image-volume snippets and libraries load with no pull Secret. Static OCI mounts and operator mode are mutually exclusive in one release, so a given install uses either these mounts or the sourceRef path above, not both.

The jsonnet-oci-images (JOI) project enforces this same single-layer rule for every image it publishes, so its images are ready-made single-layer OCIRepository sources. Reference a JOI image directly when you need a shared Jsonnet library tree (grafonnet, the jsonnet-libs catalog) rather than building and maintaining your own OCI source.

Bucket

A Bucket source mirrors objects from an S3- or GCS-compatible bucket. source-controller fetches the matching objects, packs them into the tarball the operator fetches, and there is no layer constraint — the only requirement is that the objects laid out under the bucket prefix form the file tree your snippet expects.

1. Produce the files to upload. Either upload the individual .jsonnet / .libsonnet files under a prefix, or pack them into a single archive — both work, source-controller flattens the mirrored objects into the file tree:

   dashboards/
   ├── main.jsonnet
   └── lib/
       └── panels.libsonnet
   

2. Upload the files to the bucket under a prefix. With the AWS CLI against an S3-compatible endpoint:

   aws s3 cp dashboards/ s3://example-bucket/dashboards/ \
     --recursive \
     --endpoint-url=https://s3.example.com
   

3. Create the Bucket source. With the Flux CLI:

   flux create source bucket dashboards-source \
     --bucket-name=example-bucket \
     --endpoint=s3.example.com \
     --provider=generic \
     --secret-ref=bucket-credentials \
     --interval=5m \
     --namespace=default \
     --export
   

   The equivalent CR YAML, which is authoritative:

   apiVersion: source.toolkit.fluxcd.io/v1
   kind: Bucket
   metadata:
     name: dashboards-source
     namespace: default
   spec:
     interval: 5m
     provider: generic
     bucketName: example-bucket
     endpoint: s3.example.com
     secretRef:
       name: bucket-credentials
   

   The referenced Secret carries the bucket credentials (accesskey / secretkey). See the Flux documentation for the Secret layout and provider-specific fields.

4. Point a snippet’s spec.sourceRef at the source with kind: Bucket. Use path: to extract only the prefix that holds your Jsonnet:

   apiVersion: jaas.metio.wtf/v1
   kind: JsonnetSnippet
   metadata:
     name: api-latency-dashboard
     namespace: default
   spec:
     serviceAccountName: dashboards-tenant
     entryFile: main.jsonnet
     sourceRef:
       kind: Bucket
       name: dashboards-source
       path: dashboards/
   

Which source should I use?

Symptom

READY=False, REASON=CrossNamespaceRefRejected. The Message names the offending reference (a library or a sourceRef).

Cause

The operator is running with --no-cross-namespace-refs=true (the chart default) and the snippet references a library or Flux source in a different namespace.

This is a deliberate isolation control — it mirrors Flux’s --no-cross-namespace-refs and stops a tenant in namespace A from reaching libraries / sources in namespace B without an explicit relationship.

Diagnosis

Inspect the spec and identify which reference points outside the snippet’s namespace:

kubectl --namespace <ns> get jsonnetsnippet <name> --output yaml | grep -E "namespace:|sourceRef:|libraries:"

Remediation

Three options, by isolation strength:

1. (recommended) Duplicate the library / source CR into the snippet’s namespace.
2. Promote the library to an OCI volume — mount via the chart’s additionalLibraries map. Becomes part of the operator’s filesystem, available to every snippet without a cross-namespace CR ref.
3. (loosen isolation, cluster-wide) Set --no-cross-namespace-refs=false on the operator. Affects every tenant in the cluster — only do this when tenants are mutually trusting.

Symptom

READY=False, REASON=DependencyCycle. The Message names the snippet that closes the cycle.

Cause

The snippet’s spec.sourceRef chain transitively points back at the snippet itself. The reconciler detects this and refuses to publish so chained snippets don’t loop forever (each republish would trigger every downstream snippet to re-render, which would re-trigger the upstream, and so on).

Two cycle shapes:

1. Direct sourceRef cycle: A.spec.sourceRef → ExternalArtifact/A. A snippet sourcing from its own published artifact.
2. Library-mediated cycle: A.spec.libraries → JsonnetLibrary/L, where L.spec.sourceRef → ExternalArtifact/A (or a longer chain back to A).

The validating webhook (--enable-webhook) rejects new CRs that introduce a cycle at admission; the reconciler check is a fallback for when admission is bypassed or the cycle is introduced retroactively (e.g., adding a new library that closes a loop with existing snippets).

Diagnosis

Walk the chain manually:

kubectl get jsonnetsnippet <name> --output jsonpath='{.spec.sourceRef}' && echo
# Then inspect what that sourceRef points at, and what it sources from in turn.

For library-mediated cycles, the chain is:

snippet A.spec.libraries[i] → JsonnetLibrary L → L.spec.sourceRef → ExternalArtifact X → snippet that publishes X

If the publishing snippet at the end is A, you have a cycle.

Remediation

Break the cycle by removing the back-edge. Common fixes:

• detach a library from its sourceRef (inline its files instead, if small)
• have the upstream snippet publish a smaller artifact the downstream doesn’t need to re-consume
• restructure so the shared data lives in a static ConfigMap referenced as a sourceRef-equivalent, not in a snippet output

JaaS pairs with stageset-controller to deploy Kubernetes manifests as code: you author the manifests in Jsonnet, the JaaS operator renders and publishes them as a Flux ExternalArtifact, and stageset-controller rolls that artifact out across ordered, gated stages.

This tutorial covers the JaaS side — authoring the manifests with top-level arguments and external variables, and publishing the rendered JSON. The rollout side (the StageSet resource, its stages, gates, and actions) lives on the stageset-controller site and is linked at the end.

Prerequisites

• The JaaS operator installed and a tenant ServiceAccount granted the externalartifacts write verbs. The Quickstart covers both.
• stageset-controller installed, if you intend to follow the handoff section and roll the manifests out.

This tutorial uses the namespace default and the tenant ServiceAccount manifests-tenant.

Step 1 — Grant the tenant ServiceAccount its verbs

The snippet publishes an ExternalArtifact, so the tenant needs the externalartifacts write verbs:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: manifests-tenant
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: manifests-tenant
rules:
  - apiGroups: [source.toolkit.fluxcd.io]
    resources: [externalartifacts]
    verbs: [get, create, update, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default
  name: manifests-tenant
subjects:
  - kind: ServiceAccount
    name: manifests-tenant
    namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: manifests-tenant
EOF

Verify the ServiceAccount and binding:

kubectl --namespace default get serviceaccount manifests-tenant
kubectl --namespace default get rolebinding manifests-tenant

Step 2 — Author and apply the manifest snippet

The snippet renders a List of a Deployment and a Service from top-level arguments (spec.tlas) and an external variable (spec.externalVariables). A single-value TLA arrives as a string; the snippet parses the replica count with std.parseInt. spec.output stays at its default rendered so the artifact carries the evaluated manifest JSON:

cat <<EOF | kubectl apply -f -
apiVersion: jaas.metio.wtf/v1
kind: JsonnetSnippet
metadata:
  name: web-app
  namespace: default
spec:
  serviceAccountName: manifests-tenant
  output: rendered
  files:
    main.jsonnet: |
      function(name='web', replicas='2')
        local image = std.extVar('image');
        local labels = { 'app.kubernetes.io/name': name };
        {
          apiVersion: 'v1',
          kind: 'List',
          items: [
            {
              apiVersion: 'apps/v1',
              kind: 'Deployment',
              metadata: { name: name, labels: labels },
              spec: {
                replicas: std.parseInt(replicas),
                selector: { matchLabels: labels },
                template: {
                  metadata: { labels: labels },
                  spec: {
                    containers: [{
                      name: name,
                      image: image,
                      ports: [{ containerPort: 8080 }],
                    }],
                  },
                },
              },
            },
            {
              apiVersion: 'v1',
              kind: 'Service',
              metadata: { name: name, labels: labels },
              spec: {
                selector: labels,
                ports: [{ port: 80, targetPort: 8080 }],
              },
            },
          ],
        }
  tlas:
    name: [web]
    replicas: ["3"]
  externalVariables:
    image: "ghcr.io/example/web:1.4.0"
EOF

Each spec.tlas value is a list, matching the HTTP query-parameter convention: a single element becomes a string TLA, multiple elements a JSON-encoded array. External variables seed std.extVar lookups.

Step 3 — Confirm the manifests rendered

kubectl --namespace default get jsonnetsnippet web-app
# NAME      READY   URL                                                                                     AGE
# web-app   True    http://jaas-storage.jaas-system.svc.cluster.local:8082/default/web-app/<sha256>.tar.gz  5s

If READY is False, describe the snippet — the Ready condition’s Reason and Message name the cause:

kubectl --namespace default describe jsonnetsnippet web-app

Step 4 — Inspect the published manifests

Fetch the artifact from a one-shot pod to see the rendered manifests:

URL=$(kubectl --namespace default get jsonnetsnippet web-app -o jsonpath='{.status.artifactURL}')
kubectl run --rm -i --restart=Never --image=docker.io/curlimages/curl:8.10.1 fetch -- \
    sh -c "curl -fsSL '$URL' | tar -xzO rendered.json"
# {
#    "apiVersion": "v1",
#    "kind": "List",
#    "items": [ { "kind": "Deployment", ... }, { "kind": "Service", ... } ]
# }

rendered.json is the manifest set a Flux consumer applies to the cluster.

Handoff: roll the manifests out with StageSet

The published ExternalArtifact is now ready for a Flux consumer. A consumer references it in one of two ways:

• Directly — name the ExternalArtifact (which shares the snippet’s name and namespace) in a sourceRef.
• Producer-aware — name the producing JsonnetSnippet and let the consumer resolve it to the ExternalArtifact. JaaS writes a three-field back-pointer (apiVersion, kind, name) under the artifact’s spec.sourceRef for this, which is the contract producer-aware resolvers match on.

stageset-controller consumes the published artifact the producer-aware way and rolls it out across ordered, gated stages. The StageSet resource, its stages, gates, and actions live on the stageset-controller documentation:

• stageset-controller producer-aware sources guide: https://stageset.projects.metio.wtf/usage/producer-aware-sources/
• stageset-controller project: https://stageset.projects.metio.wtf/

Follow that guide for the rollout side; it picks up exactly where this tutorial leaves off — at the published ExternalArtifact.

Where to go next

• Operator mode — the full operator reference, including the ExternalArtifact spec.sourceRef back-pointer contract that producer-aware consumers match on.
• Snippet sources — back the manifests with a GitRepository or OCIRepository instead of inline spec.files.

Not tied to a single Reason — this page covers what to do when the global concurrent-eval cap (--max-concurrent-evals) is full and JaaS is shedding new evaluations. The cap exists because the synchronous go-jsonnet API has no context-aware cancellation: once an eval starts it runs to natural completion, so an unbounded queue lets a runaway snippet pile up goroutines that outlive every caller’s deadline.

Symptom

One or more of:

• JaaSEvalSaturation is firing — jaas_eval_in_flight / jaas_eval_max_concurrent has been above the threshold (default 0.9) for the alert window.
• JaaSEvalRejected is firing — rate(jaas_eval_unavailable_total[5m]) has been above the threshold.
• HTTP clients see 503 Service Unavailable with body {"error": "evaluation_unavailable", "message": "concurrent-eval cap is full; retry after backoff"}.
• kubectl describe jsonnetsnippet shows recurring Warning EvalUnavailable events with message reconcile deferred for 1s by --max-concurrent-evals. Ready condition stays untouched (backpressure is not failure).
• jaas_eval_outstanding_timed_out is also elevated — confirms the runaway-snippet diagnosis: orphaned evals are pinning slots while their parents have already given up.

Diagnosis: why is the cap full?

The cap fills for two distinct reasons. The right remediation depends on which.

Path A — runaway snippet (goroutines outliving their ctx)

Read the leak gauge. If it’s non-zero and trending up, evaluations are starting but not finishing — almost always a single snippet whose work dwarfs --evaluation-timeout.

# Live count of evals whose parent reconcile already timed out:
kubectl --namespace <jaas-ns> exec deploy/jaas -- \
  wget -qO- http://localhost:8083/metrics | grep jaas_eval_outstanding_timed_out

To find the culprit, scan recent reconcile logs for Jsonnet evaluation timed out followed by repeated EvalUnavailable warnings on the same snippet:

kubectl --namespace <jaas-ns> logs deploy/jaas --since=15m \
  | grep -E 'EvaluationTimeout|EvalUnavailable' \
  | sort | uniq -c | sort -rn | head

The snippet whose name dominates that list is the culprit. Common causes:

• Deep recursion that takes seconds-to-minutes to complete naturally even after the parent deadline fires.
• Pathological library import that triggers go-jsonnet’s worst-case eval order.
• A std.foldl over a generated array of millions of entries.

Path B — genuine load above the cap

Leak gauge is at zero (or steady, not growing), jaas_eval_in_flight is pegged near the cap, and many distinct snippets show EvalUnavailable events. The cap is sized too small for the workload.

# Distribution of which snippets are seeing rejections — a flat
# distribution across many snippets is path B; a single dominant
# snippet is path A.
kubectl --namespace <jaas-ns> exec deploy/jaas -- \
  wget -qO- http://localhost:8083/metrics \
  | grep jaas_snippet_eval_unavailable_total

Remediation

Path A — runaway snippet

1. Suspend the offender to stop new evals while you fix the snippet:

   kubectl --namespace <ns> patch jsonnetsnippet <name> --type merge \
     --patch '{"spec":{"suspend":true}}'
   

2. Inspect the snippet to understand the cost. Lower --max-stack is a blunt clamp that rejects pathological recursion before it can leak. The chart’s operator.maxStack defaults to 500; pull it down to ~200 if the snippet doesn’t legitimately need deeper recursion.

3. Tighten --evaluation-timeout if the snippet’s natural completion time is the load-bearing factor. A 5s default lets a 60s pathological eval leak for nearly a minute; dropping to 1s shrinks the worst-case leak window.

4. Re-enable after the snippet spec is fixed:

   kubectl --namespace <ns> patch jsonnetsnippet <name> --type merge \
     --patch '{"spec":{"suspend":false}}'
   

Path B — genuine load

1. Raise the cap if the operator has CPU headroom. The default is max(GOMAXPROCS*4, 16); double it via the chart:

   helm upgrade <release> <chart> --reuse-values \
     --set arguments.maxConcurrentEvals=64
   

   Each in-flight eval pins roughly one CPU when actively running, so the practical ceiling is bounded by node CPU. Past 2-3× GOMAXPROCS the gains drop sharply — more contention, same throughput.

2. Tune the per-snippet rate limiter if a small number of snippets dominate the request rate. --rerender-rate + --rerender-burst cap each snippet’s reconcile frequency independent of the global eval cap.

3. Scale horizontally if a single replica can’t keep up even at the raised cap. The chart’s replicas.max controls the HPA ceiling; combined with the storage layer’s leader election (S3 backend) you get multi-replica HA where every replica reads but only the lease-holder writes.

When NOT to raise the cap

If the leak gauge is non-zero AND growing, raising the cap lets more goroutines pile up before the next saturation event. Diagnose path A first. The cap is a backpressure boundary, not a throughput knob.

Disable the gate (not recommended)

--max-concurrent-evals=0 disables the gate entirely. The leak gauge keeps working, but rejections never fire — a single runaway snippet can OOM the pod. Use only if you’ve sized the workload precisely and want to surface saturation purely via the leak gauge.

JaaS runs Jsonnet on the server and returns the result over HTTP. Three caps bound each evaluation, and a small security model governs what a snippet and its callers can reach. Review and tune both sections before exposing the service to a wider audience.

Evaluation caps

./jaas \
  --snippet-directory examples/snippets/dashboards \
  --evaluation-timeout 2s \
  --max-stack 1000 \
  --max-concurrent-evals 32

The default for --max-concurrent-evals bounds worst-case goroutine pile-up under a runaway snippet. Each in-flight evaluation pins roughly one CPU for its working set, so raising the cap far above the available parallelism queues work without adding throughput.

Security model

Library paths are an unrestricted read scope. Any file reachable under a configured --library-path, or under a snippet’s own directory, can be import-ed or importstr-ed by any snippet — go-jsonnet’s importer does not sandbox per snippet. Scope these directories tightly. Never point them at /, /etc, or anywhere holding credentials.

Snippets are operator-controlled, not caller-controlled. Callers supply only top-level arguments through the query string. Jsonnet’s import and importstr require string-literal paths, so a TLA or external variable cannot construct an import path. Deploying a snippet authored by someone you do not trust is equivalent to running their code on the server.

Snippet name resolution is sandboxed. The URL’s snippet segment resolves through Go’s os.Root, which rejects .. traversal and symlinks that escape the configured snippet directory. A URL like /jsonnet/../etc/passwd returns 404, even though the OS would otherwise resolve the path.

Evaluation has caps but no mid-flight cancellation. --evaluation-timeout bounds wall-clock time and --max-stack bounds call-stack depth, but go-jsonnet cannot abort an evaluation already running. A slow snippet keeps consuming CPU until it finishes naturally or the timeout fires the HTTP response. Size container CPU and memory limits to absorb that worst case.

The Prometheus metrics jaas_eval_in_flight (gauge: live in-flight count), jaas_eval_unavailable_total (counter: cumulative cap rejections), and jaas_eval_outstanding_timed_out (gauge: evals still running after their request timed out) surface how close evaluation runs to these caps. See Observability for detail.

The HTTP status codes these caps produce are documented in the rendering endpoint error contract.

Symptom

READY=False, REASON=EvaluationFailed. The Message contains the raw go-jsonnet diagnostic — file name, line, column, and the underlying error.

Cause

The snippet failed to evaluate. Three broad categories:

• Syntax error — unclosed brace, missing comma, bad indent.
• Runtime error — std.extVar('missing') for an unset variable, division by zero, error '...' thrown explicitly.
• Import error — import 'missing.libsonnet' resolves to nothing in the snippet’s file map or library imports.

Diagnosis

Read the Message — it names the file and line. Reproduce locally:

# Pull the snippet's files into a tempdir, then evaluate.
kubectl get jsonnetsnippet <name> --output json | jq -r '.spec.files["main.jsonnet"]' > /tmp/main.jsonnet
jsonnet /tmp/main.jsonnet

For sourceRef-based snippets, fetch the tarball:

SOURCE_URL=$(kubectl get gitrepository <name> --output jsonpath='{.status.artifact.url}')
curl -sL "$SOURCE_URL" | tar -xz -C /tmp/snippet
jsonnet /tmp/snippet/<entry-file>

Remediation

Fix the snippet (or its libraries / source) and re-apply.

The diagnostic message can leak the on-disk path of the snippet — fine in-cluster, worth gating behind a flag if exposed to untrusted callers in the future.

Symptom

READY=False, REASON=EvaluationTimeout. The snippet’s eval ran longer than the operator’s --evaluation-timeout.

Cause

Snippets are evaluated synchronously per reconcile. The deadline is wall-clock, not CPU — but go-jsonnet has no mid-evaluation cancellation, so a snippet that runs over the deadline still keeps consuming CPU on the operator pod until it returns naturally.

Common triggers:

• a snippet recursing deeper than necessary (try lowering --max-stack to surface this as a stack-limit error instead, then optimize)
• a snippet that loads a huge sourceRef tarball and walks it
• a snippet that calls std.set / std.uniq over a very large array

Diagnosis

Time it locally:

time jsonnet /tmp/snippet/<entry-file>

If it takes seconds locally, the operator’s bound is too tight. If it takes minutes locally, the snippet itself is the problem.

Remediation

Two paths:

1. Optimize the snippet. Memoize repeated work into local bindings, narrow the input set, avoid std.flattenDeepArrays over deep trees.
2. Raise the operator’s bound. --evaluation-timeout=30s (default 5s) gives more headroom. Pair with resources.cpu headroom in the chart so the slow snippet doesn’t starve other reconciles.

For pathological inputs, consider splitting the snippet — render the slow part less often via a separate snippet others source from (see examples/operator/chained-snippets.yaml).

JaaS feeds two kinds of input into an evaluation: external variables, set by the process owner at startup, and top-level arguments, supplied per request through the URL query string.

External variables

External variables come from two sources. The environment mechanism reads every variable prefixed with JAAS_EXT_VAR_ — the suffix is the variable name:

JAAS_EXT_VAR_name=Alice \
JAAS_EXT_VAR_key=secret \
  ./jaas --snippet-directory examples/snippets/dashboards

The --ext-var KEY=VALUE flag does the same and is repeatable. On a key conflict, the flag takes precedence over the environment value:

./jaas \
  --snippet-directory examples/snippets/dashboards \
  --ext-var name=Alice \
  --ext-var key=secret

A snippet reads a variable with std.extVar:

{
  person1: {
    name: std.extVar('name'),
    external: std.extVar('key'),
  },
}

Fetching example1 with those variables set produces:

curl http://127.0.0.1:8080/jsonnet/example1

{
  "person1": {
    "external": "secret",
    "name": "Alice",
    "welcome": "Hello Alice!"
  }
}

External variables are fixed at startup. Callers cannot set them per request — that is what top-level arguments are for.

Top-level arguments

A snippet that evaluates to a function receives top-level arguments (TLAs) from the URL query string. The tla-example snippet is such a function:

function(something="value", other="more", required)
  {
    person1: {
      welcome: 'Hello ' + something + '!',
      key: other,
      required: std.parseJson(required),
    },
  }

Each query parameter sets a TLA. A single value becomes a string:

curl 'http://127.0.0.1:8080/jsonnet/tla-example?something=Ada&required=42'
# {"person1":{"key":"more","required":42,"welcome":"Hello Ada!"},...}

A repeated parameter becomes a list. The multi-tla snippet joins whatever it receives:

function(tags=["default"])
  {
    count: std.length(tags),
    list: tags,
    joined: std.join(", ", tags),
  }

curl 'http://127.0.0.1:8080/jsonnet/multi-tla?tags=blue&tags=green'
# {"count":2,"joined":"blue, green","list":["blue","green"]}

A bare parameter with no value sets the TLA to an empty string:

curl 'http://127.0.0.1:8080/jsonnet/tla-example?something&required=0'

For the request and response shape these examples ride on, see the rendering endpoint .

For every successfully evaluated JsonnetSnippet, the JaaS operator upserts a Flux ExternalArtifact CR (source.toolkit.fluxcd.io/v1) in the same namespace as the snippet. JaaS does not own the ExternalArtifact CRD — it is defined and installed by Flux’s source-controller. The full CRD schema is in the Flux ExternalArtifact reference ; below is the subset JaaS writes and the invariants downstream consumers can rely on.

For task-oriented context, see Operator mode .

What JaaS writes

spec.sourceRef

JaaS stamps a back-pointer to the originating snippet on spec.sourceRef:

spec:
  sourceRef:
    apiVersion: jaas.metio.wtf/v1
    kind: JsonnetSnippet
    name: <snippet-name>

The namespace is always the snippet’s own namespace — JaaS never publishes an ExternalArtifact to a different namespace. The three fields (apiVersion, kind, name) are wire-stable: downstream consumers that do producer-aware reverse lookup (such as stageset-controller’s RFC-0012 resolution) match on this triple. Renaming any field is a breaking change.

status.artifact

After a successful publish, JaaS writes the following fields under status.artifact:

status.conditions

JaaS writes a single Ready condition on every successful publish:

status:
  conditions:
    - type: Ready
      status: "True"
      reason: Succeeded
      message: artifact published
      lastTransitionTime: <RFC3339>
      observedGeneration: <generation>

lastTransitionTime is preserved across steady-state republishes (same Ready=True, new revision) so the timestamp does not churn. It advances only when the condition transitions (e.g. from False to True after a failure clears).

What downstream consumers rely on

Gate on Ready=True before fetching. Every Flux consumer — including kustomize-controller, helm-controller, and JaaS’s own chained-snippet sourceRef resolver — treats an ExternalArtifact as not-yet-consumable until status.conditions[Ready].status == "True". A snippet that has not yet completed its first successful reconcile will have no Ready condition (or Ready=False) and leaves chained snippets blocked with reason SourceNotReady.

URL is revision-addressed and byte-stable. The URL published in status.artifact.url has the form <storage-base-url>/<namespace>/<name>/<sha256-hex>.tar.gz. The bytes at that URL are immutable for as long as the revision is in the snippet’s keep-set (spec.history). Consumers can safely re-fetch a pinned URL (e.g. during rollback) and verify it against the recorded digest. Once a revision leaves the keep-set it is garbage-collected after the operator’s GC grace period; a fetch after that point returns 404.

Revision identifies content, not time. Two publishes that produce identical content (same evaluated JSON or same source files) yield the same revision. Consumers that cache by revision can skip a re-fetch when the revision has not changed.

The snippet mirrors status.artifactURL. To avoid a second lookup, the originating JsonnetSnippet also carries the URL in its own status.artifactURL. kubectl describe jsonnetsnippet therefore surfaces the artifact location directly.

Tarball contents

The tarball layout depends on spec.output of the originating JsonnetSnippet:

All tarballs are produced deterministically: entries are sorted by path and ModTime is zeroed. Two publishes from the same input produce byte-identical .tar.gz files and therefore the same revision and digest.

Symptom

READY=False, REASON=ExternalVariableConflict. The Message names the conflicting key.

Cause

The snippet’s spec.externalVariables declares a key that the operator already owns via --ext-var (cluster operator-level). Operator keys win by design — they’re how the cluster admin pins cluster-scoped values like cluster, region, environment so a tenant snippet can’t override them.

Diagnosis

# Which keys does the operator own?
kubectl --namespace <jaas-ns> get pod --selector app.kubernetes.io/name=jaas --output yaml | grep -A1 "\--ext-var="

Cross-reference with the snippet’s spec.externalVariables.

Remediation

Rename the conflicting key in the snippet, or remove it from the snippet entirely (the operator-level value flows through automatically).

If the snippet legitimately needs a different value, that’s a structural problem — the snippet shouldn’t ship with an opinion that overrides a cluster-wide invariant. Re-discuss with the cluster admin.

The validating webhook (--enable-webhook) catches this at admission so kubectl apply rejects it before it lands. The reconciler enforcing the same rule is a fallback for when admission is bypassed.

JaaS pairs with the grafana-operator to manage Grafana dashboards as code: you author the dashboard in Jsonnet, the JaaS operator renders it and publishes the dashboard JSON as a Flux ExternalArtifact, and the grafana-operator reconciles that artifact into a live Grafana instance.

This tutorial covers the JaaS side — authoring the dashboard, importing grafonnet as a JsonnetLibrary, and publishing the rendered JSON. The grafana-operator side (the GrafanaDashboard CR, datasources, folders) lives on their site and is linked at the end.

Prerequisites

• The JaaS operator installed and a tenant ServiceAccount granted the externalartifacts write verbs. The Quickstart covers both.
• The grafana-operator installed, if you intend to follow the handoff section and reconcile the dashboard into Grafana.

This tutorial uses the namespace default and the tenant ServiceAccount dashboards-tenant.

Step 1 — Grant the tenant ServiceAccount its verbs

The snippet imports a JsonnetLibrary, so on top of the externalartifacts write verbs the tenant needs get on jsonnetlibraries:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboards-tenant
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: dashboards-tenant
rules:
  - apiGroups: [source.toolkit.fluxcd.io]
    resources: [externalartifacts]
    verbs: [get, create, update, patch]
  - apiGroups: [jaas.metio.wtf]
    resources: [jsonnetlibraries]
    verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default
  name: dashboards-tenant
subjects:
  - kind: ServiceAccount
    name: dashboards-tenant
    namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: dashboards-tenant
EOF

Verify the ServiceAccount and binding:

kubectl --namespace default get serviceaccount dashboards-tenant
kubectl --namespace default get rolebinding dashboards-tenant

Step 2 — Publish the dashboard helpers as a JsonnetLibrary

A JsonnetLibrary holds reusable .libsonnet files that snippets in the same namespace import by alias. The example below carries a minimal set of dashboard constructors. In a production setup this is where grafonnet lives — see Jsonnet libraries for serving the full grafonnet tree from an OCIRepository.

cat <<EOF | kubectl apply -f -
apiVersion: jaas.metio.wtf/v1
kind: JsonnetLibrary
metadata:
  name: grafana-helpers
  namespace: default
spec:
  files:
    dashboard.libsonnet: |
      {
        new(title): {
          title: title,
          schemaVersion: 38,
          panels: [],
        },
      }
    panel.libsonnet: |
      {
        timeseries(title, expr): {
          type: 'timeseries',
          title: title,
          targets: [{ expr: expr }],
        },
        stat(title, expr): {
          type: 'stat',
          title: title,
          targets: [{ expr: expr }],
        },
      }
EOF

Verify the library:

kubectl --namespace default get jsonnetlibrary grafana-helpers

Step 3 — Author and apply the dashboard snippet

The JsonnetSnippet imports the library by the alias declared in spec.libraries[*].importPath, composes a dashboard from its constructors, and leaves spec.output at its default rendered so the published artifact carries the evaluated dashboard JSON:

cat <<EOF | kubectl apply -f -
apiVersion: jaas.metio.wtf/v1
kind: JsonnetSnippet
metadata:
  name: api-latency
  namespace: default
spec:
  serviceAccountName: dashboards-tenant
  output: rendered
  files:
    main.jsonnet: |
      local dashboard = import 'grafana/dashboard.libsonnet';
      local panel = import 'grafana/panel.libsonnet';
      dashboard.new('API Latency') + {
        panels: [
          panel.timeseries('p99 by route', 'histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))'),
          panel.stat('error rate', 'sum(rate(http_requests_total{code=~"5.."}[5m]))'),
        ],
      }
  libraries:
    - kind: JsonnetLibrary
      name: grafana-helpers
      importPath: grafana
EOF

The importPath: grafana ties import 'grafana/dashboard.libsonnet' to the grafana-helpers library. It defaults to the library’s metadata.name, so naming the library grafana would let you drop the field. kind is always JsonnetLibrary.

Step 4 — Confirm the dashboard rendered

kubectl --namespace default get jsonnetsnippet api-latency
# NAME          READY   URL                                                                                         AGE
# api-latency   True    http://jaas-storage.jaas-system.svc.cluster.local:8082/default/api-latency/<sha256>.tar.gz  5s

If READY is False, describe the snippet — the Ready condition’s Reason and Message name the cause (an RBAC gap on the library, an import alias collision, or a Jsonnet error):

kubectl --namespace default describe jsonnetsnippet api-latency

Step 5 — Inspect the published dashboard JSON

Fetch the artifact from a one-shot pod to see the rendered dashboard:

URL=$(kubectl --namespace default get jsonnetsnippet api-latency -o jsonpath='{.status.artifactURL}')
kubectl run --rm -i --restart=Never --image=docker.io/curlimages/curl:8.10.1 fetch -- \
    sh -c "curl -fsSL '$URL' | tar -xzO rendered.json"
# {
#    "panels": [ ... ],
#    "schemaVersion": 38,
#    "title": "API Latency"
# }

rendered.json is the Grafana dashboard model — the exact JSON the grafana-operator hands to Grafana’s dashboard API.

Use real grafonnet instead of the toy helpers

grafana-helpers kept this tutorial self-contained, but in production you import the real grafonnet library from a JOI image rather than hand-rolling constructors. Install it as a JsonnetLibrary with the joi Helm chart :

helm upgrade --install joi oci://ghcr.io/metio/helm-charts/joi \
  --namespace default \
  --set libraries.grafonnet.enabled=true

That renders an OCIRepository plus a JsonnetLibrary named grafonnet, sourcing ghcr.io/metio/joi-grafana-grafonnet. The snippet then references that library in place of grafana-helpers and imports the real grafonnet API by its full jb-vendor path:

apiVersion: jaas.metio.wtf/v1
kind: JsonnetSnippet
metadata:
  name: api-latency
  namespace: default
spec:
  serviceAccountName: dashboard-renderer
  libraries:
    - kind: JsonnetLibrary
      name: grafonnet
  files:
    main.jsonnet: |
      local g = import 'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet';
      g.dashboard.new('API Latency')
      + g.dashboard.withUid('api-latency')

Everything downstream is unchanged — it reconciles and publishes an ExternalArtifact exactly as in Steps 4–5; only the source of the library differs.

Handoff: reconcile the dashboard into Grafana

The published ExternalArtifact is now ready for the grafana-operator to consume. The grafana-operator reconciles a JaaS-published dashboard into Grafana through a GrafanaDashboard CR that references the artifact. That configuration — the GrafanaDashboard resource, the datasource and folder wiring, and the Grafana instance — lives on the grafana-operator’s own documentation:

• grafana-operator JaaS example: https://grafana.github.io/grafana-operator/docs/examples/dashboard/jaas/readme/
• grafana-operator project: https://grafana.github.io/grafana-operator/

Follow that example for the Grafana side; it picks up exactly where this tutorial leaves off — at the published ExternalArtifact.

Where to go next

• Jsonnet libraries — serve the full grafonnet tree as a JsonnetLibrary backed by an OCIRepository, with the empty-path whole-vendor-tree pattern.
• Snippet sources — back the dashboard with a GitRepository or OCIRepository instead of inline spec.files, and point spec.entryFile at one dashboard in a multi-dashboard tree.

The jaas Helm chart lives in the metio/helm-charts monorepo and is published at oci://ghcr.io/metio/helm-charts/jaas. The tables below are generated from each chart’s values.yaml, so they track the chart’s current values rather than a hand-maintained copy.

For how the values map onto the binary’s runtime behaviour, see the Configuration reference — every arguments.* value drives the corresponding --flag.

jaas chart

joi library chart

The joi chart publishes Jsonnet OCI Images as JsonnetLibrary + OCIRepository pairs, so snippets can import vendored libraries (grafonnet, k8s-libsonnet, …) without bundling them. Deploy it alongside jaas when snippets reference shared libraries.