Skip to content

Add per-tenant alert generator URL template for customizable alert source links#7302

Open
CharlieTLe wants to merge 12 commits intocortexproject:masterfrom
CharlieTLe:809
Open

Add per-tenant alert generator URL template for customizable alert source links#7302
CharlieTLe wants to merge 12 commits intocortexproject:masterfrom
CharlieTLe:809

Conversation

@CharlieTLe
Copy link
Copy Markdown
Member

@CharlieTLe CharlieTLe commented Feb 27, 2026

Summary

  • Replace Grafana-specific config fields with a single generic ruler_alert_generator_url_template field that accepts a Go text/template string
  • Users can construct any URL format (Grafana Explore, Perses, or any metrics explorer) without Cortex needing to understand specific UI formats
  • Keep per-tenant ruler_external_url override so tenants can have different external URLs without changing the global ruler config

When ruler_alert_generator_url_template is set, the ruler evaluates the template with .ExternalURL and .Expression variables to produce the alert's GeneratorURL. Built-in Go template functions like urlquery are available for URL encoding. If the template is empty, the default Prometheus /graph format is used.

New per-tenant config options

Setting Description Default
ruler_external_url Per-tenant external URL override for the ruler "" (uses global)
ruler_alert_generator_url_template Go text/template for alert generator URLs "" (Prometheus format)

Example runtime config

overrides:
  # Grafana Explore links
  tenant-a:
    ruler_external_url: "http://grafana:3000"
    ruler_alert_generator_url_template: >-
      {{ .ExternalURL }}/explore?schemaVersion=1&panes=%7B%22default%22:%7B%22datasource%22:%22my-ds%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22{{ urlquery .Expression }}%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1

  # Perses explore links
  tenant-b:
    ruler_external_url: "http://perses:8080"
    ruler_alert_generator_url_template: >-
      {{ .ExternalURL }}/explore?expr={{ urlquery .Expression }}

  # Default Prometheus /graph format (no template needed)
  tenant-c: {}

Getting-started docker-compose example

The PR includes a working docker-compose example in docs/getting-started/ with:

  • Cortex with runtime config enabling per-tenant URL templates
  • Grafana with per-tenant Alertmanager datasources (tenant-a, tenant-b)
  • Perses v0.53.1 with explorer enabled and per-tenant Prometheus datasources
  • Demo alert rules that fire immediately for both tenants
  • Clicking "See source" on tenant-a alerts opens Grafana Explore; tenant-b alerts open Perses explore

Test plan

  • Unit tests for executeGeneratorURLTemplate with various expressions, urlquery, and invalid templates
  • Unit tests for userExternalURL tracking per-tenant URL changes
  • Unit tests for SendAlerts with custom generator URL function (default + template formats)
  • Template parse validation in Limits.Validate() catches invalid templates at config load time
  • Validated exporter test updated for removed Grafana-specific limit fields
  • Manual e2e test: Docker Compose with Cortex + Grafana + Perses, always-firing alert rules for tenant-a and tenant-b, confirmed "See source" links open the correct explorer with correct datasource and query

🤖 Generated with Claude Code

Add support for tenants to configure alert GeneratorURL to use Grafana
Explore format instead of the default Prometheus /graph format. This is
controlled by three new per-tenant settings: ruler_alert_generator_url_format,
ruler_grafana_datasource_uid, and ruler_grafana_org_id.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
@dosubot dosubot Bot added the component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. label Feb 27, 2026
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
@CharlieTLe CharlieTLe linked an issue Mar 1, 2026 that may be closed by this pull request
Copy link
Copy Markdown
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rajagopalanand Can you help take a look?

Copy link
Copy Markdown
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave a review, but I think we do not care what parameters are send to the url. We don't care about grafana org IDs

I think we should make this general enough to be used by any metrics explorer.
Maybe consider renaming the function as explore and not have any grafana

I am thinking of something that would support for example:
explore

what do you think?

Comment thread pkg/util/validation/exporter_test.go Outdated
Comment thread pkg/ruler/compat.go Outdated
Comment thread pkg/ruler/compat.go Outdated
Comment thread docs/configuration/config-file-reference.md Outdated
Comment thread pkg/ruler/ruler.go Outdated
CharlieTLe and others added 2 commits March 30, 2026 11:51
…nerator URLs

Replace the 3 Grafana-specific per-tenant config fields
(ruler_alert_generator_url_format, ruler_grafana_datasource_uid,
ruler_grafana_org_id) with a single generic field:
ruler_alert_generator_url_template.

This field accepts a Go text/template string with .ExternalURL and
.Expression variables, plus built-in functions like urlquery. Users
can construct any URL format (Grafana, Perses, etc.) without Cortex
needing to understand specific UI formats.

The ruler_external_url per-tenant override and SendAlerts signature
(func(expr string) string) are kept unchanged.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
Add per-tenant Alertmanager datasources (tenant-a, tenant-b) to Grafana
provisioning so alerts are visible in Grafana's alerting UI.

Add runtime-config.yaml with per-tenant overrides:
- tenant-a: Grafana Explore URL template with full pane JSON
- tenant-b: Perses explore URL template with PrometheusTimeSeriesQuery

Update Perses from v0.49 to v0.53.1 and enable the explorer feature
(frontend.explorer.enable: true). Rename project from "default" to
"cortex" to match template URLs.

Add Step 7 to the getting-started guide with instructions for:
- Configuring per-tenant alert generator URL templates
- Loading alertmanager configs and demo alert rules
- Viewing alerts in Grafana at /alerting/groups?groupBy=alertname
- Verifying generator URLs via the API

Also configure ruler.alertmanager_url and ruler.external_url, and set
an explicit UID on the Grafana Cortex datasource for use in templates.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
@CharlieTLe CharlieTLe requested a review from friedrichg March 30, 2026 22:23
@CharlieTLe CharlieTLe changed the title Add per-tenant Grafana Explore URL format for alert GeneratorURL Add per-tenant alert generator URL template for customizable alert source links Mar 30, 2026
CharlieTLe and others added 8 commits March 30, 2026 15:54
Add support for tenants to configure alert GeneratorURL to use Grafana
Explore format instead of the default Prometheus /graph format. This is
controlled by three new per-tenant settings: ruler_alert_generator_url_format,
ruler_grafana_datasource_uid, and ruler_grafana_org_id.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
…nerator URLs

Replace the 3 Grafana-specific per-tenant config fields
(ruler_alert_generator_url_format, ruler_grafana_datasource_uid,
ruler_grafana_org_id) with a single generic field:
ruler_alert_generator_url_template.

This field accepts a Go text/template string with .ExternalURL and
.Expression variables, plus built-in functions like urlquery. Users
can construct any URL format (Grafana, Perses, etc.) without Cortex
needing to understand specific UI formats.

The ruler_external_url per-tenant override and SendAlerts signature
(func(expr string) string) are kept unchanged.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
Add per-tenant Alertmanager datasources (tenant-a, tenant-b) to Grafana
provisioning so alerts are visible in Grafana's alerting UI.

Add runtime-config.yaml with per-tenant overrides:
- tenant-a: Grafana Explore URL template with full pane JSON
- tenant-b: Perses explore URL template with PrometheusTimeSeriesQuery

Update Perses from v0.49 to v0.53.1 and enable the explorer feature
(frontend.explorer.enable: true). Rename project from "default" to
"cortex" to match template URLs.

Add Step 7 to the getting-started guide with instructions for:
- Configuring per-tenant alert generator URL templates
- Loading alertmanager configs and demo alert rules
- Viewing alerts in Grafana at /alerting/groups?groupBy=alertname
- Verifying generator URLs via the API

Also configure ruler.alertmanager_url and ruler.external_url, and set
an explicit UID on the Grafana Cortex datasource for use in templates.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
Resolve conflict in schemas/cortex-config-schema.json: keep both
the upstream results_cache_ttl entry and our ruler_alert_generator_url_template.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
Resolve conflict in schemas/cortex-config-schema.json: keep both
results_cache_ttl and ruler_alert_generator_url_template entries.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Alert GeneratorURLs generated by the ruler do not indicate tenant

3 participants