At 10,000 feet

Setting: Grafana Loki alerting rules using LogQL regex line filters (|~ "..."), commonly wildcard alternations like argv=.*chmod|argv=.*chown.

Problem: when the branches share a literal prefix (argv=) followed by .*, Loki silently drops the .* and matches only the contiguous literal: argv=chmod yes, argv=/usr/bin/chmod no. No error or warning. The rule never fires. Confirmed on 3.4.1 through the current 3.7.7.

What this note establishes (measured, reproduced): (1) the correctness bug with its minimal trigger A.*B|A.*C, traced to Loki’s line-filter simplifier, (2) a conditional perf cliff: a correct wildcard alternation ran ~6x slower than pure-literal, but only when its implied literal is common in the stream.

Takeaway: never share a literal prefix followed by .* across alternation branches. Filter on a cheap literal first, then the precise regex (|= "argv=" |~ "argv=.*(?:chmod|chown)"): fixes both problems.

Notes. First seen on a Loki-backed detection rule set, reproduced on throwaway Loki 3.4.1 and 3.7.7 with synthetic datasets, root cause traced in source. Re-tests changed the story twice: “~20x slower” did not hold, and the trigger is a shared literal prefix, not missing parentheses. Corrections welcome.

Prior art: Loki’s literal prefilter is documented, these failure edges are not

Grafana’s 2020 “300x faster” writeup describes the optimization itself: a pure-literal alternation like err|panic becomes a bytes.Contains prescan. Credit goes there. New here are the failure edges (silent .* deletion and the conditional perf penalty), in neither that writeup, Loki’s LogQL docs, nor (as of 2026-09-09) any Loki GitHub issue. The nearest issue is #23892 (2026-08-10), a different bug in the same simplifier: a label filter such as | name=~"al.*" matches a substring instead of the whole value. Its fix, PR #24421 (opened 2026-09-08, unmerged), adds a guard that also covers the line-filter case here. See Validation.

The correctness bug: .* silently deleted after a shared prefix (measured)

460-line synthetic set, 110 true matches (ground truth confirmed with Python re and Go regexp. This is Loki’s filter layer, not regex semantics):

{job="x"} |~ `argv=.*chmod|argv=.*chown`          → 10 of 110   (only contiguous "argv=chmod" lines)
{job="x"} |~ `arg.*chmod|arg.*chown`              → 0 of 110    (cleanest zero-result repro)
{job="x"} |~ `(argv=.*chmod)|(argv=.*chown)`      → 110 of 110  (capturing parens mask it)
{job="x"} |~ `argv=.*(?:chmod|chown)`             → 110 of 110  (different parse shape, fine)
{job="x"} |~ `(?:argv=.*chmod)|(?:argv=.*chown)`  → 10 of 110   (non-capturing groups do NOT mask it)
{job="x"} |~ `argv=.*chmod`                       → 60 of 60    (single branch, fine)
{job="x"} |~ `usr.*chmod|bin.*chown`              → correct     (no shared prefix, fine)
{job="x"} |~ `argv=.+chmod|argv=.+chown`          → correct     (`.+` unaffected)

Root cause (measured, from source). Go’s parser factors the shared prefix to argv=(?:.*chmod|.*chown) (Go’s own regexp.MatchString still matches correctly: the bug is Loki’s). In pkg/logql/log/filter.go, simplifyConcat carries baseLiteral = "argv=" into each branch, where the .* node is skipped:

if sub.Op == syntax.OpStar && sub.Sub[0].Op == syntax.OpAnyCharNotNL {
    continue
}

yielding a wrong contains("argv=chmod") filter. Skipping .* is valid for a standalone unanchored filter. Concatenating across it while carrying a base literal is unsound. The region is byte-identical on main and release-3.7.x, and unit tests cover shared prefixes only among plain literals, an unintended edge case. A rule written the natural way never fires and never errors.

The perf cliff is real but conditional (measured, reproduced)

Correct (capturing-parenthesized) forms over 1M lines, median of 5 runs:

Query shape Median Ratio
pure-literal alternation chmod\|chown\|chattr 0.678 s 1x
wildcard alternation (argv=.*chmod)\|... 4.124 s ~6x slower
two-stage: literal filter, then the regex 0.651 s recovers fully

The ~6x appeared only with argv= in most lines. At ~5% of lines the wildcard form was not slower (0.77 s vs 0.69 s, Go’s regexp itself rejects at bytes.Index speed). The original ~19x field figure did not reproduce. ~6x is the controlled number, and the direction held in every run.

The fix

{job="x"} |= `argv=` |~ `argv=.*(?:chmod|chown|chattr)`

The |= stage discards most lines before the regex runs, and the suffix-group form parses to a shape the simplifier handles. On the million-line set it matched pure-literal speed with the correct count. (If you must keep branch alternation, use capturing parens. (?:...) does not help.)

Validation (2026-08-31, Loki 3.4.1 and 3.7.7)

Throwaway single-binary Docker instances, filesystem storage, full-window query_range:

  • On 3.4.1 (1M-line auditd-style stream, 500 planted matches): the natural rule returned 0 of 500. Probes showed the effective filter was contains("argv=chmod"). Correct forms returned 500. Timings as tabled.
  • On 3.4.1 and 3.7.7 (460-line set): identical wrong counts on both versions (10 of 110, 0 for arg.*chmod|arg.*chown), not fixed upstream. Shared prefix + .* is the isolated trigger. No shared prefix, .+, \S*, or single branch are correct. Capturing parens mask it (they block Go’s prefix factoring), (?:...) does not.
  • Not documented, not reported: LogQL docs describe |~ only as Go RE2, “not fully anchored”. No Loki issue reports the line-filter case. #23892 covers the label-filter cousin.
  • Fix in flight, traced from the diff and not executed (2026-09-09): PR #24421 makes simplifyConcat return nil, false when a literal follows a .* inside a branch that already carries a base literal. For argv=(?:.*chmod|.*chown) that sends the whole pattern to the plain regexp fallback, which is correct for a line filter. The PR’s TestLineFilterRegexStaysUnanchored covers a(.*c|d), the same shape. Unmerged, so 3.7.7 and every earlier release still carry the bug.

Verdict: reproduced end-to-end on all current Loki versions, mechanism traced in source.

Licensed under the site footer’s CC BY 4.0. If this saved you a debugging session, the optional thanks link in the footer is appreciated, no obligation.