Notes. First seen on a Loki-backed detection rule set, reproduced on throwaway Loki 3.4.1 and 3.7.7 with synthetic datasets, root cause traced in source. Re-tests changed the story twice: “~20x slower” did not hold, and the trigger is a shared literal prefix, not missing parentheses. Corrections welcome.

At 10,000 feet

Setting: Grafana Loki alerting rules using LogQL regex line filters (|~ "..."), commonly wildcard alternations like argv=.*chmod|argv=.*chown.

Problem: when the branches share a literal prefix (argv=) followed by .*, Loki silently drops the .* and matches only the contiguous literal — argv=chmod yes, argv=/usr/bin/chmod no. No error or warning; the rule never fires. Confirmed on 3.4.1 through the current 3.7.7.

What this note establishes (measured, reproduced): (1) the correctness bug with its minimal trigger A.*B|A.*C, traced to Loki’s line-filter simplifier; (2) a conditional perf cliff — a correct wildcard alternation ran ~6x slower than pure-literal, but only when its implied literal is common in the stream.

Takeaway: never share a literal prefix followed by .* across alternation branches. Filter on a cheap literal first, then the precise regex (|= "argv=" |~ "argv=.*(?:chmod|chown)") — fixes both problems.

Prior art: Loki’s literal prefilter is documented; these failure edges are not

Grafana’s 2020 “300x faster” writeup describes the optimization itself: a pure-literal alternation like err|panic becomes a bytes.Contains prescan. Credit goes there. New here are the failure edges — silent .* deletion and the conditional perf penalty — in neither that writeup, Loki’s LogQL docs, nor (as of 2026-08-31) any Loki GitHub issue.

The correctness bug: .* silently deleted after a shared prefix (measured)

460-line synthetic set, 110 true matches (ground truth confirmed with Python re and Go regexp — this is Loki’s filter layer, not regex semantics):

{job="x"} |~ `argv=.*chmod|argv=.*chown`          → 10 of 110   (only contiguous "argv=chmod" lines)
{job="x"} |~ `arg.*chmod|arg.*chown`              → 0 of 110    (cleanest zero-result repro)
{job="x"} |~ `(argv=.*chmod)|(argv=.*chown)`      → 110 of 110  (capturing parens mask it)
{job="x"} |~ `argv=.*(?:chmod|chown)`             → 110 of 110  (different parse shape, fine)
{job="x"} |~ `(?:argv=.*chmod)|(?:argv=.*chown)`  → 10 of 110   (non-capturing groups do NOT mask it)
{job="x"} |~ `argv=.*chmod`                       → 60 of 60    (single branch, fine)
{job="x"} |~ `usr.*chmod|bin.*chown`              → correct     (no shared prefix, fine)
{job="x"} |~ `argv=.+chmod|argv=.+chown`          → correct     (`.+` unaffected)

Root cause (measured, from source). Go’s parser factors the shared prefix to argv=(?:.*chmod|.*chown) (Go’s own regexp.MatchString still matches correctly — the bug is Loki’s). In pkg/logql/log/filter.go, simplifyConcat carries baseLiteral = "argv=" into each branch, where the .* node is skipped:

if sub.Op == syntax.OpStar && sub.Sub[0].Op == syntax.OpAnyCharNotNL {
    continue
}

yielding a wrong contains("argv=chmod") filter. Skipping .* is valid for a standalone unanchored filter; concatenating across it while carrying a base literal is unsound. The region is byte-identical on main and release-3.7.x, and unit tests cover shared prefixes only among plain literals — an unintended edge case. A rule written the natural way never fires and never errors.

The perf cliff is real but conditional (measured, reproduced)

Correct (capturing-parenthesized) forms over 1M lines, median of 5 runs:

Query shape Median Ratio
pure-literal alternation chmod\|chown\|chattr 0.678 s 1x
wildcard alternation (argv=.*chmod)\|... 4.124 s ~6x slower
two-stage: literal filter, then the regex 0.651 s recovers fully

The ~6x appeared only with argv= in most lines; at ~5% of lines the wildcard form was not slower (0.77 s vs 0.69 s — Go’s regexp itself rejects at bytes.Index speed). The original ~19x field figure did not reproduce; ~6x is the controlled number, the direction the robust result.

The fix

{job="x"} |= `argv=` |~ `argv=.*(?:chmod|chown|chattr)`

The |= stage discards most lines before the regex runs, and the suffix-group form parses to a shape the simplifier handles — on the million-line set it matched pure-literal speed with the correct count. (If you must keep branch alternation, use capturing parens; (?:...) does not help.)

Validation (2026-08-31, Loki 3.4.1 and 3.7.7)

Throwaway single-binary Docker instances, filesystem storage, full-window query_range:

  • Round 1 (3.4.1, 1M-line auditd-style stream, 500 planted matches): the natural rule returned 0 of 500; probes showed the effective filter was contains("argv=chmod"). Correct forms returned 500. Timings as tabled.
  • Round 2 (3.4.1 and 3.7.7, 460-line set): identical wrong counts on both versions (10 of 110; 0 for arg.*chmod|arg.*chown) — not fixed upstream. Shared prefix + .* is the isolated trigger; no shared prefix, .+, \S*, or single branch are correct; capturing parens mask it (they block Go’s prefix factoring), (?:...) does not.
  • Not documented, not reported: LogQL docs describe |~ only as Go RE2, “not fully anchored”; a GitHub issue search found no existing report.

Verdict: reproduced end-to-end on all current Loki versions, mechanism traced in source.

Licensed under the site footer’s CC BY 4.0. If this saved you a debugging session, the optional thanks link in the footer is appreciated, no obligation.