awh::regex and Grok
Our own regular expression engine with machine code generation, and a log-parsing layer on top of it. It was written to remove PCRE2 from AWH — and it did: the standard build doesn't know about it, and the reference stays only for verification.
Faster than PCRE2 almost everywhere — and several times faster on Elbrus
Five ways to apply an expression to text
The engine picks a strategy from the properties of the expression and the text rather than using one for everything.
| Strategy | When | What it does |
|---|---|---|
| Prefilter | before any automaton | the set of admissible first bytes and a required literal: stretches of text are skipped without running the automaton |
| DFA | a yes/no answer is enough | states are built on demand, one pass over the text |
| Pike | group boundaries are needed | all states at once, captures in linear time — no blow-up on crafted expressions |
| Backtrack | outside the regular subset | backreferences, lookahead and lookbehind, atomic groups, recursion |
| Codegen | ARM64, x86-64 and e2k | the expression program is turned into machine code |
A compiled expression never changes after compilation and is shared between threads without locks; each thread has its own matching state.
Relocatable code — decided before the first line
Everything outside the generated code — tables, parsing subroutines, character classes — is reached by an offset from a single pointer. The code contains no absolute addresses, so it is stored together with the expression: on a million expressions that is twelve seconds not spent at startup.
A memory region is never writable and executable at once; on Apple ARM64 via MAP_JIT, on OpenBSD with BTI landing pads.
The gain of code over our own interpreter shrinks with every interpreter fix, and that is a measure of success: it used to reach ×153, now the largest is ×34.7. The interpreter has caught up with the code where it used to lag tens of times behind.
| Expression | Codegen gain |
|---|---|
| \w+(?=@) | ×34.7 |
| (\w+)@(\w+)\.(\w+) | ×32.6 |
| (?>\w+)@\w+ | ×28.0 |
| .*?needle | ×14.0 |
| Content-Length | prefilter leads |
Forty-five scenarios, two measures
Matches per second; ARM64 (Apple M4 Max), both sides built with -O3. Two pairs are compared separately: our machine code against PCRE2's machine code, and our interpreter against its interpreter. They must not be mixed: the single measure we used before — code against code — hid the interpreter's lag, and it only surfaced on Elbrus.
| Machine code vs machine code | AWH | PCRE2 | Ratio |
|---|---|---|---|
| .*needle | 113,314 | 8,935 | 12.68 |
| (?>\w+)@\w+ | 9,086,432 | 2,429,420 | 3.74 |
| (?:[a-z]* ?)*dog | 42,179,841 | 11,491,479 | 3.67 |
| alpha|bravo|charlie|delta|echo|foxtrot | 11,405 | 6,201 | 1.84 |
| ^(GET|POST) (\S+) HTTP/(\d)\.(\d)$ | 105,297,795 | 59,989,513 | 1.76 |
| (\w+) \1 | 7,014,364 | 4,372,819 | 1.60 |
| \((?:[^()]|(?R))*\) | 28,721,849 | 25,997,930 | 1.10 |
| (?:HT|TP)/1 | 71,259,965 | 84,772,703 | 0.84 |
| Interpreter vs interpreter | AWH | PCRE2 | Ratio |
|---|---|---|---|
| (?:fox|dog)trap | 212,390 | 2,629 | 80.77 |
| \bneedle\b | 210,046 | 10,329 | 20.33 |
| .*needle | 73,076 | 5,104 | 14.32 |
| Content-Length | 122,768,379 | 24,091,306 | 5.10 |
| \((?:[^()]|(?R))*\) | 2,409,010 | 1,652,545 | 1.46 |
| (?:[a-z]+/)+v1 | 16,252,673 | 22,992,913 | 0.71 |
Machine code: faster than PCRE2 in 37 of 41 scenarios; of the remaining four, three are at 0.99, and only (?:HT|TP)/1 is noticeably behind; median ratio 1.18. Interpreter: ahead in 21 of 45, level in 24, behind in none; median ratio 1.46. "Ahead" means a ratio of 1.5 or more, "behind" means below 0.67. Compiling an expression costs us 2.3× more (0.44 of PCRE2): we build two programs — forward and reverse — and analyse the expression more deeply; it is compiled once and applied millions of times.
Machine code for Elbrus, which PCRE2 doesn't have
The code generator speaks the e2k instruction set (Elbrus-8C2, lcc): 23 back-end methods, with the encoder checked against the machine's assembler byte for byte. PCRE2 cannot generate code for this architecture at all, so here our code is compared with its interpreter.
| Expression | Our code | PCRE2 | Ratio |
|---|---|---|---|
| (?:fox|dog)trot | 3,753 | 109 | 34.4 |
| \w+(?=@) | 132,249 | 5,765 | 22.9 |
| (?>\w+)@\w+ | 210,074 | 12,330 | 17.0 |
| (\w+)@(\w+)\.(\w+) | 17,223 | 1,030 | 16.7 |
| .*needle | 3,007 | 216 | 13.9 |
| [0-9]{3,5} | 1,032,221 | 1,131,070 | 0.91 |
Overall: ahead in 39 of 41 scenarios, median ratio 6.6. Interpreter against interpreter: ahead in 22, level in 23, behind in none.
Compiled expressions without recompiling
Memory image
The expression program is written as is; restoring it is setting a view over the record, with no allocation or copying.
Distrust of the record
Every jump target, class, cell and group number is checked: a corrupted record is a failure, not a walk through memory.
Encryption and expiry
Encryption and compression via consumer handlers, record lifetime, machine and instruction-set identification.
Twenty verification stands and one deliberate difference
| Stand | Comparisons | Mismatches |
|---|---|---|
| Unicode properties, 1,225 properties | 1,362,278,400 | 0 |
| shorthand character classes | 26,689,536 | 0 |
| match boundaries on an expression corpus | 112,261 | 0 |
| verdict and reverse pass | 236,498 | 0 |
| acceptance and rejection of expressions | 200,000 patterns | 0 |
| grapheme clusters | 48,218,624 | 69,131 |
Graphemes follow the current edition of UAX #29, while PCRE2's table lags it in three rules. Two separate stands show the difference is limited to those rules: both report zero mismatches on the same 48 million comparisons.
Log parsing with named patterns
A pattern registry, expansion of %{NAME:field:type} references into a regular expression, and field extraction to JSON.
grok.build("%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{INT:code:int}"); // 192.168.1.10 GET /api/v1/orders?id=17 200 {"client":"192.168.1.10","method":"GET","request":"/api/v1/orders?id=17","code":200}
- References expand as text, not as subroutine calls: the compiler sees the whole tree, so both first-byte filtering and code generation work.
- Field names are wider than group names — hyphens, dots, repeats: %{IP:src-ip}:%{INT:src.port}.
- Numbers only by declared type. A version number or a code with a leading zero stays text, and the output stays valid JSON.
- Errors fail the build: unknown patterns and circular references never pass silently.
Grok patterns are also available in the online converter ACU: turn a log line into JSON, XML or YAML right in the browser.