[Bug] KQL does not properly escape leading forward slash

[eric-forte-elastic]

Pull Request

Issue link(s):

Resolves #441

Summary - What I changed

Update to align kql.to_dsl wildcard conversion with Kibana's toQueryStringQuery.

ToDsl._walk_wildcard emitted tree.value verbatim as a Lucene query_string.query, under-escaping Lucene-reserved characters — most notably a leading /, which Lucene treats as a regex delimiter.

Input:    process.args:/lockscreenurl\:http*
Before:   {"query_string": {"query": "/lockscreenurl\\:http*"}}    # leading / unescaped
After:    {"query_string": {"query": "\\/lockscreenurl\\:http*"}}  # matches Kibana

Compounding the issue, Wildcard.value had inconsistent semantics: the parser stored raw KQL text (with KQL escape sequences), while eql2kql and Value.from_python stored plain Python literals.

Note

A main decision point is whether or not we are ok with modfying the type that is KQL parser value function returns. If so this PR can be small, otherwise it is a bit larger.

Difference in values passed from parser.py

More complex fix that does not require parser change: eeb1c76

Less complex fix (current PR). The less complex fix addresses a long time mismatch in behavior and as such is my preference for the fix.

Howver, this does introduce risk for any external caller that constructed Wildcard("...") directly with a raw, KQL-escaped string and depended on the old over-escaping behavior of to_dsl/_render would see different output. That being said, the bug is in effect the difference, so the change is a forseeable consequence of fixing it.

How To Test

Unit tests should pass and run the following Python one lcommand and/or script to identify that the issue is resolved.

python -c 'import kql; q = kql.to_dsl(r"process.args:/lockscreenurl\:http*")["bool"]["filter"][0]["query_string"]["query"]; print(repr(q)); assert q == r"\/lockscreenurl\:http*", "issue #441 still present"; print("OK")'

detection-rules on  441-bug-kql-does-not-properly-escape-leading-forward-slash [!] is  v1.6.28 via  v3.12.13 (detection-rules-build) on  eric.forte 
❯ python -c 'import kql; q = kql.to_dsl(r"process.args:/lockscreenurl\:http*")["bool"]["filter"][0]["query_string"]["query"]; print(repr(q)); assert q == r"\/lockscreenurl\:http*", "
issue #441 still present"; print("OK")'
'\\/lockscreenurl\\:http*'
OK

Script Version

import kql

query = r"process.args:/lockscreenurl\:http*"
expected = r"\/lockscreenurl\:http*"

dsl = kql.to_dsl(query)
actual = dsl["bool"]["filter"][0]["query_string"]["query"]

print(f"KQL:      {query}")
print(f"DSL:      {dsl}")
print(f"query:    {actual!r}")
print(f"expected: {expected!r}")
assert actual == expected, "Issue #441 not fixed: leading `/` is not escaped"
print("OK — matches Kibana's DSL conversion.")

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Bug - Guidelines

These guidelines serve as a reminder set of considerations when addressing a bug in the code.

Documentation and Context

• Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
• Include additional context or details about the problem.
• Ensure the fix includes necessary updates to the release documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the bug fix or edge cases.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
• Validate that any rules affected by the bug are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the bug fix works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[shashank-elastic]

Tested Locally works as expected.

Testing 🟢

detection-rules on  441-bug-kql-does-not-properly-escape-leading-forward-slash [$?] is 📦 v1.6.29 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ pip3 install lib/kql           
Looking in indexes: https://pypi.org/simple, https://shashank.suryanarayana%40elastic.co:****@artifactory.elastic.dev/artifactory/api/pypi/pypi-endgame/simple
Processing ./lib/kql
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: eql==0.9.19 in ./.venv/lib/python3.12/site-packages (from detection-rules-kql==0.1.11) (0.9.19)
Requirement already satisfied: lark-parser>=0.12.0 in ./.venv/lib/python3.12/site-packages (from detection-rules-kql==0.1.11) (0.12.0)
Building wheels for collected packages: detection-rules-kql
  Building wheel for detection-rules-kql (pyproject.toml) ... done
  Created wheel for detection-rules-kql: filename=detection_rules_kql-0.1.11-py3-none-any.whl size=19164 sha256=98f119e2b61b6ebacaa1f46d7487a190ebb72cb07fc2b46ab5644fb67416a8bd
  Stored in directory: /private/var/folders/jk/t_tlgnwx4w998xqw3_kjzyx00000gn/T/pip-ephem-wheel-cache-fztha8z4/wheels/a5/04/40/23c256d5f994741fa8afe9042023a31c1bafcf883c8be68ab6
Successfully built detection-rules-kql
Installing collected packages: detection-rules-kql
  Attempting uninstall: detection-rules-kql
    Found existing installation: detection-rules-kql 0.1.10
    Uninstalling detection-rules-kql-0.1.10:
      Successfully uninstalled detection-rules-kql-0.1.10
Successfully installed detection-rules-kql-0.1.11

[notice] A new release of pip is available: 25.1.1 -> 26.1
[notice] To update, run: pip install --upgrade pip

detection-rules on  441-bug-kql-does-not-properly-escape-leading-forward-slash [$?] is 📦 v1.6.29 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 21s 
❯ python -c 'import kql; q = kql.to_dsl(r"process.args:/lockscreenurl\:http*")["bool"]["filter"][0]["query_string"]["query"]; print(repr(q)); assert q == r"\/lockscreenurl\:http*", "issue #441 still present"; print("OK")'
'\\/lockscreenurl\\:http*'
OK

detection-rules on  441-bug-kql-does-not-properly-escape-leading-forward-slash [$?] is 📦 v1.6.29 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯