feat(policy-devel): extend eval command to show raw evaluation of messages by Piskoo · Pull Request #2357 · chainloop-dev/chainloop

Piskoo · 2025-08-13T13:47:54Z

This PR changes the format of policy devel eval and adds additional feature to --debug flag.

policy devel eval output is now an object instead of array of objects:

Before

[
   {
      "violations": [
         "test"
      ],
      "skip_reasons": [],
      "skipped": false
   }
]

After

{
   "result": {
      "skipped": false,
      "skipReasons": [],
      "violations": [
         "test"
      ]
   }
}

--debug flag adds additional information to output in a debug_info field. It contains inputs field that holds json input material enriched with passed input args and chainloop metadata. Also raw_results field that contains full output out of opa eval which includes all rules.

{
   "result": {
      "skipped": false,
      "skipReasons": [],
      "violations": [
         "test"
      ]
   },
   "debug_info": {
      "inputs": [
         {
            "args": {
               "licenses": "AGPL"
            },
            "bomFormat": "CycloneDX",
            "chainloop_metadata": {
               "annotations": {
                  "chainloop.material.cas.inline": true,
                  "chainloop.material.name": "material",
                  "chainloop.material.type": "EVIDENCE"
               },
               "content": 
               (...) material content
         }
      ],
      "raw_results": [
         {
            "data.pck": {
              (...) #results
         },
         {
            "data.main": {
            (...) #results
            }
         }
      ]
   }
}

Closes #2353

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

Piskoo · 2025-08-18T08:21:42Z

raw_results is missing requested script and result due to limitations of naming rego scripts. data.<package_name> returned by opa eval is used for verification, otherwise the output scripts could be labeled by their filename for non embedded type, but that causes problem for embedded scripts since there's no name attached to them.

migmartri

Thanks @Piskoo

migmartri · 2025-08-18T11:42:26Z

app/cli/cmd/policy_develop_eval.go

 	cmd.Flags().StringVarP(&policyPath, "policy", "p", "policy.yaml", "Path to custom policy file")
 	cmd.Flags().StringSliceVar(&inputs, "input", []string{}, "Key-value pairs of policy inputs (key=value)")
 	cmd.Flags().StringSliceVar(&allowedHostnames, "allowed-hostnames", []string{}, "Additional hostnames allowed for http.send requests in policies")
+	cmd.Flags().BoolVarP(&debug, "debug", "", false, "Enable debug/verbose output and logging mode")


We could probably be more explicit here, currently, the description seems like a generic debug mode, when in reality it does more than that.

migmartri · 2025-08-18T11:43:16Z

app/cli/internal/action/policy_develop_eval.go

-	SkipReasons []string `json:"skip_reasons"`
-	Skipped     bool     `json:"skipped"`
-	Ignored     bool     `json:"ignored,omitempty"`
+	Violations  []string                 `json:"violations"`


in your example in the description it seems that the json keys where capitalized, aren't they using these annotations?

migmartri · 2025-08-18T11:45:13Z

pkg/policies/engine/rego/rego.go

+
 	// Try the main rule first
-	if err := executeQuery(mainRule, r.operatingMode == EnvironmentModeRestrictive); err != nil {
+	if err := executeQuery(fmt.Sprintf("%v.%s\n", parsedModule.Package.Path, mainRule), r.operatingMode == EnvironmentModeRestrictive); err != nil {


fmt.Sprintf("%v.%s\n", parsedModule.Package.Path, mainRule looks likke it could be in a function

migmartri · 2025-08-18T11:45:52Z

pkg/policies/engine/rego/rego.go

 		}

-		return parseViolationsRule(res, policy)
+		return parseViolationsRule(res, policy, rawData)


should we make this optional? Meaning that you can initialize the engine to return raw data too but optionally?

Makes sense due to only policy devel eval making use of it, thanks

… desc Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

app/cli/internal/policydevel/eval.go

migmartri · 2025-08-19T15:56:28Z

app/cli/internal/policydevel/eval.go

-	for _, policyEv := range policyEvs {
-		result := &EvalResult{
+	// Only one evaluation expected for a single policy attachment
+	policyEv := policyEvs[0]


you sure about this? Can't an evaluation contain info from multiple paths?

Results from a single policy attachment, evaluated against a single material, are merged into one evaluation result. So results from each path in the policy will be merged in the end.

nitpick. ok, should we protect the code?

migmartri

double check my comment about number of evaluations. Thanks!

migmartri

Awesome

migmartri · 2025-08-20T08:34:38Z

app/cli/internal/policydevel/eval.go

-	for _, policyEv := range policyEvs {
-		result := &EvalResult{
+	// Only one evaluation expected for a single policy attachment
+	policyEv := policyEvs[0]


nitpick. ok, should we protect the code?

migmartri · 2025-08-20T08:35:34Z

pkg/attestation/crafter/api/attestation/v1/crafting_state.proto

    string org_name = 4;
  }
+
+  repeated google.protobuf.Struct raw_results = 19;


add a comment explainign what this is please

migmartri · 2025-08-20T08:36:13Z

pkg/policies/engine/engine.go

+	RawData    *RawData
+}
+type RawData struct {
+	Input  interface{}


check json rawmessage

Updated types for RawData in the engine and also in proto file

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

Piskoo added 10 commits August 13, 2025 15:45

add raw flag

c903ccc

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

add input info

5e5f99b

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

add formated output

45c927a

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

fix tests

d844a69

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

fix lint and gen ref

4c7a319

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

fix tests

c3001cf

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

fix tests

42055c6

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

fix linter

768b0d2

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

show only one input

b8e9fa3

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

fix non debug output

4e4a6e2

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

Piskoo marked this pull request as ready for review August 18, 2025 08:08

Piskoo requested review from javirln and migmartri August 18, 2025 08:08

migmartri approved these changes Aug 18, 2025

View reviewed changes

Piskoo added 3 commits August 19, 2025 13:35

fix json keys; extract rego get rule name function; update debug flag…

be9dc94

… desc Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

make raw result optional for engine

b1a71ca

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

update cli ref

c920fda

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

migmartri reviewed Aug 19, 2025

View reviewed changes

migmartri requested changes Aug 19, 2025

View reviewed changes

migmartri approved these changes Aug 20, 2025

View reviewed changes

Piskoo added 2 commits August 20, 2025 11:28

add eval result nil check and proto raw results comment

ba1b906

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

add explicit types to raw results

c90e1b5

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>

Piskoo requested a review from migmartri August 20, 2025 11:01

Piskoo merged commit 2cc25f3 into chainloop-dev:main Aug 20, 2025
13 checks passed

Conversation

Piskoo commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Piskoo commented Aug 18, 2025

Uh oh!

migmartri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

migmartri left a comment

Choose a reason for hiding this comment

Uh oh!

migmartri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Piskoo commented Aug 13, 2025 •

edited

Loading