feat(policy): Return definition when attr value is missing

[c-r33d]

Proposed Changes

Feature goal:

• If any of the attribute values being requested are inactive return an inactive error
• If the attribute value is missing, return the attribute definition IFF the attribute definition is active
• If the attribute definition or namespace is inactive for the value return an error.
• Fix authorization v1, in the case that a customer updates a pre-existing attribute definition to allow_traversal
  that we fail if the value is not returned, bc direct entitlements is not used here.
• Fix edge case where older attribute values are http scheme and our new parse library returns an empty string.

1.) Return attribute definition when attribute value is missing and not-inactive.
2.) Populate attribute value inactive error.

Checklist

• I have added or updated unit tests
• I have added or updated integration tests (if appropriate)
• I have added or updated documentation

Testing Instructions

[gemini-code-assist]

Summary of Changes

Hello @c-r33d, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the attribute retrieval mechanism within the service layer, specifically for queries involving Fully Qualified Names (FQNs). The primary goal is to provide more granular and accurate responses when attribute values are queried. Instead of a generic 'not found' error for missing values, the system now intelligently returns the attribute definition if the value is absent but the definition is active. This change also introduces a distinct error for inactive attribute values, enhancing error handling clarity for API consumers. These improvements lead to a more predictable and user-friendly attribute management system.

Highlights

• Enhanced Attribute Value Retrieval: The GetAttributesByValueFqns function now returns the attribute definition when a requested attribute value is missing, provided the attribute definition itself is active. This improves the API's behavior for partial matches.
• New Error for Inactive Values: A new error, ErrAttributeValueInactive, has been introduced to specifically indicate when an attribute value is inactive, differentiating it from a truly non-existent value.
• Flexible Querying for Inactive Values: The ListAttributesByFqns function now includes a parameter to explicitly request inactive attribute values, offering more control over data retrieval.
• Updated SQL Regex Pattern: Several SQL queries have been updated to use a more specific REGEXP_REPLACE pattern, changing ^https?:// to ^https:// for FQN matching.
• Comprehensive Test Coverage: New integration tests have been added to cover various scenarios, including mixed found/missing values, inactive definitions, and inactive values with active definitions, ensuring robust behavior.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

A value sought, but not yet found, Its definition, on solid ground. Inactive, yet still known, Through code, its status shown.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a valuable enhancement to attribute retrieval by FQN, allowing the service to return an attribute definition when a specific value is missing. It also adds a specific error for inactive attribute values, which improves clarity for clients. The implementation is logical, and the new integration tests provide good coverage for the new functionality and edge cases. I have one suggestion to refactor a loop to avoid modifying an input slice, which is a good practice for preventing unexpected side effects.

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	185.655762ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	90.905753ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	372.250831ms
Throughput	268.64 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	38.73296035s
Average Latency	385.629065ms
Throughput	129.09 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	27.973787977s
Average Latency	278.841222ms
Throughput	178.74 requests/second

[github-actions]

X-Test Results

opentdfplatformP36XS1.dockerbuild
cukes-report
✅ js-v0.4.34
✅ java-v0.4.34
✅ go-v0.4.34
bats-test-results
test-cases-mapping-report
✅ js-pull-3012
✅ java-pull-3012
✅ go-pull-3012

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	192.155572ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	97.433754ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	369.437513ms
Throughput	270.68 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	39.195674035s
Average Latency	390.837972ms
Throughput	127.57 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	27.486895831s
Average Latency	273.943856ms
Throughput	181.90 requests/second

[github-actions]

X-Test Results

opentdfplatformL98S48.dockerbuild
cukes-report
✅ java-v0.4.34
✅ js-v0.4.34
✅ go-v0.4.34
bats-test-results
test-cases-mapping-report
✅ js-pull-3012
✅ java-pull-3012
✅ go-pull-3012

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	190.22465ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	106.603571ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	363.374433ms
Throughput	275.20 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	40.041107314s
Average Latency	397.987522ms
Throughput	124.87 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	28.744444975s
Average Latency	286.318579ms
Throughput	173.95 requests/second

[github-actions]

X-Test Results

opentdfplatformKSLKUV.dockerbuild
cukes-report
✅ java-v0.4.34
✅ js-v0.4.34
✅ go-v0.4.34
bats-test-results
✅ js-pull-3012
test-cases-mapping-report
✅ java-pull-3012
✅ go-pull-3012

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	176.624642ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	101.955596ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	379.164665ms
Throughput	263.74 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	38.471377482s
Average Latency	383.262346ms
Throughput	129.97 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	27.576164986s
Average Latency	274.909158ms
Throughput	181.32 requests/second

[github-actions]

X-Test Results

opentdfplatformFHQ29E.dockerbuild
cukes-report
✅ java-v0.4.34
✅ js-v0.4.34
✅ go-v0.4.34
bats-test-results
test-cases-mapping-report
✅ js-pull-3012
✅ java-pull-3012
✅ go-pull-3012

[c-r33d]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant change in how attribute values are retrieved by FQN. The new logic correctly handles cases where attribute values are missing or inactive, returning attribute definitions for missing values and a specific error for inactive ones. The changes are well-supported by new integration tests that cover various scenarios. The database queries have been updated to allow for fetching inactive values when necessary, which is a good addition. I have one suggestion to improve the code by avoiding modification of input parameters, which is a good practice for preventing unexpected side effects.

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	209.105965ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	112.310987ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	381.679857ms
Throughput	262.00 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	44.892549058s
Average Latency	447.15506ms
Throughput	111.38 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	32.806674729s
Average Latency	326.851326ms
Throughput	152.41 requests/second

[github-actions]

X-Test Results

opentdfplatformQ52ZSB.dockerbuild
✅ js-v0.4.34
cukes-report
✅ java-v0.4.34
✅ go-v0.4.34
bats-test-results
test-cases-mapping-report
✅ js-pull-3012
✅ java-pull-3012
✅ go-pull-3012

[c-r33d]

Code Review

This pull request introduces a significant change in how attribute values are retrieved by FQN. The new logic correctly handles cases where attribute values are missing or inactive, returning attribute definitions for missing values and a specific error for inactive ones. The changes are well-supported by new integration tests that cover various scenarios. The database queries have been updated to allow for fetching inactive values when necessary, which is a good addition. I have one suggestion to improve the code by avoiding modification of input parameters, which is a good practice for preventing unexpected side effects.

/gemini ask what is the suggestion?

[gemini-code-assist]

My apologies for the confusion in my initial summary! Upon closer review of the GetAttributesByValueFqns function in service/policy/db/attribute_fqn.go, I see that the input parameter r.GetFqns() is not directly modified. Instead, new data structures (definitionFqns, queryFqnsSet, and queryFqns) are created to build the list of FQNs used for querying.

Therefore, the code already adheres to the good practice of avoiding modification of input parameters, and no specific suggestion is needed for that point. Thank you for pointing out the ambiguity in my previous comment!

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	195.05163ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	104.429598ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	385.550999ms
Throughput	259.37 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	39.539411843s
Average Latency	392.707711ms
Throughput	126.46 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	27.948386499s
Average Latency	278.134802ms
Throughput	178.90 requests/second

[github-actions]

X-Test Results

opentdfplatformUKVHCW.dockerbuild
cukes-report
✅ js-v0.4.34
✅ java-v0.4.34
✅ go-v0.4.34
bats-test-results
test-cases-mapping-report
✅ js-pull-3012
✅ java-pull-3012
✅ go-pull-3012

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	200.251697ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	99.086689ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	357.788392ms
Throughput	279.49 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	39.766213789s
Average Latency	395.978326ms
Throughput	125.73 requests/second

[github-actions]

X-Test Results

opentdfplatform2S15K6.dockerbuild
cukes-report
✅ js-v0.4.34
✅ java-v0.4.34
✅ go-v0.4.34
bats-test-results
test-cases-mapping-report
✅ js-pull-3012
✅ java-pull-3012
✅ go-pull-3012

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	202.47041ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	108.545235ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	354.197534ms
Throughput	282.33 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	38.705626518s
Average Latency	385.235555ms
Throughput	129.18 requests/second

[github-actions]

X-Test Results

opentdfplatformMIS3PA.dockerbuild
cukes-report
✅ js-v0.9.0
✅ js-pull-3012
bats-test-results
test-cases-mapping-report
✅ java-pull-3012
✅ java-v0.9.0
✅ go-v0.9.0
✅ go-pull-3012

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	191.012812ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	105.269171ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	373.057501ms
Throughput	268.06 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	40.079695154s
Average Latency	398.962693ms
Throughput	124.75 requests/second

[github-actions]

X-Test Results

opentdfplatformOAFQTT.dockerbuild
cukes-report
✅ js-v0.9.0
✅ js-pull-3012
✅ java-v0.9.0
bats-test-results
test-cases-mapping-report
✅ java-pull-3012
✅ go-v0.9.0
✅ go-pull-3012

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	185.563789ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	101.302471ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	356.762322ms
Throughput	280.30 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	39.177271155s
Average Latency	389.91241ms
Throughput	127.63 requests/second

[github-actions]

X-Test Results

opentdfplatform2EFYCQ.dockerbuild
cukes-report
✅ js-v0.9.0
✅ js-pull-3012
✅ java-v0.9.0
bats-test-results
test-cases-mapping-report
✅ java-pull-3012
✅ go-v0.9.0
✅ go-pull-3012

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	205.569776ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	106.370132ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	371.343943ms
Throughput	269.29 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	40.213801988s
Average Latency	400.058958ms
Throughput	124.34 requests/second

[github-actions]

X-Test Failure Report

opentdfplatformO7PR3D.dockerbuild
cukes-report
✅ js-v0.9.0

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	192.977424ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	108.924589ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	370.37957ms
Throughput	269.99 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	39.758000121s
Average Latency	395.980197ms
Throughput	125.76 requests/second

[github-actions]

X-Test Results

opentdfplatform7951NT.dockerbuild
cukes-report
✅ js-v0.9.0
bats-test-results
test-cases-mapping-report
✅ js-pull-3012
✅ java-v0.9.0
✅ java-pull-3012
✅ go-v0.9.0
✅ go-pull-3012