side effects of failed modification by omursahin · Pull Request #1452 · WebFuzzing/EvoMaster

omursahin · 2026-02-19T08:12:24Z

No description provided.

arcuri82 · 2026-02-26T09:39:03Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+        }
+
+        // otherwise compare entire bodies
+        return bodyBefore != bodyAfter


i think we should return a false here, as otherwise too prone to false positives due to non-deterministic fields

arcuri82 · 2026-02-26T09:42:00Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+    /**
+     * Compare only the fields that were sent in the PUT/PATCH request.
+     * Returns true if any of those fields changed between before and after GET responses.
+     */


add comment stating this only works where the payload is a JSON object matching the resource structure. it will not work for cases like JSON Patch RFC6902

arcuri82 · 2026-02-26T09:44:27Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+    internal fun hasChangedModifiedFields(
+        bodyBefore: String,
+        bodyAfter: String,
+        fieldNames: Set<String>


to avoid flakiness, i think we should pass as well the bodyModify, and do check on that

arcuri82 · 2026-02-26T09:44:47Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+
+            if(!jsonBefore.isJsonObject || !jsonAfter.isJsonObject){
+                // not JSON objects, fallback to full comparison
+                return bodyBefore != bodyAfter


return false. see previous explanations

arcuri82 · 2026-02-26T09:46:27Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+                val valueAfter = objAfter.get(field)
+
+                if(valueBefore != valueAfter){
+                    return true


besides being different from valueBefore, we should make sure that valueAfter is equal to valueModify

hmmmm, maybe this is too broad. should narrow down. the modification might be just partial. eg, the PUT/PATCH might change 2 fields, and only 1 is applied before failure. or it is a JSON Merge Patch, and only a subset of fields is present. so, valueModify == valueAfter is incorrect. we should check, field by filed, if what is different between valueBefore and valueAfter is equal in valueModify. For example, for each field x, if valueBefore.x != valueAfter.x then we need to make sure valueAfter.x = valueModify.x

also, add comment that this would be done to deal with possible flakiness issues

arcuri82 · 2026-02-26T09:46:44Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+            return false
+        } catch (e: Exception) {
+            // JSON parsing failed, fallback to full comparison
+            return bodyBefore != bodyAfter


return false

arcuri82 · 2026-02-26T09:48:56Z

core/src/main/kotlin/org/evomaster/core/problem/rest/service/HttpSemanticsService.kt


+    /**
+     * Checking bugs like:
+     * POST|PUT  /X  2xx  (create resource)


this first call is not strictly necessary. resources could be already existing or created with database insertions

arcuri82 · 2026-02-26T09:49:51Z

core/src/main/kotlin/org/evomaster/core/problem/rest/service/HttpSemanticsService.kt

+     *
+     * If a PUT/PATCH fails with 4xx, it should have no side-effects.
+     * A GET before and after should return the same resource state.
+     */


add to comment that we need to take into account the non-determinism of the fields, eg like timestamps and UUIDs

arcuri82 · 2026-02-26T10:51:53Z

core/src/main/kotlin/org/evomaster/core/problem/rest/service/HttpSemanticsService.kt

+                    return@forEach
+                }
+
+                // among those, find one that also has a successful creation step


thanks. after reading this code i realized there are quite a few edge cases we need to handle. code here needs to be changed. i update the algorithm description in notes.txt

arcuri82 · 2026-03-24T09:20:37Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+    private fun extractRequestBody(modify: RestCallAction): String? {
+        val bodyParam = modify.parameters.find { it is BodyParam } as BodyParam?
+            ?: return null
+        return bodyParam.getValueAsPrintableString(mode = GeneUtils.EscapeMode.JSON)


why JSON? is the code here making the assumption that only JSON is involved? in EvoMaster we technically support other formats as well, like url-encoded and XML. unless there are some issues in checking them, we should support them as well. if not, need to clarify here and put checks on JSON. in BodyParam, recall the presence of contentTypeGene

Now, we support XML and URL-encoded, too.

arcuri82 · 2026-03-24T09:22:10Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+        if(!StatusGroup.G_4xx.isInGroup(resModify.getStatusCode())) {
+            return false
+        }
+


if we are only dealing with JSON, add a check here, and leave a TODO comment for other formats

arcuri82 · 2026-03-24T09:30:33Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+    ): Boolean {
+
+        try {
+            val jsonBefore = JsonParser.parseString(bodyBefore)


we shall not use GSON any more in EvoMaster, it has way too many problems. should rather use Jackson, eg see use and comments in JSON_FORMATTER

I used OutputFormatter for this. And also added XML support and a new function named readFields. I can read a property with that function.

arcuri82 · 2026-03-24T09:32:37Z

core/src/main/kotlin/org/evomaster/core/problem/rest/service/fitness/AbstractRestFitness.kt

        }
+
+        if(!config.isEnabledFaultCategory(ExperimentalFaultCategory.HTTP_SIDE_EFFECTS_FAILED_MODIFICATION)) {
+            LoggingUtil.uniqueUserInfo("Skipping experimental security test for repeated PUT after CREATE, as it has been disabled via configuration")


are these loggins really necessary? wouldn't this always be printed until this oracle is promoted from experimental?

arcuri82 · 2026-03-24T09:38:05Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+                val valueAfter = objAfter.get(field)
+
+                if(valueBefore != valueAfter){
+                    return true


hmmmm, maybe this is too broad. should narrow down. the modification might be just partial. eg, the PUT/PATCH might change 2 fields, and only 1 is applied before failure. or it is a JSON Merge Patch, and only a subset of fields is present. so, valueModify == valueAfter is incorrect. we should check, field by filed, if what is different between valueBefore and valueAfter is equal in valueModify. For example, for each field x, if valueBefore.x != valueAfter.x then we need to make sure valueAfter.x = valueModify.x

arcuri82 · 2026-03-24T10:00:06Z

core/src/main/kotlin/org/evomaster/core/problem/rest/service/HttpSemanticsService.kt

+        val getDef = actionDefinitions.find { it.verb == HttpVerb.GET && it.path.isEquivalent(path) }
+            ?: return
+
+        // T: smallest clean individual ending with GET 2xx (no prior PUT/PATCH on same path)


any specific reason to enforce no prior? for example, what if:

PUT /data/42 -> 201 (created)

GET /data/42 -> 200

PATCH /data/42 -> 4xx (failed with partial modifications)

would we still be able to handle to find those cases?

arcuri82 · 2026-03-24T10:25:14Z

core/src/main/kotlin/org/evomaster/core/problem/rest/oracle/HttpSemanticsOracle.kt

+                val valueAfter = objAfter.get(field)
+
+                if(valueBefore != valueAfter){
+                    return true


also, add comment that this would be done to deal with possible flakiness issues

arcuri82 · 2026-03-24T10:31:24Z

core/src/main/kotlin/org/evomaster/core/problem/rest/service/HttpSemanticsService.kt

+     *   PUT|PATCH /path  [k]
+     *   GET       /path  (same auth)
+     */
+    private fun addGetAroundFailedModification(


i don't understand this function... what is it for? was it for: otherwise: normal case (eg 400 and 409). find a 4xx PUT/PATCH action, copy it, and add it after the GET with same auth? if so, it is not doing this

you are right. I updated the algorithm and split it into two pieces: 404 cases and others.

arcuri82 · 2026-03-24T10:33:34Z

...g/evomaster/e2etests/spring/openapi/v3/httporacle/failmodification/FailModificationEMTest.kt

+
+        runTestHandlingFlakyAndCompilation(
+                "FailedModificationEM",
+                2000


do you really need so many iterations for this kind of E2E test?

arcuri82 · 2026-03-24T10:34:10Z

...er/e2etests/spring/openapi/v3/httporacle/failmodification/FailModificationForbiddenEMTest.kt

+
+        runTestHandlingFlakyAndCompilation(
+                "FailedModificationForbiddenEM",
+                3000


see previous comment

I updated all of them with the minimum number of iterations they required.

omursahin added 3 commits February 17, 2026 17:02

failed modification starter

88a0537

failed modification create ind

b69d9ce

add more tests

3823ee2

omursahin requested a review from arcuri82 February 19, 2026 11:54

arcuri82 requested changes Feb 26, 2026

View reviewed changes

omursahin and others added 8 commits March 1, 2026 11:50

Merge branch 'master' into oracle-failed-modification

e8d7c42

minor updates

418948f

checking modified body

be9571e

body extract

d5d244a

updated algorithm

81e48f2

not found detection

b59881c

failed modification 403 case

cc9850b

fix local id bug

7099e92

omursahin requested a review from arcuri82 March 9, 2026 19:30

arcuri82 requested changes Mar 24, 2026

View reviewed changes

omursahin and others added 4 commits March 27, 2026 10:19

Merge branch 'master' into oracle-failed-modification

b05432d

fixing failed modification

6cd618c

update brittle tests and extend

97375ed

increase iteration

99d0898

omursahin requested a review from arcuri82 March 30, 2026 06:57

Conversation

omursahin commented Feb 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants