Skip to content

Fix REST path encoding, Java Unicode escaping, and monitor initialization#1653

Open
hoco-scy wants to merge 5 commits into
WebFuzzing:masterfrom
hoco-scy:fix/uri-surrogate-and-process-monitor-20260718
Open

Fix REST path encoding, Java Unicode escaping, and monitor initialization#1653
hoco-scy wants to merge 5 commits into
WebFuzzing:masterfrom
hoco-scy:fix/uri-surrogate-and-process-monitor-20260718

Conversation

@hoco-scy

@hoco-scy hoco-scy commented Jul 22, 2026

Copy link
Copy Markdown

Summary

This PR fixes four independent failures observed while generating and running EvoMaster tests across the WebFuzzing SUT suite:

  • Resolved REST path fragments can fail or be corrupted for lone surrogates, supplementary Unicode characters, and non-ASCII characters.
  • Java Mongo initialization source can contain literal backslash-u sequences that interact with Java Unicode preprocessing.
  • Java request-body source can collapse literal backslash-u sequences into invalid or value-changing Unicode escapes.
  • Disabled process monitoring still initializes and deletes shared monitor output state, creating a concurrent filesystem race.

Changes

  • Encode non-ASCII resolved REST path fragments by Unicode code point and use URI raw paths, preserving valid surrogate pairs and handling isolated surrogates safely.
  • Escape Java-only Mongo EJSON literal backslash-u runs with octal backslash escapes while preserving genuine Unicode escapes and runtime values.
  • Preserve doubled source backslashes for Java BODY output so literal backslash-u values survive Java Unicode preprocessing.
  • Initialize process-monitor output only when process monitoring is enabled.

Scope

  • REST encoding applies to resolved/static path fragments, including fragments around OpenAPI link variables. Runtime link-variable values remain outside this change.
  • Mongo octal escaping applies only to Java output. Kotlin keeps its existing escaping behavior.
  • BODY escaping changes only Java JUnit 4/5 output. Kotlin, JavaScript, and Python behavior is unchanged.
  • Process-monitor behavior changes only when the monitor is disabled.

Tests

  • Added and extended regression coverage for BMP, supplementary, and isolated-surrogate REST path values; static fragments around runtime REST link variables; literal and genuine Unicode sequences in Java Mongo EJSON; Java BODY escaping through the JSON writer; and disabled process-monitor output isolation.
  • Local targeted suite: 64 tests passed with no failures, errors, or skips.
  • Additional local core coverage passed for all modified areas. Cases requiring unavailable external LLM/HTTP services or uncached Docker images are left to the fork CI environment.
  • Experimental validation confirmed the affected SUT generation and execution paths.

@arcuri82

Copy link
Copy Markdown
Collaborator

hi @hoco-scy !
you are a student of @man-zhang isn't it?
thanks for this PR ;)
as Man told you about docs/for_msc_bsc_students.md? once CI passes, ask @man-zhang for a review, and, when she will accept your updates/changes, she will ask for my review here on GitHub.
also, no need to create entries under issues (that is mainly for external users), you can just discuss them here directly in the PR description
thanks ;)

@hoco-scy

Copy link
Copy Markdown
Author

Hi @arcuri82,

Yes, I am a student of @man-zhang.

Thank you for the guidance. I apologize that I had not read docs/for_msc_bsc_students.md before opening this PR, so I followed my usual personal development workflow and created separate issues for the fixes.

I have now read the document and understand the expected internal contribution process. I will close the issues, keep the technical context in the PR description, make sure the CI process is completed on my fork, and then follow the review flow through @man-zhang.

Thank you again for your time and help.

@hoco-scy
hoco-scy force-pushed the fix/uri-surrogate-and-process-monitor-20260718 branch from 12a0f4d to 6a526ce Compare July 26, 2026 07:35
@hoco-scy hoco-scy changed the title Fix REST path, process monitor, and Mongo EJSON test generation Fix REST path encoding, Java Unicode escaping, and monitor initialization Jul 26, 2026
@@ -0,0 +1,36 @@
@file:Suppress("DEPRECATION")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove unnecessary annotation

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the unnecessary @file:Suppress("DEPRECATION") annotation in fe79f8e. I also confirmed that the affected tests compile and pass without it.

Thank you.

.replace("\b", "\\b")
.replace("\t", "\\t")

if (format.isJava()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handle needs to move with the format.isKotlin().

This fix seems to relate to ret.replace("\\\\u", "\\u").
Does it also happen in Kotlin? can you add a test for Kotlin as well.

In addition, can you add an example in the comment to explain this fix?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for pointing this out. I investigated the Kotlin behavior and addressed it in fe79f8e.

Kotlin was affected by the same issue. The previous Kotlin branch called ret.replace("\\\\u", "\\u"), which reduced the doubled source backslash to a single backslash-u sequence. As a result:

  • a malformed sequence such as \uquWOnj could make the generated Kotlin source fail to compile;
  • a valid sequence such as \u0041 could compile but change the runtime value to A, instead of preserving the literal \u0041 text.

I verified the fix in the following ways:

  1. Extended bodyEscapesKeepLiteralBackslashUForJvmFormats to cover Java JUnit 4/5 and Kotlin JUnit 4/5.
  2. Added kotlinBodyWriterPreservesLiteralBackslashUAtRuntime, which generates an actual Kotlin class, compiles it with CompilerForTestGenerated, loads the compiled class, invokes its methods, and checks the runtime string values.
  3. Covered malformed backslash-u text, valid \u0041 text, repeated backslashes, and Kotlin dollar-sign escaping.
  4. Kept the existing non-JVM behavior for JavaScript and Python unchanged.

The verification confirms that the generated Kotlin source now compiles successfully and preserves the original runtime strings, including literal \u0041, rather than converting it to A.

I also added concrete malformed and valid examples to the source comment as requested.

@hoco-scy

Copy link
Copy Markdown
Author

Hi @man-zhang,

I have addressed both review comments in fe79f8e:

  • removed the unnecessary suppression annotation;
  • investigated the Kotlin behavior and confirmed that it was affected by the same backslash-u issue;
  • extended the fix to Kotlin output while preserving Kotlin dollar-sign escaping;
  • added Kotlin JUnit 4/5 coverage and an actual compile-and-runtime regression test;
  • added concrete examples to the source comment.

The latest full CI run for the same PR head commit has completed successfully: 22 jobs passed, with only the conditional debug job skipped.

CI run:
https://github.com/hoco-scy/EvoMaster/actions/runs/30246503337

Could you please take another look when convenient?

Thank you.

@hoco-scy
hoco-scy requested a review from man-zhang July 27, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants