Skip to content

Enhance benchmark results and optimize instrumentation performance#1943

Draft
Bertk wants to merge 9 commits into
coverlet-coverage:masterfrom
Bertk:refactor-performance
Draft

Enhance benchmark results and optimize instrumentation performance#1943
Bertk wants to merge 9 commits into
coverlet-coverage:masterfrom
Bertk:refactor-performance

Conversation

@Bertk
Copy link
Copy Markdown
Collaborator

@Bertk Bertk commented May 24, 2026

This pull request updates the benchmark history documentation and improves the PowerShell script used to process benchmark results. The main focus is on making the benchmark table clearer by only showing user-defined parameter columns, handling different number formats, and skipping invalid or infrastructure-related data. Additionally, the script is enhanced for robustness and flexibility.

Key improvements include:

Benchmark Table and Documentation:

  • The benchmark table in BenchmarkHistory.md now displays only user-configured [Params] values in the "Options" column, omitting infrastructure columns like Job, Toolchain, and others. Explanatory notes have been added to clarify this behavior.

Coverlet.core performance improvement proposals

# Proposal Effort Expected Impact Risk
2 Cache type-exclusion results Low High Low
6 Pre-compile filter expressions Low High Low
3 Replace LINQ hot paths with loops Low Medium Low
4 Pool StringBuilder in reporters Low Medium Low
1 Avoid double module read Medium High Medium
5 Parallelise PrepareModules High Very High High

@Bertk
Copy link
Copy Markdown
Collaborator Author

Bertk commented May 24, 2026

**Phase 1 results

Signal Verdict
Mean timing Flat — differences of 1–10 ms are within ShortRun noise (3 iterations). Not statistically significant.
Max / worst-case timing Clear improvement in p1 — spikes up to 328 ms in p0 collapse to ≤185 ms in p1. The caches eliminate the cold-path JIT / first-iteration outliers.
Allocation Consistent −0.24–0.25 MB per instrumentation run across all option combinations. The compiled regex cache and type-filter caches reduce per-run allocations reliably.
Report formats No change — not on the optimised path.

The most important takeaway is that with ShortRun (3 iterations), a single slow outlier can dominate the Max and inflate the Mean. This is exactly why the switch to MediumRun (15 iterations) was applied — the p1 Max values already show the stabilising effect even within 3 iterations once the caches eliminate cold-start spikes.

@Bertk
Copy link
Copy Markdown
Collaborator Author

Bertk commented May 24, 2026

Signal p0 p2 Verdict
Mean timing (instrumentation) 165–171 ms 165–177 ms Flat — within measurement noise
Max timing (instrumentation) 176–328 ms 168–188 ms ✅ Clear improvement — worst-case reduced by up to 156 ms
Max spread across all option combos 152 ms 17 ms ✅ 9× more consistent worst-case
Instrumentation allocation 44.68 MB 44.41 MB ✅ −0.27 MB (−0.6%) per run
Report allocation (cobertura/lcov/opencover) baseline −3–25% ✅ Proposal 4 visible
Mean timing (workflow) 468 ms 484 ms Flat / noise-dominated at ShortRun p0 baseline

The cumulative effect of all five proposals is primarily a stability gain (worst-case latency and iteration-to-iteration variance significantly reduced) plus a consistent allocation saving of ~0.28 MB per instrumentation run. Mean times are stable, confirming the optimisations have no regression risk.

@Bertk Bertk marked this pull request as ready for review May 24, 2026 16:06
Copilot AI review requested due to automatic review settings May 24, 2026 16:06
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves Coverlet’s performance benchmarking workflow and implements several runtime performance optimizations in coverlet.core, primarily targeting instrumentation hot paths and reporter allocations.

Changes:

  • Optimize instrumentation performance by caching type include/exclude decisions, avoiding a second module read for reachability analysis, and caching compiled include/exclude filter regexes.
  • Reduce reporter allocations by pooling buffers (MemoryStream / StringBuilder) for OpenCover/Cobertura/Lcov report generation.
  • Update benchmark suite and documentation: refine which benchmark columns are captured, improve benchmark history update scripting, and refresh benchmark history content.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
test/coverlet.core.benchmark.tests/Simulator.cs Adjust benchmark attributes (diagnoser configuration).
test/coverlet.core.benchmark.tests/Program.cs Update BenchmarkDotNet job settings and benchmark set executed.
test/coverlet.core.benchmark.tests/PerformanceImprovementProposal.md Add an implementation plan section for the proposals.
test/coverlet.core.benchmark.tests/InstrumenterBenchmarks.cs Rework benchmark to instrument a clean per-iteration DLL/PDB copy and isolate setup overhead.
test/coverlet.core.benchmark.tests/InstrumentationOptionsBenchmarks.cs Similar per-iteration clean binary setup; remove benchmarks deemed noise-only.
test/coverlet.core.benchmark.tests/HowTo.md Add guidance to update benchmark history documentation via script.
src/coverlet.core/Reporters/OpenCoverReporter.cs Pool MemoryStream and avoid per-call ToArray() allocation.
src/coverlet.core/Reporters/LcovReporter.cs Switch to pooled StringBuilder to reduce allocations during report generation.
src/coverlet.core/Reporters/CoberturaReporter.cs Pool MemoryStream and avoid per-call ToArray() allocation.
src/coverlet.core/Instrumentation/Instrumenter.cs Add caching and hot-path loop optimizations; attempt to avoid double module reads.
src/coverlet.core/Helpers/InstrumentationHelper.cs Cache compiled filter regex pairs for include/exclude evaluation.
scripts/Update-BenchmarkHistory.ps1 Improve parsing/normalization and add upsert/dedup + version guard logic.
Documentation/BenchmarkHistory.md Update benchmark history format notes and add new benchmark rows.

Comment thread src/coverlet.core/Instrumentation/Instrumenter.cs Outdated
Comment thread scripts/Update-BenchmarkHistory.ps1 Outdated
Comment thread src/coverlet.core/Helpers/InstrumentationHelper.cs
Comment thread src/coverlet.core/Reporters/OpenCoverReporter.cs Outdated
Comment thread src/coverlet.core/Reporters/CoberturaReporter.cs Outdated
Comment thread src/coverlet.core/Instrumentation/Instrumenter.cs
Comment thread src/coverlet.core/Reporters/LcovReporter.cs
Comment thread scripts/Update-BenchmarkHistory.ps1 Outdated
Comment thread scripts/Update-BenchmarkHistory.ps1
@Bertk Bertk marked this pull request as draft May 25, 2026 09:10
@Bertk Bertk added skip-changelog ignore PR for change.log document (release-drafter chore PR label for maintenance (dependencies, build, scripts) coverlet-core labels May 25, 2026
@Bertk Bertk force-pushed the refactor-performance branch 2 times, most recently from 93e2586 to 23c94bc Compare May 26, 2026 18:41
Bertk added 9 commits June 7, 2026 08:41
- BenchmarkHistory script now deduplicates, normalizes numbers (locale-aware), and only includes user [Params] in "Options"
- Skips unmeasurable BDN rows; prevents accidental version downgrades unless forced
- Benchmarks use temp working dirs for clean DLL/PDB per iteration; robust setup/teardown with [IterationSetup]/[GlobalCleanup]
- Removed DeterministicAndSourceLink/ExcludeAssembliesHeuristic benchmarks (not meaningful with current test subject)
- Updated docs and code style for clarity and maintainability
- Improved error handling for missing files and instrumentation failures
Improves performance by caching regex filters and type exclusion/inclusion checks, reducing redundant computations. Replaces LINQ .Any() with allocation-free foreach loops in hot paths, adds helper methods for attribute checks, and updates all IsTypeIncluded usages to leverage the new cache. Also includes minor code style and copyright updates.
Updated BenchmarkHistory.md with new results for 2026-05-24 (10.0.2-p1), replacing previous entries and reflecting new performance metrics. Changed benchmark job in Program.cs from ShortRun to MediumRun for more robust measurements. Also updated opencover ReportFormatBenchmarks entry.
Refactor reporters to use thread-local buffers (MemoryStream/StringBuilder) for reduced allocations. Instrumenter now avoids double file reads by passing module bytes to the reachability helper. Updates implementation plan to reflect completed and deferred tasks.
Optimize memory usage and consistency in reporters

- Cap filter regex cache in InstrumentationHelper to prevent unbounded memory growth; add CompileFilter helper.
- Add 2GB guard and use ReadExactly in Instrumenter for efficient module reading.
- Replace per-thread MemoryStream pooling with short-lived streams and ArrayPool<byte> in Cobertura/OpenCover reporters to reduce memory retention.
- Improve culture-invariant formatting in LcovReporter.
- Update comments in Update-BenchmarkHistory.ps1 for consistency.
Introduce test/coverlet.benchmark.subject with diverse async, iterator, switch, lambda, generic, exclusion, and deep-nesting workloads to better exercise coverlet's instrumentation and reporting. Update all benchmarks and infrastructure to use this new subject instead of coverlet.testsubject. Adjust solution, project references, and filter expressions accordingly. Update documentation to reflect the new subject and its purpose. No production code changes; improves benchmark coverage and regression detection.
Refactor Coverage.cs to use HashSet for branch lookups and pre-group HitCandidates by document index, improving lookup and loop efficiency. Replace LINQ with single-pass loops in CoverageSummary.cs for line/method coverage calculations. Cache filter validation regex and optimize bracket counting in InstrumentationHelper.cs. Pre-group branches by line in CoberturaReporter.cs to speed up XML generation. Update BenchmarkHistory.md with new results and mark related performance proposals as complete.
Prevent possible NullReferenceException by skipping iteration when r.BranchesInCompiledGeneratedClass is null before adding its elements to the HashSet. This ensures safer handling of instrumenter results.
Introduce Program.cs to exercise all benchmark workloads for coverage.
Set OutputType to Exe in coverlet.benchmark.subject.
Update Simulator to use AppContext.BaseDirectory for artifact paths.
Add error handling for missing DLL/PDB and non-zero exit codes.
@Bertk Bertk force-pushed the refactor-performance branch from b19ed54 to bd5cfc2 Compare June 7, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore PR label for maintenance (dependencies, build, scripts) coverlet-core skip-changelog ignore PR for change.log document (release-drafter

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants