Enhance benchmark results and optimize instrumentation performance#1943
Enhance benchmark results and optimize instrumentation performance#1943Bertk wants to merge 9 commits into
Conversation
|
**Phase 1 results
The most important takeaway is that with ShortRun (3 iterations), a single slow outlier can dominate the Max and inflate the Mean. This is exactly why the switch to MediumRun (15 iterations) was applied — the p1 Max values already show the stabilising effect even within 3 iterations once the caches eliminate cold-start spikes. |
The cumulative effect of all five proposals is primarily a stability gain (worst-case latency and iteration-to-iteration variance significantly reduced) plus a consistent allocation saving of ~0.28 MB per instrumentation run. Mean times are stable, confirming the optimisations have no regression risk. |
There was a problem hiding this comment.
Pull request overview
This PR improves Coverlet’s performance benchmarking workflow and implements several runtime performance optimizations in coverlet.core, primarily targeting instrumentation hot paths and reporter allocations.
Changes:
- Optimize instrumentation performance by caching type include/exclude decisions, avoiding a second module read for reachability analysis, and caching compiled include/exclude filter regexes.
- Reduce reporter allocations by pooling buffers (
MemoryStream/StringBuilder) for OpenCover/Cobertura/Lcov report generation. - Update benchmark suite and documentation: refine which benchmark columns are captured, improve benchmark history update scripting, and refresh benchmark history content.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| test/coverlet.core.benchmark.tests/Simulator.cs | Adjust benchmark attributes (diagnoser configuration). |
| test/coverlet.core.benchmark.tests/Program.cs | Update BenchmarkDotNet job settings and benchmark set executed. |
| test/coverlet.core.benchmark.tests/PerformanceImprovementProposal.md | Add an implementation plan section for the proposals. |
| test/coverlet.core.benchmark.tests/InstrumenterBenchmarks.cs | Rework benchmark to instrument a clean per-iteration DLL/PDB copy and isolate setup overhead. |
| test/coverlet.core.benchmark.tests/InstrumentationOptionsBenchmarks.cs | Similar per-iteration clean binary setup; remove benchmarks deemed noise-only. |
| test/coverlet.core.benchmark.tests/HowTo.md | Add guidance to update benchmark history documentation via script. |
| src/coverlet.core/Reporters/OpenCoverReporter.cs | Pool MemoryStream and avoid per-call ToArray() allocation. |
| src/coverlet.core/Reporters/LcovReporter.cs | Switch to pooled StringBuilder to reduce allocations during report generation. |
| src/coverlet.core/Reporters/CoberturaReporter.cs | Pool MemoryStream and avoid per-call ToArray() allocation. |
| src/coverlet.core/Instrumentation/Instrumenter.cs | Add caching and hot-path loop optimizations; attempt to avoid double module reads. |
| src/coverlet.core/Helpers/InstrumentationHelper.cs | Cache compiled filter regex pairs for include/exclude evaluation. |
| scripts/Update-BenchmarkHistory.ps1 | Improve parsing/normalization and add upsert/dedup + version guard logic. |
| Documentation/BenchmarkHistory.md | Update benchmark history format notes and add new benchmark rows. |
93e2586 to
23c94bc
Compare
- BenchmarkHistory script now deduplicates, normalizes numbers (locale-aware), and only includes user [Params] in "Options" - Skips unmeasurable BDN rows; prevents accidental version downgrades unless forced - Benchmarks use temp working dirs for clean DLL/PDB per iteration; robust setup/teardown with [IterationSetup]/[GlobalCleanup] - Removed DeterministicAndSourceLink/ExcludeAssembliesHeuristic benchmarks (not meaningful with current test subject) - Updated docs and code style for clarity and maintainability - Improved error handling for missing files and instrumentation failures
Improves performance by caching regex filters and type exclusion/inclusion checks, reducing redundant computations. Replaces LINQ .Any() with allocation-free foreach loops in hot paths, adds helper methods for attribute checks, and updates all IsTypeIncluded usages to leverage the new cache. Also includes minor code style and copyright updates.
Updated BenchmarkHistory.md with new results for 2026-05-24 (10.0.2-p1), replacing previous entries and reflecting new performance metrics. Changed benchmark job in Program.cs from ShortRun to MediumRun for more robust measurements. Also updated opencover ReportFormatBenchmarks entry.
Refactor reporters to use thread-local buffers (MemoryStream/StringBuilder) for reduced allocations. Instrumenter now avoids double file reads by passing module bytes to the reachability helper. Updates implementation plan to reflect completed and deferred tasks.
Optimize memory usage and consistency in reporters - Cap filter regex cache in InstrumentationHelper to prevent unbounded memory growth; add CompileFilter helper. - Add 2GB guard and use ReadExactly in Instrumenter for efficient module reading. - Replace per-thread MemoryStream pooling with short-lived streams and ArrayPool<byte> in Cobertura/OpenCover reporters to reduce memory retention. - Improve culture-invariant formatting in LcovReporter. - Update comments in Update-BenchmarkHistory.ps1 for consistency.
Introduce test/coverlet.benchmark.subject with diverse async, iterator, switch, lambda, generic, exclusion, and deep-nesting workloads to better exercise coverlet's instrumentation and reporting. Update all benchmarks and infrastructure to use this new subject instead of coverlet.testsubject. Adjust solution, project references, and filter expressions accordingly. Update documentation to reflect the new subject and its purpose. No production code changes; improves benchmark coverage and regression detection.
Refactor Coverage.cs to use HashSet for branch lookups and pre-group HitCandidates by document index, improving lookup and loop efficiency. Replace LINQ with single-pass loops in CoverageSummary.cs for line/method coverage calculations. Cache filter validation regex and optimize bracket counting in InstrumentationHelper.cs. Pre-group branches by line in CoberturaReporter.cs to speed up XML generation. Update BenchmarkHistory.md with new results and mark related performance proposals as complete.
Prevent possible NullReferenceException by skipping iteration when r.BranchesInCompiledGeneratedClass is null before adding its elements to the HashSet. This ensures safer handling of instrumenter results.
Introduce Program.cs to exercise all benchmark workloads for coverage. Set OutputType to Exe in coverlet.benchmark.subject. Update Simulator to use AppContext.BaseDirectory for artifact paths. Add error handling for missing DLL/PDB and non-zero exit codes.
b19ed54 to
bd5cfc2
Compare
This pull request updates the benchmark history documentation and improves the PowerShell script used to process benchmark results. The main focus is on making the benchmark table clearer by only showing user-defined parameter columns, handling different number formats, and skipping invalid or infrastructure-related data. Additionally, the script is enhanced for robustness and flexibility.
Key improvements include:
Benchmark Table and Documentation:
BenchmarkHistory.mdnow displays only user-configured[Params]values in the "Options" column, omitting infrastructure columns like Job, Toolchain, and others. Explanatory notes have been added to clarify this behavior.Coverlet.core performance improvement proposals
StringBuilderin reportersPrepareModules