ITADN

`forge test --mutate`: all mutants silently reported as INVALID (0% score) for certain --match-contract selections; underlying compile error is swallowed

#15101Closedgrandizzy 创建于 2026-06-08
T-bugA-testingCmd-forge-testC-forge
G
grandizzycommented
### Component Forge (`forge test --mutate`) ### Describe the bug When the test selection passed to mutation testing includes certain test contracts, **every** mutant is classified as `INVALID` ("mutation produced a compilation error") and the run reports a meaningless **Mutation Score: 0.0% (0/0)** — even though the exact same mutations are valid and killable with a slightly different selection, and the project compiles and tests green normally. The per-mutant compile is scoped to the file set from `TestArgs::get_sources_to_compile()` (with `dynamic_test_linking` force-enabled for mutation). For some selected test contracts this scoped compile returns `Err`, and `crates/forge/src/mutation/runner.rs` maps that `Err` directly to `MutationResult::Invalid` **without surfacing the compiler error**. The result is a silent, misleading 0% score with no way to tell a tooling/compile failure apart from genuinely non-compiling mutants. ### To Reproduce ```bash git clone --recurse-submodules https://github.com/morpho-org/morpho-blue cd morpho-blue # BROKEN: includes MorphoStorageLibTest -> 92/92 INVALID, score 0.0% FOUNDRY_PROFILE=no_via_ir FOUNDRY_OPTIMIZER_RUNS=200 \ forge test --mutate src/libraries/SharesMathLib.sol \ --mc "SupplyIntegrationTest|WithdrawIntegrationTest|MorphoStorageLibTest" \ --mutation-timeout 50 # WORKS: drop MorphoStorageLibTest -> mutants killed/survived correctly (~95%) FOUNDRY_PROFILE=no_via_ir FOUNDRY_OPTIMIZER_RUNS=200 \ forge test --mutate src/libraries/SharesMathLib.sol \ --mc "SupplyIntegrationTest|WithdrawIntegrationTest" \ --mutation-timeout 50 ``` The selected contracts run fine as a normal test (no mutation): ```bash FOUNDRY_PROFILE=no_via_ir forge test \ --mc "SupplyIntegrationTest|WithdrawIntegrationTest|MorphoStorageLibTest" # -> all green ``` Other selections also trigger it (e.g. any allowlist including `MorphoStorageLibTest`, and at least one of the `Authorization`/`Callbacks`/`OnlyOwner` integration tests); `--no-match-contract` denylists that leave those contracts in the set hit it too. `MathLibTest`, `MarketParamsLibTest`, `ExtSloadIntegrationTest`, etc. do **not** trigger it. ### Broken output ``` ╭──────────┬───────────┬────────────╮ │ Status ┆ # Mutants ┆ % of Total │ ╞══════════╪═══════════╪════════════╡ │ Survived ┆ 0 ┆ 0.0% │ │ Killed ┆ 0 ┆ 0.0% │ │ Invalid ┆ 92 ┆ 100.0% │ │ Skipped ┆ 0 ┆ 0.0% │ ╰──────────┴───────────┴────────────╯ Mutation Score: 0.0% (0/0 mutants killed) ``` ### Expected behavior - Mutants should be classified identically regardless of which (compiling, passing) test contracts are selected; valid mutations must not be reported as `INVALID`. - If a per-mutant compile genuinely fails, the compiler error should be surfaced (at least under `-v`/`-vvv`), so an infra/tooling compile failure is distinguishable from a non-typechecking mutation. - A run that produces `0/0` real results should not be presented as a `0.0%` score; it should warn that no mutant was actually evaluated. ### Environment - `forge 1.7.2-dev` (commit `eba698dcf`, master @ the native-symbolic merge) - macOS (arm64), solc 0.8.19 ### Additional context Relevant code: `TestArgs::get_sources_to_compile()` in `crates/forge/src/cmd/test/mod.rs` and the `Err(_) => MutationResult::Invalid` mapping in `crates/forge/src/mutation/runner.rs` (`compile_and_test`). `config.dynamic_test_linking` is force-enabled for mutation runs, which may be implicated in the scoped-compile failure.
关闭于 2026-06-09 0 条评论