Tracking: @test_broken tests audit and status

# Broken Tests Summary - Current State This document summarizes tests that are currently marked as `@test_broken` in the Catalyst test suite, organized by category and root cause. **Last updated:** 2026-02-10 --- ## Summary Statistics - **Total `@test_broken` occurrences:** 54 - **Upstream MTK issues:** ~19 - **Upstream Symbolics issues:** ~10 - **Conservation law issues:** ~10 - **Catalyst internal issues:** ~3 - **Other (extensions, visualization, misc):** ~12 --- ## Tests Blocked by MTK Issues ### Misformatted Events Not Throwing Errors ([MTK #4167](https://github.com/SciML/ModelingToolkit.jl/issues/4167)) Some malformed event syntax doesn't produce errors. | File | Lines | Description | |------|-------|-------------| | [events.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/events.jl) | 160 | Misformatted events don't yield errors | ### SteadyStateProblem Issues ([MTK #4174](https://github.com/SciML/ModelingToolkit.jl/issues/4174)) SteadyStateProblem solving fails in certain cases. | File | Lines | Description | |------|-------|-------------| | [coupled_equation_crn_systems.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/coupled_equation_crn_systems.jl) | 177 | SteadyStateProblem solving fails | ### Rational Parameter Equivalence ([MTK #4163](https://github.com/SciML/ModelingToolkit.jl/issues/4163)) Rational numbers with equivalent values but different representations aren't considered equal. | File | Lines | Description | |------|-------|-------------| | [coupled_equation_crn_systems.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/coupled_equation_crn_systems.jl) | 400 | `2//10` vs equivalent rational | ### Higher-Order Differentials ([MTK #4186](https://github.com/SciML/ModelingToolkit.jl/issues/4186)) Cannot generate systems with higher-order differentials. | File | Lines | Description | |------|-------|-------------| | [coupled_equation_crn_systems.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/coupled_equation_crn_systems.jl) | 563 | Entire test block broken | ### IV Mismatch Error Handling ([MTK #4092](https://github.com/SciML/ModelingToolkit.jl/issues/4092)) Error handling for independent variable mismatch doesn't work as expected. | File | Lines | Description | |------|-------|-------------| | [reactionsystem.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/reactionsystem.jl) | 669 | `@test_throws` as broken | ### save_idxs for JumpProblems `save_idxs` argument doesn't work for jump problems. | File | Lines | Description | |------|-------|-------------| | [mtk_structure_indexing.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/upstream/mtk_structure_indexing.jl) | 215-228 | Various `save_idxs` forms fail | ### Problem Indexing ([MTK #4187](https://github.com/SciML/ModelingToolkit.jl/issues/4187)) | File | Lines | Description | |------|-------|-------------| | [mtk_structure_indexing.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/upstream/mtk_structure_indexing.jl) | 319 | Entire test block broken | ### NonlinearProblem with Vector Variables | File | Lines | Description | |------|-------|-------------| | [mtk_problem_inputs.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/upstream/mtk_problem_inputs.jl) | 342 | Vector variable handling in NonlinearProblem | ### mtkcompile Equation Elimination for SDEs `mtkcompile` doesn't eliminate equations for SDEs like it does for ODEs. | File | Lines | Description | |------|-------|-------------| | [component_based_model_creation.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/compositional_modelling/component_based_model_creation.jl) | 105, 118, 135, 255 | Equation count after simplification | ### Solution Interpolation with Tuple ([SciMLBase #711](https://github.com/SciML/SciMLBase.jl/issues/711)) | File | Lines | Description | |------|-------|-------------| | [mtk_structure_indexing.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/upstream/mtk_structure_indexing.jl) | 249 | `sol(0.0; idxs = (XY,Y))` fails | --- ## Tests Blocked by Symbolics Issues ### Symbolic Simplification ([Symbolics.jl #1739](https://github.com/JuliaSymbolics/Symbolics.jl/issues/1739)) Expressions that should simplify to zero don't always do so. | File | Lines | Description | |------|-------|-------------| | [network_properties.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/network_analysis/network_properties.jl) | 410, 460, 464, 477-478, 485 | Rate equations don't simplify to zero | | [coupled_equation_crn_systems.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/coupled_equation_crn_systems.jl) | 807-810 | Equation equality comparisons | --- ## Conservation Law Issues (NonlinearSystem `remake`) These tests involve `remake` of `NonlinearProblem`s with `remove_conserved = true`. The conservation law parameter Gamma and eliminated species values don't update correctly. | File | Lines | Description | |------|-------|-------------| | [conservation_laws.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/network_analysis/conservation_laws.jl) | 345 | Entire test block - ODE/SDE remake with conservation laws | | [conservation_laws.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/network_analysis/conservation_laws.jl) | 503 | Entire test block - NonlinearSystem warnings and remake | | [conservation_laws.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/network_analysis/conservation_laws.jl) | 550, 555, 560, 565, 596, 597, 601, 602 | Individual assertions within the block | --- ## Catalyst Internal Issues ### Metadata in Completed System (#1353) | File | Lines | Description | |------|-------|-------------| | [reactionsystem.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/reactionsystem.jl) | 982 | `getmetadata` on completed system | ### DSL Independent Variable Test | File | Lines | Description | |------|-------|-------------| | [dsl_options.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/dsl/dsl_options.jl) | 525 | `@reaction_network ivstest` block - MTK bug in collect_vars! | ### Transport Reaction Macro with Interpolation | File | Lines | Description | |------|-------|-------------| | [spatial_reactions.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/spatial_modelling/spatial_reactions.jl) | 105 | `@transport_reaction dZ $species3` syntax | --- ## Extension/Visualization Issues ### BifurcationKit + Conservation Laws Awaiting response from maintainer. | File | Lines | Description | |------|-------|-------------| | [bifurcation_kit.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/extensions/bifurcation_kit.jl) | 20, 152 | Entire test blocks broken | ### Makie Plot Internals Changed The interface for accessing Makie plot internals has changed. | File | Lines | Description | |------|-------|-------------| | [lattice_simulation_plotting.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/extensions/lattice_simulation_plotting.jl) | 36, 45, 76 | Plot value access | ### Latexify Issues Various latexify output differences. | File | Lines | Description | |------|-------|-------------| | [latexify.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/visualisation/latexify.jl) | 169, 190, 198 | LaTeX output mismatches | --- ## Other Issues ### Jump Event Comparison | File | Lines | Description | |------|-------|-------------| | [events.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/events.jl) | 438 | Solution comparison with events in jumps - may need test redesign | ### Rare SDE Failure | File | Lines | Description | |------|-------|-------------| | [simulate_SDEs.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/simulation_and_solving/simulate_SDEs.jl) | 279 | Rare non-reproducible SDE failure | --- ## Previously Broken Tests Now Fixed The following tests from the earlier MTK v10/v11 migration were fixed: - **hybrid_models.jl:244** - Discrete events in hybrid models - **simulate_jumps.jl:195-196** - RSSA dependency graph structure - **reactionsystem.jl:208** - MassActionJump internal structure - **reactionsystem.jl:520** - Jump system with interpolated species - **symbolic_stoichiometry.jl:179** - VariableRateJump assembly - **events.jl:336-339, 363-365** - Solution parameter access after events (MTK #4030) - NOW FIXED - **dsl_options.jl:663** - Integer symtype inference - test removed/was never broken - **simulate_jumps.jl:249,265** - Integer type preservation for jumps (MTK #4170) - FIXED by `c314ed5f7` - **events.jl:145,236** - ReactionSystem equality with events - FIXED by switching to `Catalyst.isequivalent` - **dsl_options.jl:1609,1629** - isequal for systems with events - FIXED by switching to `Catalyst.isequivalent` - **custom_crn_functions.jl:201-203** - Events equality after deepcopy - FIXED by switching to `Catalyst.isequivalent`/`continuous_events_equal`/`discrete_events_equal` - **reactionsystem_serialisation.jl:358,360** - Serialised system equality with events - FIXED by switching to `Catalyst.isequivalent` - **custom_crn_functions.jl:218-219** - Expanded event affects were `nothing` - FIXED by correcting `expand_registered_functions` to properly pass affect eqs, handle `affect_neg`, and preserve `discrete_parameters` --- ## Removed/Commented Out Tests The following tests that were previously `@test_broken` have been fully commented out: - **latexify.jl:178** - Changed to comment, not `@test_broken` - **latexify.jl:279, 282** - Entire test block commented out (extend/compose issues with System+ReactionSystem) --- ## Recommendations ### Medium Priority 1. Investigate conservation law `remake` issues for NonlinearProblems 2. File/track Catalyst #1353 for metadata issue 3. Update Makie plot tests for new internals ### Low Priority 4. Latexify output differences (cosmetic) 5. BifurcationKit integration (awaiting upstream)