[Tests] Migrate from TUnit to MSTest 3 for LLM Compatibility#609
Merged
[Tests] Migrate from TUnit to MSTest 3 for LLM Compatibility#609
Conversation
Infrastructure changes only - test files/attributes not yet migrated. Project file changes: - Replace TUnit 1.13.11 + MSTest.TestFramework 2.1.1 with MSTest 3.8.3 - Enable MSTest runner with source generators (EnableMSTestRunner) - Re-enable net8.0 target (was disabled due to TUnit .NET 9+ requirement) TestCategory.cs: - Change all custom attributes to inherit from TestCategoryBaseAttribute - OpenBugsAttribute, MisalignedAttribute, WindowsOnlyAttribute, etc. - LargeMemoryTestAttribute now returns both OpenBugs and HighMemory categories GlobalUsings.cs: - Replace TUnit.Assertions.Extensions with Microsoft.VisualStudio.TestTools.UnitTesting CI workflow (.github/workflows/build-and-release.yml): - Change from TUnit's --treenode-filter to MSTest's --filter syntax - Add separate test steps for net8.0 and net10.0 - Dynamic WindowsOnly exclusion based on RUNNER_OS Documentation: - Update CLAUDE.md with MSTest filter examples and commands - Replace docs/TUNIT_FILTER_GUIDE.md with docs/MSTEST_FILTER_GUIDE.md - Update np-tests skill with MSTest test patterns Remaining work (not in this commit): - Change [Test] to [TestMethod] in ~276 test files - Add [TestClass] to test classes - Change [Arguments] to [DataRow] (~40 occurrences) - Remove 'using TUnit.Core;' from test files
Attribute conversions:
- [Test] → [TestMethod]
- [Test, ...] → [TestMethod, ...]
- [Arguments(...)] → [DataRow(...)]
- [Category(...)] → [TestCategory(...)]
- [Skip(...)] → [Ignore(...)]
- [Explicit(...)] → [TestCategory("Explicit")] with reason as comment
- [NotInParallel] → [DoNotParallelize]
- [DisplayName(...)] → removed (no MSTest equivalent)
- TUnit.Core.Timeout → Timeout
Files removed:
- TUnitCompatibility.cs (shim layer no longer needed)
- SkipOnNonWindowsAttribute.cs (TUnit-specific conditional skip)
Other changes:
- Removed all 'using TUnit.*' statements
- Removed 'global using TUnit.Core' from GlobalUsings.cs
- Removed TUnit comments from csproj
- Updated filter syntax in comments from TUnit to MSTest
Note: TUnit assertion syntax (await Assert.That(...).IsEqualTo(...))
still needs conversion to AwesomeAssertions in a follow-up commit.
Migrate remaining TUnit-style assertions to MSTest v3 compatible syntax: - Convert `await Assert.That(x).IsEqualTo(y)` → `x.Should().Be(y)` - Convert `await Assert.That(x).IsTrue/IsFalse()` → `x.Should().BeTrue/BeFalse()` - Convert `await Assert.That(x).IsEquivalentTo(y)` → `x.Should().BeEquivalentTo(y)` - Convert `await Assert.That(x).IsNotNull()` → `x.Should().NotBeNull()` - Convert `await Assert.That(() => x).Throws<T>()` → `Assert.ThrowsException<T>(() => x)` - Fix cast precedence: `(type)x.Should()` → `((type)x).Should()` Files modified: 28 test files (~3000 lines) Note: net8.0 has a pre-existing framework-specific issue in NDArray.View.Test.cs unrelated to this migration (method resolution with ToArray<T>().Reverse())
MSTest requires [TestClass] on every test class for test discovery. TUnit did not require this attribute. - Added [TestClass] to 274 test classes - Removed duplicate [TestClass] from OpenBugs partial class Test results: 5560 passed, 36 failed (exception type strictness), 11 skipped
- Convert Assert.ThrowsException<Exception> to Action.Should().Throw<Exception>() MSTest's ThrowsException requires exact type match, while AwesomeAssertions' Throw<T> accepts derived exception types (35 occurrences in 9 files) - Remove CancellationToken parameter from ExtractShape_FromArray test TUnit injected cancellation tokens for timeout support; MSTest uses [Timeout] attribute Test results: 5596 passed, 0 failed, 11 skipped
TUnit used async Assert.That() which required async methods. After migration to MSTest + AwesomeAssertions, these methods no longer await anything. Changes across 28 files: - Removed `using System.Threading.Tasks;` where no longer needed - Changed `public async Task TestName()` to `public void TestName()` - Converted 2 remaining Assert.ThrowsAsync to AwesomeAssertions .Should().Throw() Benefits: - Cleaner code without unnecessary async state machines - ~20KB smaller IL output - Follows MSTest conventions for sync tests
.NET 10 SDK changed the CLI syntax - project paths must now be specified with explicit --project flag instead of positional argument. Error was: "Specifying a project for 'dotnet test' should be via '--project'."
MSTest v3 with EnableMSTestRunner uses the Testing Platform which has different CLI options. Changed --logger "trx" to --report-trx.
.NET 10.0.201 SDK has a protocol version mismatch with MSTest 3.8.3 Testing Platform mode - the SDK looks for protocol version 8 which the current MSTest package doesn't support. This disables Testing Platform mode in CI by passing MSBuild properties: - EnableMSTestRunner=false - TestingPlatformDotnetTestSupport=false Reverts to classic VSTest adapter mode with --logger "trx" instead of --report-trx.
- Build step needs bash shell to parse multi-line commands with backslash continuations (PowerShell fails on Windows) - dotnet test requires --project flag in .NET 10 SDK
dotnet test reads csproj settings regardless of --no-build, so it still used Testing Platform mode. Pass the disable flags to test command as well to force VSTest adapter mode.
global.json enforces Microsoft.Testing.Platform for all tests, which conflicts with CI's need to use VSTest mode (via -p:EnableMSTestRunner=false) to work around .NET 10 SDK protocol version issues. The Testing Platform settings remain in the csproj for local development.
…ibility The MSTest Testing Platform has protocol version issues with .NET 10.0.201 SDK. Instead of complex workarounds, simply use the standard VSTest adapter which works reliably with all SDK versions. Removed: - EnableMSTestRunner from csproj - TestingPlatformDotnetTestSupport from csproj - global.json (removed in previous commit) - Unnecessary -p: flags from workflow
ILKernelGenerator.Unary.Vector.cs doesn't implement Round/Truncate for Vector512, causing failures on AVX-512 capable CI runners: - "Could not find Truncate for Vector512" - "Could not find Round for Vector512" These 7 tests pass on machines without AVX-512 support but fail on GitHub Actions runners which have AVX-512. Marking as OpenBugs until Vector512 support is added. Affected tests: - Trunc_NoParams_Compiles - Round_NoParams_Compiles - Around_NoParams_Compiles - NDArray_Trunc_WorksWithLargeArray - Round_HalfToEven_BankersRounding - Round_NegativeHalf - Round_BankersRounding
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete migration of the test suite from TUnit to MSTest v3, resolving #608.
Key changes:
[Test],[Arguments], etc.) with MSTest equivalents ([TestMethod],[DataRow], etc.)Assert.That()assertions to AwesomeAssertions.Should()fluent syntax[TestClass]attribute to all test classes (required by MSTest)TUnitCompatibility.csshims - now using pure MSTestTest results: 5596 passed, 0 failed, 11 skipped
Changes by commit
4ad699cbac020336[Test]→[TestMethod], etc.b1d1d543Assert.That()→.Should()e0db3c3e[TestClass]to all test classes5a4a4c0a4eea9644Git blame
Large formatting commits are tracked in
.git-blame-ignore-revs. Configure locally:Filter syntax migration
--treenode-filter "/*/*/*/*[Category!=OpenBugs]"--filter "TestCategory!=OpenBugs"--treenode-filter "/*/*/ClassName/*"--filter "ClassName~ClassName"Closes #608