[BUG]: VSTest@2 is passing while there are reported test failures when using batching

[Youssef1313]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

No response

Task version

No response

Issue Description

https://dev.azure.com/uno-platform/Uno%20Platform/_build/results?buildId=152367&view=logs&j=c814f4c4-6968-579a-5adb-420b4397838c&t=fad2c059-e56b-516e-11ce-011393560ef1

YML:

  - task: VSTest@2
    inputs:
      testAssemblyVer2: |
        **\*test*.dll
        !**\obj\**
        !**\*Wasm.Test*.dll
        !**\*UITests.dll
        !**\*.RuntimeTests.dll
        !**\Uno.UI.Tests.ViewLibraryProps.dll
        !**\Uno.UI.Tests.ViewLibrary.dll
        !**\testhost*.*
        !**\Microsoft*.dll
      vsTestVersion: toolsInstaller
      testRunTitle: $(Agent.JobName)
      testSelector: testAssemblies
      batchingBasedOnAgentsOption: customBatchSize
      rerunFailedTests: 'true'
      customBatchSizeValue: 200 # test count / 10 (https://developercommunity.visualstudio.com/content/problem/891803/vstestconsoleadapter-fails-with-outofmemory.html?childToView=896206#comment-896206)

Logs show:

Failed    Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService.HR(System.String,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Scenario,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Project[])    HR (When_Simple_Xaml_Add_xBind_Function_Property_Add,Debug,NetCore,XR:False,)
Failed    Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService.HR(System.String,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Scenario,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Project[])    HR (When_Simple_Xaml_Add_xBind_Function_Property_Add,Debug,MonoVM,XR:False,)
Failed    Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService.HR(System.String,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Scenario,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Project[])    HR (When_Simple_Xaml_Add_xBind_Function_Property_Add,Debug,NetCore,XR:True,)
Failed    Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService.HR(System.String,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Scenario,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Project[])    HR (When_Simple_Xaml_Add_xBind_Function_Property_Add,Debug,MonoVM,XR:True,)

Looking at the run via https://dev.azure.com/uno-platform/Uno%20Platform/_TestManagement/Runs#_a=resultQuery&runId=3201282&queryPath=Recent+Run%2FRun3201282 doesn't show the failure. However, downloaded the relevant trx from the attachments on the same link shows the failure. The end result is that the job is passing when there are test failures.

The job will fail correctly if batching is not used (i.e, delete batchingBasedOnAgentsOption and customBatchSizeValue from YML)

FYI @jeromelaban @MartinZikmund. Though you are no longer impacted as Uno is now using Microsoft.Testing.Platform with the proper dotnet test support.

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Relevant log output

-

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps