Performance Run-Alvalidation

[RonKoppelaar]

Not sure when this started, but I see a huge performance drop on the ALValidation
We run the validation against bc24.5 (latest container).

Validating an app which in vscode takes couple of seconds takes minutes...

I'm running this in a build pipeline on a hosted agent with very strong power...

AMD Ryzen 9 7950X3D 16-Core Processor 

Base speed:	4,20 GHz
Sockets:	1
Cores:	16
Logical processors:	32
Virtualization:	Enabled
L1 cache:	1,0 MB
L2 cache:	16,0 MB
L3 cache:	192 MB

    192Gb RAM...

On my local laptop I don't see this happening...
Other builds (and container creation ) are lightning fast

Parameters for run-alvalidation:
Container name RunAlValidation-59
Credential admin/Bara4671
MemoryLimit 8G
validateVersion 24
validateCurrent False
validateNextMinor False
validateNextMajor False
countries nl
validateCountries nl
affixes EMP,ADLSE
supportedCountries
ObsoleteTagMinAllowedMajorMinor
vsixFile
Install Apps

• None

Previous Apps

• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_Empire Foundation_24.2410.11745.0.app
• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_Empire System Application_24.2410.11745.0.app
• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_Empire Table APIs_24.2410.11745.0.app
• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_EmpireFinance_24.2410.11745.0.app
• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_EmpireHousing_24.2410.11745.0.app
• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_EmpireOpen_24.2410.11745.0.app
• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_EmpirePageExtension_24.2410.11745.0.app
• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_EmpireProfiles_24.2410.11745.0.app
• C:\agent2_work\6\a\PreviousApps\zig365-operations\Zig_EmpireRealEstate_24.2410.11745.0.app
  Apps
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_Empire Foundation_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_Empire System Application_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_Empire Table APIs_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_EmpireFinance_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_EmpireHousing_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_EmpireOpen_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_EmpirePageExtension_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_EmpireProfiles_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_EmpireRealEstate_24.2412.12656.0.app
• C:\agent2_work\6\a\CurrentApps\zig365-operations\Zig_Zig365 Operations_24.2412.12656.0.app

I'll add logfile from azure devops when its completed

[freddydk]

Isn't the performance drop simply from finding and downloading artifacts?

[RonKoppelaar]

No. I think it started when I switched to W22, using local system account on non-azure machine. I’ll back port to azure and w19 to see where the drop started. Also have a w22 azure machine.

[RonKoppelaar]

It seems to be Windows 22 related. Both servers (Azure / non-Azure) are slow. I’m currently running the same build on W22 and W19. On W19 I see compile times similar to the normal build. The first app inline takes 55secs. On W22, the same app takes 280 secs. Main difference between the tow machines: * On W19 the agents run as a local admin user. Every agent on the server has its own user * PS Modules are installed in the user scope. * On W22, agents run as local system * PSModules are save inside the Agent folder * PSModule path changed so it can find the modules. I’ll also test on the W22 machine as a normal (admin) user and see if it has same results

[RonKoppelaar]

W19.log
W22.log
time diff between AppSource validation (9 apps):
W19: 22min.
W22: 1H:38min.

Please find attached detailed logs of both

Also running it in ISE on W22 (with administrator account) compile of the first app took: 275secs..
Just ran this:

$Start = Get-date
Run-AlValidation -containerName test `
 -apps "C:\Temp\Zig_Empire System Application_24.2412.12635.0.app" `
 -countries "NL" `
 -includeWarnings:$false `
 -validateVersion "24" `
 -affixes "EMP" 

 (Get-Date)-$Start

So my first conclusion is that run-alValidation is slow in all cases when running on W22. Tested on Azure and non-Azure machine.
Local environment (W11) also slow... : 250secs. to compile first app.

[RonKoppelaar]

Freddy, I think found the issue on W22, its related to PS7 sessions. If I set the parameter:
$bcContainerHelperConfig.usePsSession = $false

All works with same performance. Any clue?

Also on W2019, the performance is faster using the $bcContainerHelperConfig.usePsSession = $false
But the total penalty is less: 93secs vs 123secs to validate one app

On Windows22 its: 41secs vs 551secs!

[RonKoppelaar]

After implementing some improvements in pipeline and using the UsePSSession switch the overall performance increased dramatically.
See attached screenshots of 4 builds and their total duration time:

Going from bottom to top:
#204929.1 - Original build W22 ==> 2h:27m
#204848.6 - Improved code on our end to not validate apps twice (There are multiple offerings in this build) ==> 1h:42m
#204848.7 - Running the improved code on W19 ==> 22m:15s
#204848.8 - Running the improved code on W22 with UsePSSession = false ==> 9m:11s

[RonKoppelaar]

So with this workaround (UsePSSession=false) my problem is solved, but I do think there's a small bug somewhere...