[Bug]: Poor performance of Vue component tests in 1.49.x version

[pbrzosko]

Version

1.49.1

Steps to reproduce

1. Clone repo https://github.com/pbrzosko/playwright-lock-issue
2. npm i
3. npm run test-ct (tests executed with version 1.48.0)
4. npm i @playwright/experimental-ct-vue@latest
5. npm run test-ct (tests executed with version 1.49.x)

Expected behavior

Expect to see at least same running time in newer version.

Actual behavior

Running time in 1.49.x version is bigger by 50%. This has a significant impact when you run a lot of tests and is currently blocking us from upgrading.

Additional context

1.48.0

Running 21 tests using 8 workers

✓ 1 [chromium] › TestComponent.spec.ts:33:7 › TestComponent.vue › Should fill in form 4 (859ms)
✓ 2 [chromium] › TestComponent.spec.ts:21:7 › TestComponent.vue › Should fill in form 1 (776ms)
✓ 3 [chromium] › TestComponent.spec.ts:17:7 › TestComponent.vue › Should fill in form (860ms)
✓ 4 [chromium] › TestComponent.spec.ts:41:7 › TestComponent.vue › Should fill in form 6 (756ms)
✓ 5 [chromium] › TestComponent.spec.ts:25:7 › TestComponent.vue › Should fill in form 2 (929ms)
✓ 6 [chromium] › TestComponent.spec.ts:37:7 › TestComponent.vue › Should fill in form 5 (951ms)
✓ 7 [chromium] › TestComponent.spec.ts:45:7 › TestComponent.vue › Should fill in form 7 (854ms)
✓ 8 [chromium] › TestComponent.spec.ts:29:7 › TestComponent.vue › Should fill in form 3 (844ms)
✓ 9 [chromium] › TestComponent.spec.ts:49:7 › TestComponent.vue › Should fill in form 8 (492ms)
✓ 10 [chromium] › TestComponent.spec.ts:53:7 › TestComponent.vue › Should fill in form 9 (476ms)
✓ 11 [chromium] › TestComponent.spec.ts:57:7 › TestComponent.vue › Should fill in form 10 (463ms)
✓ 12 [chromium] › TestComponent.spec.ts:61:7 › TestComponent.vue › Should fill in form 11 (455ms)
✓ 13 [chromium] › TestComponent.spec.ts:65:7 › TestComponent.vue › Should fill in form 12 (453ms)
✓ 14 [chromium] › TestComponent.spec.ts:69:7 › TestComponent.vue › Should fill in form 13 (472ms)
✓ 15 [chromium] › TestComponent.spec.ts:73:7 › TestComponent.vue › Should fill in form 14 (462ms)
✓ 16 [chromium] › TestComponent.spec.ts:77:7 › TestComponent.vue › Should fill in form 15 (429ms)
✓ 17 [chromium] › TestComponent.spec.ts:81:7 › TestComponent.vue › Should fill in form 16 (424ms)
✓ 18 [chromium] › TestComponent.spec.ts:85:7 › TestComponent.vue › Should fill in form 17 (443ms)
✓ 19 [chromium] › TestComponent.spec.ts:89:7 › TestComponent.vue › Should fill in form 18 (424ms)
✓ 20 [chromium] › TestComponent.spec.ts:93:7 › TestComponent.vue › Should fill in form 19 (412ms)
✓ 21 [chromium] › TestComponent.spec.ts:97:7 › TestComponent.vue › Should fill in form 20 (426ms)

21 passed (3.2s)

1.49.x

Running 21 tests using 8 workers

✓ 1 [chromium] › TestComponent.spec.ts:37:7 › TestComponent.vue › Should fill in form 5 (1.5s)
✓ 2 [chromium] › TestComponent.spec.ts:25:7 › TestComponent.vue › Should fill in form 2 (1.5s)
✓ 3 [chromium] › TestComponent.spec.ts:45:7 › TestComponent.vue › Should fill in form 7 (1.6s)
✓ 4 [chromium] › TestComponent.spec.ts:17:7 › TestComponent.vue › Should fill in form (1.5s)
✓ 5 [chromium] › TestComponent.spec.ts:33:7 › TestComponent.vue › Should fill in form 4 (1.5s)
✓ 6 [chromium] › TestComponent.spec.ts:41:7 › TestComponent.vue › Should fill in form 6 (1.5s)
✓ 7 [chromium] › TestComponent.spec.ts:29:7 › TestComponent.vue › Should fill in form 3 (1.5s)
✓ 8 [chromium] › TestComponent.spec.ts:21:7 › TestComponent.vue › Should fill in form 1 (1.5s)
✓ 9 [chromium] › TestComponent.spec.ts:49:7 › TestComponent.vue › Should fill in form 8 (450ms)
✓ 10 [chromium] › TestComponent.spec.ts:53:7 › TestComponent.vue › Should fill in form 9 (444ms)
✓ 11 [chromium] › TestComponent.spec.ts:57:7 › TestComponent.vue › Should fill in form 10 (441ms)
✓ 12 [chromium] › TestComponent.spec.ts:61:7 › TestComponent.vue › Should fill in form 11 (440ms)
✓ 13 [chromium] › TestComponent.spec.ts:65:7 › TestComponent.vue › Should fill in form 12 (443ms)
✓ 14 [chromium] › TestComponent.spec.ts:69:7 › TestComponent.vue › Should fill in form 13 (435ms)
✓ 15 [chromium] › TestComponent.spec.ts:73:7 › TestComponent.vue › Should fill in form 14 (468ms)
✓ 16 [chromium] › TestComponent.spec.ts:77:7 › TestComponent.vue › Should fill in form 15 (432ms)
✓ 17 [chromium] › TestComponent.spec.ts:81:7 › TestComponent.vue › Should fill in form 16 (445ms)
✓ 18 [chromium] › TestComponent.spec.ts:85:7 › TestComponent.vue › Should fill in form 17 (426ms)
✓ 19 [chromium] › TestComponent.spec.ts:89:7 › TestComponent.vue › Should fill in form 18 (429ms)
✓ 20 [chromium] › TestComponent.spec.ts:93:7 › TestComponent.vue › Should fill in form 19 (427ms)
✓ 21 [chromium] › TestComponent.spec.ts:97:7 › TestComponent.vue › Should fill in form 20 (427ms)

21 passed (4.8s)

Environment

System:
    OS: macOS 15.2
    CPU: (10) arm64 Apple M2 Pro
    Memory: 250.13 MB / 32.00 GB
  Binaries:
    Node: 20.14.0 - /usr/local/bin/node
    npm: 10.8.3 - /usr/local/bin/npm
  Languages:
    Bash: 3.2.57 - /bin/bash

[Skn0tt]

These numbers don't reproduce on my machine. I'm running npm run test-ct -- --workers=1 --repeat-each=10, so i'm making it serial execution and repeating it a couple times to reduce variance. My results are 1.5min on both v1.49.1 and v1.48.0.

Could you try running the same on your end, to see if it's a matter of variance? Both of your test runs concluded within seconds, so on this repro it might just be randomness.

You mention this is blocking you from upgrading. When you first saw that 1.49.1 slows down your execution, how often did you repeat that measurement? Maybe it the CI server was packed and slowed down the execution. For context, we haven't received any related slowdown reports for this version yet.

[pbrzosko]

No, it was my pc. I did multiple runs on this small repro repository, and indeed the average is the same on 1.48 and 1.49. Although I saw a big difference when I executed our tests. Let me check the average for our tests and I will get back to you.