OWD BCD project: increase mdn-bcd-collector coverage to 90%

[queengooborg]

Problem statement

Projects such as #85 and #160 that work to synchronize collector results with BCD are great. However, if the collector only covers a small set of the features BCD tracks, we can never guarantee complete accuracy.

And if we run the collector for new browser releases (#168), then ideally we should be able to update as many BCD data points as possible.

Command to get coverage stats: npm run feature-coverage -- -p api -p css -p html -p javascript -p mathml -p svg -c

Proposed solutions

This project is all about writing new tests to increase the collector's coverage of features tracked in BCD. The goal should be to get above 90% of overall coverage for testable features.

Task list

• Increase test coverage for APIs
• Increase test coverage for CSS
• Increase test coverage for HTML
• Increase test coverage for javascript
• If possible, add tests for MathML
• Increase test coverage for SVG

Progress Report

v10.10.2

api, css, html, javascript, mathml, svg: collector covers 12236 (90.02%) of 13593 entries tracked in bcd (1357 (9.98%) missing)

api: collector covers 8201 (95.15%) of 8619 entries tracked in bcd (418 (4.85%) missing)
css: collector covers 1841 (81.64%) of 2255 entries tracked in bcd (414 (18.36%) missing)
html: collector covers 549 (76.36%) of 719 entries tracked in bcd (170 (23.64%) missing)
javascript: collector covers 1202 (87.42%) of 1375 entries tracked in bcd (173 (12.58%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 414 (83.98%) of 493 entries tracked in bcd (79 (16.02%) missing)

v10.10.0 / v10.10.1

api, css, html, javascript, mathml, svg: collector covers 12202 (89.48%) of 13637 entries tracked in bcd (1435 (10.52%) missing)

api: collector covers 8167 (94.29%) of 8662 entries tracked in bcd (495 (5.71%) missing)
css: collector covers 1841 (81.64%) of 2255 entries tracked in bcd (414 (18.36%) missing)
html: collector covers 549 (76.25%) of 720 entries tracked in bcd (171 (23.75%) missing)
javascript: collector covers 1202 (87.42%) of 1375 entries tracked in bcd (173 (12.58%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 414 (83.98%) of 493 entries tracked in bcd (79 (16.02%) missing)

v10.9.0

api, css, html, javascript, mathml, svg: collector covers 11642 (88.30%) of 13184 entries tracked in bcd (1542 (11.70%) missing)

api: collector covers 8133 (94.34%) of 8621 entries tracked in bcd (488 (5.66%) missing)
css: collector covers 1369 (76.95%) of 1779 entries tracked in bcd (410 (23.05%) missing)
html: collector covers 549 (76.57%) of 717 entries tracked in bcd (168 (23.43%) missing)
javascript: collector covers 1173 (85.43%) of 1373 entries tracked in bcd (200 (14.57%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 389 (69.22%) of 562 entries tracked in bcd (173 (30.78%) missing)

v10.8.0

api, css, html, javascript, mathml, svg: collector covers 11613 (88.08%) of 13184 entries tracked in bcd (1571 (11.92%) missing)

api: collector covers 8128 (94.28%) of 8621 entries tracked in bcd (493 (5.72%) missing)
css: collector covers 1369 (76.95%) of 1779 entries tracked in bcd (410 (23.05%) missing)
html: collector covers 525 (73.22%) of 717 entries tracked in bcd (192 (26.78%) missing)
javascript: collector covers 1173 (85.43%) of 1373 entries tracked in bcd (200 (14.57%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 389 (69.22%) of 562 entries tracked in bcd (173 (30.78%) missing)

v10.7.2

api, css, html, javascript, mathml, svg: collector covers 11381 (84.83%) of 13416 entries tracked in bcd (2035 (15.17%) missing)

api: collector covers 8110 (92.06%) of 8809 entries tracked in bcd (699 (7.94%) missing)
css: collector covers 1174 (64.65%) of 1816 entries tracked in bcd (642 (35.35%) missing)
html: collector covers 524 (73.39%) of 714 entries tracked in bcd (190 (26.61%) missing)
javascript: collector covers 1155 (83.51%) of 1383 entries tracked in bcd (228 (16.49%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 389 (69.22%) of 562 entries tracked in bcd (173 (30.78%) missing)

v10.7.1

2227/13378 (16.65%) entries missing from collector that are in bcd for api, css, html, javascript, mathml, svg

727/8777 (8.28%) entries missing from collector that are in bcd for api
660/1810 (36.46%) entries missing from collector that are in bcd for css
190/715 (26.57%) entries missing from collector that are in bcd for html
347/1384 (25.07%) entries missing from collector that are in bcd for javascript
132/132 (100.00%) entries missing from collector that are in bcd for mathml
171/560 (30.54%) entries missing from collector that are in bcd for svg

v10.7.0

all: 2324/13356 (17.40%) entries missing from collector that are in bcd for api, css, html, javascript, mathml, svg

api: 724/8758 (8.27%) entries missing from collector that are in bcd
css: 670/1811 (37.00%) entries missing from collector that are in bcd
html: 257/711 (36.15%) entries missing from collector that are in bcd
javascript: 370/1384 (26.73%) entries missing from collector that are in bcd
mathml: 132/132 (100.00%) entries missing from collector that are in bcd
svg: 171/560 (30.54%) entries missing from collector that are in bcd

[queengooborg]

As of mdn-bcd-collector v10.10.2, we have now reached over 90% feature coverage! 🎉

[Elchi3]

@queengooborg can we update the feature coverage script to list untestable features along with the other stats?

So something like:

api, css, html, javascript, mathml, svg: collector covers 12236 (90.02%) of 13593 entries, 1357 (9.98%) missing, and XXX (x.xx%) are untestable.

[queengooborg]

I can update the script, though I'd have to change the output format (untestable features are omitted from the stats)!

[Elchi3]

We've finished this project. Thanks again to funding from the Sovereign Tech Fund making this work possible.

From our final report:

Deliverable 2: Increase the coverage of the mdn-bcd-collector to 90% (completed)

OWD’s mdn-bcd-collector currently monitors 90.15% of all web platform features recorded in browser-compat-data. At the beginning of the second round of the challenge, our compatibility data collector monitored 82% of all web platform features in the browser-compat-data project. We’ve surpassed the 90% coverage deliverable, by expanding the scope of our test suite. Notably, the collector now includes tests for all CSS property values in addition to CSS properties.

While the ultimate goal would be 100% coverage, we discovered some features (2.27% of all features) that can not be covered. Unfortunately, there are a limited set of features that can not be tested programmatically, either because they require specific hardware or they require user interactions that are not machine replicable. We created a list of untestable features that includes the features limiting factors. We will be addressing the remaining (7.58%) features in the near future; including global events and worker support for API interfaces.