You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In #805 I once proposed to test the nightly stage on the master branch of all dependencies, it could help catch some errors before releasing a new version without requiring new CI resources. It turned out that the proposal isn't really useful in practice if we choose to tag our versions frequently.
That's not good news for both users and developers. Code reviews can help catch some of them but to guarantee none of them passes silently, we might need a really thorough test on it.
The plan is simple:
generating all possible Manifest.tomls for all versions of dependencies we claim compatible
test all of them using CI
The only and really big challenge is we don't have enough CI resources to support this.
There could be thousands of possible combinations if we literally test all of them and it's impractical, hence we need to take some strategy to reduce the number, for example:
versions that older than three months are not under consideration
if there're many patch versions, only test the latest, e.g., test v0.8.7 and ignore v0.8.0-v0.8.6.
only trigger the CI when we trigger the registrator
I need to do some statistics on this strategy and see if it reduces the number to a practical one.
Right now, this isn't an urgent issue, and it might not be the best time to tackle this.
The text was updated successfully, but these errors were encountered:
johnnychen94
changed the title
[RFC] Add a thorough test on **all** compatible versions
Add a thorough test on **all** compatible versions
Mar 28, 2021
In #805 I once proposed to test the nightly stage on the master branch of all dependencies, it could help catch some errors before releasing a new version without requiring new CI resources. It turned out that the proposal isn't really useful in practice if we choose to tag our versions frequently.
Recent devs showed that many bugs are due to incompatible versions that we claim compatible. E.g., JuliaImages/ImageCore.jl#110, JuliaGraphics/Colors.jl#390, JuliaGraphics/Colors.jl#383
That's not good news for both users and developers. Code reviews can help catch some of them but to guarantee none of them passes silently, we might need a really thorough test on it.
The plan is simple:
Manifest.toml
s for all versions of dependencies we claim compatibleThe only and really big challenge is we don't have enough CI resources to support this.
There could be thousands of possible combinations if we literally test all of them and it's impractical, hence we need to take some strategy to reduce the number, for example:
I need to do some statistics on this strategy and see if it reduces the number to a practical one.
Right now, this isn't an urgent issue, and it might not be the best time to tackle this.
The text was updated successfully, but these errors were encountered: