Services with ConditionNeedsUpdate= are still executed at first boot

[septatrix]

mkosi commit the issue has been seen with

main

Used host distribution

Fedora 41

Used target distribution

Arch

Linux kernel version used

6.12.7-200.fc41.x86_64

CPU architectures issue was seen on

x86_64

Unexpected behaviour you saw

On the first boot services which have ConditionNeedsUpdate=/var (or /etc) set are still started even though they are fresh images. This delays the bootup unnecessarily. For images with an immutable /etc this might even result in them being started each boot (as the /etc/.updated file cannot be created) though I did not explicitly test that.

One example of such a service is systemd-journal-catalog-update even though the catalog is freshly installed. systemd-hwdb-update would be a similar candidate, however, it has additional checks which do not trigger because mkosi manually invokes systemd-hwdb --usr. For other services these optimizations do not exist.

mkosi should likely start all services which use ConditionNeedsUpdate= as part of the build process and call /usr/lib/systemd/systemd-update-done at the very end. Doing it this way should be safe as either: The /etc and/or /var directories are included in the image in which case their generated files would also be included, or, they are not included in which case the .updated files would also be absent resulting in them being invoked again as expected.

---

(I noticed there is also another issue if a device reboots from v1 to v2 after v3 is built but not yet installed. In that case the .updated files will have timestamps later than v3's /usr resulting in ConditionNeedsUpdate= not being triggered after a subsequent update from v2 to v3. This, however, is a technical problem in the way these .updated files currently work and not a problem of mkosi.)

Used mkosi config

Default config from this repo

mkosi output

No response

[DaanDeMeyer]

I'll certainly review a PR to run systemd-journal-catalog-update as part of the image build process. I'm not sure we should be calling systemd-update-done though, that seems like something to fix in systemd.

[septatrix]

I'll certainly review a PR to run systemd-journal-catalog-update as part of the image build process.

Just running the catalog update manually won't improve things because it would simply be run again on first boot. This would improve if it had a similar mechanism to hwdb/udev where it does not run if a precompiled catalog exists under /usr. Even if that were implemented there are still several other services which will get invoked on first boot unnecessarily.

I'm not sure we should be calling systemd-update-done though, that seems like something to fix in systemd.

Which way do you envision for fixing this? I though about maybe letting sd-firstboot take care of this?

[septatrix]

Even if that were implemented there are still several other services which will get invoked on first boot unnecessarily.

One example is ldconfig to populate /etc/ld.so.cache. Here it also will not be possible to support the mechanism from hwdb/udev of checking if /usr/lib/ld.so.cache exists because it will be very hard to change to logic of ld.so.

One possible solution could be to add support for a more fine grained needs-update condition on a file basis like ConditionNeedsUpdate=/etc/ld.so.cache on the systemd side and compare that files timestamp with the one from /usr. In that case the service would be responsible for updating the timestamp on that file on its own. The drawback is that we would lose the TIMESTAMP_NSEC fallback for filesystems which do not support NS accuracy.

[septatrix]

I created systemd/systemd#36046 and systemd/systemd#36045 now to track possible improvements/solutions over at systemd