Parallel compilation is broken with GNU Make 4.4 #375
Comments
|
I think the best way would be to simply update servo's rust version. There is already PR for that: servo/servo#29993 |
|
Yep, I've called that out in the PR in servo to pin GNU Make :) |
|
It seems moving to Rust 1.74 was not enough to get make 4.4 working well :( When building mozjs_sys as part of servo/servo@3555671 with nixpkgs pinned to nixos-23.05 (NixOS/nixpkgs@70bdade), the build script takes 634 seconds (cargo-timing-20240110T071341Z.html.gz). When gnumake is downgraded from 4.4.1 to 4.3 (NixOS/nixpkgs@6adf48f), the build script takes only 71 seconds on the same 16-core 32-thread machine (cargo-timing-20240110T074057Z.html.gz). diff --git a/etc/shell.nix b/etc/shell.nix
index 306b25acb9..81deee77fe 100644
--- a/etc/shell.nix
+++ b/etc/shell.nix
@@ -2,7 +2,7 @@
# NOTE: This does not work offline or for nix-build
with import (builtins.fetchTarball {
- url = "https://github.com/NixOS/nixpkgs/archive/6adf48f53d819a7b6e15672817fa1e78e5f4e84f.tar.gz";
+ url = "https://github.com/NixOS/nixpkgs/archive/70bdadeb94ffc8806c0570eb5c2695ad29f0e421.tar.gz";
}) {
overlays = [
(import (builtins.fetchTarball {
@@ -17,6 +17,9 @@ let
cargo = rustToolchain;
rustc = rustToolchain;
};
+ pkgs_gnumake_4_3 = import (builtins.fetchTarball {
+ url = "https://github.com/NixOS/nixpkgs/archive/6adf48f53d819a7b6e15672817fa1e78e5f4e84f.tar.gz";
+ }) {};
in
clangStdenv.mkDerivation rec {
name = "servo-env";
@@ -42,7 +45,7 @@ clangStdenv.mkDerivation rec {
# functionality in mozjs and causes builds to be extremely
# slow as it behaves as if -j1 was passed.
# See https://github.com/servo/mozjs/issues/375
- gnumake
+ pkgs_gnumake_4_3.gnumake
# crown needs to be in our Cargo workspace so we can test it with `mach test`. This means its
# dependency tree is listed in the main Cargo.lock, making it awkward to build with Nix because |
|
Could this be problematic? mozjs/mozjs-sys/makefile.cargo Line 12 in 7184f65 |
|
I printed the value of It looks like cargo for now only supports inheriting and forwarding named fifo arguments when invoked under make but new jobservers that a top-level cargo itself creates will still use pipes: From https://docs.rs/jobserver/0.1.27/jobserver/
|
Not familiar with the mozjs build system so maybe I’m doing it wrong, but I tried other values of --build-backends that I saw on searchfox and couldn’t get them working. With With @mukilan, it looks like that followup PR hasn’t happened yet. Can the upstream Makefile be fixed, or can we run a fifo-based jobserver with jobslot? |
I tried a couple of approaches to fix the makefile back when I was investigating this issue, but I didn't find any working solutions. Admittedly my knowledge of Make and SM build system is also limited, so in theory a fix to the Makefile might be still be possible.
That's an interesting idea! I wasn't aware of jobslot. For the whole Servo build to use the same job server, we'd have to build a binary that runs the jobserver and spawns cargo with correct CARGO_MAKEFLAGS (during `./mach build). But since make already has the jobserver functionality, it might be simpler be to wrap the cargo invocation in a dummy GNU Make invocation in a way that forwards all arguments to cargo. But these approaches seem less than ideal to me as they add more moving parts to the build system. I guess pushing for/contributing to upstream fix (either in SM or cargo adopting jobslot) and keeping make pinned via nix seems better, now that we have the ability to use nix on other distros with your recent PR! |
GNU Make 4.4 breaks the parallel builds in mozjs and behaves as though
-j1was given, leading to long build times.Relevant logs from
cargo build -vv:Th release notes for GNU Make 4.4 and the linked bugs note that when using 'old-style' anonymous pipes for jobserver, the file descriptors are closed when parent make invokes a recursive sub-make when the sub-make call is not direct
SpiderMonkey's Makefile seem to implement recursive invocations that are not direct $(MAKE) invocations, so these sub-makes fail to open the closed jobserver pipes and fallback to -j1 mode.
This issue can be fixed by using a version of Cargo that has support for the new 'named fifo' based jobserver as GNU Make 4.4 introduced them to allow any process to integrate with the jobserver and thus should work with both direct and indirect sub-make invocations.
This affects Servo (on distros that ship GNU Make 4.4 such as Fedora 39, Arch, NixOS 23.05 etc) since it currently uses 2023-02-01 build of rust toolchain where the Cargo doesn't have support for named fifo jobserver.
The text was updated successfully, but these errors were encountered: