Compiling node 23.1.0 from Homebrew's source, when 23.1.0_1 is current — a conflict?

[GwynethLlewelyn]

Output of brew config

HOMEBREW_VERSION: 4.4.4-76-g40f4ab2
ORIGIN: https://github.com/Homebrew/brew
HEAD: 40f4ab25468bce1640b5ec7ff34fed271cad6e4f
Last commit: 12 hours ago
Core tap HEAD: 8882e044fcaf5030618028b759e23bceb563c2da
Core tap last commit: 3 weeks ago
Core tap JSON: 10 Nov 21:02 UTC
Core cask tap HEAD: 02906511a584a468911b491f18079ed18965cf6a
Core cask tap last commit: 3 weeks ago
Core cask tap JSON: 10 Nov 21:03 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.pEy9xd0jj5/org.xquartz:0
HOMEBREW_EDITOR: /usr/local/bin/jmacs
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
http_proxy: http://10.69.69.250:3128/
https_proxy: http://10.69.69.250:3128/
no_proxy: localhost,127.0.0.1,localaddress,.localdomain.com
Homebrew Ruby: 3.3.6 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.6/bin/ruby
CPU: octa-core 64-bit haswell
Clang: 13.0.0 build 1300
Git: 2.47.0 => /usr/local/bin/git
Curl: 8.1.2 => /usr/bin/curl
macOS: 11.7.9-x86_64
CLT: 13.2.0.0.1.1638488800
Xcode: 13.2.1

Output of brew doctor

You really, really, really don't want to see that...

Description of issue

Hi! Before writing this, let me tell you that I have both my bulletproof vest and my fireproof NBC suit on, so flame away if you wish :)

TL;DR: Cannot get Node.js 23.1.0**_1** properly compiled from the sources? Have no fear, allow Node.js 23.1.0 to be compiled instead (this is what the existing formula does anyway), and, after installation and post-installation, just add a symlink for 23.1.0_1, pointing to 23.1.0. That's enough to make Homebrew happy!

Long version follows...

(Estimated reading time: 8m30s)

I'm part of the 95% of Mac users who cannot go beyond macOS Big Sur. Aye, 95%, according to Statista, which produces impeccable statistics with all sources fully documented (the 95% include Catalina and Big Sur, which gets — erroneously — reported by Apple as "Catalina" as well). This is my personal choice, of course, but any forceful upgrade (it can be done) comes at the price of lacking Nvidia card support, and, for the kind of things I use my vintage Mac for, the Nvidia card is not merely a "nice thing to have", but an essential bit of hardware which I cannot live without. macOS Big Sur is the last Apple OS which supports Nvidia cards, so, I get no option here.

That said...

It means that there are no more bottled formulae for my ancient, used-by-95%-of-Mac-users-worldwide-but-who-cares macOS version, and everything must be compiled from the sources.

I don't mind it in the least, even taking into account that it takes eternities (I can always do it overnight). It's just because I was more than used to do the same under FreeBSD using their ports tool, and did that for a decade or so. Now if I have to do it on macOS, sure, why not, macOS is a close cousin to FreeBSD anyway.

What I do mind, of course, is when the source code is deliberately broken because, well, those 5% of lucky macOS users who can run more recent versions of macOS couldn't care less if compiling from source works or not. Again, that's ok. We got ample warning — and still get it every day we attempt to do a brew upgrade — so it's not as we are kept in the dark about walking on the edge, without safety net, and where Things Get Broken™.

Usually, however, almost everything is simple enough to fix, and requires little less than using --keep-tmp to keep the compiled sources around, tweak with a parameter or two in the configuration, make install (or whatever the command might be), and that's it — in 99.99% of the cases, such workarounds will work fine. It's been quite a while since something was so broken that I couldn0t find a way to get it properly compiled; and even in those cases (OpenSSL comes to mind...) it was just a question of waiting a few releases — eventually more recent versions will fix whatever broke earlier ones during compilation, and things will run smoothly again.

One common culprit is the node formula — which, as you know, is one of those core, central packages, which is at the root of hundreds, if not thousands, of dependencies (I never bothered to count them). In fact, I would claim that Node.js is the main framework these days that must be around in some form that is usable by everything else in the system. If Node.js breaks down, the system is next-to-unusable. I'm being overly dramatic here — you can always use a locally installed version of Node.js for your personal projects, of course — but I'm sure you understand where I'm coming from.

Because Node.js is so crucial, it's also quite sensitive to the infinity of compilation parameters that it must have, or else, at some point, something will break down. Even a successful compilation is not necessarily 'good' — if it fails to address certain issues that only happen in some obscure dependent packages, which, however, seem to be more popular than one might imagine. Thus, Node.js must be compiled flawlessly according to Homebrew's mandatory configuration parameters — that's the only way to make sure that everything works as intended.

Node.js also releases two main branches, a bleeding edge version (currently 23.1.0), and a long-term stable version (currently 22.10.something). As it is traditional with Homebrew (and, mind you, I'm totally in favour of that approach!), anything "bleeding edge" is Cool™. In fact, I usually know well in advance when something has just been released, because, among all package managers I use across several different flavours of Unix and Unix-like OSes, Homebrew is almost invariably the first to have an update (until last year or so, the sole exception had been the team updating the Go programming language, which would lag a bit behind the official release; but this is currently not the case any more).

That said... we are merely humans, and humans make mistakes. Fortunately, Homebrew, although it's not "mistake-proof" (nothing is), can handle most hiccups very elegantly.

There was a hiccup with Node.js 23.1.0 recently. It's not fixed yet, but it's annoying. There is a workaround, which I will gladly share with the 95% fellow macOS-upgrade-impaired users who have no other choice but to hope that everything compiles well under the source.

In a short interval, a lot of Node.js releases came out in quick succession (well, comparatively speaking), and with Node 23 officially released, almost all dependents were immediately switched to that instead (as expected!). On old Macs, Node.js is painfully slow in compiling, and it just happened that when I finally managed to get a version to finish compilation, a new one was just around the corner, and it was time for a new round of compilations. Those 5% who are lucky to be able to use the bottled versions would not be much affected — even if they just upgrade once per day, that's just a minute or two to get the brand new version upgraded, and if you get a new one the next day, hooray, who cares anyway, so long as it gets properly updated, right?

Well, 23.1.0 came out, and while it didn't compile straight from the sources (there was just one missing parameter on the ./compile script, which was not very hard to track down and fix), you could use --keep-tmp, add the missing parameter, compile everything again, run brew postinstall node, and that was it — Node 23.1.0 would get in place and fully operational, and you could now spend your time in the endless compilations of all other dependents.

But at some point someone noticed that the bottled version has a glitch. I don't know what it was, or how it was fixed, or why this only affected the bottled version (a rare case where the self-compiled formula would actually have no issues!), but a new fix followed quite quickly: it changed only the slightest bits of the Ruby formula (as far as I can tell from the history), and, obviously, this required an appropriate bump to a new version, called... 23.1.0_1 — since the real 23.1.1 isn't out (yet).

This was more than enough for Homebrew to detect that 23.1.0 was now marked as being 'obsolete' and the 23.1.0_1 should be used as a bottled replacement — all fine, mind you, if you used a bottled version. However, this is not the case for the source-compiling crowd: the formula doesn't really "compile" version 23.1.0_1, because no such Node.js version exists. It's just an internal "patch", so to speak, to allow a re-bottled version to get distributed. When running the formula for 23.1.0_1, what will be downloaded is the source for Node.js 23.1.0 — because that's all that exists on GitHub. It gets placed in the 23.1.0 directory for node. And it is, for all purposes, the 23.1.0 version (since no other more recent Node.js version exists yet).

What that means is that, when Node.js finally gets into its proper place, brew outdated will show that there is supposed to be a newer version of Node.js, called 23.1.0_1, and promptly follows the formula... and downloads the source for 23.1.0 again (what else should it download, anyway? After all, that's the only version that exists). And then gets flagged as outdated, and everything starts from scratch again, and so forth...

Now, I admit that I'm not a very bright person, and it took me the better part of two weeks to figure out why Homebrew persisted in compiling the same source code, over and over again, and always complaining that it wasn't the "latest and greatest" version. I should have noticed that the issue is in the formula itself: it can only download what actually exists on GitHub. It just happens to be mismatched with what Homebrew thinks it's the latest version.

I'm fully aware that no amount of pleading will reach the ear of a friendly code maintainer who, taking pity of those 95% of frustrated Mac users who cannot use Node.js and any dependent package, would be so kind as to add two characters somewhere in the package (namely, _1...) so that the compilation will, indeed, reflect the latest and greatest version as counted by Homebrew.

Therefore, all I did was to cd /usr/local/Cellar/node and create a symlink, ln -s 23.1.0 23.1.0_1.

That's all it takes. This tricks Homebrew in believing that there is a formula called 23.1.0_1, so it satisfies the dependency tree. Whatever 'outdated' packages exist that depend on Node.js will never notice the difference — they will go ahead and purr happily along, since, from their perspective, there is no difference between what Homebrew calls the "latest" version and what actually is downloaded. Whatever Node.js 23.1.0 is called — you could even call it 666.6.6, for all it matters — it is still 23.1.0.

A better approach, of course, would be to change the directory prefix during the configuration stage to correctly point to /usr/local/Cellar/node/23.1.0_1 instead. The only reason I did not do that was because I didn't know if that wouldn't break a few links here and there — again, I didn't check. The 'safest' way is just to have an additional "fake" symlink. Unless Homebrew changes its version checking overnight, this should be good enough (there is no problem of having several different versions side-by-side — think Python!), until Node.js 23.1.1 is officially released, which will fix everything again.

There is a slight risk that an aggressive brew cleanup might delete /usr/local/Cellar/node/23.1.0 by mistake, wrongly assuming that this is an old, outdated version, and that only /usr/local/Cellar/node/23.1.0_1 should be kept around — but once 23.1.0 is deleted, 23.1.0_1 will just be a broken symlink with no content whatsoever, so everything will collapse. Tough! We know we live on the edge. Again, recompiling everything but using /usr/local/Cellar/node/23.1.0_1 as the intended installation destination might fix things, but I haven't (yet) tried it out myself. There is no good reason why it shouldn't work, though.

How do other package managers handle this issue?

I'm just familiar with a very small number of package managing systems, Ubuntu/Debian's apt being one of those. What these package managers do — as well all derivatives, or systems directly inspired by apt — is to create what they call a 'virtual release'. This is a pseudo-release version which doesn't contain any files — it is deliberately kept empty! — and only serves as a stub for the actuallyeee existing packages. In a sense, this is like a "placeholder" instruction to the package manager, telling it that it should assume that a certain package is installed under some version, but do not trust that to be the case (and never assume anything!). This "workaround" has its own set of issues, too, and makes package management a tad more difficult, which is probably why it's not so popular, specially among self-compiled packages. This is especially true of those cases where there are really very substantial changes since the "old" versions were released...

Anyway, please ignore my rant. 5% of you guys, after all, won't have the slightest issue with this "internal" release of Node.js. It's just for the 95% of us that this discussion might be useful for...

[SMillerDev]

So you're saying that the roughly 16 million installs (together 80% of Homebrew usage) on supported macOS versions somehow account for 5% of all macOS users?

Man that's a lot of macOS users. I think it's much more likely that your source is bad at tracking macOS versions since Apple stopped updating the user agent.

[p-linnane]

As @SMillerDev stated, you are making a number of incorrect assumptions here. Let me also recommend you drop all the snark about "95% of macOS users are on Big Sur or older", when such a claim can be debunked with some simple common sense. This project would not be running if 95% of the user base was on unsupported versions of macOS. macOS Catalina (10.15.7) is the last time Apple updated the macOS user agent, which is why those sites are incorrect.

Node was rebuilt to link against v76 of icu4c. That is what the _1 is at the end of the version number. If you want to avoid being "flamed" as you put it in your main post: fully research your claims before bringing them up to a project, truncate your overly verbose post because no one is going to spend 9 minutes reading your inaccurate claims.

The brew doctor output is also not optional, and I'm closing this discussion since you failed to provide it.