Remove Lerna

[mfedderly]

Removes the Lerna dependency entirely.

• Positive: CI gets much faster (see the table below)
• Positive: A large tree of dependencies gets pruned (20% of pnpm-lock)
• Negative: Lerna's task caching for test commands goes away

lerna version is only partially replaced by pnpm version, so there's a small wrapper script to commit and tag the specified version ./scripts/version.sh

pnpm test timings on CI:

Node	Lerna	pnpm
22	5:15	3:48
24	4:53	3:13
26	5:05	3:37

Local dev experience trying to run test in all of the packages:

Cache	Cold	Hot
Lerna	316.37s user 42.10s system 711% cpu 50.414 total	1.03s user 0.33s system 99% cpu 1.361 total
PNPM	153.72s user 11.96s system 304% cpu 54.362 total	n/a

pnpm does about half the work of lerna, but does not cache its results leading to slower reruns. It may be worth considering adopting turborepo. Using turbo implies not using project references, and I'm somewhat scared of introducing consistency bugs with an improper turbo.json configuration.

I'm considering merging this to get the CI wins, and then revisiting a migration to turborepo down the road if we decide that the risk of caching issues is worth further speeding up CI and local devex with more caching. As it stands the removal of Lerna in favor of pnpm and TypeScript project references seems to have made CI take about half the time (10->5min).

I think I'm personally de-weighting the impact of a full pnpm test run because its already pretty fast for me, and I expect that most of the local testing will be in a specific package instead of repo-wide. One factor worth considering is that although CI can run the tests for pretty quick feedback, outside contributors currently need their CI runs approved by a maintainer before they run.

[mfedderly]

We don't have any automation but this (entire file) also passes over at https://www.shellcheck.net/

[mfedderly]

turborepo has the potential to be really fast locally and in CI if we persist the .turbo caches along the way, that's pretty compelling.

Timings below from a half-baked turbo.json and no project references. It does more overall work in the uncached case, but its in the ballpark of the project reference + pnpm version, and the cached builds become incredibly fast.

local build cold cache
    Tasks:    115 successful, 115 total
    Cached:    0 cached, 115 total
    Time:    19.689s 

    pnpm turbo run build  148.07s user 14.53s system 777% cpu 20.924 total

local build hot cache
    Tasks:    115 successful, 115 total
    Cached:    115 cached, 115 total
    Time:    31ms >>> FULL TURBO

    pnpm turbo run build  0.48s user 0.29s system 171% cpu 0.452 total

local build test cold cache (but build is cached)
    Tasks:    236 successful, 236 total
    Cached:    115 cached, 236 total
    Time:    28.48s 

    pnpm turbo run test  201.79s user 23.07s system 777% cpu 28.912 total

local build hot cache (test and build cached)
    Tasks:    236 successful, 236 total
    Cached:    236 cached, 236 total
    Time:    232ms >>> FULL TURBO

    pnpm turbo run test  0.60s user 0.55s system 170% cpu 0.672 total

Update: turbo watch is a bit of a regression against project references for builds, taking 18 seconds to build the world after a @turf/helpers change.

[mfedderly]

I'm going to merge this for the wins, and push the turbo changes in a draft PR in case someone is interested in picking that up and trying to fix the typescript build speed for local dev. I definitely prioritize typechecking at dev time over the speed of running tests across the entire repo. You can run typechecking across the entire repo, and watch mode tests in the package you're changing for instant feedback. The thing you'd miss is any packages that depend on yours having test failures because of the changes.