Skip to content

refactor!: unify entire codebase logic - #2002

Open
vhyrro wants to merge 39 commits into
mainfrom
push-ywksqwqmssvq
Open

vhyrro wants to merge 39 commits into
mainfrom
push-ywksqwqmssvq

Conversation

@vhyrro

@vhyrro vhyrro commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Overview

This PR is a general overhaul of the codebase, ready to prepare it for a 1.0 release. This contains a lot of refactorings and a lot of code-moving. However, the code itself is generally untouched. Almost all modules have simply been cut up and copy+pasted into new areas, so the logic of Lux remains identical across the refactor.

The exception to this is the vendoring logic, which has had its flow refactored a bit to work with the rest of the codebase. We might want to perform some serious tests there.

The goal of this PR is unification: taking a bunch of duplicated logic and code and putting it all into single structs and builders so that they can be reused everywhere.

Renames

Basic renames include:

  • LocalPackage -> LockedPackage - the earlier name was actually misleading. A LockedPackage is one which is ready to be recorded in a Lockfile, hence the name Locked. Local can mean too many things.
  • RemotePackageDB -> PackageDB - the PackageDB can now ingest a lockfile as well as local packages, allowing it to pull from multiple sources. This means it's no longer "remote".

Pipeline

I've created a new module called the "pipeline", which is the heart of Lux's logic. It describes the following stages:

  • Discover - the act of contacting the server to ensure that a given package exists (DiscoveredPackage struct)
  • Resolve - downloads all dependencies recursively, just the rockspecs, not the sources, unless it's a .src.rock (ResolvedPackage)
  • DownloadAndHash - downloads all sources and hashes everything, produces a DownloadedPackage
  • Build - performs the actual building of each package, placing it all into a Tree. This last step is actually a two in one: it builds the package and installs it into the Tree. Once we implement transactions in Lux, we will want to break this apart into two stages. Produces LockedPackages.

Each stage produces its own Artifacts<T> - because luarocks can sometimes return a source (e.g. binary rocks or src rocks), these can be stored as intermediate artifacts as we move through the pipeline.

Drivers

These drive the pipeline in some desired fashion. This is where a lot of the logic becomes unified.

  • InstallPackages - converts a set of PackageInstallSpecs into fully installed LockedPackages, as well as a LockfileHandle so everything can be committed to the correct lockfile (workspace lockfile vs tree lockfile).
  • Sync - sync gained a large overhaul. It now operates in two modes, Open and Frozen.
    • Open - sync the current workspace with the lockfile, pulling any unresolved packages from the luarocks manifest, but otherwise preferring the lockfile (this prevents packages with a lockfile constraint from being rebuilt with a newer version). Also builds the project as a package in the tree, so it can be executed or other things. Does not remove unused packages.
    • Frozen - the equivalent of the old Sync's behaviour. syncs the current workspace so that it matches the lockfile exactly. Installs or removes all packages so that they match the lockfile. Name mirrors Cargo's behaviour.

The sync overhaul needs extra context: it is used by essentially all commands now to ensure the project is up-to-date and properly built before executing any given command (for example, lx lua now ensures the project is properly built before running a REPL in a potentially broken environment).

This is important for Lux 1.0: imagine someone's Lux project involves generating a .so file with mlua. Earlier, none of the Lux commands would ensure that the project got built properly, or that it was available. Now, whenever running something like lx run or lx lua, the project is built and ready to be used directly.

Extra Changes

  • There are now three BuildBehaviours - Ignore, Conflict, Force. Ignore will simply ignore any package that is already installed. Conflict will throw an error whenever a package of the same name but with a different version is trying to be installed (useful for lx add). Force works exactly as it used to.
  • Projects can now be built without needing a remote url field. The project's codebase will be used instead.
  • Projects only get rebuilt when their hashes change, i.e. whenever the source directory is modified.

Multiple Modules Deleted

A bunch of code got outright removed thanks to the massive deduplication effort, making Lux a leaner codebase.

@vhyrro
vhyrro requested a review from mrcjkb as a code owner October 5, 2026 10:41
@vhyrro
vhyrro marked this pull request as draft October 5, 2026 10:41
@vhyrro
vhyrro force-pushed the push-ywksqwqmssvq branch 5 times, most recently from 799fc84 to b9e6a34 Compare October 7, 2026 16:41
Comment thread lux-cli/src/outdated.rs Outdated
Comment thread lux-cli/src/outdated.rs
Comment thread lux-cli/src/uninstall.rs Outdated
Comment thread lux-lib/src/build/mod.rs
Comment thread lux-lib/src/drivers/sync/frozen.rs
Comment thread lux-lib/src/drivers/sync/frozen.rs
Comment thread lux-lib/src/drivers/sync/frozen.rs Outdated
@vhyrro
vhyrro force-pushed the push-ywksqwqmssvq branch 5 times, most recently from 0018767 to ba78d09 Compare October 10, 2026 15:38
Comment thread lux-lib/src/fs/sync.rs
Comment thread lux-lib/src/drivers/sync/mod.rs Outdated
Comment thread lux-lib/src/drivers/sync/mod.rs Outdated
Comment thread lux-lib/src/operations/test.rs Outdated
Comment thread lux-lib/src/workspace/mod.rs Outdated
@vhyrro
vhyrro marked this pull request as ready for review October 10, 2026 17:17
@vhyrro

vhyrro commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

This should be ready for initial review. I'm aware that there are conflicts. There will probably be some cargo clippy errors too.

I assume that this PR will break things. That is the nature of such a sizeable change. Our tests can catch almost everything, but not the craziest edge cases. We will have to rediscover those as we go.

Feel free to be as critical as necessary with the changes :)

@mrcjkb mrcjkb linked an issue Oct 10, 2026 that may be closed by this pull request
vhyrro and others added 26 commits October 11, 2026 12:14
(allows overwrites and prevents conflicting entrypoints by removing the
old version of the package and replacing it with a new one)
@vhyrro
vhyrro force-pushed the push-ywksqwqmssvq branch from 9f6d320 to 0c39628 Compare October 11, 2026 10:37
@vhyrro

vhyrro commented Oct 11, 2026

Copy link
Copy Markdown
Member Author

Fixed all the conflicts and hopefully all CI should start passing :)

@vhyrro
vhyrro added this pull request to stack #2015 October 11, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] cache builds

2 participants