From e7fc832ae09e31cf93251127c131bac6543c1435 Mon Sep 17 00:00:00 2001 From: Jesse Geens Date: Tue, 21 Jul 2026 16:21:01 +0200 Subject: [PATCH 1/4] Write implementation plan --- plan.md | 379 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 379 insertions(+) create mode 100644 plan.md diff --git a/plan.md b/plan.md new file mode 100644 index 0000000000..c8432f6e64 --- /dev/null +++ b/plan.md @@ -0,0 +1,379 @@ +# Share reconciliation implementation plan + +## Context + +We run a service called CERNBox, similar to Google Drive. It uses a backend server called Reva. Reva consists of multiple microservices, which are interconnected via standardized API's called the CS3APIs. To do operator-only operations, we have a dedicated tool, called cernboxcop. + +The problem we are trying to solve is the following. Users can share resources with other users. This results in an ACL entry on the storage, as well as an entry in the database. For multiple reasons, these can diverge. The goal is to implement algorithms that fix this divergence. + +Some background info: storage in CERNBox is split up across "spaces". Spaces are completely disjunct. There are two types of spaces to take into account: personal ones (one per user account), and project spaces (for collaborative work between users). + +Shares can also go to different recipients. Specifically, there are +1) CERN user acounts. These go directly in the actual ACLs. +2) Groups. Also go in the ACLs. In the share definition, recipient type (user or group) is determined by share_with_is_group +3) External accounts. Do not go in the native EOS acls, but have a dedicated attribute: `sys.reva.lwshare.=` + +Users can be resolved via the reva gateway. ACLs (native ones, but also lightweight) should all be set via the CS3API. Note that even lightweight ACLs should be set via AddACL. + +We use a three-level approach. +1. The first is to detect shares in the database that are no longer valid. This can be because the file has been deleted, the recipient no longer exists, etc. We mark these invalid shares as "orphan". + +2. The second is to list all spaces. Then for every space, we reconstruct what the ACLs *should* be on all the paths that are shared. Then, we check these paths and set the correctly if there is any difference. + +3. Check the whole namespace. We once again list all the spaces. But then we use `eos-ns-inspect` and compare the whole namespace against what the database tells us + +Note that spaces will have default ACLs. There are global default ACLs that are *allowed* to be anywhere but don't have to (`cbackeosro` and `cboxexternal`). Then for user spaces the owner of the space HAS to be everywhere. And for project spaces, there are three groups (one for readers, writers and admins each) that also have to be everywhere. + +We should have an extremely extensive test suite. This test suite should cover the three levels. Every time, we need tests for: +* every recipient type +* all combo's of ACLs (see share hierarchy for possibilities) +* real eos-ns-inspect output parsing + +Everything should also be very configurable. For example, we want to be able to set path_prefixes and map these to default ACL's, and we want to be able to say if they *can* be there or *should* be there. + +Implementation details: +* We want three levels to run as three different jobs +* Please think about where the best place would be to put the code for the jobs. Perhaps together with the share hierarchy? Or under EOS (although this should also work for other storage drivers)? +* Use Reva's built-in jobs framework for running the jobs +* We already have a half-baked implementation under ~/Code/cernboxcop. This might be useful to inspect for the eos-ns-inspect code. +* Implement a dry_run mode so we can do dry runs on production data without modifying anything + +## Implementation strategy + +### Where the code lives + +Put the reconciliation engine in a new top-level package `pkg/reconciliation`, not under +`pkg/storage/fs/eos` and not folded into `pkg/sharehierarchy`. + +Reasoning: +* The three jobs are cross-cutting: they read the share DB (`pkg/share/manager/sql`), + resolve identities through the gateway, list spaces, and mutate ACLs. None of those + belong to a single storage driver, so `pkg/storage/fs/eos` is the wrong home. The + requirement that this "should also work for other storage drivers" makes the driver + package a dead end. +* `pkg/sharehierarchy` already owns the permission-ordering algebra (`PermLevel`, + `PermLevelFromCS3`, ancestor/descendant resolution). Reconciliation *reuses* that, but + it is a much bigger surface (namespace scanning, orphan detection, ACL diffing, jobs). + Keep `sharehierarchy` as the pure algorithm library and let `pkg/reconciliation` depend + on it. Do not grow `sharehierarchy` into a jobs package. + +The one genuinely EOS-specific piece is reading the EOS namespace for level 3, which is what +`eos-ns-inspect` does. That does **not** live under `pkg/reconciliation`. It belongs with the +rest of the EOS driver under `pkg/storage/fs/eos`, exactly like the existing eos client, +grant, and recycle code. `pkg/reconciliation` only defines the `NamespaceScanner` interface +and a small registry; the EOS driver provides the concrete scanner and registers it from its +own `init()`, the same `Register(name, NewFunc)` + `loader.go` pattern reva already uses for +storage backends (`pkg/storage/fs/registry`). This keeps EOS code with EOS and keeps the +engine free of any EOS import. + +What "read the EOS namespace" means, and why it is not the MGM gRPC API (established while +writing this plan, from the EOS source under `~/Code/eos`): +* `eos-ns-inspect` reads **QDB directly** (QuarkDB: RocksDB behind a Redis-protocol server), + not the MGM. `Find` / the reva `EOSClient.ListWithRegex` gRPC call goes through the running + MGM and is a different operation; it is not a substitute for a whole-namespace audit and + would load the live MGM. So level 3 uses a QDB-reading scanner, not gRPC. +* In QDB the namespace is protobuf blobs: all container MDs under one "locality hash" key, + all file MDs under another, read with QDB-custom commands (`LHGET`/`LHSCAN`), with + parent/child links in standard Redis hashes `:map_conts` and `:map_files` + (`HSCAN`/`HGET`), root container id = 1. Each `FileMdProto` / `ContainerMdProto` + (`proto/namespace/ns_quarkdb/{FileMd,ContainerMd}.proto`) carries the `xattrs` map, where + `sys.acl` and the lightweight `sys.reva.lwshare.` entries live, plus uid/gid/name. + +Decision (see "Open questions"): define the `NamespaceScanner` interface now and, for now, +ship a single EOS implementation behind it that shells out to the binary. A native QDB reader +is a possible future option, not part of this work: +* `eos-nsinspect-binary` (what we build): exec the version-matched `eos-ns-inspect scan ... --json` + binary and parse its output, as cernboxcop does. Least code, no coupling to the QDB on-disk + schema, fastest path to a working level 3. Needs the binary and keytab on the host running + the job. This is the only scanner we implement now. +* Native QDB reader (**could consider later**, not scheduled): a Go QDB reader that does what + the binary does in-process: a redis-protocol client issuing `LHSCAN`/`HSCAN`/`LHGET`, the QDB + **HMAC challenge-response handshake** for auth (QDB does not use plain redis `AUTH`; this is + the fiddliest part, would be ported from `qclient`), generated Go types from the two `.proto` + files, and the flat or tree scan. It would remove the external-binary dependency and be fully + unit-testable with recorded QDB responses, at the cost of tracking the EOS QDB schema across + releases. Worth revisiting only if the on-host binary dependency becomes a real operational + problem. +The point of the `NamespaceScanner` interface is that if that native reader is ever built, it +drops in behind the same interface: level 3 and its tests do not change, only the registered +scanner name in config would. + +Decisions taken (see "Open questions" answers): +* Levels 2 and 3 are separate jobs. They share only the "what should the ACL be" computation + (the `expected_acls.go` + `planner.go` pair below), not their scan or scheduling. +* Orphan detection (level 1) resolves recipients and resources through the gateway (CS3), + not by reading EOS or the DB directly. Driver-agnostic and consistent with the rest. + +Proposed layout. Two files carry the naming that needs explaining up front: +* `expected_acls.go` is the pure function "given the shares and defaults for a space, what + ACL entries *should* exist on each path". This is the piece shared between levels 2 and 3. +* `planner.go` diffs those expected ACLs against the *observed* ACLs and produces a `Plan`: + the ordered list of add/remove/update actions. `applier.go` then executes a `Plan`. + +``` +pkg/reconciliation/ + reconcile.go // shared types: Space, Recipient, ExpectedACL, Plan, Action, Outcome + config.go // Config + ApplyDefaults, path_prefix -> default-ACL rules (can/should) + defaults.go // default-ACL computation per space type (owner, project egroups, globals) + expected_acls.go // pure: (shares for a space) + defaults -> expected ACL set per path + // (shared by levels 2 and 3; wraps sharehierarchy) + planner.go // pure: expected ACLs vs observed ACLs -> Plan of add/remove/update + applier.go // executes a Plan via the CS3 grant API; honours dry_run + identity.go // recipient resolution + classification (user / group / lightweight) + scanner.go // NamespaceScanner interface + Register/registry (driver-agnostic) + orphans.go // level 1 + space_acls.go // level 2 + namespace.go // level 3 (depends on the interface, never on the EOS impl) + jobs/ + orphans_job.go // rjobs on-demand + periodic registration for level 1 + spaceacls_job.go // level 2 + namespace_job.go // level 3 + +pkg/storage/fs/eos/ // EOS-specific scanner lives with the EOS driver + nsscan_loader.go // registers the scanner in init() (Register pattern) + nsscan_binary.go // eos-nsinspect-binary: exec eos-ns-inspect + JSON parser + // (ported from cernboxcop); the only scanner we build now + testdata/nsinspect/ // captured real eos-ns-inspect JSON (binary scanner tests) + // A native QDB reader (nsscan_qdb.go + a qdb/ client package) could be added here later + // behind the same interface, but is out of scope for now. +``` + +The scanner sits behind the `NamespaceScanner` interface defined in `pkg/reconciliation`. +Level 3 depends only on that interface and looks the scanner up by name from the registry +(config: `scanner = "eos-nsinspect-binary"`). The concrete implementation lives under +`pkg/storage/fs/eos` and registers itself at init time, so `pkg/reconciliation` never imports +the EOS driver. A storage driver that cannot enumerate its namespace simply registers no +scanner, and level 3 is a no-op for its spaces; levels 1 and 2 stay driver-agnostic by going +through CS3. + +### Reuse from cernboxcop + +Port, do not import, from `~/Code/cernboxcop/pkg`: +* `eos/ns_inspect.go`: the `eos-ns-inspect scan ... --json` command builder and the + `CommonEntry` / `DirEntry` / `FileEntry` parser. This is the `eos-nsinspect-binary` scanner + and lands in `pkg/storage/fs/eos` (`nsscan_binary.go`), not in `pkg/reconciliation`. Keep + the `prefetchedData` path: it is what makes real-output parsing testable and enables dry + runs against a captured snapshot. +* `reconciliation/set_operations.go` and `acl_change_set.go`: the ACL set-diff + (add / remove / update) logic is sound and becomes the core of `planner.go`. +* `reconciliation/permission_store.go` and `deep_fs.go`: the "reconstruct expected ACLs by + walking parent shares" idea, reworked to use `sharehierarchy` for the permission ordering + instead of the ad-hoc `rx`/`rwx` string comparison, and to be space-scoped from the start. + +Fixes to make while porting: +* Replace `os/user.Lookup` and hardcoded `/eos/user/...` path math with gateway-based + identity resolution and the space's own root path. +* Do not shell to EOS for mutations. `acl_change_set.go` currently calls the EOS client + directly; route every mutation through the CS3 grant API (see below). +* Space isolation is mandatory. Every DB read filters by `space_id` (`SpaceIDFilter`), and + hierarchy never crosses a space boundary. See the existing memory on share space isolation. + +### Setting ACLs: always through CS3 + +All mutations go through the gateway grant API, never the EOS binary directly: +* Native user/group ACLs and lightweight (external) ACLs are all set with `AddGrant` / + `RemoveGrant` / `UpdateGrant` / `DenyGrant` on the storage provider. The EOS driver + (`pkg/storage/fs/eos/grant.go`) already routes lightweight accounts to the + `sys.reva.lwshare.` xattr and everything else to `sys.acl`, so the reconciler does + not need to know the on-disk encoding. This is what keeps it driver-agnostic and is also + the project rule (lightweight ACLs still go via AddGrant). +* Recipient classification drives the CS3 `Grantee`: + * `share_with_is_group == true` -> `GranteeType_GROUP`. + * external account (lightweight) -> user grantee whose id the driver recognises as + lightweight; the driver picks the xattr path. + * otherwise -> `GranteeType_USER`. +* Reconstruct `ResourcePermissions` from the DB `permissions` (OCS uint8) exactly as + `model.Share.AsCS3Share` does, then map through `sharehierarchy.PermLevelFromCS3` when we + need to compare levels. Permissions=0 is an active deny, not "no share" (see `PermDeny`). + +### The three jobs + +Each level is its own `rjobs` job, registered both on-demand (operator triggers a run, +optionally scoped to one space or user) and periodic (`ScopeLeader`, since they mutate +shared state and must fire once across replicas). Register under stable names, e.g. +`reconciliation.orphans`, `reconciliation.space_acls`, `reconciliation.namespace`. Config +per job comes from `[serverless.services.jobs.on_demand."reconciliation.namespace"]`. + +**Level 1: orphan detection (`orphans.go`).** +List DB shares (`ListModelShares`, including orphans) per space. A share is an orphan when +its resource no longer resolves (gateway Stat returns not-found / in recycling), or its +recipient no longer exists (gateway user/group lookup), or the space is gone. All three +checks go through the gateway (CS3), never a direct EOS or DB read, so this stays +driver-agnostic. Mark with `ShareMgr.MarkAsOrphaned`. No ACL writes, so it is cheap and safe +to run frequently. Public links reuse the same pass via `PublicShareMgr`. + +**Level 2: per-space expected-ACL reconstruction (`space_acls.go`).** +For each space: gather its non-orphan shares, group by grantee, and use `sharehierarchy` to +collapse each grantee's shares into the minimal correct ACL set per path (nearest-ancestor +wins, children with higher perms are re-applied). Add the space's default ACLs (below). +Stat each shared path, diff observed grants against expected with the planner, and apply. +This corrects the shared paths only; it does not walk the whole tree, so it is the routine +reconciler. + +**Level 3: full-namespace sweep (`namespace.go`).** +List spaces, then for each look up the configured `NamespaceScanner` from the registry (the +EOS one shells out to the `eos-ns-inspect` binary, which reads QDB) and scan the whole space +subtree. For every node compute expected ACLs = default ACLs for the space + inherited +share ACLs from the permission store, diff against the scanned `sys.acl` (and lightweight +xattrs), and apply. This is the expensive, authoritative sweep; schedule it `@daily`/`@weekly` +with jitter and `Skip` overlap. It catches drift on paths that no share touches anymore. + +Levels 2 and 3 are separate jobs but share `expected_acls.go` (what the ACLs should be) plus +`planner.go` and `applier.go` (diff and execute). They differ only in how they gather the +observed state (gateway Stat on shared paths vs. full eos-ns-inspect scan) and which node set +they cover. Level 1 shares none of this; it only reads and marks the DB. + +### Default ACLs and configuration + +`Config` (decoded from the job's config section) holds an ordered list of path-prefix rules: + +``` +[[path_prefix]] + prefix = "/eos/user" + space_type = "personal" # personal | project | global + [[path_prefix.default_acl]] + entry = "cbackeosro" # or an egroup / owner template + enforcement = "may" # "may" (allowed anywhere) | "must" (required everywhere) +``` + +Semantics, matching the spec: +* Global defaults (`cbackeosro`, `cboxexternal`): `enforcement = "may"`. Present is fine, + absent is fine; never added, never removed by the reconciler. +* Personal space owner: `enforcement = "must"`, template resolves to the space owner uid. + Missing => add; the planner never removes a "must" entry. +* Project spaces: the readers/writers/admins egroups are three `must` entries, templated + from the project name. + +`defaults.go` resolves templates (`{owner}`, `{project}`) against the space. The planner +treats `must` entries as always-expected and `may` entries as never-diffed (neither added +nor flagged), so a `may` entry present on disk is left untouched. + +### dry_run mode + +`dry_run` is a `Config` bool threaded into `applier.go`. When set, the applier logs and +records each intended `Action` (path, grantee, before/after) into the job's result `Params` +and the run status, and skips the CS3 call entirely. Level 3 additionally accepts a +`prefetched_scan` path so a captured `eos-ns-inspect` snapshot can be replayed offline, so we +can dry-run against production data without touching EOS or the DB. + +### Test suite + +Tests live beside each file plus an integration layer in `pkg/reconciliation`. Coverage per +the spec, driven by table tests: +* Every recipient type: CERN user, group (`share_with_is_group`), external/lightweight. + Assert each maps to the correct CS3 `Grantee` and, for lightweight, that the driver would + target the `sys.reva.lwshare.` xattr. +* Every ACL combination from the hierarchy: all ordered pairs of `{R, RW, Deny}` for + parent/child on nested paths, plus the re-apply and delete cases already covered by + `sharehierarchy` tests, now asserted end to end as `Plan` actions. +* Default-ACL rules: `may` present/absent (untouched), `must` present/absent (added), and + wrong-perms `must` (updated), for personal and project spaces. +* Real `eos-ns-inspect` output: commit captured JSON under + `pkg/storage/fs/eos/testdata/nsinspect` (personal and project space, files and folders, + sys entries, lightweight xattrs). Assert the binary scanner's parser (EOS-driver test) and, + feeding the scanner output into the engine, that the level-3 planner produces the expected + `Plan` (reconciliation test). Reuse the `prefetchedData` path so no QDB or MGM is needed in + CI. (If a native QDB scanner is ever added, it would get its own recorded-response tests and + a cross-check asserting it yields the identical node set and ACLs as the binary scanner.) +* Orphan detection: deleted resource, recycled resource, missing recipient, missing space. +* dry_run: assert no mutation is issued and the recorded actions match what a live run would + have applied (run planner once, apply in both modes, compare). + +## Work breakdown + +Ordered by dependency. Each step is a self-contained unit: it builds, has its own tests, and +is meant to be one reviewable PR / commit. Steps 1 to 6 are pure and need no live services, so +they land fast and de-risk the engine before any job or EOS code. Steps 7 to 11 wire real +dependencies. Nothing after a step depends on a later step. + +**Step 1: core types and config.** +`reconcile.go` (`Space`, `Recipient`, `ExpectedACL`, `Plan`, `Action`, `Outcome`) and +`config.go` (`Config` + `ApplyDefaults`, the `path_prefix` -> default-ACL rules with +`may`/`must` enforcement). No logic beyond decoding and validation. +Tests: config decode/defaults, rule validation (bad enforcement, missing prefix). +Depends on: nothing. + +**Step 2: default-ACL computation.** +`defaults.go`: given a `Space` and the configured rules, produce the default ACL entries, +resolving `{owner}`/`{project}` templates. Encodes the personal-owner and project-egroup +rules and the global `may` entries. +Tests: personal owner `must`, project readers/writers/admins `must`, global `may`, template +resolution, wrong space type. +Depends on: 1. + +**Step 3: permission model bridge.** +Map the DB `permissions` (OCS uint8) and grantee fields to CS3 `ResourcePermissions` / +`Grantee`, and to `sharehierarchy.PermLevel`. Small file (`permmap.go`) plus recipient +classification stubs (user / group / lightweight) that do not yet call the gateway. +Tests: every recipient type maps to the right `Grantee`; permissions=0 is `PermDeny`, not +absent; OCS round-trips match `model.Share.AsCS3Share`. +Depends on: 1. + +**Step 4: expected-ACL reconstruction.** +`expected_acls.go`: given a space's shares plus its default ACLs, compute the expected ACL +set per path, wrapping `sharehierarchy` for the nearest-ancestor / reapply ordering. Pure. +Tests: all ordered `{R, RW, Deny}` parent/child pairs on nested paths, reapply and delete +cases, defaults merged in, space isolation (never crosses `space_id`). +Depends on: 2, 3. + +**Step 5: planner.** +`planner.go`: diff expected vs observed ACLs into an ordered `Plan` of add/remove/update. +Port and adapt cernboxcop's `set_operations.go` / `acl_change_set.go` diff. Pure. +Tests: pure add, pure remove, update (same grantee different perms), `may` present left +untouched, `must` wrong-perms updated, ordering (shallowest first). +Depends on: 4. + +**Step 6: applier with dry_run.** +`applier.go`: execute a `Plan` through the CS3 grant API (`AddGrant`/`RemoveGrant`/ +`UpdateGrant`/`DenyGrant`), honouring `dry_run`. Define a small gateway-client interface so +tests use a fake. +Tests: each action issues the right grant call; dry_run issues none and records the actions; +recorded actions in dry_run equal those applied live (run planner once, apply both ways). +Depends on: 5. + +**Step 7: identity resolution against the gateway.** +`identity.go`: resolve and validate recipients and resources through the gateway (exists / +not-found / recycled), replacing the step-3 stubs. This is the first step that talks to a +live service, still behind an interface with a fake in tests. +Tests: user/group/lightweight resolution, missing recipient, resource not-found vs recycled. +Depends on: 3. + +**Step 8: NamespaceScanner interface + EOS binary scanner.** +`scanner.go` (interface + `Register`/registry) in `pkg/reconciliation`; `nsscan_binary.go` + +`nsscan_loader.go` in `pkg/storage/fs/eos`, porting `ns_inspect.go` (command builder, JSON +parser, `prefetchedData` path) and registering under `eos-nsinspect-binary`. +Tests: parse captured JSON from `testdata/nsinspect` (personal + project, files + folders, +sys entries, lightweight xattrs); `prefetchedData` prefix/depth filtering. +Depends on: 1 (interface types only); independent of 4 to 7, so can proceed in parallel. + +**Step 9: level 1 orphan job.** +`orphans.go` + `jobs/orphans_job.go`: per-space DB scan via `ListModelShares`, gateway-based +validity checks, `MarkAsOrphaned`; register as on-demand + `ScopeLeader` periodic. Public +links via `PublicShareMgr`. +Tests: deleted / recycled resource, missing recipient, missing space; on-demand run scoped to +one space; idempotent re-run. +Depends on: 7. + +**Step 10: level 2 space-ACL job.** +`space_acls.go` + `jobs/spaceacls_job.go`: gather a space's non-orphan shares, build expected +ACLs, gateway-Stat the shared paths for observed grants, plan, apply. Register on-demand + +periodic. +Tests: end-to-end per recipient type and ACL combo with fakes producing a `Plan`; dry_run; +space scoping. +Depends on: 4, 6, 7. + +**Step 11: level 3 namespace job.** +`namespace.go` + `jobs/namespace_job.go`: list spaces, run the scanner over each space tree, +compute expected ACLs per node (defaults + inherited shares), plan, apply. Register on-demand ++ periodic (`@daily`/`@weekly`, jitter, `Skip`). Support `prefetched_scan` for offline dry +runs. +Tests: feed `testdata/nsinspect` output through the engine and assert the produced `Plan`; +default-ACL `may`/`must` on untouched paths; dry_run against a snapshot. +Depends on: 4, 6, 8. + +**Step 12: config wiring and docs.** +Register the three jobs' config sections under the jobs serverless service, add an example +config block, and document the `path_prefix` rules, scanner selection, and `dry_run`. No new +logic. +Tests: config decodes end to end; example config validates. +Depends on: 9, 10, 11. \ No newline at end of file From 0e4da8ae160a350d016e336f52d22eddb74fedad Mon Sep 17 00:00:00 2001 From: Jesse Geens Date: Tue, 21 Jul 2026 16:43:02 +0200 Subject: [PATCH 2/4] implement core types and config --- pkg/reconciliation/config.go | 151 +++++++++++++++++++++++++++ pkg/reconciliation/config_test.go | 162 ++++++++++++++++++++++++++++ pkg/reconciliation/reconcile.go | 168 ++++++++++++++++++++++++++++++ plan.md | 43 +++++--- 4 files changed, 509 insertions(+), 15 deletions(-) create mode 100644 pkg/reconciliation/config.go create mode 100644 pkg/reconciliation/config_test.go create mode 100644 pkg/reconciliation/reconcile.go diff --git a/pkg/reconciliation/config.go b/pkg/reconciliation/config.go new file mode 100644 index 0000000000..51d253b560 --- /dev/null +++ b/pkg/reconciliation/config.go @@ -0,0 +1,151 @@ +// Copyright 2018-2026 CERN +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// In applying this license, CERN does not waive the privileges and immunities +// granted to it by virtue of its status as an Intergovernmental Organization +// or submit itself to any jurisdiction. + +package reconciliation + +import ( + "github.com/cs3org/reva/v3/pkg/storage/utils/acl" + "github.com/pkg/errors" +) + +// Enforcement decides how the reconciler treats a default ACL entry that +// diverges from what is on the storage. +type Enforcement string + +const ( + // EnforcementMay means the entry is allowed to be present but is not + // required. The reconciler never adds it and never removes it: a "may" + // entry on disk is left untouched. Used for the global defaults + // (cbackeosro, cboxexternal) that may live anywhere. + EnforcementMay Enforcement = "may" + // EnforcementMust means the entry has to be present everywhere in scope. + // The reconciler adds it when missing and corrects it when its permissions + // differ, and never removes it. Used for the personal-space owner and the + // project reader/writer/admin egroups. + EnforcementMust Enforcement = "must" +) + +// DefaultScanner is the default NamespaceScanner used for the full-namespace +// sweep. It shells out to the eos-ns-inspect binary. +const DefaultScanner = "eos-nsinspect-binary" + +// Config configures the reconciliation engine. It is decoded from the job's own +// configuration section. +type Config struct { + // DryRun, when set, makes every job compute and report the ACL changes it + // would make without applying any of them. + DryRun bool `mapstructure:"dry_run"` + // Scanner selects the registered NamespaceScanner used by the + // full-namespace sweep (level 3). Defaults to DefaultScanner. + Scanner string `mapstructure:"scanner"` + // PathPrefixes maps filesystem path prefixes to the default ACL entries + // that apply under them. Rules are evaluated in order; a space is governed + // by the first rule whose prefix matches its root and whose space_type + // matches (or is "global"). + PathPrefixes []PathPrefixRule `mapstructure:"path_prefix"` +} + +// PathPrefixRule associates a path prefix and space type with a set of default +// ACL entries. +type PathPrefixRule struct { + // Prefix is the filesystem path prefix the rule applies to, e.g. + // "/eos/user" or "/eos/project". + Prefix string `mapstructure:"prefix"` + // SpaceType restricts the rule to a space kind: "personal", "project", or + // "global" to apply regardless of kind. + SpaceType SpaceType `mapstructure:"space_type"` + // DefaultACLs are the default entries that apply under Prefix. + DefaultACLs []DefaultACLRule `mapstructure:"default_acl"` +} + +// DefaultACLRule is a single default ACL entry and how strictly it is enforced. +// The qualifier may contain the templates "{owner}" and "{project}", resolved +// per space when the defaults are computed. +type DefaultACLRule struct { + // Type is the ACL entry type: "u" (user), "egroup" (group) or "lw" + // (lightweight). See package acl. + Type string `mapstructure:"type"` + // Qualifier identifies the grantee. May contain "{owner}" or "{project}". + Qualifier string `mapstructure:"qualifier"` + // Permissions is the EOS permission string, e.g. "rx", "rwx" or "rwx+d". + Permissions string `mapstructure:"permissions"` + // Enforcement is "must" or "may". + Enforcement Enforcement `mapstructure:"enforcement"` +} + +// ApplyDefaults implements cfg.Setter. +func (c *Config) ApplyDefaults() { + if c.Scanner == "" { + c.Scanner = DefaultScanner + } +} + +// Validate checks that the configuration is internally consistent. It is +// separate from ApplyDefaults so a caller can decode, default and validate in +// that order and surface a precise error. +func (c *Config) Validate() error { + for i := range c.PathPrefixes { + if err := c.PathPrefixes[i].validate(); err != nil { + return errors.Wrapf(err, "reconciliation: path_prefix[%d]", i) + } + } + return nil +} + +func (r *PathPrefixRule) validate() error { + if r.Prefix == "" { + return errors.New("prefix must not be empty") + } + switch r.SpaceType { + case SpaceTypePersonal, SpaceTypeProject, SpaceTypeAny: + case "": + return errors.New("space_type must not be empty") + default: + return errors.Errorf("invalid space_type %q", r.SpaceType) + } + for j := range r.DefaultACLs { + if err := r.DefaultACLs[j].validate(); err != nil { + return errors.Wrapf(err, "default_acl[%d]", j) + } + } + return nil +} + +func (d *DefaultACLRule) validate() error { + switch d.Type { + case acl.TypeUser, acl.TypeGroup, acl.TypeLightweight: + case "": + return errors.New("type must not be empty") + default: + return errors.Errorf("invalid type %q", d.Type) + } + if d.Qualifier == "" { + return errors.New("qualifier must not be empty") + } + if d.Permissions == "" { + return errors.New("permissions must not be empty") + } + switch d.Enforcement { + case EnforcementMay, EnforcementMust: + case "": + return errors.New("enforcement must not be empty") + default: + return errors.Errorf("invalid enforcement %q", d.Enforcement) + } + return nil +} diff --git a/pkg/reconciliation/config_test.go b/pkg/reconciliation/config_test.go new file mode 100644 index 0000000000..0a243e3138 --- /dev/null +++ b/pkg/reconciliation/config_test.go @@ -0,0 +1,162 @@ +// Copyright 2018-2026 CERN +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// In applying this license, CERN does not waive the privileges and immunities +// granted to it by virtue of its status as an Intergovernmental Organization +// or submit itself to any jurisdiction. + +package reconciliation + +import ( + "testing" + + "github.com/cs3org/reva/v3/pkg/utils/cfg" +) + +func TestConfigDecodeAndDefaults(t *testing.T) { + in := map[string]any{ + "dry_run": true, + "path_prefix": []map[string]any{ + { + "prefix": "/eos/user", + "space_type": "personal", + "default_acl": []map[string]any{ + {"type": "u", "qualifier": "{owner}", "permissions": "rwx", "enforcement": "must"}, + {"type": "egroup", "qualifier": "cbackeosro", "permissions": "rx", "enforcement": "may"}, + }, + }, + }, + } + + var c Config + if err := cfg.Decode(in, &c); err != nil { + t.Fatalf("decode: %v", err) + } + + if !c.DryRun { + t.Errorf("DryRun = false, want true") + } + if c.Scanner != DefaultScanner { + t.Errorf("Scanner = %q, want default %q", c.Scanner, DefaultScanner) + } + if len(c.PathPrefixes) != 1 { + t.Fatalf("PathPrefixes = %d, want 1", len(c.PathPrefixes)) + } + rule := c.PathPrefixes[0] + if rule.Prefix != "/eos/user" || rule.SpaceType != SpaceTypePersonal { + t.Errorf("rule = %+v, unexpected prefix/space_type", rule) + } + if len(rule.DefaultACLs) != 2 { + t.Fatalf("DefaultACLs = %d, want 2", len(rule.DefaultACLs)) + } + if got := rule.DefaultACLs[0]; got.Qualifier != "{owner}" || got.Enforcement != EnforcementMust { + t.Errorf("DefaultACLs[0] = %+v, unexpected", got) + } + if err := c.Validate(); err != nil { + t.Errorf("Validate: %v", err) + } +} + +func TestConfigScannerOverride(t *testing.T) { + var c Config + if err := cfg.Decode(map[string]any{"scanner": "custom"}, &c); err != nil { + t.Fatalf("decode: %v", err) + } + if c.Scanner != "custom" { + t.Errorf("Scanner = %q, want %q", c.Scanner, "custom") + } +} + +func TestConfigValidate(t *testing.T) { + valid := DefaultACLRule{Type: "u", Qualifier: "{owner}", Permissions: "rwx", Enforcement: EnforcementMust} + + tests := []struct { + name string + cfg Config + wantErr bool + }{ + { + name: "ok", + cfg: Config{PathPrefixes: []PathPrefixRule{ + {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{valid}}, + }}, + }, + { + name: "global space type ok", + cfg: Config{PathPrefixes: []PathPrefixRule{ + {Prefix: "/eos", SpaceType: SpaceTypeAny, DefaultACLs: []DefaultACLRule{valid}}, + }}, + }, + { + name: "missing prefix", + cfg: Config{PathPrefixes: []PathPrefixRule{{SpaceType: SpaceTypePersonal}}}, + wantErr: true, + }, + { + name: "empty space type", + cfg: Config{PathPrefixes: []PathPrefixRule{{Prefix: "/eos/user"}}}, + wantErr: true, + }, + { + name: "invalid space type", + cfg: Config{PathPrefixes: []PathPrefixRule{{Prefix: "/eos/user", SpaceType: "bogus"}}}, + wantErr: true, + }, + { + name: "invalid acl type", + cfg: Config{PathPrefixes: []PathPrefixRule{ + {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{ + {Type: "x", Qualifier: "q", Permissions: "rx", Enforcement: EnforcementMay}, + }}, + }}, + wantErr: true, + }, + { + name: "missing qualifier", + cfg: Config{PathPrefixes: []PathPrefixRule{ + {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{ + {Type: "u", Permissions: "rx", Enforcement: EnforcementMay}, + }}, + }}, + wantErr: true, + }, + { + name: "missing permissions", + cfg: Config{PathPrefixes: []PathPrefixRule{ + {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{ + {Type: "u", Qualifier: "q", Enforcement: EnforcementMay}, + }}, + }}, + wantErr: true, + }, + { + name: "invalid enforcement", + cfg: Config{PathPrefixes: []PathPrefixRule{ + {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{ + {Type: "u", Qualifier: "q", Permissions: "rx", Enforcement: "sometimes"}, + }}, + }}, + wantErr: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := tt.cfg.Validate() + if (err != nil) != tt.wantErr { + t.Errorf("Validate() error = %v, wantErr %v", err, tt.wantErr) + } + }) + } +} diff --git a/pkg/reconciliation/reconcile.go b/pkg/reconciliation/reconcile.go new file mode 100644 index 0000000000..9e9737dd3a --- /dev/null +++ b/pkg/reconciliation/reconcile.go @@ -0,0 +1,168 @@ +// Copyright 2018-2026 CERN +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// In applying this license, CERN does not waive the privileges and immunities +// granted to it by virtue of its status as an Intergovernmental Organization +// or submit itself to any jurisdiction. + +// Package reconciliation reconciles the share database against the ACLs stored +// on the storage. It runs as three independent jobs: detecting orphaned shares, +// correcting the ACLs on shared paths per space, and sweeping the whole +// namespace of every space. The engine is storage-driver agnostic: it reads +// shares from the database, resolves identities and mutates ACLs through the +// gateway (CS3), and only reaches driver-specific code through the +// NamespaceScanner interface for the full-namespace sweep. +package reconciliation + +import ( + "github.com/cs3org/reva/v3/pkg/storage/utils/acl" +) + +// SpaceType is the kind of a storage space. +type SpaceType string + +const ( + // SpaceTypePersonal is a personal space, one per user account. + SpaceTypePersonal SpaceType = "personal" + // SpaceTypeProject is a project space, shared for collaborative work. + SpaceTypeProject SpaceType = "project" + // SpaceTypeAny matches any space type. It is valid only in configuration + // rules (see PathPrefixRule), never as the type of an actual Space. + SpaceTypeAny SpaceType = "global" +) + +// Space is a storage space to reconcile. Spaces are disjoint: reconciliation +// never crosses a space boundary. +type Space struct { + // ID is the CS3 space id. + ID string + // Type is the space kind. Always SpaceTypePersonal or SpaceTypeProject for + // a real space. + Type SpaceType + // StorageID is the storage provider (EOS instance) hosting the space. + StorageID string + // Root is the filesystem path of the space root. + Root string + // Owner is the owner's username. Set for personal spaces. + Owner string + // Project is the project name. Set for project spaces. + Project string +} + +// RecipientKind classifies the target of a share, which determines how its ACL +// is encoded on the storage. +type RecipientKind int + +const ( + // RecipientUser is a CERN user account. Goes into the native ACLs. + RecipientUser RecipientKind = iota + // RecipientGroup is a group. Goes into the native ACLs. + RecipientGroup + // RecipientLightweight is an external account. Does not go into the native + // ACLs but into a dedicated sys.reva.lwshare. attribute, handled by + // the storage driver. + RecipientLightweight +) + +// String returns a human readable name for the recipient kind. +func (k RecipientKind) String() string { + switch k { + case RecipientUser: + return "user" + case RecipientGroup: + return "group" + case RecipientLightweight: + return "lightweight" + default: + return "unknown" + } +} + +// Recipient is the target of a share. +type Recipient struct { + // Kind is the recipient classification. + Kind RecipientKind + // ID is the username, group name, or external account email, depending on + // Kind. + ID string +} + +// ExpectedACL is an ACL entry that should exist at Path, as reconstructed from +// the shares and default rules for a space. +type ExpectedACL struct { + // Path is the filesystem path the entry applies to. + Path string + // Entry is the ACL entry (type, qualifier, permissions). + Entry *acl.Entry + // Enforcement decides how a divergence is treated: a "must" entry is added + // or corrected when missing or wrong, a "may" entry is left untouched + // whether present or absent. + Enforcement Enforcement +} + +// ActionKind is the type of an ACL mutation. +type ActionKind int + +const ( + // ActionAdd adds an ACL entry that is missing. + ActionAdd ActionKind = iota + // ActionRemove removes an ACL entry that should not be present. + ActionRemove + // ActionUpdate changes the permissions of an existing ACL entry. + ActionUpdate +) + +// String returns a human readable name for the action kind. +func (k ActionKind) String() string { + switch k { + case ActionAdd: + return "add" + case ActionRemove: + return "remove" + case ActionUpdate: + return "update" + default: + return "unknown" + } +} + +// Action is a single ACL mutation on a path, the unit the applier executes +// through the CS3 grant API. +type Action struct { + // Kind is the mutation type. + Kind ActionKind + // Path is the filesystem path to mutate. + Path string + // Entry is the ACL entry to add, remove or update. + Entry *acl.Entry +} + +// Plan is the ordered set of ACL mutations that brings the observed ACLs in +// line with the expected ones. Actions are ordered so that parent paths are +// mutated before their children. +type Plan struct { + Actions []Action +} + +// Len returns the number of actions in the plan. +func (p *Plan) Len() int { return len(p.Actions) } + +// Outcome records the result of applying (or, in dry-run, simulating) a Plan. +type Outcome struct { + // Applied lists the actions that were carried out, or that would have been + // carried out when DryRun is true. + Applied []Action + // DryRun reports whether the outcome is a simulation. + DryRun bool +} diff --git a/plan.md b/plan.md index c8432f6e64..69f5169fd2 100644 --- a/plan.md +++ b/plan.md @@ -230,12 +230,23 @@ they cover. Level 1 shares none of this; it only reads and marks the DB. ``` [[path_prefix]] prefix = "/eos/user" - space_type = "personal" # personal | project | global + space_type = "personal" # personal | project | global [[path_prefix.default_acl]] - entry = "cbackeosro" # or an egroup / owner template - enforcement = "may" # "may" (allowed anywhere) | "must" (required everywhere) + type = "u" # u | egroup | lw (see package acl) + qualifier = "{owner}" # may contain {owner} / {project} + permissions = "rwx" + enforcement = "must" # "may" (allowed anywhere) | "must" (required everywhere) + [[path_prefix.default_acl]] + type = "egroup" + qualifier = "cbackeosro" + permissions = "rx" + enforcement = "may" ``` +The default ACL entry is given as explicit `type` / `qualifier` / `permissions` rather than a +single opaque token, so it is unambiguous and validatable at config load. `defaults.go` +resolves the `{owner}` / `{project}` templates in the qualifier per space. + Semantics, matching the spec: * Global defaults (`cbackeosro`, `cboxexternal`): `enforcement = "may"`. Present is fine, absent is fine; never added, never removed by the reconciler. @@ -286,14 +297,16 @@ is meant to be one reviewable PR / commit. Steps 1 to 6 are pure and need no liv they land fast and de-risk the engine before any job or EOS code. Steps 7 to 11 wire real dependencies. Nothing after a step depends on a later step. -**Step 1: core types and config.** +Progress is tracked with a `[x]` (done) or `[ ]` (todo) marker on each step heading. + +**Step 1: core types and config.** `[x]` done. `reconcile.go` (`Space`, `Recipient`, `ExpectedACL`, `Plan`, `Action`, `Outcome`) and `config.go` (`Config` + `ApplyDefaults`, the `path_prefix` -> default-ACL rules with `may`/`must` enforcement). No logic beyond decoding and validation. Tests: config decode/defaults, rule validation (bad enforcement, missing prefix). Depends on: nothing. -**Step 2: default-ACL computation.** +**Step 2: default-ACL computation.** `[ ]` `defaults.go`: given a `Space` and the configured rules, produce the default ACL entries, resolving `{owner}`/`{project}` templates. Encodes the personal-owner and project-egroup rules and the global `may` entries. @@ -301,7 +314,7 @@ Tests: personal owner `must`, project readers/writers/admins `must`, global `may resolution, wrong space type. Depends on: 1. -**Step 3: permission model bridge.** +**Step 3: permission model bridge.** `[ ]` Map the DB `permissions` (OCS uint8) and grantee fields to CS3 `ResourcePermissions` / `Grantee`, and to `sharehierarchy.PermLevel`. Small file (`permmap.go`) plus recipient classification stubs (user / group / lightweight) that do not yet call the gateway. @@ -309,21 +322,21 @@ Tests: every recipient type maps to the right `Grantee`; permissions=0 is `PermD absent; OCS round-trips match `model.Share.AsCS3Share`. Depends on: 1. -**Step 4: expected-ACL reconstruction.** +**Step 4: expected-ACL reconstruction.** `[ ]` `expected_acls.go`: given a space's shares plus its default ACLs, compute the expected ACL set per path, wrapping `sharehierarchy` for the nearest-ancestor / reapply ordering. Pure. Tests: all ordered `{R, RW, Deny}` parent/child pairs on nested paths, reapply and delete cases, defaults merged in, space isolation (never crosses `space_id`). Depends on: 2, 3. -**Step 5: planner.** +**Step 5: planner.** `[ ]` `planner.go`: diff expected vs observed ACLs into an ordered `Plan` of add/remove/update. Port and adapt cernboxcop's `set_operations.go` / `acl_change_set.go` diff. Pure. Tests: pure add, pure remove, update (same grantee different perms), `may` present left untouched, `must` wrong-perms updated, ordering (shallowest first). Depends on: 4. -**Step 6: applier with dry_run.** +**Step 6: applier with dry_run.** `[ ]` `applier.go`: execute a `Plan` through the CS3 grant API (`AddGrant`/`RemoveGrant`/ `UpdateGrant`/`DenyGrant`), honouring `dry_run`. Define a small gateway-client interface so tests use a fake. @@ -331,14 +344,14 @@ Tests: each action issues the right grant call; dry_run issues none and records recorded actions in dry_run equal those applied live (run planner once, apply both ways). Depends on: 5. -**Step 7: identity resolution against the gateway.** +**Step 7: identity resolution against the gateway.** `[ ]` `identity.go`: resolve and validate recipients and resources through the gateway (exists / not-found / recycled), replacing the step-3 stubs. This is the first step that talks to a live service, still behind an interface with a fake in tests. Tests: user/group/lightweight resolution, missing recipient, resource not-found vs recycled. Depends on: 3. -**Step 8: NamespaceScanner interface + EOS binary scanner.** +**Step 8: NamespaceScanner interface + EOS binary scanner.** `[ ]` `scanner.go` (interface + `Register`/registry) in `pkg/reconciliation`; `nsscan_binary.go` + `nsscan_loader.go` in `pkg/storage/fs/eos`, porting `ns_inspect.go` (command builder, JSON parser, `prefetchedData` path) and registering under `eos-nsinspect-binary`. @@ -346,7 +359,7 @@ Tests: parse captured JSON from `testdata/nsinspect` (personal + project, files sys entries, lightweight xattrs); `prefetchedData` prefix/depth filtering. Depends on: 1 (interface types only); independent of 4 to 7, so can proceed in parallel. -**Step 9: level 1 orphan job.** +**Step 9: level 1 orphan job.** `[ ]` `orphans.go` + `jobs/orphans_job.go`: per-space DB scan via `ListModelShares`, gateway-based validity checks, `MarkAsOrphaned`; register as on-demand + `ScopeLeader` periodic. Public links via `PublicShareMgr`. @@ -354,7 +367,7 @@ Tests: deleted / recycled resource, missing recipient, missing space; on-demand one space; idempotent re-run. Depends on: 7. -**Step 10: level 2 space-ACL job.** +**Step 10: level 2 space-ACL job.** `[ ]` `space_acls.go` + `jobs/spaceacls_job.go`: gather a space's non-orphan shares, build expected ACLs, gateway-Stat the shared paths for observed grants, plan, apply. Register on-demand + periodic. @@ -362,7 +375,7 @@ Tests: end-to-end per recipient type and ACL combo with fakes producing a `Plan` space scoping. Depends on: 4, 6, 7. -**Step 11: level 3 namespace job.** +**Step 11: level 3 namespace job.** `[ ]` `namespace.go` + `jobs/namespace_job.go`: list spaces, run the scanner over each space tree, compute expected ACLs per node (defaults + inherited shares), plan, apply. Register on-demand + periodic (`@daily`/`@weekly`, jitter, `Skip`). Support `prefetched_scan` for offline dry @@ -371,7 +384,7 @@ Tests: feed `testdata/nsinspect` output through the engine and assert the produc default-ACL `may`/`must` on untouched paths; dry_run against a snapshot. Depends on: 4, 6, 8. -**Step 12: config wiring and docs.** +**Step 12: config wiring and docs.** `[ ]` Register the three jobs' config sections under the jobs serverless service, add an example config block, and document the `path_prefix` rules, scanner selection, and `dry_run`. No new logic. From d99ac54e4aa7a3172cc7254cdc8e5de1be756fef Mon Sep 17 00:00:00 2001 From: Jesse Geens Date: Tue, 21 Jul 2026 17:03:08 +0200 Subject: [PATCH 3/4] implement tooling for default ACLs --- pkg/reconciliation/config.go | 19 +-- pkg/reconciliation/config_test.go | 35 ++---- pkg/reconciliation/defaults.go | 108 ++++++++++++++++ pkg/reconciliation/defaults_test.go | 183 ++++++++++++++++++++++++++++ pkg/reconciliation/reconcile.go | 3 - plan.md | 14 ++- 6 files changed, 312 insertions(+), 50 deletions(-) create mode 100644 pkg/reconciliation/defaults.go create mode 100644 pkg/reconciliation/defaults_test.go diff --git a/pkg/reconciliation/config.go b/pkg/reconciliation/config.go index 51d253b560..607f445615 100644 --- a/pkg/reconciliation/config.go +++ b/pkg/reconciliation/config.go @@ -54,21 +54,17 @@ type Config struct { // full-namespace sweep (level 3). Defaults to DefaultScanner. Scanner string `mapstructure:"scanner"` // PathPrefixes maps filesystem path prefixes to the default ACL entries - // that apply under them. Rules are evaluated in order; a space is governed - // by the first rule whose prefix matches its root and whose space_type - // matches (or is "global"). + // that apply under them. A space is governed by the single rule whose prefix + // is a path prefix of its root. Prefixes may not overlap, so at most one rule + // matches a space. See Config.DefaultACLs. PathPrefixes []PathPrefixRule `mapstructure:"path_prefix"` } -// PathPrefixRule associates a path prefix and space type with a set of default -// ACL entries. +// PathPrefixRule associates a path prefix with a set of default ACL entries. type PathPrefixRule struct { // Prefix is the filesystem path prefix the rule applies to, e.g. // "/eos/user" or "/eos/project". Prefix string `mapstructure:"prefix"` - // SpaceType restricts the rule to a space kind: "personal", "project", or - // "global" to apply regardless of kind. - SpaceType SpaceType `mapstructure:"space_type"` // DefaultACLs are the default entries that apply under Prefix. DefaultACLs []DefaultACLRule `mapstructure:"default_acl"` } @@ -111,13 +107,6 @@ func (r *PathPrefixRule) validate() error { if r.Prefix == "" { return errors.New("prefix must not be empty") } - switch r.SpaceType { - case SpaceTypePersonal, SpaceTypeProject, SpaceTypeAny: - case "": - return errors.New("space_type must not be empty") - default: - return errors.Errorf("invalid space_type %q", r.SpaceType) - } for j := range r.DefaultACLs { if err := r.DefaultACLs[j].validate(); err != nil { return errors.Wrapf(err, "default_acl[%d]", j) diff --git a/pkg/reconciliation/config_test.go b/pkg/reconciliation/config_test.go index 0a243e3138..b8756b9f40 100644 --- a/pkg/reconciliation/config_test.go +++ b/pkg/reconciliation/config_test.go @@ -29,8 +29,7 @@ func TestConfigDecodeAndDefaults(t *testing.T) { "dry_run": true, "path_prefix": []map[string]any{ { - "prefix": "/eos/user", - "space_type": "personal", + "prefix": "/eos/user", "default_acl": []map[string]any{ {"type": "u", "qualifier": "{owner}", "permissions": "rwx", "enforcement": "must"}, {"type": "egroup", "qualifier": "cbackeosro", "permissions": "rx", "enforcement": "may"}, @@ -54,8 +53,8 @@ func TestConfigDecodeAndDefaults(t *testing.T) { t.Fatalf("PathPrefixes = %d, want 1", len(c.PathPrefixes)) } rule := c.PathPrefixes[0] - if rule.Prefix != "/eos/user" || rule.SpaceType != SpaceTypePersonal { - t.Errorf("rule = %+v, unexpected prefix/space_type", rule) + if rule.Prefix != "/eos/user" { + t.Errorf("rule = %+v, unexpected prefix", rule) } if len(rule.DefaultACLs) != 2 { t.Fatalf("DefaultACLs = %d, want 2", len(rule.DefaultACLs)) @@ -89,34 +88,18 @@ func TestConfigValidate(t *testing.T) { { name: "ok", cfg: Config{PathPrefixes: []PathPrefixRule{ - {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{valid}}, - }}, - }, - { - name: "global space type ok", - cfg: Config{PathPrefixes: []PathPrefixRule{ - {Prefix: "/eos", SpaceType: SpaceTypeAny, DefaultACLs: []DefaultACLRule{valid}}, + {Prefix: "/eos/user", DefaultACLs: []DefaultACLRule{valid}}, }}, }, { name: "missing prefix", - cfg: Config{PathPrefixes: []PathPrefixRule{{SpaceType: SpaceTypePersonal}}}, - wantErr: true, - }, - { - name: "empty space type", - cfg: Config{PathPrefixes: []PathPrefixRule{{Prefix: "/eos/user"}}}, - wantErr: true, - }, - { - name: "invalid space type", - cfg: Config{PathPrefixes: []PathPrefixRule{{Prefix: "/eos/user", SpaceType: "bogus"}}}, + cfg: Config{PathPrefixes: []PathPrefixRule{{DefaultACLs: []DefaultACLRule{valid}}}}, wantErr: true, }, { name: "invalid acl type", cfg: Config{PathPrefixes: []PathPrefixRule{ - {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{ + {Prefix: "/eos/user", DefaultACLs: []DefaultACLRule{ {Type: "x", Qualifier: "q", Permissions: "rx", Enforcement: EnforcementMay}, }}, }}, @@ -125,7 +108,7 @@ func TestConfigValidate(t *testing.T) { { name: "missing qualifier", cfg: Config{PathPrefixes: []PathPrefixRule{ - {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{ + {Prefix: "/eos/user", DefaultACLs: []DefaultACLRule{ {Type: "u", Permissions: "rx", Enforcement: EnforcementMay}, }}, }}, @@ -134,7 +117,7 @@ func TestConfigValidate(t *testing.T) { { name: "missing permissions", cfg: Config{PathPrefixes: []PathPrefixRule{ - {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{ + {Prefix: "/eos/user", DefaultACLs: []DefaultACLRule{ {Type: "u", Qualifier: "q", Enforcement: EnforcementMay}, }}, }}, @@ -143,7 +126,7 @@ func TestConfigValidate(t *testing.T) { { name: "invalid enforcement", cfg: Config{PathPrefixes: []PathPrefixRule{ - {Prefix: "/eos/user", SpaceType: SpaceTypePersonal, DefaultACLs: []DefaultACLRule{ + {Prefix: "/eos/user", DefaultACLs: []DefaultACLRule{ {Type: "u", Qualifier: "q", Permissions: "rx", Enforcement: "sometimes"}, }}, }}, diff --git a/pkg/reconciliation/defaults.go b/pkg/reconciliation/defaults.go new file mode 100644 index 0000000000..830e97a199 --- /dev/null +++ b/pkg/reconciliation/defaults.go @@ -0,0 +1,108 @@ +// Copyright 2018-2026 CERN +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// In applying this license, CERN does not waive the privileges and immunities +// granted to it by virtue of its status as an Intergovernmental Organization +// or submit itself to any jurisdiction. + +package reconciliation + +import ( + "strings" + + "github.com/cs3org/reva/v3/pkg/storage/utils/acl" + "github.com/pkg/errors" +) + +// DefaultEntry is a default ACL entry that must or may exist on every path in a +// space, independent of any particular node. The full-namespace and per-space +// jobs turn each DefaultEntry into an ExpectedACL at every node they visit. +type DefaultEntry struct { + // Entry is the resolved ACL entry (templates already substituted). + Entry *acl.Entry + // Enforcement is "must" or "may". + Enforcement Enforcement +} + +// DefaultACLs computes the default ACL entries for space from the configured +// path-prefix rules. It returns the entries of the rule whose prefix is a path +// prefix of the space root. Prefixes may not overlap, so at most one rule +// matches; the qualifier templates {owner} and {project} are resolved against +// the space. Returns nil if no rule matches. +func (c *Config) DefaultACLs(space *Space) ([]DefaultEntry, error) { + if space == nil { + return nil, errors.New("reconciliation: nil space") + } + switch space.Type { + case SpaceTypePersonal, SpaceTypeProject: + default: + return nil, errors.Errorf("reconciliation: space %q has invalid type %q", space.ID, space.Type) + } + + for i := range c.PathPrefixes { + rule := &c.PathPrefixes[i] + if !pathHasPrefix(space.Root, rule.Prefix) { + continue + } + out := make([]DefaultEntry, 0, len(rule.DefaultACLs)) + for j := range rule.DefaultACLs { + d := &rule.DefaultACLs[j] + qualifier, err := resolveTemplate(d.Qualifier, space) + if err != nil { + return nil, errors.Wrapf(err, "reconciliation: path_prefix[%d].default_acl[%d]", i, j) + } + out = append(out, DefaultEntry{ + Entry: &acl.Entry{ + Type: d.Type, + Qualifier: qualifier, + Permissions: d.Permissions, + }, + Enforcement: d.Enforcement, + }) + } + return out, nil + } + return nil, nil +} + +// pathHasPrefix reports whether path lies at or under prefix, comparing whole +// path components so that "/eos/user" is a prefix of "/eos/user/j/jdoe" but not +// of "/eos/username". An empty prefix matches any path. +func pathHasPrefix(path, prefix string) bool { + path = strings.TrimRight(path, "/") + prefix = strings.TrimRight(prefix, "/") + if prefix == "" { + return true + } + return path == prefix || strings.HasPrefix(path, prefix+"/") +} + +// resolveTemplate substitutes the {owner} and {project} placeholders in a +// qualifier with the space's owner and project. It is an error to use a +// placeholder the space does not populate. +func resolveTemplate(qualifier string, space *Space) (string, error) { + if strings.Contains(qualifier, "{owner}") { + if space.Owner == "" { + return "", errors.Errorf("qualifier %q uses {owner} but space has no owner", qualifier) + } + qualifier = strings.ReplaceAll(qualifier, "{owner}", space.Owner) + } + if strings.Contains(qualifier, "{project}") { + if space.Project == "" { + return "", errors.Errorf("qualifier %q uses {project} but space has no project", qualifier) + } + qualifier = strings.ReplaceAll(qualifier, "{project}", space.Project) + } + return qualifier, nil +} diff --git a/pkg/reconciliation/defaults_test.go b/pkg/reconciliation/defaults_test.go new file mode 100644 index 0000000000..9367b2fb27 --- /dev/null +++ b/pkg/reconciliation/defaults_test.go @@ -0,0 +1,183 @@ +// Copyright 2018-2026 CERN +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// In applying this license, CERN does not waive the privileges and immunities +// granted to it by virtue of its status as an Intergovernmental Organization +// or submit itself to any jurisdiction. + +package reconciliation + +import ( + "testing" + + "github.com/cs3org/reva/v3/pkg/storage/utils/acl" +) + +// entryKey renders a DefaultEntry as a stable string for comparison. +func entryKey(d DefaultEntry) string { + return string(d.Enforcement) + " " + d.Entry.Type + ":" + d.Entry.Qualifier + ":" + d.Entry.Permissions +} + +func keys(ds []DefaultEntry) map[string]bool { + m := make(map[string]bool, len(ds)) + for _, d := range ds { + m[entryKey(d)] = true + } + return m +} + +// personalRule is the single rule governing personal spaces: the owner "must" +// entry alongside the global "may" entries. +func personalRule() PathPrefixRule { + return PathPrefixRule{ + Prefix: "/eos/user", + DefaultACLs: []DefaultACLRule{ + {Type: acl.TypeUser, Qualifier: "{owner}", Permissions: "rwx", Enforcement: EnforcementMust}, + {Type: acl.TypeGroup, Qualifier: "cbackeosro", Permissions: "rx", Enforcement: EnforcementMay}, + {Type: acl.TypeGroup, Qualifier: "cboxexternal", Permissions: "rx", Enforcement: EnforcementMay}, + }, + } +} + +// projectRule is the single rule governing project spaces: the reader/writer/ +// admin "must" egroups alongside the global "may" entries. +func projectRule() PathPrefixRule { + return PathPrefixRule{ + Prefix: "/eos/project", + DefaultACLs: []DefaultACLRule{ + {Type: acl.TypeGroup, Qualifier: "cernbox-project-{project}-admins", Permissions: "rwx+d", Enforcement: EnforcementMust}, + {Type: acl.TypeGroup, Qualifier: "cernbox-project-{project}-writers", Permissions: "rwx+d", Enforcement: EnforcementMust}, + {Type: acl.TypeGroup, Qualifier: "cernbox-project-{project}-readers", Permissions: "rx", Enforcement: EnforcementMust}, + {Type: acl.TypeGroup, Qualifier: "cbackeosro", Permissions: "rx", Enforcement: EnforcementMay}, + {Type: acl.TypeGroup, Qualifier: "cboxexternal", Permissions: "rx", Enforcement: EnforcementMay}, + }, + } +} + +func TestDefaultACLsPersonal(t *testing.T) { + // Both rules are configured; only the personal one governs a personal space. + c := &Config{PathPrefixes: []PathPrefixRule{personalRule(), projectRule()}} + + space := &Space{ID: "s1", Type: SpaceTypePersonal, Root: "/eos/user/j/jdoe", Owner: "jdoe"} + got, err := c.DefaultACLs(space) + if err != nil { + t.Fatalf("DefaultACLs: %v", err) + } + + want := map[string]bool{ + "must u:jdoe:rwx": true, + "may egroup:cbackeosro:rx": true, + "may egroup:cboxexternal:rx": true, + } + if g := keys(got); !equalSet(g, want) { + t.Errorf("entries = %v, want %v", g, want) + } +} + +func TestDefaultACLsProject(t *testing.T) { + c := &Config{PathPrefixes: []PathPrefixRule{personalRule(), projectRule()}} + + space := &Space{ID: "s2", Type: SpaceTypeProject, Root: "/eos/project/c/cernbox", Project: "cernbox"} + got, err := c.DefaultACLs(space) + if err != nil { + t.Fatalf("DefaultACLs: %v", err) + } + + want := map[string]bool{ + "must egroup:cernbox-project-cernbox-admins:rwx+d": true, + "must egroup:cernbox-project-cernbox-writers:rwx+d": true, + "must egroup:cernbox-project-cernbox-readers:rx": true, + "may egroup:cbackeosro:rx": true, + "may egroup:cboxexternal:rx": true, + } + if g := keys(got); !equalSet(g, want) { + t.Errorf("entries = %v, want %v", g, want) + } +} + +func TestDefaultACLsNoMatch(t *testing.T) { + c := &Config{PathPrefixes: []PathPrefixRule{personalRule()}} + // A project space with no project rule configured: no defaults apply. + space := &Space{ID: "s3", Type: SpaceTypeProject, Root: "/eos/project/c/cernbox", Project: "cernbox"} + got, err := c.DefaultACLs(space) + if err != nil { + t.Fatalf("DefaultACLs: %v", err) + } + if len(got) != 0 { + t.Errorf("entries = %v, want none", keys(got)) + } +} + +func TestDefaultACLsErrors(t *testing.T) { + ownerRule := PathPrefixRule{ + Prefix: "/eos/user", + DefaultACLs: []DefaultACLRule{ + {Type: acl.TypeUser, Qualifier: "{owner}", Permissions: "rwx", Enforcement: EnforcementMust}, + }, + } + + t.Run("invalid space type", func(t *testing.T) { + c := &Config{PathPrefixes: []PathPrefixRule{ownerRule}} + if _, err := c.DefaultACLs(&Space{ID: "x", Type: "bogus", Root: "/eos/user/j/jdoe"}); err == nil { + t.Error("expected error for invalid space type") + } + }) + + t.Run("nil space", func(t *testing.T) { + c := &Config{} + if _, err := c.DefaultACLs(nil); err == nil { + t.Error("expected error for nil space") + } + }) + + t.Run("unresolved owner template", func(t *testing.T) { + c := &Config{PathPrefixes: []PathPrefixRule{ownerRule}} + // Personal space matching the rule but without an owner set. + if _, err := c.DefaultACLs(&Space{ID: "x", Type: SpaceTypePersonal, Root: "/eos/user/j/jdoe"}); err == nil { + t.Error("expected error for missing owner") + } + }) +} + +func TestPathHasPrefix(t *testing.T) { + tests := []struct { + path, prefix string + want bool + }{ + {"/eos/user/j/jdoe", "/eos/user", true}, + {"/eos/user", "/eos/user", true}, + {"/eos/user/", "/eos/user", true}, + {"/eos/username/x", "/eos/user", false}, + {"/eos/project/c/cernbox", "/eos/user", false}, + {"/eos/anything", "", true}, + {"/eos/user/j/jdoe", "/eos/user/", true}, + } + for _, tt := range tests { + if got := pathHasPrefix(tt.path, tt.prefix); got != tt.want { + t.Errorf("pathHasPrefix(%q, %q) = %v, want %v", tt.path, tt.prefix, got, tt.want) + } + } +} + +func equalSet(a, b map[string]bool) bool { + if len(a) != len(b) { + return false + } + for k := range a { + if !b[k] { + return false + } + } + return true +} diff --git a/pkg/reconciliation/reconcile.go b/pkg/reconciliation/reconcile.go index 9e9737dd3a..74674a4a05 100644 --- a/pkg/reconciliation/reconcile.go +++ b/pkg/reconciliation/reconcile.go @@ -37,9 +37,6 @@ const ( SpaceTypePersonal SpaceType = "personal" // SpaceTypeProject is a project space, shared for collaborative work. SpaceTypeProject SpaceType = "project" - // SpaceTypeAny matches any space type. It is valid only in configuration - // rules (see PathPrefixRule), never as the type of an actual Space. - SpaceTypeAny SpaceType = "global" ) // Space is a storage space to reconcile. Spaces are disjoint: reconciliation diff --git a/plan.md b/plan.md index 69f5169fd2..751f28649e 100644 --- a/plan.md +++ b/plan.md @@ -229,8 +229,7 @@ they cover. Level 1 shares none of this; it only reads and marks the DB. ``` [[path_prefix]] - prefix = "/eos/user" - space_type = "personal" # personal | project | global + prefix = "/eos/user" [[path_prefix.default_acl]] type = "u" # u | egroup | lw (see package acl) qualifier = "{owner}" # may contain {owner} / {project} @@ -243,9 +242,12 @@ they cover. Level 1 shares none of this; it only reads and marks the DB. enforcement = "may" ``` -The default ACL entry is given as explicit `type` / `qualifier` / `permissions` rather than a -single opaque token, so it is unambiguous and validatable at config load. `defaults.go` -resolves the `{owner}` / `{project}` templates in the qualifier per space. +A space is governed by the single rule whose prefix is a path prefix of its root. Prefixes may +not overlap (e.g. `/eos/user` vs `/eos/project`), so at most one rule matches and there is no +space_type or priority to reason about. The default ACL entry is given as explicit `type` / +`qualifier` / `permissions` rather than a single opaque token, so it is unambiguous and +validatable at config load. `defaults.go` resolves the `{owner}` / `{project}` templates in +the qualifier per space. Semantics, matching the spec: * Global defaults (`cbackeosro`, `cboxexternal`): `enforcement = "may"`. Present is fine, @@ -306,7 +308,7 @@ Progress is tracked with a `[x]` (done) or `[ ]` (todo) marker on each step head Tests: config decode/defaults, rule validation (bad enforcement, missing prefix). Depends on: nothing. -**Step 2: default-ACL computation.** `[ ]` +**Step 2: default-ACL computation.** `[x]` done. `defaults.go`: given a `Space` and the configured rules, produce the default ACL entries, resolving `{owner}`/`{project}` templates. Encodes the personal-owner and project-egroup rules and the global `may` entries. From 9180c236d2d6835e1d256277d00bfbf844ac84d3 Mon Sep 17 00:00:00 2001 From: Jesse Geens Date: Wed, 22 Jul 2026 14:16:35 +0200 Subject: [PATCH 4/4] implement orphan job --- .../{defaults.go => default_acls.go} | 0 ...{defaults_test.go => default_acls_test.go} | 0 pkg/reconciliation/orphan.go | 230 +++++++++++++ pkg/reconciliation/orphan_test.go | 320 ++++++++++++++++++ plan.md | 144 +++----- 5 files changed, 593 insertions(+), 101 deletions(-) rename pkg/reconciliation/{defaults.go => default_acls.go} (100%) rename pkg/reconciliation/{defaults_test.go => default_acls_test.go} (100%) create mode 100644 pkg/reconciliation/orphan.go create mode 100644 pkg/reconciliation/orphan_test.go diff --git a/pkg/reconciliation/defaults.go b/pkg/reconciliation/default_acls.go similarity index 100% rename from pkg/reconciliation/defaults.go rename to pkg/reconciliation/default_acls.go diff --git a/pkg/reconciliation/defaults_test.go b/pkg/reconciliation/default_acls_test.go similarity index 100% rename from pkg/reconciliation/defaults_test.go rename to pkg/reconciliation/default_acls_test.go diff --git a/pkg/reconciliation/orphan.go b/pkg/reconciliation/orphan.go new file mode 100644 index 0000000000..7746f7528b --- /dev/null +++ b/pkg/reconciliation/orphan.go @@ -0,0 +1,230 @@ +// Copyright 2018-2026 CERN +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// In applying this license, CERN does not waive the privileges and immunities +// granted to it by virtue of its status as an Intergovernmental Organization +// or submit itself to any jurisdiction. + +package reconciliation + +import ( + "context" + "strconv" + + gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1" + grouppb "github.com/cs3org/go-cs3apis/cs3/identity/group/v1beta1" + userpb "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1" + rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1" + collaboration "github.com/cs3org/go-cs3apis/cs3/sharing/collaboration/v1beta1" + provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1" + "github.com/cs3org/reva/v3/pkg/appctx" + "github.com/cs3org/reva/v3/pkg/rjobs" + "github.com/cs3org/reva/v3/pkg/share/manager/sql/model" + "github.com/pkg/errors" +) + +// OrphanJobName is the stable identity of the orphan job. +const OrphanJobName = "reconciliation.orphans" + +// OrphanReason says why a share was marked orphaned. +type OrphanReason string + +const ( + // ReasonResourceMissing means the shared resource no longer exists. Its + // storage space being gone shows up here too, since the resource Stat then + // fails. + ReasonResourceMissing OrphanReason = "resource-missing" + // ReasonRecipientMissing means the user or group the share is for no longer + // exists. + ReasonRecipientMissing OrphanReason = "recipient-missing" +) + +// ShareStore is the subset of the share manager the orphan job needs. +// *sql.ShareMgr satisfies it. +type ShareStore interface { + // ListModelShares returns the shares matching the filters. Pass a nil user + // to list across all owners and hideOrphans=true to skip already-orphaned + // shares. + ListModelShares(u *userpb.User, filters []*collaboration.Filter, hideOrphans bool) ([]model.Share, error) + // MarkAsOrphaned flags the referenced share as orphaned. + MarkAsOrphaned(ctx context.Context, ref *collaboration.ShareReference) error +} + +// OrphanJob marks shares whose resource or recipient is gone as orphaned. It is +// idempotent: a share already orphaned is skipped by the hideOrphans filter, and +// re-running never marks a valid share. +type OrphanJob struct { + // Shares is the share store to scan and mutate. + Shares ShareStore + // Gateway resolves resource and recipient existence. + Gateway gateway.GatewayAPIClient + // DryRun, when set, reports what would be orphaned without mutating. + DryRun bool +} + +// OrphanedShare records one share the job orphaned (or, in dry-run, would have). +type OrphanedShare struct { + // ShareID is the CS3 opaque id of the share. + ShareID string + // Reason is why it was orphaned. + Reason OrphanReason + // ResourceID is the shared resource, for logging. + ResourceID *provider.ResourceId + // ShareWith is the recipient (username, group name or external id), for + // logging. + ShareWith string +} + +// OrphanReport summarises a run. +type OrphanReport struct { + // Checked is the number of non-orphan shares examined. + Checked int + // Skipped is the number of shares left undecided because a lookup failed. + Skipped int + // Orphaned lists the shares marked (or, in dry-run, that would be marked). + Orphaned []OrphanedShare + // DryRun reports whether the run was a simulation. + DryRun bool +} + +// Run scans the share store and orphans shares whose resource or recipient is +// gone. A per-share lookup failure is logged and the share is skipped, never +// orphaned, so a flaky gateway can never produce a false orphan. The run itself +// only fails if the shares cannot be listed at all. +func (j *OrphanJob) Run(ctx context.Context) (OrphanReport, error) { + log := appctx.GetLogger(ctx) + + shares, err := j.Shares.ListModelShares(nil, nil, true) + if err != nil { + return OrphanReport{}, errors.Wrap(err, "reconciliation: listing shares") + } + + report := OrphanReport{DryRun: j.DryRun} + for i := range shares { + s := &shares[i] + report.Checked++ + + reason, orphaned, err := j.classify(ctx, s) + if err != nil { + report.Skipped++ + log.Error().Err(err).Uint("share_id", s.Id).Msg("reconciliation: existence check failed, skipping share") + continue + } + if !orphaned { + continue + } + + rec := OrphanedShare{ + ShareID: strconv.FormatUint(uint64(s.Id), 10), + Reason: reason, + ResourceID: &provider.ResourceId{StorageId: s.Instance, OpaqueId: s.Inode}, + ShareWith: s.ShareWith, + } + report.Orphaned = append(report.Orphaned, rec) + + if j.DryRun { + log.Info().Str("share_id", rec.ShareID).Str("reason", string(reason)).Msg("reconciliation: would mark share orphaned (dry_run)") + continue + } + + if err := j.Shares.MarkAsOrphaned(ctx, shareRefByID(s.Id)); err != nil { + log.Error().Err(err).Str("share_id", rec.ShareID).Msg("reconciliation: marking share orphaned failed") + continue + } + log.Info().Str("share_id", rec.ShareID).Str("reason", string(reason)).Msg("reconciliation: marked share orphaned") + } + + return report, nil +} + +// classify decides whether a share is orphaned and why, using gateway lookups. +// It returns an error only on a lookup failure, which the caller treats as +// "undecided", not as absence. +func (j *OrphanJob) classify(ctx context.Context, s *model.Share) (OrphanReason, bool, error) { + statRes, err := j.Gateway.Stat(ctx, &provider.StatRequest{ + Ref: &provider.Reference{ResourceId: &provider.ResourceId{StorageId: s.Instance, OpaqueId: s.Inode}}, + }) + if err != nil { + return "", false, errors.Wrap(err, "reconciliation: stat") + } + if exists, err := existsFromStatus(statRes.GetStatus()); err != nil { + return "", false, err + } else if !exists { + return ReasonResourceMissing, true, nil + } + + var st *rpc.Status + if s.SharedWithIsGroup { + res, err := j.Gateway.GetGroupByClaim(ctx, &grouppb.GetGroupByClaimRequest{ + Claim: "group_name", Value: s.ShareWith, SkipFetchingMembers: true, + }) + if err != nil { + return "", false, errors.Wrap(err, "reconciliation: get group") + } + st = res.GetStatus() + } else { + res, err := j.Gateway.GetUserByClaim(ctx, &userpb.GetUserByClaimRequest{ + Claim: "username", Value: s.ShareWith, SkipFetchingUserGroups: true, + }) + if err != nil { + return "", false, errors.Wrap(err, "reconciliation: get user") + } + st = res.GetStatus() + } + if exists, err := existsFromStatus(st); err != nil { + return "", false, err + } else if !exists { + return ReasonRecipientMissing, true, nil + } + + return "", false, nil +} + +// existsFromStatus maps a CS3 status to existence: OK is present, NOT_FOUND is a +// confirmed absence, anything else is a real error (undecided, not absent). +func existsFromStatus(s *rpc.Status) (bool, error) { + switch s.GetCode() { + case rpc.Code_CODE_OK: + return true, nil + case rpc.Code_CODE_NOT_FOUND: + return false, nil + default: + return false, errors.Errorf("reconciliation: unexpected status %s: %s", s.GetCode(), s.GetMessage()) + } +} + +// shareRefByID builds a share reference addressing a share by its numeric id. +func shareRefByID(id uint) *collaboration.ShareReference { + return &collaboration.ShareReference{ + Spec: &collaboration.ShareReference_Id{ + Id: &collaboration.ShareId{OpaqueId: strconv.FormatUint(uint64(id), 10)}, + }, + } +} + +// Periodic wraps the job as an rjobs.Periodic. It runs on the leader because it +// mutates shared database state, and skips a fire if the previous run is still +// going. +func (j *OrphanJob) Periodic(schedule string) rjobs.Periodic { + return rjobs.Periodic{ + Name: OrphanJobName, + Schedule: schedule, + Scope: rjobs.ScopeLeader, + Overlap: rjobs.Skip, + Run: func(ctx context.Context) error { + _, err := j.Run(ctx) + return err + }, + } +} diff --git a/pkg/reconciliation/orphan_test.go b/pkg/reconciliation/orphan_test.go new file mode 100644 index 0000000000..da4160ab52 --- /dev/null +++ b/pkg/reconciliation/orphan_test.go @@ -0,0 +1,320 @@ +// Copyright 2018-2026 CERN +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// In applying this license, CERN does not waive the privileges and immunities +// granted to it by virtue of its status as an Intergovernmental Organization +// or submit itself to any jurisdiction. + +package reconciliation + +import ( + "context" + "errors" + "sort" + "testing" + + gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1" + grouppb "github.com/cs3org/go-cs3apis/cs3/identity/group/v1beta1" + userpb "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1" + rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1" + collaboration "github.com/cs3org/go-cs3apis/cs3/sharing/collaboration/v1beta1" + provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1" + "github.com/cs3org/reva/v3/pkg/share/manager/sql/model" + "google.golang.org/grpc" +) + +// fakeStore is an in-memory ShareStore recording which shares were marked. +type fakeStore struct { + shares []model.Share + marked []string + listErr error + markErr error +} + +func (f *fakeStore) ListModelShares(u *userpb.User, filters []*collaboration.Filter, hideOrphans bool) ([]model.Share, error) { + if f.listErr != nil { + return nil, f.listErr + } + if !hideOrphans { + return f.shares, nil + } + var out []model.Share + for _, s := range f.shares { + if !s.Orphan { + out = append(out, s) + } + } + return out, nil +} + +func (f *fakeStore) MarkAsOrphaned(ctx context.Context, ref *collaboration.ShareReference) error { + if f.markErr != nil { + return f.markErr + } + f.marked = append(f.marked, ref.GetId().GetOpaqueId()) + return nil +} + +// fakeGateway is a gateway client driven by presence sets. Only the three +// methods the orphan job calls are implemented; the embedded interface makes any +// other call panic, which keeps the fake honest. +type fakeGateway struct { + gateway.GatewayAPIClient + resources map[string]bool + users map[string]bool + groups map[string]bool + statErr error + userErr error + groupErr error +} + +func status(present bool) *rpc.Status { + if present { + return &rpc.Status{Code: rpc.Code_CODE_OK} + } + return &rpc.Status{Code: rpc.Code_CODE_NOT_FOUND} +} + +func (f *fakeGateway) Stat(ctx context.Context, in *provider.StatRequest, _ ...grpc.CallOption) (*provider.StatResponse, error) { + if f.statErr != nil { + return nil, f.statErr + } + id := in.GetRef().GetResourceId() + return &provider.StatResponse{Status: status(f.resources[id.StorageId+"/"+id.OpaqueId])}, nil +} + +func (f *fakeGateway) GetUserByClaim(ctx context.Context, in *userpb.GetUserByClaimRequest, _ ...grpc.CallOption) (*userpb.GetUserByClaimResponse, error) { + if f.userErr != nil { + return nil, f.userErr + } + return &userpb.GetUserByClaimResponse{Status: status(f.users[in.GetValue()])}, nil +} + +func (f *fakeGateway) GetGroupByClaim(ctx context.Context, in *grouppb.GetGroupByClaimRequest, _ ...grpc.CallOption) (*grouppb.GetGroupByClaimResponse, error) { + if f.groupErr != nil { + return nil, f.groupErr + } + return &grouppb.GetGroupByClaimResponse{Status: status(f.groups[in.GetValue()])}, nil +} + +// share builds a model.Share with the fields the orphan job reads. +func share(id uint, instance, inode, shareWith string, isGroup, orphan bool) model.Share { + var s model.Share + s.Id = id + s.Orphan = orphan + s.Instance = instance + s.Inode = inode + s.ShareWith = shareWith + s.SharedWithIsGroup = isGroup + return s +} + +func sortedMarked(f *fakeStore) []string { + out := append([]string(nil), f.marked...) + sort.Strings(out) + return out +} + +func TestOrphanResourceMissing(t *testing.T) { + store := &fakeStore{shares: []model.Share{ + share(1, "eosuser", "inode-1", "jdoe", false, false), + }} + gw := &fakeGateway{ + resources: map[string]bool{}, // resource gone + users: map[string]bool{"jdoe": true}, + } + job := &OrphanJob{Shares: store, Gateway: gw} + + report, err := job.Run(context.Background()) + if err != nil { + t.Fatalf("Run: %v", err) + } + if report.Checked != 1 || len(report.Orphaned) != 1 { + t.Fatalf("report = %+v, want 1 checked / 1 orphaned", report) + } + if report.Orphaned[0].Reason != ReasonResourceMissing { + t.Errorf("reason = %q, want %q", report.Orphaned[0].Reason, ReasonResourceMissing) + } + if got := sortedMarked(store); len(got) != 1 || got[0] != "1" { + t.Errorf("marked = %v, want [1]", got) + } +} + +func TestOrphanUserRecipientMissing(t *testing.T) { + store := &fakeStore{shares: []model.Share{ + share(2, "eosuser", "inode-2", "ghost", false, false), + }} + gw := &fakeGateway{ + resources: map[string]bool{"eosuser/inode-2": true}, + users: map[string]bool{}, // user gone + } + job := &OrphanJob{Shares: store, Gateway: gw} + + report, err := job.Run(context.Background()) + if err != nil { + t.Fatalf("Run: %v", err) + } + if len(report.Orphaned) != 1 || report.Orphaned[0].Reason != ReasonRecipientMissing { + t.Fatalf("report = %+v, want 1 recipient-missing", report) + } + if got := sortedMarked(store); len(got) != 1 || got[0] != "2" { + t.Errorf("marked = %v, want [2]", got) + } +} + +func TestOrphanGroupRecipientMissing(t *testing.T) { + store := &fakeStore{shares: []model.Share{ + share(3, "eosproject", "inode-3", "defunct-group", true, false), + }} + gw := &fakeGateway{ + resources: map[string]bool{"eosproject/inode-3": true}, + groups: map[string]bool{}, // group gone + users: map[string]bool{"defunct-group": true}, // must be ignored: it is a group + } + job := &OrphanJob{Shares: store, Gateway: gw} + + report, err := job.Run(context.Background()) + if err != nil { + t.Fatalf("Run: %v", err) + } + if len(report.Orphaned) != 1 || report.Orphaned[0].Reason != ReasonRecipientMissing { + t.Fatalf("report = %+v, want 1 recipient-missing", report) + } + if got := sortedMarked(store); len(got) != 1 || got[0] != "3" { + t.Errorf("marked = %v, want [3]", got) + } +} + +func TestOrphanAllPresentMarksNothing(t *testing.T) { + store := &fakeStore{shares: []model.Share{ + share(4, "eosuser", "inode-4", "jdoe", false, false), + share(5, "eosproject", "inode-5", "cern-users", true, false), + }} + gw := &fakeGateway{ + resources: map[string]bool{"eosuser/inode-4": true, "eosproject/inode-5": true}, + users: map[string]bool{"jdoe": true}, + groups: map[string]bool{"cern-users": true}, + } + job := &OrphanJob{Shares: store, Gateway: gw} + + report, err := job.Run(context.Background()) + if err != nil { + t.Fatalf("Run: %v", err) + } + if report.Checked != 2 || len(report.Orphaned) != 0 { + t.Fatalf("report = %+v, want 2 checked / 0 orphaned", report) + } + if len(store.marked) != 0 { + t.Errorf("marked = %v, want none", store.marked) + } +} + +func TestOrphanDryRunMarksNothing(t *testing.T) { + store := &fakeStore{shares: []model.Share{ + share(6, "eosuser", "inode-6", "jdoe", false, false), + }} + gw := &fakeGateway{resources: map[string]bool{}, users: map[string]bool{"jdoe": true}} + job := &OrphanJob{Shares: store, Gateway: gw, DryRun: true} + + report, err := job.Run(context.Background()) + if err != nil { + t.Fatalf("Run: %v", err) + } + if !report.DryRun || len(report.Orphaned) != 1 { + t.Fatalf("report = %+v, want dry-run with 1 would-orphan", report) + } + if len(store.marked) != 0 { + t.Errorf("dry_run marked %v, want none", store.marked) + } +} + +func TestOrphanLookupErrorSkips(t *testing.T) { + store := &fakeStore{shares: []model.Share{ + share(7, "eosuser", "inode-7", "jdoe", false, false), + }} + gw := &fakeGateway{statErr: errors.New("gateway down")} + job := &OrphanJob{Shares: store, Gateway: gw} + + report, err := job.Run(context.Background()) + if err != nil { + t.Fatalf("Run must not fail on a per-share lookup error: %v", err) + } + if report.Skipped != 1 || len(report.Orphaned) != 0 { + t.Fatalf("report = %+v, want 1 skipped / 0 orphaned", report) + } + if len(store.marked) != 0 { + t.Errorf("marked %v on lookup error, want none (no false orphan)", store.marked) + } +} + +func TestOrphanAlreadyOrphanExcluded(t *testing.T) { + store := &fakeStore{shares: []model.Share{ + share(8, "eosuser", "inode-8", "jdoe", false, true), // already orphan, resource also gone + }} + gw := &fakeGateway{resources: map[string]bool{}, users: map[string]bool{}} + job := &OrphanJob{Shares: store, Gateway: gw} + + report, err := job.Run(context.Background()) + if err != nil { + t.Fatalf("Run: %v", err) + } + if report.Checked != 0 || len(report.Orphaned) != 0 { + t.Fatalf("report = %+v, want 0 checked (already-orphan filtered out)", report) + } + if len(store.marked) != 0 { + t.Errorf("marked %v, want none", store.marked) + } +} + +func TestOrphanMixedBatch(t *testing.T) { + store := &fakeStore{shares: []model.Share{ + share(10, "eosuser", "inode-10", "jdoe", false, false), // valid + share(11, "eosuser", "inode-11", "ghost", false, false), // recipient gone + share(12, "eosproject", "inode-gone", "cern-users", true, false), // resource gone + share(13, "eosuser", "inode-13", "jdoe", false, true), // already orphan, excluded + }} + gw := &fakeGateway{ + resources: map[string]bool{"eosuser/inode-10": true, "eosuser/inode-11": true}, + users: map[string]bool{"jdoe": true}, + groups: map[string]bool{"cern-users": true}, + } + job := &OrphanJob{Shares: store, Gateway: gw} + + report, err := job.Run(context.Background()) + if err != nil { + t.Fatalf("Run: %v", err) + } + if report.Checked != 3 || len(report.Orphaned) != 2 { + t.Fatalf("report = %+v, want 3 checked / 2 orphaned", report) + } + if got, want := sortedMarked(store), []string{"11", "12"}; len(got) != len(want) || got[0] != want[0] || got[1] != want[1] { + t.Errorf("marked = %v, want %v", got, want) + } +} + +func TestOrphanListErrorFails(t *testing.T) { + store := &fakeStore{listErr: errors.New("db down")} + job := &OrphanJob{Shares: store, Gateway: &fakeGateway{}} + + if _, err := job.Run(context.Background()); err == nil { + t.Fatal("Run must fail when shares cannot be listed") + } +} + +func TestShareRefByID(t *testing.T) { + ref := shareRefByID(42) + if got := ref.GetId().GetOpaqueId(); got != "42" { + t.Errorf("opaque id = %q, want %q", got, "42") + } +} diff --git a/plan.md b/plan.md index 751f28649e..f78b8e77a1 100644 --- a/plan.md +++ b/plan.md @@ -114,7 +114,7 @@ Proposed layout. Two files carry the naming that needs explaining up front: pkg/reconciliation/ reconcile.go // shared types: Space, Recipient, ExpectedACL, Plan, Action, Outcome config.go // Config + ApplyDefaults, path_prefix -> default-ACL rules (can/should) - defaults.go // default-ACL computation per space type (owner, project egroups, globals) + default_acls.go // default-ACL computation per space type (owner, project egroups, globals) expected_acls.go // pure: (shares for a space) + defaults -> expected ACL set per path // (shared by levels 2 and 3; wraps sharehierarchy) planner.go // pure: expected ACLs vs observed ACLs -> Plan of add/remove/update @@ -246,7 +246,7 @@ A space is governed by the single rule whose prefix is a path prefix of its root not overlap (e.g. `/eos/user` vs `/eos/project`), so at most one rule matches and there is no space_type or priority to reason about. The default ACL entry is given as explicit `type` / `qualifier` / `permissions` rather than a single opaque token, so it is unambiguous and -validatable at config load. `defaults.go` resolves the `{owner}` / `{project}` templates in +validatable at config load. `default_acls.go` resolves the `{owner}` / `{project}` templates in the qualifier per space. Semantics, matching the spec: @@ -257,7 +257,7 @@ Semantics, matching the spec: * Project spaces: the readers/writers/admins egroups are three `must` entries, templated from the project name. -`defaults.go` resolves templates (`{owner}`, `{project}`) against the space. The planner +`default_acls.go` resolves templates (`{owner}`, `{project}`) against the space. The planner treats `must` entries as always-expected and `may` entries as never-diffed (neither added nor flagged), so a `may` entry present on disk is left untouched. @@ -292,103 +292,45 @@ the spec, driven by table tests: * dry_run: assert no mutation is issued and the recorded actions match what a live run would have applied (run planner once, apply in both modes, compare). + ## Work breakdown -Ordered by dependency. Each step is a self-contained unit: it builds, has its own tests, and -is meant to be one reviewable PR / commit. Steps 1 to 6 are pure and need no live services, so -they land fast and de-risk the engine before any job or EOS code. Steps 7 to 11 wire real -dependencies. Nothing after a step depends on a later step. - -Progress is tracked with a `[x]` (done) or `[ ]` (todo) marker on each step heading. - -**Step 1: core types and config.** `[x]` done. -`reconcile.go` (`Space`, `Recipient`, `ExpectedACL`, `Plan`, `Action`, `Outcome`) and -`config.go` (`Config` + `ApplyDefaults`, the `path_prefix` -> default-ACL rules with -`may`/`must` enforcement). No logic beyond decoding and validation. -Tests: config decode/defaults, rule validation (bad enforcement, missing prefix). -Depends on: nothing. - -**Step 2: default-ACL computation.** `[x]` done. -`defaults.go`: given a `Space` and the configured rules, produce the default ACL entries, -resolving `{owner}`/`{project}` templates. Encodes the personal-owner and project-egroup -rules and the global `may` entries. -Tests: personal owner `must`, project readers/writers/admins `must`, global `may`, template -resolution, wrong space type. -Depends on: 1. - -**Step 3: permission model bridge.** `[ ]` -Map the DB `permissions` (OCS uint8) and grantee fields to CS3 `ResourcePermissions` / -`Grantee`, and to `sharehierarchy.PermLevel`. Small file (`permmap.go`) plus recipient -classification stubs (user / group / lightweight) that do not yet call the gateway. -Tests: every recipient type maps to the right `Grantee`; permissions=0 is `PermDeny`, not -absent; OCS round-trips match `model.Share.AsCS3Share`. -Depends on: 1. - -**Step 4: expected-ACL reconstruction.** `[ ]` -`expected_acls.go`: given a space's shares plus its default ACLs, compute the expected ACL -set per path, wrapping `sharehierarchy` for the nearest-ancestor / reapply ordering. Pure. -Tests: all ordered `{R, RW, Deny}` parent/child pairs on nested paths, reapply and delete -cases, defaults merged in, space isolation (never crosses `space_id`). -Depends on: 2, 3. - -**Step 5: planner.** `[ ]` -`planner.go`: diff expected vs observed ACLs into an ordered `Plan` of add/remove/update. -Port and adapt cernboxcop's `set_operations.go` / `acl_change_set.go` diff. Pure. -Tests: pure add, pure remove, update (same grantee different perms), `may` present left -untouched, `must` wrong-perms updated, ordering (shallowest first). -Depends on: 4. - -**Step 6: applier with dry_run.** `[ ]` -`applier.go`: execute a `Plan` through the CS3 grant API (`AddGrant`/`RemoveGrant`/ -`UpdateGrant`/`DenyGrant`), honouring `dry_run`. Define a small gateway-client interface so -tests use a fake. -Tests: each action issues the right grant call; dry_run issues none and records the actions; -recorded actions in dry_run equal those applied live (run planner once, apply both ways). -Depends on: 5. - -**Step 7: identity resolution against the gateway.** `[ ]` -`identity.go`: resolve and validate recipients and resources through the gateway (exists / -not-found / recycled), replacing the step-3 stubs. This is the first step that talks to a -live service, still behind an interface with a fake in tests. -Tests: user/group/lightweight resolution, missing recipient, resource not-found vs recycled. -Depends on: 3. - -**Step 8: NamespaceScanner interface + EOS binary scanner.** `[ ]` -`scanner.go` (interface + `Register`/registry) in `pkg/reconciliation`; `nsscan_binary.go` + -`nsscan_loader.go` in `pkg/storage/fs/eos`, porting `ns_inspect.go` (command builder, JSON -parser, `prefetchedData` path) and registering under `eos-nsinspect-binary`. -Tests: parse captured JSON from `testdata/nsinspect` (personal + project, files + folders, -sys entries, lightweight xattrs); `prefetchedData` prefix/depth filtering. -Depends on: 1 (interface types only); independent of 4 to 7, so can proceed in parallel. - -**Step 9: level 1 orphan job.** `[ ]` -`orphans.go` + `jobs/orphans_job.go`: per-space DB scan via `ListModelShares`, gateway-based -validity checks, `MarkAsOrphaned`; register as on-demand + `ScopeLeader` periodic. Public -links via `PublicShareMgr`. -Tests: deleted / recycled resource, missing recipient, missing space; on-demand run scoped to -one space; idempotent re-run. -Depends on: 7. - -**Step 10: level 2 space-ACL job.** `[ ]` -`space_acls.go` + `jobs/spaceacls_job.go`: gather a space's non-orphan shares, build expected -ACLs, gateway-Stat the shared paths for observed grants, plan, apply. Register on-demand + -periodic. -Tests: end-to-end per recipient type and ACL combo with fakes producing a `Plan`; dry_run; -space scoping. -Depends on: 4, 6, 7. - -**Step 11: level 3 namespace job.** `[ ]` -`namespace.go` + `jobs/namespace_job.go`: list spaces, run the scanner over each space tree, -compute expected ACLs per node (defaults + inherited shares), plan, apply. Register on-demand -+ periodic (`@daily`/`@weekly`, jitter, `Skip`). Support `prefetched_scan` for offline dry -runs. -Tests: feed `testdata/nsinspect` output through the engine and assert the produced `Plan`; -default-ACL `may`/`must` on untouched paths; dry_run against a snapshot. -Depends on: 4, 6, 8. - -**Step 12: config wiring and docs.** `[ ]` -Register the three jobs' config sections under the jobs serverless service, add an example -config block, and document the `path_prefix` rules, scanner selection, and `dry_run`. No new -logic. -Tests: config decodes end to end; example config validates. -Depends on: 9, 10, 11. \ No newline at end of file +We build the simplest thing that works first, then deepen. The strategy above describes the +eventual full system; the phases below are the build order. Each phase is a self-contained, +reviewable unit that compiles and has its own tests, and each is useful on its own. A later +phase never blocks an earlier one. + +Progress marker: `[x]` done, `[ ]` todo. + +**Phase 1: orphan job.** `[x]` done. +`orphan.go`: a periodic job that scans the share DB and marks a share orphaned +when its resource or its recipient no longer exists. It lists non-orphan shares via +`ListModelShares(nil, nil, hideOrphans=true)`; for each it checks the resource +(`gateway.Stat` on `{Instance, Inode}`) and the recipient (`GetUserByClaim` for users, +`GetGroupByClaim` for groups), then marks via `MarkAsOrphaned`. A lookup error is never +treated as absence: the share is skipped, never orphaned on uncertainty. `dry_run` reports +what would be marked without mutating. Runs `ScopeLeader` because it mutates shared DB state. +Consumer-defined `ShareStore` and `ExistenceChecker` interfaces keep the logic unit-testable: +`*sql.ShareMgr` satisfies the first; the concrete CS3 gateway-backed `ExistenceChecker` is +built at service-startup wiring time (with the gateway address from config), not in this +package, so phase 1 carries no dead wiring. +Missing-space is folded into the resource check for now (if the space is gone the resource +Stat fails); a dedicated space check can come later. +Tests: resource missing, user recipient missing, group recipient missing, all present, +dry_run marks nothing, lookup error skips (no false orphan), already-orphan shares excluded, +mixed batch, share-reference by id. + +**Phase 2: shallow check (DB only).** `[ ]` +Reconcile the ACLs implied by the share DB against what is actually set on each shared path, +without a full-namespace scan. For each non-orphan share, resolve its path and read the +current grants on that single node through the gateway, then add or fix the missing/wrong +grant. Reuses the default-ACL config (`config.go` / `default_acls.go`) and the permission ordering +from `sharehierarchy`. `dry_run`. Targeted and per-share, so cost scales with the number of +shares, not the size of the namespace. + +**Phase 3: deep FS check (eos-ns-inspect).** `[ ]` +The whole-namespace sweep. Enumerate every node of a space directly from QuarkDB via +`eos-ns-inspect` (which reads the namespace, not the MGM), compare each node's actual +`sys.acl` against what the DB says it should be, and correct drift, including stray entries +that no share justifies. Behind a `NamespaceScanner` interface with the EOS binary scanner as +the first implementation; a native QDB reader could come later. `dry_run`.