missing remote objects (404/NoSuchKey) burn the full retry budget per file — fail fast + optional allow_missing_files_on_download salvage mode

[minguyen9988]

Problem

Backuper.Classify (pkg/backup/backuper.go:72) returns retrier.Retry for every non-nil error:

func (b *Backuper) Classify(err error) retrier.Action {
	if err == nil {
		return retrier.Succeed
	}
	log.Warn().Err(err).Msgf("Will wait near %s and retry", ...)
	return retrier.Retry
}

A permanently-missing remote object — GCS storage: object doesn't exist, S3 NoSuchKey/404, Azure BlobNotFound — is retried through the full retries_on_failure x retries_duration exponential backoff budget per file, even though a 404 will never heal. On a backup with many missing objects (e.g., remote storage that suffered partial data loss, or object-disk keys removed by lifecycle rules), a download burns hours in pointless retries before failing.

Second, when the operator knows the backup is partially corrupt, there is currently no way to salvage the intact tables/parts: download hard-fails on the first missing object.

Proposal

1. Add storage.IsNotFoundErr(err) that recognizes not-found errors across backends (GCS/S3/Azure/SFTP/FTP + generic 404/NoSuchKey/BlobNotFound strings), and make Classify return retrier.Fail for those → fail fast instead of burning the retry budget.
2. Add opt-in allow_missing_files_on_download (general section, default false): when enabled, a not-found data file/part archive is skipped with a loud per-file error-level log and a final summary count, letting operators salvage everything that still exists in a partially-corrupt backup. Metadata files are never skipped — only part data. Default behavior (fail on missing) is unchanged.

Extracted from a production fork used to back up multi-TiB ClickHouse clusters, where salvage mode has been used for disaster recovery of partially-corrupted remote backups.

I have a PR ready to submit.