pycore: don't stop source folder detection at the first package

[quazardous]

The defect

_find_source_folders returns as soon as a folder has a package among its children:

def _find_source_folders(self, folder):
    for resource in folder.get_folders():
        if self._is_package(resource):
            return [folder]          # every sibling folder goes unscanned

So on a project whose root holds a package, detection stops at the root and no other source folder is ever found.

Why this looks like a defect rather than a design choice

Two existing tests already describe side-by-side roots:

test	expects
test_multi_source_folders	src and test
test_multi_source_folders2	root and src

Both pass today only because in their layouts no package sits directly at the root, so the early return never fires. Multi-root detection is clearly intended; the early return silently disables it for any repository that grew a package at its top level.

Why it matters

The consequence is silent, which is the worst part. Imports that would resolve through the unscanned folders simply do not resolve, so Rename skips those occurrences rather than reporting them — a repository-wide rename comes back having quietly renamed a subset. There is no error and no unsure occurrence to review.

The change

Apply the same rule at every level instead of stopping at the first match, and skip package subfolders when recursing — descending into a package would make package.sub importable as plain sub.

Two tests added. The first fails on master; the second guards the subpackage boundary and passes either way.

Measurements

On a real project whose root holds two packages with further code under src/:

	before	after
source folders detected	1	15
from pkg import mod	unresolved	src/pkg/mod.py
a rename that was already correct	3.2s	17.0s

ropetest/: 2122 passed, 7 skipped, 5 xfailed.

On the cost, plainly: this spends time where detection previously skipped work, so a project that gains source folders pays for analysing code that was invisible before. It is not overhead, but it is real, and on a project where the old single root was the right answer there is nothing to gain and time to lose. source_folders in the prefs still overrides detection for anyone who wants to narrow it back.

I measured a second case where the coverage actually changes — an occurrence resolving 0 of 2 references before and 2 of 2 after — but that rename needs #868 to run at all on this codebase, so I am not claiming it for this PR alone.

Relation to #868

Independent, and based on master. #868 fixes the walker matching tokens inside string literals; this one fixes which folders are searched. Either can land first.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V5GqJKaXZzFzmFEYgo4pPR

[lieryan]

The original implementation has the advantage that for pure python projects where the root directory is the actual source folder, it would be really fast. The comment in get_source_folders seems to imply that this is an intentional goal of why it did things the way it does.

rope/rope/base/project.py

Lines 79 to 90 in 7e04460

	# INFO: It was decided not to cache source folders, since:
	# - Does not take much time when the root folder contains
	# packages, that is most of the time
	# - We need a separate resource observer; `self.observer`
	# does not get notified about module and folder creations
	def get_source_folders(self):
	"""Returns project source folders"""
	if self.root is None:
	return []
	result = list(self._custom_source_folders)
	result.extend(self.pycore._find_source_folders(self.root))
	return result

This change of always recursing on a package's subdirectories would cause all subfolders in a source folder to have to be scanned all the time to look for potential additional source folders, which potentially can cause some performance issues for projects that contains a huge directory with lots of non-python files, for example, if the project contains node_modules or directories of compilation output/generated files.

That said, this performance issue would have also already existed on projects that uses a non-package src directory for all their python files and also has huge non-python directories on the root directory. Projects that has unusual project structure probably already expected to have to modify their config file to have to tweak source_folders/ignored_resources prefs. But now pure python projects whose root directories are the sole source folder would potentially be getting unexpected performance degradation where they previously got the fast path.

I'm not really quite sure what the right solution here should be. On one hand, the fixed behavior is definitely more correct, but springing performance trap to unsuspecting project also doesn't seem ideal either.

[quazardous]

Thanks — that's a fair concern. Two facts that may narrow it, then I'll happily follow your lead.

The walk is already bounded twice. Folder.get_children() filters through project.is_ignored(), and get_folders() is built on it, so recursion already skips ignored_resources. And this patch never descends into a package. So the exposure isn't "all subfolders, all the time" — it's exactly the large directories rope doesn't ignore by default. node_modules is the right example: the shipped list is *.pyc, *~, .ropeproject, .hg, .svn, _svn, .git, .tox, .venv, venv, .mypy_cache, .pytest_cache. Adding node_modules, build, dist there seems defensible on its own, independent of this PR.

One caveat on "unusual layouts can set their prefs": _init_source_folders, which reads the source_folders pref, is currently marked @utils.deprecated. If declaring layouts is meant to be the escape hatch, that path probably needs to stay supported.

On the design itself — I'd rather follow your call than push my patch. But I don't think the current code is cleanly either option: the early return [folder] implements "the root layout wins", while the recursion below it implements "search everything". That's why a sibling src/ is silently invisible while a deeper non-package folder is happily searched. Either rule is defensible; the mix is what surprises.

• exhaustive — descend unless package or ignored (this PR); the cost is paid by the ignore list.
• root wins — if the root directly contains a package, it is the sole source folder; fast, and correct by definition, but mixed layouts must then declare themselves (see the caveat above).

I'm glad to rewrite this for the second option if you prefer it. I'll rebase either way — the conflict is only CHANGELOG/tests, pycore.py is untouched on master.