ProjectDB PgBouncer pool sizing

[esoergel]

These three stacked PRs introduce a database-level safeguard that keeps a Project DB query from reading another domain's data, even if the query layer above it has a bug or is handed hostile SQL.

Part I: HQ Query Users (#6962)

Each domain gets its own postgres role, which has read-only access to only that domain's tables. Queries on behalf of that domain are authenticated as that user, so postgres enforces separation between tenants.

HQ provisions these query users using a constrained SECURITY DEFINER function called projectdb_provision_role which narrowly defines the action. This function is created during database provisioning. Passwords are derived from the domain name via django.utils.crypto.salted_hmac, never stored.

Part II: PgBouncer

PgBouncer connection pooling complicates things in two ways

Part IIa: PGBouncer Authentication (#6963)

PgBouncer authenticates queries itself, so it has to know user passwords. Currently, it validates against a file called userlist.txt which is managed by commcare-cloud. Adding a new user requires amending that file, which just isn't doable in this context. Instead, we give pgbouncer an auth_query it can run to get the username and password to authenticate a user that's not in userlist.txt.

To do this, we need a few things in place:

• A user_lookup function as described in the pgbouncer docs (https://www.pgbouncer.org/config.html), supporting the auth_query
• An auth_user pgbouncer can authenticate as when running the auth_query, and a password for that user
• A role for the pgbouncer auth user with only LOGIN permissions by default
  • This role must also be granted access to the user_lookup function

Part IIb: (This PR)

PgBouncer keys its pools by (user, database) pair, meaning that each domain will have it's own pool in pgbouncer. Right now, we allow up to 490 open connections per pool (default_pool_size). Since ProjectDB uses many pools, we can instead set a max_db_connections as a limit across all ProjectDB pools, and set a much lower pool_size to prevent one domain from claiming all that for itself.

Environments Affected

Staging, since it has project_db configured already. Should only affect other environments when that's enabled.

Announce New Release

No. Shouldn't require action by anyone but me.