[#9418] fix(iceberg): inject GCS FileIO token from gcs-service-account-file by lasdf1234 · Pull Request #12962 · apache/gravitino

lasdf1234 · 2026-09-07T10:11:35Z

What changes were proposed in this pull request?

Iceberg's GCSFileIO does not understand Gravitino's gcs-service-account-file.
When that property is set, load the service account at catalog initialization and
inject Iceberg gcs.oauth2.token / gcs.oauth2.token-expires-at so server-side
FileIO can authenticate (same property-injection style as S3/OSS/ADLS key mapping).
Also update GCS docs that previously required GOOGLE_APPLICATION_CREDENTIALS
even when the catalog property was set.

Why are the changes needed?

gcs-service-account-file reached the credential provider (vending worked) but not
FileIO. Table create wrote metadata with Application Default Credentials and failed
with 401 unless GOOGLE_APPLICATION_CREDENTIALS was set on the process.

Fix: #9418

Does this PR introduce any user-facing change?

Configuring gcs-service-account-file is sufficient for Iceberg GCS FileIO;
GOOGLE_APPLICATION_CREDENTIALS is only a fallback when the property is unset.
No new public API or property keys.

How was this patch tested?

./gradlew :iceberg:iceberg-common:test --tests org.apache.gravitino.iceberg.common.utils.TestIcebergCatalogUtil -PskipITs

Made with Cursor

…account-file Iceberg's GCSFileIO ignores gcs-service-account-file; mint gcs.oauth2.token at catalog load so server-side FileIO works from catalog properties alone. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

…load Co-authored-by: Cursor <cursoragent@cursor.com>

roryqi · 2026-09-07T11:37:35Z

+   */
+  @VisibleForTesting
+  static void applyGcsServiceAccountCredentials(Map<String, String> properties) {
+    String serviceAccountFile = properties.get(GCSProperties.GRAVITINO_GCS_SERVICE_ACCOUNT_FILE);


How to refresh the GCP access token? I will recommend that we add a expire hook for the catalog cache.
You can recreate the catalog before the access token expiration.

Good catch — addressed in the latest commit.

IRC catalog wrappers now keep the minted gcs.oauth2.token-expires-at on IcebergConfig, and IcebergCatalogWrapperManager uses a Caffeine Expiry that caps expire-after-access by token expiry minus a 5-minute buffer. When the entry is evicted, the next request recreates the catalog and remints the token.

Follow-up: we dropped the catalog-cache expiry approach.

IRC catalogWrapperCache is back to plain expireAfterAccess. Token refresh now lives in a separate GcsAccessTokenCache + GravitinoGCSFileIO, which remints from gcs-service-account-file when the cached token is about to expire — independent of catalog-wrapper eviction.

You can't. Because you can't handle ResolvingFileIO cases.ResolvingFile will use origin GCSFileIO.

Got resolved.

…cache expiry Cap catalog-wrapper cache lifetime by gcs.oauth2.token-expires-at so hot catalogs recreate and remint OAuth2 tokens before they expire. Co-authored-by: Cursor <cursoragent@cursor.com>

…d FileIO Keep IRC catalog expire-after-access unchanged. Remint OAuth2 tokens in GcsAccessTokenCache and GravitinoGCSFileIO so credential lifetime is independent of catalog-wrapper eviction. Co-authored-by: Cursor <cursoragent@cursor.com>

…logUtil Co-authored-by: Cursor <cursoragent@cursor.com>

Rewrite GravitinoGCSFileIO to the Iceberg GCSFileIO class name via a string constant so iceberg-rest-server does not need iceberg-gcp at compile time. Co-authored-by: Cursor <cursoragent@cursor.com>

…tests Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-09-07T14:28:14Z

Code Coverage Report

Overall Project	69.67% `-0.16%`	🟢
Files changed	56.47%	🔴

Module	Coverage
aliyun	19.74%	🔴
api	51.57%	🟢
authorization-common	85.96%	🟢
authorization-ranger	4.38%	🔴
aws	53.54%	🟢
azure	32.1%	🔴
catalog-common	25.76% `-0.04%`	🔴
catalog-fileset	82.17%	🟢
catalog-glue	69.8%	🟢
catalog-hive	82.96%	🟢
catalog-jdbc-common	45.09%	🟢
catalog-jdbc-doris	82.69%	🟢
catalog-jdbc-mysql	79.33%	🟢
catalog-jdbc-postgresql	83.83%	🟢
catalog-jdbc-starrocks	79.16%	🟢
catalog-kafka	76.99%	🟢
catalog-lakehouse-generic	60.88%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	85.9%	🟢
catalog-lakehouse-paimon	84.29%	🟢
catalog-model	77.99%	🟢
cli	44.51%	🟢
client-java	77.5%	🟢
common	57.75%	🟢
core	84.29% `-0.77%`	🟢
filesystem-hadoop3	76.48%	🟢
flink	0.0%	🔴
flink-common	53.22%	🟢
flink-runtime	0.0%	🔴
gcp	32.2%	🔴
hadoop-auth	68.0%	🟢
hadoop-common	17.84%	🔴
hive-metastore-common	53.5%	🟢
iceberg-aliyun-bundle	0.0%	🔴
iceberg-common	66.21% `-0.73%`	🟢
iceberg-rest-server	76.77% `-1.78%`	🟢
idp-basic	86.75%	🟢
integration-test-common	0.0%	🔴
jobs	62.92%	🟢
lance-common	32.52%	🔴
lance-rest-server	68.12%	🟢
lineage	59.39%	🟢
optimizer	83.24%	🟢
optimizer-api	21.95%	🔴
server	89.62%	🟢
server-common	81.35%	🟢
spark	56.27%	🟢
tencent	81.78%	🟢
trino-connector	58.36%	🟢

Files

Module	File	Coverage
catalog-common	IcebergConstants.java	0.0%	🔴
core	SchemaHookDispatcher.java	92.31%	🟢
	ViewHookDispatcher.java	86.21%	🟢
	FilesetHookDispatcher.java	85.29%	🟢
	TopicHookDispatcher.java	82.61%	🟢
	FunctionHookDispatcher.java	81.48%	🟢
	TableHookDispatcher.java	80.49%	🟢
	CapabilityHelpers.java	79.24%	🟢
	ModelHookDispatcher.java	56.82%	🔴
	AccessControlHookDispatcher.java	26.72%	🔴
	GravitinoEnv.java	26.61%	🔴
iceberg-common	IcebergCatalogUtil.java	61.25%	🟢
iceberg-rest-server	IcebergCatalogWrapperManager.java	91.95%	🟢
	RESTService.java	5.17%	🔴

Resolve IcebergConstants conflict by keeping both ADLS token-provider constants from main and GCS OAuth2 token constants from this branch. Co-authored-by: Cursor <cursoragent@cursor.com>

…expiry Mint gcs.oauth2.token from gcs-service-account-file and expire IRC catalog wrappers before token expiry, instead of a dedicated GravitinoGCSFileIO cache. Co-authored-by: Cursor <cursoragent@cursor.com>

roryqi

No better solution. If GCS FileIO API provides better solution, we should refactor the code.

…er close Caffeine may retire a missed cache entry asynchronously after close(), so asserting pool size immediately can flake with size 1. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

lasdf1234 · 2026-09-09T08:49:31Z

I deployed irc and used the gravitino.iceberg-rest.gcs-service-account-file configuration for testing. It worked fine. I can see log : Injected gcs.oauth2.token from gcs-service-account-file for Iceberg GCSFileIO. And create table and schema is OK.

roryqi · 2026-09-09T08:51:26Z

Do u test the refresh case?

lasdf1234 · 2026-09-09T09:07:18Z

#9418

As shown in the above figure, after the cache expires, the catalog is removed and the token is re-minted.

roryqi

LGTM.

…account-file (apache#12962) Iceberg's `GCSFileIO` does not understand Gravitino's `gcs-service-account-file`. When that property is set, load the service account at catalog initialization and inject Iceberg `gcs.oauth2.token` / `gcs.oauth2.token-expires-at` so server-side FileIO can authenticate (same property-injection style as S3/OSS/ADLS key mapping). Also update GCS docs that previously required `GOOGLE_APPLICATION_CREDENTIALS` even when the catalog property was set. `gcs-service-account-file` reached the credential provider (vending worked) but not FileIO. Table create wrote metadata with Application Default Credentials and failed with 401 unless `GOOGLE_APPLICATION_CREDENTIALS` was set on the process. Fix: apache#9418 - Configuring `gcs-service-account-file` is sufficient for Iceberg GCS FileIO; `GOOGLE_APPLICATION_CREDENTIALS` is only a fallback when the property is unset. - No new public API or property keys. ``` ./gradlew :iceberg:iceberg-common:test --tests org.apache.gravitino.iceberg.common.utils.TestIcebergCatalogUtil -PskipITs ``` Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>

…t-file (#12962) Iceberg's `GCSFileIO` does not understand Gravitino's `gcs-service-account-file`. When that property is set, load the service account at catalog initialization and inject Iceberg `gcs.oauth2.token` / `gcs.oauth2.token-expires-at` so server-side FileIO can authenticate (same property-injection style as S3/OSS/ADLS key mapping). Also update GCS docs that previously required `GOOGLE_APPLICATION_CREDENTIALS` even when the catalog property was set. `gcs-service-account-file` reached the credential provider (vending worked) but not FileIO. Table create wrote metadata with Application Default Credentials and failed with 401 unless `GOOGLE_APPLICATION_CREDENTIALS` was set on the process. Fix: #9418 - Configuring `gcs-service-account-file` is sufficient for Iceberg GCS FileIO; `GOOGLE_APPLICATION_CREDENTIALS` is only a fallback when the property is unset. - No new public API or property keys. ``` ./gradlew :iceberg:iceberg-common:test --tests org.apache.gravitino.iceberg.common.utils.TestIcebergCatalogUtil -PskipITs ``` Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>

lasdf1234 self-assigned this Sep 7, 2026

lasdf1234 added the branch-1.3 Automatically cherry-pick commit to branch-1.3 label Sep 7, 2026

lasdf1234 and others added 2 commits September 7, 2026 19:16

[apache#9418] docs: align GCS configuration table columns

d0bee57

Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#9418] refactor(iceberg): drop test-only GCS token loader over…

e9d3973

…load Co-authored-by: Cursor <cursoragent@cursor.com>

roryqi reviewed Sep 7, 2026

View reviewed changes

lasdf1234 and others added 5 commits September 7, 2026 19:49

[apache#9418] fix(iceberg): refresh GCS FileIO token via IRC catalog …

b094546

…cache expiry Cap catalog-wrapper cache lifetime by gcs.oauth2.token-expires-at so hot catalogs recreate and remint OAuth2 tokens before they expire. Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#9418] fix(iceberg): restore IOException import in IcebergCata…

1d2ca62

…logUtil Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#9418] fix(iceberg): avoid iceberg-gcp compile dep in rest-server

9aae30a

Rewrite GravitinoGCSFileIO to the Iceberg GCSFileIO class name via a string constant so iceberg-rest-server does not need iceberg-gcp at compile time. Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#9418] fix(iceberg): suppress JavaUtilDate in GCS token cache …

51a2ea8

…tests Co-authored-by: Cursor <cursoragent@cursor.com>

lasdf1234 and others added 2 commits September 8, 2026 20:14

Merge branch 'main' into fix/gcs-service-account-file-fileio

4873382

Resolve IcebergConstants conflict by keeping both ADLS token-provider constants from main and GCS OAuth2 token constants from this branch. Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#9418] fix(iceberg): refresh GCS tokens via IRC catalog cache …

8b93fe0

…expiry Mint gcs.oauth2.token from gcs-service-account-file and expire IRC catalog wrappers before token expiry, instead of a dedicated GravitinoGCSFileIO cache. Co-authored-by: Cursor <cursoragent@cursor.com>

roryqi previously approved these changes Sep 8, 2026

View reviewed changes

[apache#9418] test(core): wait for ClassLoader pool drain after manag…

fdd7610

…er close Caffeine may retire a missed cache entry asynchronously after close(), so asserting pool size immediately can flake with size 1. Co-authored-by: Cursor <cursoragent@cursor.com>

lasdf1234 dismissed roryqi’s stale review via fdd7610 September 9, 2026 01:41

[apache#9418] chore: retrigger CI after BackendIT mysql timeout

15f2d51

Co-authored-by: Cursor <cursoragent@cursor.com>

roryqi approved these changes Sep 9, 2026

View reviewed changes

roryqi merged commit 5c8f5bf into apache:main Sep 9, 2026
38 checks passed

lasdf1234 deleted the fix/gcs-service-account-file-fileio branch September 9, 2026 09:10

lasdf1234 mentioned this pull request Sep 10, 2026

[Cherry-pick to branch-1.3] Resolve #13051 #13052 #13054 #13034 #13060

Merged

Navigation Menu

Uh oh!

[#9418] fix(iceberg): inject GCS FileIO token from gcs-service-account-file - #12962

[#9418] fix(iceberg): inject GCS FileIO token from gcs-service-account-file#12962
roryqi merged 12 commits into
apache:mainfrom
lasdf1234:fix/gcs-service-account-file-fileio

lasdf1234 commented Sep 7, 2026

Uh oh!

roryqi Sep 7, 2026

Uh oh!

lasdf1234 Sep 7, 2026

Uh oh!

lasdf1234 Sep 7, 2026

Uh oh!

roryqi Sep 8, 2026

Uh oh!

lasdf1234 Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 7, 2026 •

edited

Loading

Uh oh!

roryqi left a comment

Uh oh!

lasdf1234 commented Sep 9, 2026 •

edited

Loading

Uh oh!

roryqi commented Sep 9, 2026

Uh oh!

lasdf1234 commented Sep 9, 2026

Uh oh!

roryqi left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lasdf1234 commented Sep 7, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

roryqi Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

lasdf1234 Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

lasdf1234 Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

roryqi Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

lasdf1234 Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Uh oh!

roryqi left a comment

Choose a reason for hiding this comment

Uh oh!

lasdf1234 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roryqi commented Sep 9, 2026

Uh oh!

lasdf1234 commented Sep 9, 2026

Uh oh!

roryqi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Sep 7, 2026 •

edited

Loading

lasdf1234 commented Sep 9, 2026 •

edited

Loading