What would you like to be improved?
The Glue catalog does not release its isolated ClassLoader when it is dropped, so its classes stay in Metaspace for the life of the process. Every other provider I could run locally releases its loader after #12987; Glue is the one that does not, and its cause is different in kind, which is why it is filed on its own.
Verified against LocalStack on a packaged server started with -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=512m: create a glue catalog, list and create a schema, drop the catalog, then force a full GC. jcmd <pid> VM.classloader_stats still lists the catalog's IsolatedClassLoader$CustomURLClassLoader with ~2265 classes, and jcmd <pid> GC.heap_info shows Metaspace unchanged.
Tracing the heap dump back to a GC root gives:
JAVA_FRAME (a live thread's stack)
-> software.amazon.awssdk.http.apache.internal.conn.IdleConnectionReaper$ReaperTask
-> <class> --loader--> the catalog's ClassLoader
The thread is idle-connection-reaper, running IdleConnectionReaper$ReaperTask.run. The AWS SDK keeps that reaper as a singleton per ClassLoader and stops it only when the last connection manager is deregistered, that is, when every Apache-HTTP-backed client built by that loader has been closed. It also swallows interrupts, so the ClassLoader cleanup cannot stop it from the outside: I confirmed the cleanup does interrupt the thread and the thread keeps running.
GlueCatalogOperations.close() does close its glueClient, so the reaper staying alive says some other AWS client the catalog builds is never closed. The warehouse is an s3a:// location, so an S3 client is the obvious candidate, but I did not chase which one.
How should we improve?
Find the AWS client that outlives GlueCatalogOperations.close() and close it there, so the SDK deregisters its connection manager and the reaper stops on its own.
Please do not paper over this in ClassLoaderResourceCleanerUtils by reflecting into IdleConnectionReaper and forcing the thread down. That would hide a client that is leaking sockets and connection pools for as long as the server runs, which is a problem in its own right, and it would depend on SDK internals that are not API.
A regression test in the same shape as the manual check would be worth having: drop the catalog, force a full GC, assert that no CustomURLClassLoader for it remains.
Found while working on #12986 / #12987, which fix the same symptom for hive, lakehouse-iceberg, lakehouse-paimon, the JDBC catalogs and fileset on s3a, gs and abfss.
What would you like to be improved?
The Glue catalog does not release its isolated ClassLoader when it is dropped, so its classes stay in Metaspace for the life of the process. Every other provider I could run locally releases its loader after #12987; Glue is the one that does not, and its cause is different in kind, which is why it is filed on its own.
Verified against LocalStack on a packaged server started with
-Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=512m: create agluecatalog, list and create a schema, drop the catalog, then force a full GC.jcmd <pid> VM.classloader_statsstill lists the catalog'sIsolatedClassLoader$CustomURLClassLoaderwith ~2265 classes, andjcmd <pid> GC.heap_infoshows Metaspace unchanged.Tracing the heap dump back to a GC root gives:
The thread is
idle-connection-reaper, runningIdleConnectionReaper$ReaperTask.run. The AWS SDK keeps that reaper as a singleton per ClassLoader and stops it only when the last connection manager is deregistered, that is, when every Apache-HTTP-backed client built by that loader has been closed. It also swallows interrupts, so the ClassLoader cleanup cannot stop it from the outside: I confirmed the cleanup does interrupt the thread and the thread keeps running.GlueCatalogOperations.close()does close itsglueClient, so the reaper staying alive says some other AWS client the catalog builds is never closed. The warehouse is ans3a://location, so an S3 client is the obvious candidate, but I did not chase which one.How should we improve?
Find the AWS client that outlives
GlueCatalogOperations.close()and close it there, so the SDK deregisters its connection manager and the reaper stops on its own.Please do not paper over this in
ClassLoaderResourceCleanerUtilsby reflecting intoIdleConnectionReaperand forcing the thread down. That would hide a client that is leaking sockets and connection pools for as long as the server runs, which is a problem in its own right, and it would depend on SDK internals that are not API.A regression test in the same shape as the manual check would be worth having: drop the catalog, force a full GC, assert that no
CustomURLClassLoaderfor it remains.Found while working on #12986 / #12987, which fix the same symptom for
hive,lakehouse-iceberg,lakehouse-paimon, the JDBC catalogs andfileseton s3a, gs and abfss.