Create deferrable SqlExecuteQueryOperator compatible with Postgres by karenbraganz · Pull Request #65618 · apache/airflow

karenbraganz · 2026-04-21T17:58:19Z

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

This PR modifies the SqlExecuteQueryOperator to be deferrable. I have also created a SqlExecuteQueryTrigger and added asynchronous functions to the DbApiHook for use with the SqlExecuteQueryOperator when it is deferred.

I plan on making this compatible with several database types. This PR only adds compatibility with Postgres.

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Srabasti · 2026-06-03T05:39:59Z

Thanks for incorporating the review changes for import of conf @karenbraganz !!!

Seeing that the 3 failing tests are related to docker failures and files not being found as below.

https://github.com/apache/airflow/actions/runs/26853708158/job/79193370724?pr=65618
https://github.com/apache/airflow/actions/runs/26853708158/job/79193370736?pr=65618
https://github.com/apache/airflow/actions/runs/26853708158/job/79193370689?pr=65618

Can you try rebasing and pushing the changes to repo, so as to ensure any bug fixes for unrelated failures are resolved by another open source contributor, automatically?

I observed @jscheffl pushed no files however CI ran fine, from log above last on May 29th. Looks to me, possibly some change happened after that to cause these errors to start failing.

FYI - @potiuk I see same error in failed CI run below. Looks to me the same issue.
https://github.com/apache/airflow/actions/runs/26858010209/job/79206609169?pr=67929

karenbraganz · 2026-06-30T17:41:56Z

Sorry, I accidentally tangled up my branch and pushed the wrong commits, which is why all these reviews were requested. I have reverted the mistake.

RNHTTR · 2026-06-30T19:27:11Z

+            if self.fetch_results:
+                # Fetch the raw rows with the built-in handler and return them with the cursor
+                # descriptions; the operator applies any user handler on the worker.
+                results = await hook.run_async(


What happens if the triggerer is killed in the middle of a run? Will this trigger be re-executed? Is there a chance that multiple instances of the same query are invoked?

yes, this could result in a duplicate query run if the trigger is killed. I can look into whether this can be avoided.

I wonder if we can initiate on the worker and then poll on the triggerer? Not sure there's a straightforward way to do that.

We could also potentially block non-read-only queries by default and include a parameter to do things like INSERT, UPDATE, DELETE, etc with a docstring explaining the risks?

RNHTTR · 2026-06-30T19:30:20Z


-class SQLExecuteQueryTrigger(BaseTrigger):
+
+class SQLGenericTransferTrigger(BaseTrigger):


This is a breaking change, right? If so, we'll need to bump the major version of this provider.

yes, this would qualify as a breaking change, so we should bump the major version FYI @kacpermuda

I do not anticipate a lot of disruption since only the trigger name is changed- not the operator name. This would only affect people who use this trigger in a custom operator.

Are there any other breaking changes in this PR? I think we discussed that for the default deferrable value we'll have a workaround to avoid breaking changes for now, so doing it just so that we can rename a class seems a bit ... pointless ? Not sure.

I don't think it's pointless -- If someone built their own deferrable operator with SQLExecuteQueryTrigger, it will break.

I'm not saying making a major release is pointless, just wondering if renaming a class is needed, trying to think of a way we can add this deferrable functionality as purely additive

@kacpermuda previously, the GenericTransfer operator used a trigger called the SqlExecuteQueryTrigger. Now that I am making the SqlExecuteQueryOperator deferrable, I thought it would be confusing to have a SqlExecuteQueryTrigger that is actually used by a different operator. I renamed the trigger used by the GenericTransfer operator to SqlGenericTransferTrigger and used the SqlExecuteQueryTrigger name for the trigger used by the SqlExecuteQueryOperator.

This name change is not required, but is good for consistency between the trigger and operator.

Sure sure, that makes sense, just wanted to make sure we're doing all the changes consciously since they lead to major release 😄

dabla · 2026-07-01T16:00:19Z

            yield TriggerEvent({"status": "failure", "message": str(e)})
+
+
+class SQLExecuteQueryTrigger(BaseTrigger):


I would prefer to see the GenericTransfer also use this SQLExecuteQueryTrigger, which is better suited due to it's more async native nature. I don't like the fact that we have now a dedicated trigger for the GenericTransfer, I think it should be possible to reuse the new SQLExecuteQueryTrigger.

Makes sense, I will look into how this can be implemented. @RNHTTR @kacpermuda we might be able to avoid releasing this in a major version if I do this since the trigger name for GenericTransfer won't change.

This to me stays a major change independently of this.

dabla · 2026-07-01T16:02:50Z

+        :return: DbApiHook for this connection
+        """
+        connection = await sync_to_async(BaseHook.get_connection)(self.conn_id)
+        hook = await sync_to_async(connection.get_hook)()


I could maybe also add a get_async_hook helper method in common.compat, so that you can use it here as well just like the get_async_connection, that would be much cleaner as newer Airflow will now also support the native async BaseHook.aget_hook method which was added in this PR.

I created a PR which adds this get_async_hook method in common.compat.

Update the dependency to common.compat in the sql provider and use the new get_async_hook method instead of sync_to_async(connection.get_hook).

I have made changes to use the functions you suggested. You mentioned that I would have to update the dependency. Does this mean that I would update this line in providers/common/sql/pyproject.toml?

"apache-airflow-providers-common-compat>=1.14.1",

Would I have to update this to whatever the next version released would be (1.16.0 I assume) so that the function you added is available when people use the SEQO in deferrable mode?

I think you need to put a certain comment that specifies that version should be updated to the next version the release manager will apply.

boring-cyborg Bot added area:providers provider:common-sql provider:postgres labels Apr 21, 2026

karenbraganz mentioned this pull request Apr 21, 2026

Make SQLExecuteQuery deferrable #61742

Closed

1 task

karenbraganz requested review from amoghrajesh, ashb, bugraoz93, gopidesupavan, jason810496, jscheffl and potiuk as code owners April 28, 2026 17:43

Srabasti suggested changes May 4, 2026

View reviewed changes

Comment thread providers/common/sql/src/airflow/providers/common/sql/operators/sql.py Outdated

jscheffl requested a review from Srabasti May 28, 2026 20:35

Srabasti approved these changes Jun 3, 2026

View reviewed changes

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 3, 2026

kacpermuda reviewed Jun 8, 2026

View reviewed changes

Comment thread providers/common/sql/src/airflow/providers/common/sql/operators/sql.py Outdated

karenbraganz requested review from RNHTTR, bolkedebruin, choo121600, dabla, ephraimbuddy, hussein-awala, jedcunningham, kaxil, mobuchowski, shahar1, uranusjr, vatsrahul1001 and vincbeck as code owners June 30, 2026 17:18

karenbraganz requested review from Fokko, Lee-W, XD-DENG, bbovenzi, dheerajturaga, dstandish, eladkal, guan404ming, josh-fell, o-nikolas, pierrejeambrun, rawwar, ryanahamilton and shubhamraj-git as code owners June 30, 2026 17:18

Add async methods to DbApiHook to support deferrable SQL operators

70704b7

karenbraganz force-pushed the seqo_defer branch from 71c7b65 to 70704b7 Compare June 30, 2026 17:38

RNHTTR requested changes Jun 30, 2026

View reviewed changes

Remove dead code for importing handler

d055fae

dabla requested changes Jul 1, 2026

View reviewed changes

dabla mentioned this pull request Jul 2, 2026

Added get_async_hook in common.compat provider #69208

Merged

1 task

karenbraganz added 7 commits July 7, 2026 14:20

Rename run_async to arun

a2b5fe6

Change naming convention for async functions

1df931a

Change fucntion names in test_sql.py

e86bc2f

Fix failing tests

272e38f

Merge branch 'main' into seqo_defer

5e093ea

Use correct async methods to get hook and conn

e7ee08f

Fix failing tests

179c0d2


		class SQLExecuteQueryTrigger(BaseTrigger):

		class SQLGenericTransferTrigger(BaseTrigger):

		yield TriggerEvent({"status": "failure", "message": str(e)})


		class SQLExecuteQueryTrigger(BaseTrigger):

Uh oh!

Conversation

karenbraganz commented Apr 21, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Srabasti commented Jun 3, 2026

Uh oh!

Uh oh!

karenbraganz commented Jun 30, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants