update lb delete dag from main (so we can test it) - #120
Draft
rlittle08 wants to merge 39 commits into
Draft
Conversation
…s_to_snowflake(). (#91)
* Update changelog. * Increment version in setup.py.
* init * fix null array * looking into typeerror * safe map cast * safe map cast * list casting * list casting * debugging * debugging * change comment * update changelog * label mapped tasks * remove straggler code * Update setup.py Increment setup.py with latest version. --------- Co-authored-by: sleblanc23 <sam23leblanc@gmail.com> Co-authored-by: Jay Kaiser <jayckaiser@gmail.com>
There can be multiple column combinations equivalent to the 'highest match rate'. This ensures that we are consistently using the SAME combination of columns when that situation occurs.
* draft * chain taskgroup in * fix * type fix * remove unnecessary rerouting * oops * add total_counts args to readme * default total_counts to false * fix indent * fix optional args and logging message
* change default of pull_all_deletes to false no longer necessary * update changelog
* draft of move log helper to util and use for em fails * fix import * add back kwargs * reintroduce structured log * rework * . * making util work as executor or wrapper creator * return to old log format * only log errors * fixes to em call * simplify wrapper for both hope this works * log even on exit * need raise exception * . * remove log filter * print conn id * pass context * need to pass through logging args ew * access context correctly * filter to actual errors * combining logs * fix kwargs and error handling * context should be passed as a kwarg not arg * restructure the logs * . * Unnest serialize_argument helper. * Revert EarthmoverOperator and apply logging wrapper to operator in EarthbeamDAG. * Commit missing code from previous commit. --------- Co-authored-by: jayckaiser <jayckaiser@gmail.com>
* Add mapped_index_templates to dynamic EMLB taskgroups. * Add exit code return logic to EarthmoverOperator, even if unused in EM updates. * Update Snowflake match rate logic to only retrieve records that meet the required match rate. * Add check_new_match_rates task to gate sending tasks on non-matched assessment records and refactor taskstream logic for clarity. * Update remove files operator to skip null xcoms and only trigger on non-skipped taskgroups. --------- Co-authored-by: ejoranlienea <ejoranlien@edanalytics.org>
* refactor bulk sql queries to reflect full refresh behavior * clear deletes and keychanges records on full refresh * add change version records for deletes and keyChanges on full refresh * maintain return pattern * delete only enabled endpoints on full refresh * retrieve configured endpoints through helper function
When pulling prior max CVs on a resource, we need to exclude deletes and key_changes, not take the max across all records
…nal when not enabled in the DAG. (#112)
* add map index to log * need to pull it out * Update earthbeam_dag.py * move to the run earthmover one * Update earthbeam_dag.py * Update earthbeam_dag.py
rlittle08
marked this pull request as draft
October 20, 2025 21:58
* Prepare metadata files in preparation for new release. * Feature/storage interfaces (#125) * Move endpoint checks into their own helper. * Improve is_endpoint_specified() helper to be less dynamic. * Genericize code to use object storage instead of S3-direct logic; use temporary files as the temp local writing location (since the data is always deleted anyway on failure). * Refactor bulk operator to use same logic. * Refactor code to break out S3 logic. * Create initial dunder-new for determining which Ed-Fi transfer operator is selected during init. * Genericize S3 variables in Ed-Fi Resource DAG. * More (untested) cleanup of Ed-Fi transfer operators. * Genericise get_snowflake_params_from_conn(). * Begin refactor of genericized ObjectStorageToDatabaseOperators. * working edfi to adls! * clean value error by registry * auto delete tempfile 🤦 * working version thru database! * we switched this, will need for snowflake to work * testing has proven these lines unnecessary * rename generic database functions * simplifying object storage abstraction * make generic named copies of operators and point to those * delete now useless registry * readme for storage options * Update ADDING_STORAGE_OPTIONS.md * Remove deprecated edfi_to_s3; restore s3_to_snowflake to original version. * Refactor mixins and the generic operators that use them (TODO: add callable-integrations with database mixins, update Ed-Fi DAG to use new implementation). * Move callable database integrations into database mixin. * Remove unused code and raise a deprecation warning on code that should otherwise be removed. * Fix passing of connection IDs to mixins (todo: how do mixins parse ID types to select correct child class?) * Impove object storage mixin selection. * Remove mixin in generic transfer operators and use as helper classes (TODO: rename Mixin, since this is no longer accurate to usage). * Bugfix. * Rename databaseMixin helpers and how they are used in Obj-to-DB transfer. * Bugfix: incorrect argument name. * Bugfix: use directory path in Snowflake bulk copies. * Change S3 clean_url format to only contain key and not bucket. * Rename Mixin to Interface across entire package. * Fix bug in DatabaseInterface.delete_from_raw(). * Fix typing in DatabaseInterface.delete_from_raw(). * Bugfix: error in bulk delete from raw clauses. * Feature/object storage refactor rl v2 jk context manager database (#124) * Make DatabaseInterface use a context manager when sending all delete/copy queries at once. * Apply context-manager implementation to transfer operation. * Bugfix: return self in context manager. * Minor cleanup around package. * Revert get_database_params_from_conn and move functionality into DatabaseInterface object. * Revert EarthbeamDAG to main. * Actually revert EM DAG to main. * extend class * updated markdown * Fix bug in type of EdPiToObjectStorageOperator used in each run type. * Fix bug in arguments passed to bulk EdFiToObjectStorage operator. * Log errors for individual endpoint failures for bulk Ed-Fi loads. --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> * Feature/edfi token cache 5.0 (#127) * Move endpoint checks into their own helper. * Improve is_endpoint_specified() helper to be less dynamic. * Genericize code to use object storage instead of S3-direct logic; use temporary files as the temp local writing location (since the data is always deleted anyway on failure). * Refactor bulk operator to use same logic. * Refactor code to break out S3 logic. * Create initial dunder-new for determining which Ed-Fi transfer operator is selected during init. * Genericize S3 variables in Ed-Fi Resource DAG. * More (untested) cleanup of Ed-Fi transfer operators. * Genericise get_snowflake_params_from_conn(). * Begin refactor of genericized ObjectStorageToDatabaseOperators. * working edfi to adls! * Add EdFiResourceDAG parameter to enable token caching, passed to EdFiHook * clean value error by registry * auto delete tempfile 🤦 * working version thru database! * we switched this, will need for snowflake to work * testing has proven these lines unnecessary * rename generic database functions * simplifying object storage abstraction * make generic named copies of operators and point to those * delete now useless registry * readme for storage options * Update ADDING_STORAGE_OPTIONS.md * Remove deprecated edfi_to_s3; restore s3_to_snowflake to original version. * Refactor mixins and the generic operators that use them (TODO: add callable-integrations with database mixins, update Ed-Fi DAG to use new implementation). * Move callable database integrations into database mixin. * Remove unused code and raise a deprecation warning on code that should otherwise be removed. * Fix passing of connection IDs to mixins (todo: how do mixins parse ID types to select correct child class?) * Impove object storage mixin selection. * Remove mixin in generic transfer operators and use as helper classes (TODO: rename Mixin, since this is no longer accurate to usage). * Bugfix. * Rename databaseMixin helpers and how they are used in Obj-to-DB transfer. * Bugfix: incorrect argument name. * Bugfix: use directory path in Snowflake bulk copies. * Change S3 clean_url format to only contain key and not bucket. * Rename Mixin to Interface across entire package. * Fix bug in DatabaseInterface.delete_from_raw(). * Fix typing in DatabaseInterface.delete_from_raw(). * Bugfix: error in bulk delete from raw clauses. * Feature/object storage refactor rl v2 jk context manager database (#124) * Make DatabaseInterface use a context manager when sending all delete/copy queries at once. * Apply context-manager implementation to transfer operation. * Bugfix: return self in context manager. * Minor cleanup around package. * Revert get_database_params_from_conn and move functionality into DatabaseInterface object. * Revert EarthbeamDAG to main. * Actually revert EM DAG to main. * extend class * updated markdown * Fix bug in type of EdPiToObjectStorageOperator used in each run type. * Fix bug in arguments passed to bulk EdFiToObjectStorage operator. * Log errors for individual endpoint failures for bulk Ed-Fi loads. * Tweak EdFiHook implementation of use_token_cache to match new EdFiClient interface --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> Co-authored-by: Alex Chen <achen@edanalytics.org> * Bugfix: remove extraneous tick. * Bugfix: add token cache to database operator. * Update CHANGELOG. * Remove full pull of deletes (#106) * remove full pull of deletes * update changelog * Reenable but deprecate `pull_all_deletes` argument. --------- Co-authored-by: jayckaiser <jayckaiser@gmail.com> * Maintain python backwards compatibility (#122) Use f-string syntax with support in older python versions * allow input_filepaths to be empty (#121) * Modifies EarthbeamDAG.get_filename to handle existing directories differently (#117) * Modifies EarthbeamDAG.get_filename to handle existing directories differently `EarthbeamDAG.get_filename` is passed both filenames (from which we want to strip extensions) as well as directories (from which we may not want to, if they contain periods in the name). This change adds a check for existing directories and tests for the intended behavior. * Add CHANGELOG entry for changes --------- Co-authored-by: Jay Kaiser <jayckaiser@gmail.com> * Feature/storage interface names (#129) * Make interface type an argument to EdFiResourceDAG and a means to hardcode types of storage operators. * Update README with generic language and the new DAG arguments. * Override object storage and database storage types if deprecated connection arguments are used. * Feature/validate edfi conns (#128) * validate conns script with command line doc * couple things * Add fallback strategy for retrieving LEA IDs via leas or calendars * adjust token info to account for multiple lea * draft operator, couple pending questions * update fail to fail for mismatch, no_org_id, error, or no_mapping statuses * add init py * add detailed logging even in quite mode * ifx typo in the logging info * adjust comments and remove path comment above the script * remove quiet from operator, and opt conn prefix, make failure options a list * the tenant count makes sense in sc but is confusing in tx * clarify use of fail_list to log * allow exclusion of conns * more consistency in error reporting * conn_suffix, get rows, early exits * go back to prefix --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> Co-authored-by: Jay Kaiser <jayckaiser@gmail.com> * Add validateEdFiConnectionsOperator to CHANGELOG. --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> Co-authored-by: Alex Chen <achen@edanalytics.org> Co-authored-by: Samantha LeBlanc <56237580+sleblanc23@users.noreply.github.com> Co-authored-by: ejoranlienea <ejoranlien@edanalytics.org> Co-authored-by: Alex C. <alchenist@users.noreply.github.com> Co-authored-by: Keen Zarate <51969358+keenzarate213@users.noreply.github.com>
* Init runway * Remove extraneous code * Re-arrange code. Use absolute imports * Make send_to_runway a callable * I already wrote RunwayClient.load_files(), so use that
* Update metadata files for release. * Update edfi_api_client version to 0.3.1 * Add s3fs package to required installs.
* removed "_operator" suffix as it is deprecated syntax in Airflow 2.0. * removed apply_default decorators which was deprecated in 2.1 * remove provide_context var
* first draft * Revert "first draft" This reverts commit 41afda1. * draft * setting cursor_paging * init * edit * remove * point to cursor_paging * opt out of reverse_paging * turn of step change version * turn off step change version when cursor_paging is True * draft at partitioning * fix comma * turn off default change version * add conditional max change version * debug * revert to default True use_change_version * default to cursor_paging based on new api_client code updated * remove log * delete whitespaces * add options for deletes/key_changes * get_rows as paginator * refactor json parsing after change to get_rows (now return dict not list of dicts) * Minor cleanup of branch to align with base before merge. * Minor tweaks * Bugfix --------- Co-authored-by: gnguyen87 <gnguyen@macalester.edu> Co-authored-by: rlittle08 <rlittle@edanalytics.org>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
* Update changelog and setup files in preparation for release. * Add is_sideloaded column to COPY INTO statements, functions (#139) * Add optional is_sideloaded kwarg to raw copy functions. Add `not is_sideloaded` to delete statements * Bump version * Add is_sideloaded to the select part of the statement * Fix missing is_sideloaded from column list * Bugfix: Typo --------- Co-authored-by: Jacob Bortell <jbortell@edanalytics.org>
Hotfix: Replace `logical_date` with `dag_run.start_date`
RunwayHook to use AirflowTokenCache in runway_python_client v0.4.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.