All changes that impact users of this module are documented in this file, in the Common Changelog format with some additional specifications defined in the CONTRIBUTING file. This codebase adheres to Semantic Versioning.
Development of this release was supported by Anthelia.
- Stop aborting tracking runs with a
cannot lock ref 'HEAD'error on servers that run Git 2.47 to 2.50, such as Debian 13, by making the commit that triggers the automatic maintenance of Git wait for it instead of leaving it to a background process; the tracking run pauses meanwhile, usually about a minute on a repository of several gigabytes, and occasionally several minutes when Git consolidates its packs
Development of this release was supported by Anthelia.
- Relaunch the headless browser when it crashes, instead of failing every following fetch of the run on its closed connection; the terms being fetched when it crashes get a new attempt at the end of the tracking run
Development of this release was supported by the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Use the
OTA_ENGINE_GITHUB_TOKENenvironment variable only to publish datasets, as the engine no longer needs it to track terms - Publish datasets to GitLab when the
OTA_ENGINE_GITLAB_RELEASES_TOKENenvironment variable is defined, as it is the token that authenticates the publishing requests; theOTA_ENGINE_GITLAB_TOKENenvironment variable is no longer used and can be removed
- Breaking: Remove the reporter that created issues on GitHub or GitLab when tracking failed, along with its
reporterconfiguration; install the@opentermsarchive/issue-reportermodule in the collection, move thereporterconfiguration to its@opentermsarchive/issue-reporterkey with an additionalcollectionApi.urlentry, replacing any legacygithubIssuesentry with"type": "github"next to the samerepositories, rename the token environment variable toOTA_ISSUE_REPORTER_GITHUB_TOKENorOTA_ISSUE_REPORTER_GITLAB_TOKEN, and runota-issue-reporter sync --schedulealongside the engine, as described in how to report tracking failures - Breaking: Remove the
scripts/reporter/duplicatescript, which closed the duplicate issues that engines older than v2.3.2 could create
- Publish datasets to GitLab through the proxy defined by the lowercase
http_proxyorhttps_proxyenvironment variables, or byHTTP_PROXYalone, as the fetching of terms already does
Development of this release was supported by the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add the
runIdof the completed tracking run reflected by the/tracking-resultsand/tracking-result/{serviceId}responses of the Collection API, so that consumers paginating through them can tell when a run completed between two pages and start over
Development of this release was supported by Anthelia.
-
Create new versions, snapshots and tracking-results repositories on the
mainbranch, the default branch of GitHub and GitLab and the one used by the deployment, instead of the default branch of the Git configuration of the server, oftenmaster; this requires Git 2.28 or later; to move an existing repository tomain, stop the tracker and run in its local directory:git branch -m master main git push -u origin main # Add --force if the remote main branch only holds the files created with the repository git push origin --delete master # After making main the default branch of the remote repository
Development of this release was supported by Anthelia.
- Fix the links to the declaration, version and snapshot files in the tracking issues of services whose name differs from their ID, which pointed to nonexistent files named after the service name instead of its ID
- Fix the links to the snapshots of terms combining multiple source documents in the tracking issues, which pointed to nonexistent files
Development of this release was supported by the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add Collection API endpoints serving the latest known tracking status of each terms and the summary of the latest completed tracking run:
/tracking-results, filterable bystatus,/tracking-result/{serviceId},/tracking-result/{serviceId}/{termsType}and/tracking-results/run; the tracking results of terms removed from the declarations are kept and flagged as notdeclared; these endpoints are not exposed when tracking-results is disabled
Development of this release was supported by Anthelia.
- Refuse to start a process that writes to a Git repository already written by another running process, such as a second
ota trackstarted while the scheduled tracker runs, instead of letting both race the working tree and the index, which made one of them fail or discard the uncommitted changes of the other; stop the running writer first, the error names it and the.git/ota-writer.lockfile that holds the lock until it exits
- Remove at each tracking run the
index.lockfile left behind by a Git operation interrupted by a kill, which otherwise made the tracker fail at every restart withindex.lock: File exists
Development of this release was supported by Anthelia.
- Fix the "Latest version" and "Latest snapshot" links in tracking issues, which pointed into the declarations repository instead of the versions and snapshots repositories
Development of this release was supported by User Rights and by the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add a
tracking-resultsdata repository recording the tracking status of each terms and the lifecycle of every run in a tamper-evident Git history (RFC) - Add
@opentermsarchive/engine.tracking-resultsconfiguration defining the storage location, author and publication of the tracking-results repository; defaults record locally to./data/tracking-resultswithout publication; set it tonullto disable tracking-results - Add an
x-run-idmetadata to snapshots and versions recorded while a tracking-results run is in progress, tying each record to the run that produced it; it is a commit trailer with Git storage - Reject in
ota validatethe service IDs containing characters that cannot be used in file names on every supported platform (/,\,:,",<,>,|,*,?, control characters), as documented in the service ID guidelines
- Mention the declared URL instead of the last redirect target in the errors of fetches that fail because of redirects, so that the error stays the same from one tracking to the next
- Record as a technical upgrade the version of combined terms whose newly declared source document could only be fetched on a retry, instead of refetching all their source documents and recording a regular change that notified a false modification of the terms
- Stop stating in tracking issues that the source documents were recorded as snapshots and that missed versions may be recovered, when they could actually not be fetched
- Fix the links to the latest snapshots in the tracking issues of combined terms, which pointed to a nonexistent file with a
.nullextension for every source document that was successfully fetched; also omit these links for source documents that have never been recorded as snapshots, as there is no file to link to - Pass falsy parameters, such as
falseor0, to collection-specific filters, which received the extraction context in their place - Track only once a service whose ID is given several times to
ota track --services
Development of this release was supported by Anthelia.
- Stop the Collection API and the dataset export from resetting the working tree and rewriting the commit-graph of the repositories they only read, which raced the tracker started at the same time and made either fail with
commit-graph.lock: File exists - Read PDF versions and snapshots straight from the Git object database instead of temporarily restoring them in the working tree, which could make a concurrent tracker commit stale content
- Fail with an explicit error when the Collection API or the dataset export opens a repository that does not exist yet, instead of silently querying the enclosing repository
Development of this release was supported by Anthelia.
- Report a failure to apply the user agent override or to enable PDF interception in the full DOM fetcher as an error on the fetched document instead of exiting the whole tracking process on an unhandled promise rejection
- Register the full DOM fetcher evasions once per browser launch instead of adding a copy on every tracking run, which multiplied the scripts and commands sent to Chrome for every page
Development of this release was supported by the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Store the latest generated dataset locally in the directory defined by the new
dataset.storagePathconfiguration option (defaults to./data/datasets), alongside ametadata.jsonfile describing it; this directory is managed by the engine, which deletes any.ziparchive it does not reference at each generation, so it must not be shared with other files; see the RFC - Add
GET /dataset/latestandGET /dataset/latest/downloadendpoints to the Collection API to expose the metadata and the archive of the latest locally stored dataset, with HTTP caching and resumable downloads
- Breaking: Write the dataset archive generated by
ota datasetintodataset.storagePathinstead of the current working directory, atomically, and keep it after publishing; make sure the machine has enough disk space to keep two dataset archives during generation
- Breaking: Remove the
--remove-local-copyoption ofota dataset, as the local copy is now the reference dataset served by the Collection API; remove this option from thedataset:schedulescript of the collectionpackage.json
- Fix GitLab dataset publishing failing after the release creation because of a missing
debuglog level - Fix the dataset scheduler exiting after a failed release; the error is now logged and the next scheduled release still runs
Development of this release was made on a volunteer basis by be-student.
- Reject duplicate source document locations in
combineduring schema-only validation, as full validation already does
Development of this release was supported by Anthelia.
- Name the collection in the subject of Collection API error emails and give them the same body as engine error emails
- Prefix the subject and title of error emails with the environment name, such as
[development], whenNODE_ENVis notproduction
- Keep the Collection API running when sending an error email fails
- Send one error email per unhandled promise rejection instead of two
- Wait for the error email to be sent before exiting on an unhandled promise rejection
- Stop sending errors a second time as warnings when
logger.sendMailOnError.sendWarningsis enabled - Send error emails as HTML instead of plain text
- Start the engine when
logger.sendMailOnErroris enabled withoutsendWarnings - Start the Collection API with a warning instead of failing when
OTA_ENGINE_SMTP_PASSWORDis missing whilelogger.sendMailOnErroris enabled
Development of this release was made on a volunteer basis by be-student.
- Remove well-known tracking query parameters (
utm_*,fbclid,gclid, ...) by default when the built-inremoveQueryParamsfilter is declared without parameters; see the built-in filters documentation
Development of this release was supported by User Rights.
- Add
convertSpacesToStandardbuilt-in filter that replaces Unicode space separators (non-breaking space, narrow no-break space, ...) with a regular space; see more in the built-in filters documentation
Development of this release was supported by the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add paginated
GET /versions,GET /versions/{serviceId}andGET /versions/{serviceId}/{termsType}endpoints to the Collection API to list versions - Add
GET /version/{versionId}andGET /version/{serviceId}/{termsType}/latestendpoints to the Collection API to retrieve a single version - Expose
serviceId,termsType,isFirstRecord,isTechnicalUpgrade, sourcefetchUrls, navigationlinksand diff statistics (additionsanddeletions) in Collection API version responses
- Breaking: Return the
GET /service/{serviceId}not-found response as JSON instead of plain text
- Reject version lookups whose service ID, terms type or record ID would be parsed as a Git option, which previously let an unauthenticated caller overwrite arbitrary files through the Collection API
- Return
404instead of a server error when a version lookup contains path separators or relative path segments; the error previously exposed the repository filesystem location - Return a generic message instead of internal error details in Collection API
500responses
Development of this release was supported by the Research Chair in Content Moderation at the Conservatoire National des Arts et Métiers.
- Validate that no
combinelists the same source document location more than once within a terms type
Development of this release was supported by Reset Tech.
- Breaking: Update minimum supported Node.js version to v22
- Update supported Node.js version to v26
- Recover automatically from a stale
commit-graph.lockleft behind when acommit-graph writeis interrupted - Prevent the full DOM fetcher from hanging indefinitely when a page navigation exceeds the configured timeout
Development of this release was made on a volunteer basis by Bhavy Singhal.
- Improve error message when history file is wrongly formatted as an object instead of an array
Development of this release was supported by User Rights.
- Breaking: Set
navigator.languageandnavigator.languagesin the full DOM fetcher strictly from the tags in@opentermsarchive/engine.fetcher.languagethrough the stealth evasions registered at browser launch rather than a per-page CDP override
- Breaking: Drop support for quality factors (
;q=…) in@opentermsarchive/engine.fetcher.language; previously accepted values such asen-IE,en-GB;q=0.9,en;q=0.8now throw at launch. Replace with a plain comma-separated priority list, e.g.en-IE,en-GB,en
- Reactivate the
puppeteer-extrastealth plugin in the full DOM fetcher, which had been silently inert since v10.3.1 because it was registered afterpuppeteer.launch(), restoring removal ofnavigator.webdriverandHeadlessChromefrom the user agent
Development of this release was supported by User Rights.
- Update
@accordproject/markdown-ciceroto fixnpm cifailures in collections caused by brokenfile:..references in the deprecated transitive@accordproject/concerto-core@3.19.8
Development of this release was supported by User Rights.
- Avoid GitHub API "body is too long" error when reporting issues for declarations using
combine, by linking directly to the declaration file on GitHub instead of embedding the full declaration JSON in the contribution tool URL.
Development of this release was supported by the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add
GET /feedendpoint on the Collection API exposing an Atom feed of the latest version changes across the whole collection - Add
GET /feed/:serviceIdendpoint on the Collection API exposing an Atom feed scoped to a single service - Add
GET /feed/:serviceId/:termsTypeendpoint on the Collection API exposing an Atom feed scoped to a single service and terms type - Add
@opentermsarchive/engine.collection-api.feed.limitconfiguration option controlling the maximum number of entries returned by feed endpoints (default:100) - Add
@opentermsarchive/engine.collection-api.feed.versionUrlTemplateconfiguration option to customize thealternatelink of feed entries with a URL template (e.g.https://github.com/openTermsArchive/demo-versions/commit/%VERSION_ID); useful to point feed readers to a human-readable page instead of the default version API endpoint
- Breaking: Resolve
serviceIdpath parameter case-sensitively on theGET /service/:serviceIdendpoint, in line with the documented service ID format; clients relying on case-insensitive matching must now use the exact ID casing
Development of this release was supported by Reset Tech.
- Fix snapshot storage failure when source document URLs contain characters forbidden in filenames
- Fix duplicate file extension in snapshot filenames when source document URLs end with an extension
Development of this release was supported by Reset Tech.
- Update JSDOM to fix extraction of terms with large inline stylesheets
Development of this release was supported by the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Remove Notifier module
- Breaking: Remove all default SMTP values (
host,port,username) from logger configuration; operators enablingsendMailOnErrormust now explicitly set all three in@opentermsarchive/engine.logger.smtp
Development of this release was supported by Reset Tech.
- Fix declaration validation when a new service is added along with a filters file
Development of this release was supported by Reset Tech.
- Treat navigation timeout errors as transient, enabling automatic retry when pages take too long to load
Development of this release was supported by Reset Tech.
- Add PDF file support in full DOM fetcher
Development of this release was supported by Reset Tech.
- Make SMTP port configurable and switch from implicit SSL to STARTTLS, enabling use of alternative ports (like 2525) on servers where standard SMTP ports are blocked
Development of this release was supported by Reset Tech.
- Fix dataset publishing failure for large files by executing uploads sequentially instead of in parallel
Development of this release was supported by Reset Tech.
- Fix error messages when the start and end selectors of range selectors match elements on the page but the selected range or extracted content is empty
Development of this release was supported by Reset Tech.
- Fix browser and request language with full DOM fetcher
Development of this release was made on a volunteer basis by @Ndpnt.
- Set realistic viewport size for full DOM fetcher to avoid detection based on default Puppeteer dimensions
- Improve isolation between fetches by using isolated browser contexts
Full changeset and discussions: #1219.
Development of this release was supported by Reset Tech.
- Add
Advertising Content Policyterms type, introduced interms-types@2.2.0
Full changeset and discussions: #1217.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Validate presence of required configuration for dataset publication to data.gouv.fr
Full changeset and discussions: #1212.
Development of this release was supported by Reset Tech.
- Exclude development and local config files from the published package to avoid unintentionally overriding default configuration
Full changeset and discussions: #1209.
Development of this release was supported by the European Commission for its VLOPs/VLOSEs instance.
- Support proxies in GitLab issues reporting
Full changeset and discussions: #1202.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add support for publishing datasets to data.gouv.fr; configure
dataset.datagouv.datasetIdordataset.datagouv.organizationIdOrSlugin configuration file and setOTA_ENGINE_DATAGOUV_API_KEYenvironment variable - Add ability to publish datasets to multiple platforms simultaneously; datasets can now be published to GitHub (or GitLab) and data.gouv.fr in parallel
Full changeset and discussions: #1208.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Prevent external hosts from accessing the collection API directly
Full changeset and discussions: #1207.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add
ota apply-technical-upgradesCLI command to apply technical upgrades independently
- Breaking: Remove
--extract-onlyoption fromota trackcommand; use the newota apply-technical-upgradescommand instead
- Fix incorrect versioning that occurred when adding new source documents to combined terms declarations
Full changeset and discussions: #1204.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix email reporting on collection API errors
Full changeset and discussions: #1203.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix saving records whose content is not yet loaded when using MongoDB storage.
Full changeset and discussions: #1206.
Development of this release was supported by the European Commission for its VLOPs/VLOSEs instance.
- Fix tracking when proxy configuration is provided
Full changeset and discussions: #1173.
Development of this release was supported by the European Commission for its VLOPs/VLOSEs instance.
- Add proxy support for fetching documents behind firewalls or restricted networks; configure using
HTTP_PROXYandHTTPS_PROXY(orhttp_proxyandhttps_proxy) environment variables - Add debugging options to disable headless mode for visual troubleshooting during development; set
OTA_ENGINE_FETCHER_NO_HEADLESS=1to show browser window - Add sandbox control for improved compatibility with Docker and containerized environments; set
OTA_ENGINE_FETCHER_NO_SANDBOX=1when running in containers
Full changeset and discussions: #1199.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Update Node compatibility metadata; actual compatibility was added in v7.2.1
Full changeset and discussions: #1198.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Increase robustness of PDF content type detection
Full changeset and discussions: #1197.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Improve memory management when tracking terms with multiple source documents
- All available snapshots are now stored in multi-documents terms, even if some are unavailable.
Full changeset and discussions: #1196.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Update collection governance roles by adding
contributor,analystanddiffuserand replacingadministratorwithsysadmin; update roles in your metadata file at the root of your collection folder
Full changeset and discussions: #1193.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Update minimum supported Node.js version to v20
- Add
Content Monetisation Policyterms type, introduced interms-types@2.1.0 - Update dependencies
Full changeset and discussions: #1195.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix built-in
removeQueryParamsfilter to not modify URLs without target parameters
Full changeset and discussions: #1192.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix declarations validation when filters history is modified
Full changeset and discussions: #1191.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix validation of declaration files with filters
Full changeset and discussions: #1189.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add support for Node.js v24
Full changeset and discussions: #1178.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add support for parameters in filters; see more in the filters documentation
- Add
removeQueryParamsbuilt-in filter to remove query parameters from links and images; see more in the built-in filters documentation
Full changeset and discussions: #1188.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Improve readability of multi-document terms
Full changeset and discussions: #1175.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Update issue labels to improve clarity; existing labels will be automatically updated on the first run
Full changeset and discussions: #1176.
Development of this release was supported by the Lab Platform Governance, Media and Technology (PGMT), Centre for Media, Communication and Information Research (ZeMKI), University of Bremen as part of the project Governance: Private ordering of ComAI through corporate communication and policies in the research unit Communicative AI, funded by the German Research Foundation (DFG) (Grant No. 516511468).
- Fetch multiple source documents sequentially to prevent bot detection and improve tracking success rate
Full changeset and discussions: #1171.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Ensure labels are properly recreated after removing deprecated labels in GitHub and GitLab reporters
Full changeset and discussions: #1170.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Replace generic issue labels with more descriptive names to better understand why terms tracking was interrupted; existing labels will be automatically updated on the first run
- Add
⚠ needs interventionlabel to highlight critical issues requiring manual action to restore terms tracking
Full changeset and discussions: #1169.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Improve headless browser context isolation when fetching pages by disabling cache and clearing cookies between requests to prevent session persistence across different URLs and to improve tracking success rate
Full changeset and discussions: #1168.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix automatic labeling of GitHub issues when tracking fails due to DNS resolution errors
Full changeset and discussions: #1166.
Development of this release was supported by the Lab Platform Governance, Media and Technology (PGMT), Centre for Media, Communication and Information Research (ZeMKI), University of Bremen as part of the project Governance: Private ordering of ComAI through corporate communication and policies in the research unit Communicative AI, funded by the German Research Foundation (DFG) (Grant No. 516511468).
- Extend automatic retry mechanism for failed tracking attempts due to likely bot blocking errors to improve tracking success rate
Full changeset and discussions: #1159.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add content validation to ensure selectors contain actual text before considering them loaded to improve tracking success rate
Full changeset and discussions: #1162.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix engine version in logs, records metadata and collection API
Full changeset and discussions: #1156.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix network system error reporting
Full changeset and discussions: #1155.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Improve email error and warning reporting to provide more detailed information and better readability
Full changeset and discussions: #1154.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Handle connection resets as bot protection to improve tracking success rate
Full changeset and discussions: #1152.
Development of this release was supported by the Lab Platform Governance, Media and Technology (PGMT), Centre for Media, Communication and Information Research (ZeMKI), University of Bremen as part of the project Governance: Private ordering of ComAI through corporate communication and policies in the research unit Communicative AI, funded by the German Research Foundation (DFG) (Grant No. 516511468).
- Add an automatic fetching fallback to headless browser when bot detection is encountered to improve tracking success rate
Full changeset and discussions: #1151.
Development of this release was supported by the Lab Platform Governance, Media and Technology (PGMT), Centre for Media, Communication and Information Research (ZeMKI), University of Bremen as part of the project Governance: Private ordering of ComAI through corporate communication and policies in the research unit Communicative AI, funded by the German Research Foundation (DFG) (Grant No. 516511468).
-
Add an automatic retry mechanism for failed tracking attempts due to likely transient errors to improve tracking success rate
⚠️ Version correction notice: Version 5.1.1 was incorrectly published as a patch release, violating semantic versioning principles. This version (5.2.0) contains the same changes but is published as a minor version to properly reflect the addition of the new automatic retry mechanism feature.
Full changeset and discussions: #1150.
Development of this release was supported by the Lab Platform Governance, Media and Technology (PGMT), Centre for Media, Communication and Information Research (ZeMKI), University of Bremen as part of the project Governance: Private ordering of ComAI through corporate communication and policies in the research unit Communicative AI, funded by the German Research Foundation (DFG) (Grant No. 516511468).
- Add an automatic retry mechanism for failed tracking attempts due to likely transient errors to improve tracking success rate
Full changeset and discussions: #1149.
Development of this release was supported by the Lab Platform Governance, Media and Technology (PGMT), Centre for Media, Communication and Information Research (ZeMKI), University of Bremen as part of the project Governance: Private ordering of ComAI through corporate communication and policies in the research unit Communicative AI, funded by the German Research Foundation (DFG) (Grant No. 516511468).
- Reduce navigation timeout errors and enhance performance of headless browser fetching
Full changeset and discussions: #1148.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Update links to documentation in tracking issues reports
Full changeset and discussions: #1147.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Enable declarations linting in production environments
Full changeset and discussions: #1146.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix creation of issues labels for GitHub repositories with over 30 labels
Full changeset and discussions: #1141.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix MongoDB repository support of terms with multiple source documents
Full changeset and discussions: #1140.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix validation of the collection metadata file
Full changeset and discussions: #1139.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Validate required fields in collection metadata file
Full changeset and discussions: #1137.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
-
Breaking: Rename
documentskey totermsin service declarations for vocabulary consistency; to update your declarations:For Unix-like systems:
find . -name "*.json" -type f -exec sed -i 's/"documents":/"terms":/g' {} +
For macOS:
find . -name "*.json" -type f -exec sed -i '' 's/"documents":/"terms":/g' {} +
⚠️ Note: when updating the engine, CI checks of modified declarations will fail once. This is expected and can be safely ignored
Full changeset and discussions: #1135.
Development of this release was supported by the European Union.
- Add support for nested subgroups in Gitlab reporter repository paths
- Fix error logging in Gitlab reporter
Full changeset and discussions: #1134.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add
ota validate metadatacommand to the CLI to validate the collection metadata file
Full changeset and discussions: #1133.
Development of this release was supported by the European Union.
- Fix functionality of filtering by status for the issues list on GitLab
Full changeset and discussions: #1124.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix documentation of collection metadata in the collection API
Full changeset and discussions: #1123.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Expose collection metadata through the collection API; ensure to add the required metadata file at the root of your collection folder
- Breaking: Replace
@opentermsarchive/engine.services.declarationsPathwith@opentermsarchive/engine.collectionPath; ensure your declarations are located in./declarationsin your collection folder
Full changeset and discussions: #1122.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Updated peer dependency
@opentermsarchive/terms-typestov2.0.0; in your declarations, if you define a specific version of this library, ensure to also upgrade it tov2.0.0to avoid compatibility issues; see terms-types@2.0.0 for details
Full changeset and discussions: #1121.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Optimize performance for Git storage
Full changeset and discussions: #1120.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix timestamp prefix format in logs
Full changeset and discussions: #1118.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add configuration option to toggle timestamp prefix in logs; set
@opentermsarchive/engine.logger.timestampPrefixtotrueorfalsein your configuration file to control this feature.
Full changeset and discussions: #1116.
Development of this release was supported by the European Union and the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add support for GitLab for issue reporting
- Add support for GitLab Releases for publishing datasets
Full changeset and discussions: #1115.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add script to remove duplicate issues in GitHub reports
Full changeset and discussions: #1114.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add
Service Level Agreementterms type, introduced interms-types@1.4.0
Full changeset and discussions: #1113.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Reduce GitHub API calls in issues reporter
Full changeset and discussions: #1112.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Reduce API calls and avoid inconsistent states on GitHub issues reports
Full changeset and discussions: #1111.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Prevent duplicate issues on GitHub reports
Full changeset and discussions: #1110.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Bump
@opentermsarchive/terms-typespeer dependency to support new types introduced in v1.3.0
Full changeset and discussions: #1109.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Prevent the engine from exiting on SMTP errors from the logger
Full changeset and discussions: #1088.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Display engine version in logs
- Fix the number of terms displayed in logs
- Fix Reporter module instantiation
Full changeset and discussions: #1087.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Set default tracking frequency to two per day
Full changeset and discussions: #1086.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Make tracking schedule configurable
- Make publishing schedule configurable
- Add tracking overrun protection
Full changeset and discussions: #1085.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Corrected a missing negation in a Reporter error log
Full changeset and discussions: #1071.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Nest all configurations under the
@opentermsarchive/enginekey; wrap the content ofconfig/production.jsonlike this:{ @opentermsarchive/engine: { < your previous configuration >}} - Breaking: Rename the
apiconfiguration key tocollection-api; update this in yourconfig/production.json - Breaking: Remove default values for
portandbasePathincollection-api; specify these configurations explicitly in yourconfig/production.json - Breaking: Prefix all environment variables with
OTA_ENGINE_; rename variables such asSENDINBLUE_API_KEYtoOTA_ENGINE_SENDINBLUE_API_KEY,SMTP_PASSWORDtoOTA_ENGINE_SMTP_PASSWORD, andGITHUB_TOKENtoOTA_ENGINE_GITHUB_TOKEN
Full changeset and discussions: #1076.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Enable a service to be renamed into several services when exporting a dataset
Full changeset and discussions: #1075.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Exclude non-record files (e.g., README, LICENSE) from dataset
Full changeset and discussions: #1070.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix
ota lintcommand
Full changeset and discussions: #1069.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add label
empty contentto reported issues on GitHub when server returns empty content or when PDF content is unextractable - Add label
invalid selectorto reported issues on GitHub when CSS selector is invalid
- Consider extraction errors as expected operational errors instead of crashing the engine
- Do not crash when source documents links point to invalid URLs
- Take into account the
--typesCLI option - Fix counts of terms when they are specified via the CLI option
--types - Fix display issues for service ID and terms type associated with errors
Full changeset and discussions: #1068.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix validation of modified declarations when declaration history is modified; bug introduced in v1.1.2
Full changeset and discussions: #1066.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix validation of modified declarations when new services were added
- Fix validation of modified declarations when filters were modified
Full changeset and discussions: #1065.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Ensure to reopen associated issue if terms become inaccessible again
Full changeset and discussions: #1063.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Introduce
pluginErrorerror type to isolate plugin errors from the engine and increase robustness
Full changeset and discussions: #1061.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
No code changes were made in this release, it simply signifies that the public API can now be considered stable.
Full changeset and discussions: #1056.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix support of service names with special character in the validation CLI
Full changeset and discussions: #1054.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Minimize required privileges associated to the GitHub token to create issues in the declarations repository
Full changeset and discussions: #1053.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Log a warning in case log emails cannot be sent because of a missing config
Full changeset and discussions: #1052.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Base the loading of the Notifier module on the presence of required configuration, not on the environment; only define the
SENDINBLUE_API_KEYenvironment variable if you want the Notifier to be loaded, avoid relying onNODE_ENV=production
Full changeset and discussions: #1051.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Consider EAI_AGAIN errors as a legitimate part of the tracking lifecycle rather than causing the engine to crash
Full changeset and discussions: #1050.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix engine hang with non-existent service ID
Full changeset and discussions: #1045.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix support of special character in service names
Full changeset and discussions: #1037.
Development of this release was made on a volunteer basis by @Ndpnt.
- Improve performance for large-scale MongoDB databases
Full changeset and discussions: #1033.
Development of this release was supported by the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Expose versions data through the collection API (#1003). When using Git as storage for versions, this API relies on the assumption that the commit date matches the author date, introduced in the engine in June 2022 (#875). If your collection was created before this date, inconsistencies in the API results may arise. You can verify if your version history includes commits with commit dates differing from author dates by executing the following command at the root of your versions repository:
git log --format="%H %ad %cI" --date=iso-strict | awk '{if ($2 != $3) print "Author date", $2, "and commit date", $3, "mismatch for commit", $1 }'. You can correct the history with the command:git rebase --committer-date-is-author-date $(git rev-list --max-parents=0 HEAD). Since the entire history will be rewritten, a force push may be required for distributed repositories
- Provide a succinct JSON-formatted error message as response in API errors
Full changeset and discussions: #1032.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Update ESLint to use the stylistic plugin following deprecation of formatting rules, correcting a randomly-appearing
Configuration for rule "lines-between-class-members" is invaliderror
Full changeset and discussions: #1031.
Development of this release was supported by the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add
termsattribute to/servicesAPI response, containing declared term types for each service
Full changeset and discussions: #1026.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Ensure
Reporterconfiguration is defined before instantiating it - Ensure to add links to
versionsandsnapshotsin issues only if repositories are defined in configuration
- No longer define default configuration for
Reportermodule
Full changeset and discussions: #1025.
Development of this release was supported by GitHub Social Impact through its DPG Open Source Community Manager Program.
- Auto create labels for declarations repository on GitHub
- Assign a label based on the error type to issues managed by the engine
- Breaking: Revise the formatting of reported issues, with notable adjustments to the title, transitioning from
Fix <service_id> - <terms_type>to<service_id> ‧ <terms_type> ‧ not tracked anymore - Breaking: Rename
trackermodule intoreporter; update your configuration file by renamingtrackerintoreporter - Breaking: Change configuration for
reportermodule; update yourreporterconfiguration accordingly
- Breaking: No longer assign the label
bot-reportto issues managed by the engine
Full changeset and discussions: #1024.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Upgrade dependencies patches and ensure compatibility with Node v20
Full changeset and discussions: #1021.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Update
opentermsarchive/terms-typesdependency; update your declarations following the instructions in the changelog for validating declarations
Full changeset and discussions: #1020.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix validation for removed terms types
Full changeset and discussions: #1015.
Development of this release was supported by the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Embed Swagger UI for graphical user interface documentation of the API; access it on
/docs
- Breaking: Change path from
/services/:serviceIdto/service/:serviceIdfor direct access to aserviceresource; update paths accordingly in your own codebase - Breaking: Change path from
/specsto/docsfor OpenAPI specification; update paths accordingly in your own codebase and set the HTTP headerAccept: application/json
Full changeset and discussions: #1013.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix issue creation on GitHub tracker (#1012)
Full changeset and discussions: #1011.
Development of this release was supported by the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add collection metadata API (#1003)
- Add
ota servecommand to CLI to start the server and expose the API
- Include collection name in dataset name
- Clean up README
Full changeset and discussions: #996, #999, #1000.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Rename CLI option
--terms-typesto--typesin API; simply rename accordingly in your own codebase - Breaking: Rename CLI option
--refilter-only,-rto--extract-only,-ein API; simply rename accordingly in your own codebase - Breaking: Rename class
PageDeclarationtoSourceDocumentand its atributenoiseSelectorstoinsignificantContentSelectorsin API; simply rename accordingly in your own codebase - Breaking: Rename function and its parameters
filter({ content, mimeType, pageDeclaration })toextract(sourceDocument)in API;contentandmimeTypeare embeddedsourceDocumentattributes; rename accordingly in your own codebase and setcontentandmimeTypein thesourceDocumentpassed as a parameter to the function - Breaking: Rephrase commit messages in Git storage:
Start trackingis changed toFirst record of,RefiltertoApply technical or declaration upgrade onandUpdatetoRecord new changes of; existing data will still be loaded, but new commits will use these new messages, if you have scripts that parse commit messages directly, update them accordingly - Breaking: Rename document attribute
isRefiltertoisExtractOnlyin MongoDB storage; existing data will still be loaded, but new entries will use this new attribute, if you have scripts that query the Mongo database directly, update them accordingly - Make vocabulary consistent throughout the codebase (#971)
- Breaking: Remove
npm run extractcommand; usenpm run start -- --extract-onlyinstead
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Change URL to engine repo from
ambanum/OpenTermsArchivetoOpenTermsArchive/engine - Replace documentation with links to the documentation website to avoid redundancy
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Split validation and linting of declaration files in 2 commands.
ota validatedoes not test the linting of files anymore. - Breaking:
ota lintdoes not fix files by default.ota lint --fixmust be used for that.
Full changeset and discussions: #981.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Specification of changelog format
- Specification of changelog format is now in CONTRIBUTING
- Improve documentation for CLI commands
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Replace behaviour of
ota track --schedule. It now tracks the changes only on scheduled hours. It was before running also on launch.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Trigger a release event on CI to deploy documentation website
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Remove obsolete ansible deployment recipes as it was extracted in a dedicated repository. Look at the README to know how to deploy the engine.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Replace embedded terms types list with the one defined in the dedicated repository
@opentermsarchive/terms-types.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add dataset command to CLI; this command can be discovered in the documentation and by running
ota dataset help
Full changeset and discussions: #959.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Improved reliability and expanded coverage of email protection global filter
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add missing configuration for production
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Unify all CLI commands as subcommands of one single
otacommand and rename some options; the new CLI can be discovered in the documentation and by runningota help(#978)
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Ensure paths for given Git storage configuration are relative to current working directory instead of engine module folder
Full changeset and discussions: #979.
- Add missing configuration for production
Full changeset and discussions: #976.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Load both the configurations defined in this module and the configurations defined in the module which use it as dependency
Full changeset and discussions: #977.
- Add missing file required for CLI in the packaging process
Full changeset and discussions: #971.
- Rewrite documentation to bring it up to date with current behaviour
Full changeset and discussions: #957.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Publish package on NPM under name
@opentermsarchive/engine - Export
filter,pageDeclarationandfetchin NPM module - Add changelog