Initial implementation of Locale framework, with updates to asyncLoad to handle json files, and updates to testsuite.

[dpvc]

This PR is a replacement for #1122, which had numerous merge conflicts, as it was originally written in the summer of 2024, and things have progressed in the code base since then. I have updated the files, and made several improvements so that it works properly in node applications, among other things. I also added test files to get full coverage of the localization utility, and updated some of the test framework to properly handle loading of the locale data. Note that you will need to recompile the test scripts in order to properly handle the localization, so do

cd testsuite
pnpm install
tsc
cd ..

The MathJax-demos-node utilities need some updating to work with this branch, as they don't currently handle the locale files. I will make a branch in that repository for testing purposes.

The main file is the ts/util/Locale.ts file. It defines a Locale class that is used for registering and loading localization files. Those are JSON files that map the message ids to the message strings, with each component registering its own localization files. This is so that third-party components can register localizations, for example, and so that we don't have to transfer all the strings for components that are never loaded.

When a component file is loaded, it registers itself with the Locale object so that its messages can be loaded when needed. This is done by calling Locale.registerLocaleFiles(), which is passed the component name (like 'input/tex' or '[tex]/bbox') and the directory where the locale files are loaded (for use by node applications that are loading the MathJax modules directly rather than through components).

This is illustrated by example changes to the bbox TeX extension. The BboxConfiguration.ts file shows the registration call, and defines a bboxError() function that creates a TexError object and looks up the localized message for it before throwing the error. Eventually, this lookup will be part of TeXError itself, but I only wanted to illustrate the process here, and didn't want to go through and change every TeXError call at this point. In the end, one will call TexError(components, id, ...args) rather than needing a special bboxError() command, so this is just for now.

The rest of the changes in BboxConfiguration.ts are to replace the throw TexError() commands with corresponding bboxError() commands for now, just to show how it works.

The localization files are stored in ts/input/tex/bbox/locales, and I've only provided the en.json file there as an example. The bbox component config.json file gets a new copy section that copies the locale files to the bundle directory. Because this uses the new [bundle] and [ts] paths, this PR targets the update/copy-config branch where those are introduced (which is why I made that separate PR -- these had been combined in the original #1122).

Because the locale files are JSON files, MathJax's file loaders need to be modified to handle JSON files. Unfortunately, how that has to be done differs depending on if you are using the browser or a node application, and whether you are using import() or require(). In order to handle these variations, we introduce a new mathjax.json() function similar to the mathjax.asyncLoad() function, but for loading JSON files. The files in ts/utility/asyncLoad are modified to set up this function appropriately for the given mechanism (except for system.ts, as I'm not sure what the mechanism should be for that, and it is no longer used in any case).

To help with this, ts/util/AsyncLoad.ts has a new method, resolvePath() that gives a consistent mechanism for finding the path to a file being loaded, whether it is a component name like [tex]/bbox, a relative path, an absolute path, or a node module reference. The esm, node, node-import, and system implementations use this to guarantee that paths are treated consistently in all cases.

Because using import or import() to load JSON files in node currently is an experimental feature, which produces a warning message when used, the esm.ts implementation doesn't use import and instead uses fs.readFileSync() to avoid that. In order to resolve node module directories (like @mathjax/src), the input.meta.resolve function is used. But jest doesn't implement input.meta.resolve, so there is a check to use a default function if that isn't available. A new fs.d.ts file is added to get a type for the readFileSync() function.

The node.ts implementation is easier, as require() works to load JSON files, and also handles the node module paths, so nothing really needs to be done there. Similarly, the node-import.ts file uses require() to load JSON files (and import() for others).

The mathjax.ts file sets up mathjax.json to use fetch(), which is the mechanism used in the browser. But fetch() for file:// URLs is not allowed in node, which is why mathjax.json is overridden in node applications by loading one of the asyncLoad implementations just discussed.

The test files for AsyncLoad and its individual implementations have new tests to check the handling of JSON files.

The components/mjs/core/core.js file's implementation of mathjax.asyncLoad is modified to handle JSON file using config.json if it exists, or mathjax.json otherwise. This allows users to configure the json loader just like they configure the require function, via setting the MathJax.loader.json configuration option. For example, one could do

MathJax = {
  loader: {
    require: (file) => import(file),
    json: (file) => import(file, {with: {type: 'json'}}),
  },
};

to use import() for loading all files, including JSON files. Because this would result in a warning message when used in a node application, however, there is a new components/json.cjs file that exports a json function (and a require function) that can be used in node applications as

import {require, json} from '@mathjax/src/components/json.cjs';

MathJax = {
  loader: {
    require: require,
    json: json,
  },
};

which will use require() for both. The json.cjs file is copied to the bundle directory via the config files for the require component.

The core component also loads a new local.js file that imports the Locale utility and sets Locale.isComponent = true. This is used by the Locale object to determine where to load the locale file (either using the component name or the explicit path). In browsers or node applications that use components, this will use the Package.resolvePath() function to map component names to their proper locations. In node applications with direct imports of MathJax modules, this will use the direct paths to the locale data in the locale folders in the ts directory.

The node-main component has had its asyncLoad modified to use the resolvePath method from AsyncLoad and to handle JSON files.

The package.json file gets a new copy:locales script to copy any tex extension locale files (I forgot about that, so I guess the change to the bbox/config.json weren't needed; we can decide which way to go with that).

Then there are new test files and support files for testing loading of JSON files. I also added some .d.ts files to avoid typescript complaints about missing types. The setupTex.js file now handles setup of locales. This makes a couple of functions asynchronous, so some calls to setupTex() need to be async.

The json option is added to the ts/components/loader.ts file, and I fixed some typing mistakes there.

The ts/components/startup.ts file now loads and configures the Locale library. The Locale.setLocale() call is what loads the needed JSON files.

The require TeX extension now calls Locale.setLocale() to load any localizations needed by the newly loaded package. That way, if bbox is loaded, for example, is localizations will be loaded automatically when \require{bbox} is used, or when it is autoloaded.

That should get you through most of the changes here.

[codecov]

Codecov Report

❌ Patch coverage is 92.22374% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.93%. Comparing base (9ef060f) to head (09815fe).

Files with missing lines	Patch %	Lines
ts/core/MmlTree/MmlNode.ts	65.21%	8 Missing ⚠️
components/mjs/node-main/node-main.js	73.07%	7 Missing ⚠️
ts/core/MathDocument.ts	30.00%	7 Missing ⚠️
components/mjs/core/core.js	57.14%	6 Missing ⚠️
ts/core/MmlTree/MmlNodes/mmultiscripts.ts	0.00%	5 Missing ⚠️
components/mjs/startup/init.js	50.00%	4 Missing ⚠️
ts/components/startup.ts	42.85%	4 Missing ⚠️
ts/components/loader.ts	96.10%	3 Missing ⚠️
ts/core/MmlTree/MmlNodes/mtable.ts	0.00%	2 Missing ⚠️
ts/core/MmlTree/MmlNodes/mtr.ts	0.00%	2 Missing ⚠️
... and 9 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1407      +/-   ##
===========================================
+ Coverage    86.70%   86.93%   +0.22%     
===========================================
  Files          340      388      +48     
  Lines        86264    87570    +1306     
  Branches      3246     4966    +1720     
===========================================
+ Hits         74798    76129    +1331     
+ Misses       11466    11420      -46     
- Partials         0       21      +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[dpvc]

I've rebased this branch onto the latest develop branch and fixed the merge conflicts (mostly with the new testsuite changes).

[zorkow]

I think the tests are failing because the component copy is not yet in the branch.

[zorkow]

I did review and adjust in the later PRs. So this is fine to "undraft" and then merge.

[dpvc]

I think the tests are failing because the component copy is not yet in the branch.

It looks like you have removed the erroneous copy blocks in #1479, so when that is merged, it should take care of the issue failed test.

[dpvc]

So this is fine to "undraft" and then merge.

I'm leaving it a draft for the moment because I want to wait to merge it until after our meeting on Thursday. I want to do some more testing with node applications to make sure there are no breaking changes (which I think there still are). We may have to decide how we want to handle that.

[dpvc]

PR #1480 has caused some of the tests to fail. I will make another PR that will resolve the issue. There is a concurrency problem with the tests, since the Locale uses global properties to keep track of the loaded locales, and there are issues with the setupTex() with that. Apparently the the jest beforeEach() call doesn't wait for promises, and so the locales don't always load in time. I have the solution, but need to make the PR for it.