IMPAQTS-X

IMPAQTS-X is a comprehensive machine learning framework for text quality assessment and classification. It combines classical supervised learning models with modern LLM-based evaluation approaches, supporting both gold-standard and automatically annotated (silver) training paradigms.

The project leverages stylistic and syntactic features extracted from text to train and evaluate multiple model architectures, with extensive support for multi-metric evaluation and LLM-based judgment.

---

Data

The data directory contains the datasets used for training and evaluation.

It includes:

• IMPAQTS-X.csv Main evaluation dataset containing preprocessed text instances with target labels and associated metadata. Used for comprehensive model evaluation across all algorithms.

• train.csv Training dataset used for model fitting and hyperparameter tuning. Includes the same feature structure as the evaluation set.

---

Frozen Models

Check the frozen_models directory for pre-trained model artifacts and corresponding predictions.

This directory contains production-ready models serialized in joblib format, organized by algorithm type:

• Binary Classifiers

  • binary_ridgecl/: Ridge classifier optimized for balanced accuracy using style and POS features
  • logreg_bin_drop_standard_metrics/: Logistic regression with standard metrics removed
  • logreg_bin_frozen_only_style_plus_pos__objective_balanced_accuracy/: Logistic regression using custom feature set
  • svm_bin_only_style_plus_pos/: Support Vector Machine with style and POS features

• Ordinal Classifier

  • ordinal_ridge/: Ridge-based ordinal regression for multi-class ordered predictions

Each frozen model directory includes:

• .joblib: Serialized scikit-learn model artifact
• .csv: Predictions on silver evaluation set
• .json: Model metadata (hyperparameters, feature names, optimization objectives)

Lastly, raw_scores_on_silver.csv contains the frozen, aggregated results of all saved configurations.

LLMs as Judges

The LLMs-as-judges directory contains scripts that leverage large language models for text evaluation and judgment.

• aya_as_judge.py Implements evaluation using the Aya language model by CohereAI.

• llama4_as_judge.py Implements evaluation using the LLAMA4-Scout-Instruct language model by Meta.

• qwen_as_judge.py Implements evaluation using the Qwen 30B language model by Alibaba Cloud.

---

Evaluation Metrics

The metrics directory implements multiple complementary evaluation approaches for text quality assessment.

• bertscore_script.py Computes BERTScore, a contextual embedding-based similarity metric that correlates well with human judgment. Captures semantic similarity without exact token matching.

• bleurt_script.py Implements BLEURT (Learned Evaluation Metric Rank Tuning), a learned metric trained on human annotations for more robust evaluation than reference-based metrics.

• nli_script.py Natural Language Inference-based evaluation script using entailment relationships to assess text coherence and semantic consistency.

---

Models on Gold

The models_on_gold directory contains training scripts for models trained exclusively on manually annotated (gold) data.

Script	Model Type	Purpose
binary_LogReg.py	Logistic Regression	Binary classification using gold standard annotations
binary_Ridge.py	Ridge Regression	Binary ridge classifier optimized for balanced accuracy on gold data
Ordinal_Ridge.py	Ordinal Ridge	Multi-class ordered prediction using gold annotations
SVM_model.py	Support Vector Machine	Non-linear binary classification with gold training data

---

Models on Silver

The models_on_silver directory contains scripts to run the best performing configuration of each model presented in models_on_gold. The best performing configuration is saved in Frozen Models and retrieved automatically by each script contained in this folder.

Each script automatically assign a binary label to the explanations present in IMPAQTS-X.csv. This automatic labelling is not validated.

---

Features Used

Common features across all models include:

• style: Stylistic features capturing text tone, formality, and linguistic patterns
• pos: Part-of-speech tag distributions and syntactic patterns
• balanced_accuracy: Primary optimization objective for imbalanced classification scenarios

---

Outputs

Model training scripts generate:

• Serialized model files (.joblib)
• Prediction files (.csv)
• Performance metrics and evaluation reports

Metric scripts produce:

• Per-instance scores (.csv)
• Aggregated performance summaries
• Comparison tables across models

---

Authors & Affiliations

• Walter Paci – University of Florence

---

📄 Citation

---

🪪 License

• Code & Scripts: Released under the MIT License
• Dataset: Released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. This means you may share and adapt the dataset for non-commercial purposes, with attribution.

---

📬 Contact

For questions, issues, or collaborations, please contact the authors via GitHub.