Data

Check the data directory for the dataset description and usage.

It includes:

• Iron_IMPAQTS.csv A dataset of 400 annotated passages from Italian political speeches. Each instance contains a target sentence, speaker metadata, discourse summary, and a binary gold label (IRONY). For 200 items, an expert-written explanation of the ironic content is also provided.

• detection_task_prompt_example.txt Example prompt used in the irony detection task.

• explication_task_prompt_example.txt Example prompt used in the open-ended explication task.

• Instructions_DETECTION.pdf Annotation guidelines for the binary irony detection task.

• Instructions_EXPLICATION.pdf Guidelines provided to annotators for irony explanations.

• Instructions_EVALUATION.pdf Human evaluation protocol for assessing model-generated explanations.

---

detection_task script

detection_task.py implements a binary irony detection experiment using OpenAI models via the Responses API.

Models must output 1 if irony is present in the final sentence of the excerpt, and 0 otherwise.

At a high level, the script:

1. Loads the Iron-IMPAQTS dataset.
2. Builds prompts in Italian that include speaker, year, political role (if available), discourse summary, and target excerpt.
3. Queries OpenAI models (gpt-4.1 and o3) with pseudo-deterministic settings.
4. Repeats inference multiple times (default: 5 runs per model) to capture output variability.
5. Stores predictions for each run in the main dataframe and exports them to CSV/XLSX.
6. Computes evaluation metrics (Accuracy, Precision, Recall, F1) against the gold IRONY label.
7. Aggregates results across runs, saving per-run metrics and an overall metrics table.
8. Optionally plots metric distributions (boxplots) across runs for robustness analysis.

This design explicitly supports multi-run evaluation, enabling inspection of stability and variance in model performance.

Most Relevant Functions

Function	Purpose
load_dataset	Loads the dataset and fills missing values.
prompt_creator	Constructs the Italian detection prompt, asking explicitly for 1 (irony) or 0 (no irony).
ask	Sends a single-turn request to either gpt-4.1 or o3 using the OpenAI Responses API.
evaluate_multiple_runs	Computes Accuracy, Precision, Recall, and F1 for each run of a model.
plot_metrics_distribution	Produces boxplots showing metric variability across runs.

(Inline comments in the code provide implementation-level detail.)

---

explication_task script

explication_task.py implements the open-ended generation experiment, where models are asked to explain the ironic content of a passage in free text.

The script operates only on instances labeled as ironic (IRONY == 1) and focuses on explanation quality rather than classification.

Concretely, it:

1. Loads the dataset and filters to ironic cases.

2. Builds structured prompts incorporating:

   • speaker
   • year
   • political role (if present)
   • discourse summary
   • target excerpt

3. Explicitly instructs the model to explain the ironic content of the last sentence.

4. Queries multiple OpenAI models (currently gpt-4.1 and o3).

5. Collects responses in a structured ModelResponse dataclass with success/error tracking.

6. Appends model outputs as new columns in the dataframe.

7. Exports results to both CSV (tab-separated) and XLSX formats.

Unlike the detection script, this pipeline is single-pass per model and produces textual explanations intended for downstream qualitative or human evaluation.

Most Relevant Functions

Function	Purpose
load_dataset	Loads the dataset and fills missing values.
prompt_creator	Creates the Italian explication prompt, requesting a concise explanation of the ironic content.
ask	Sends prompts to OpenAI models, handling both GPT-4.1 and o-series reasoning models.
save_dataset	Writes the augmented dataframe to CSV and XLSX.
main	Filters ironic instances, queries models, stores explanations, and saves final outputs.

Responses are wrapped in a ModelResponse dataclass to track failures and logging information.

---

How to Use

After cloning the repository and configuring your API key and paths inside each script:

python detection_task.py

or

python explication_task.py

Outputs

After running detection_task.py, the results directory will contain:

• Per-run prediction files (*_detection_results.csv/.xlsx)
• Per-model metrics (*_all_runs_metrics.csv)
• all_models_all_runs_metrics.csv
• metrics_distribution_plot.png (if plotting is enabled)

After running explication_task.py:

• explication_results.csv — tab-separated model explanations
• explication_results.xlsx — same content in Excel format

Authors & Affiliations

• Walter Paci – University of Florence
• Valentina Saccone – University of Florence

---

📄 Citation

---

🪪 License

• Code & Prompts: Released under the MIT License
• Dataset: Released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. This means you may share and adapt the dataset for non-commercial purposes, with attribution.

---

📬 Contact

For questions or collaborations, please contact the authors via GitHub.

---