这是一个Python基础教程，展示如何定义函数。
\ndef greet(name):\n \"\"\"问候函数\"\"\"\n return f\"Hello, {name}!\"\n\n# 使用示例\nresult = greet(\"World\")\nprint(result)这个函数可以用来问候任何人。
\n ", "groundtruth_llm_webkit_md": "# Python编程教程\n\n这是一个Python基础教程，展示如何定义函数。\n\n```python\ndef greet(name):\n \"\"\"问候函数\"\"\"\n return f\"Hello, {name}!\"\n\n# 使用示例\nresult = greet(\"World\")\nprint(result)\n```\n\n这个函数可以用来问候任何人。", "groundtruth_content_list": [{"type": "heading", "content": "Python编程教程", "level": 1}, {"type": "paragraph", "content": "这是一个Python基础教程，展示如何定义函数。"}, {"type": "code", "content": "def greet(name):\n \"\"\"问候函数\"\"\"\n return f\"Hello, {name}!\"\n\n# 使用示例\nresult = greet(\"World\")\nprint(result)"}, {"type": "paragraph", "content": "这个函数可以用来问候任何人。"}], "llm_webkit_md": "# Python编程教程\n\n这是一个Python基础教程，展示如何定义函数。\n\n```python\ndef greet(name):\n \"\"\"问候函数\"\"\"\n return f\"Hello, {name}!\"\n\n# 使用示例\nresult = greet(\"World\")\nprint(result)\n```\n\n这个函数可以用来问候任何人。", "content_list": [{"type": "heading", "content": "Python编程教程", "level": 1}, {"type": "paragraph", "content": "这是一个Python基础教程，展示如何定义函数。"}, {"type": "code", "content": "def greet(name):\n \"\"\"问候函数\"\"\"\n return f\"Hello, {name}!\"\n\n# 使用示例\nresult = greet(\"World\")\nprint(result)"}, {"type": "paragraph", "content": "这个函数可以用来问候任何人。"}], "url": "https://python-tutorial.example.com/functions", "domain": null, "language": "en", "content_type": "programming", "difficulty": null, "tags": null} +{"id": "sample-002-math-formulas", "html": "\n这里展示一些基本的数学公式。
\n勾股定理：a² + b² = c²
\n二次方程的解为：
\nx = (-b ± √(b² - 4ac)) / 2a
\n欧拉公式是数学中最美丽的公式之一：e^(iπ) + 1 = 0
\n| 函数 | 导数 |
|---|---|
| x² | 2x |
| sin(x) | cos(x) |
以下是2024年第一季度的销售数据分析。
\nimport pandas as pd\nimport numpy as np\n\n# 读取数据\ndf = pd.read_csv('sales_q1_2024.csv')\n\n# 计算统计信息\nmonthly_avg = df.groupby('month')['sales'].mean()\nprint(f\"平均销售额: {monthly_avg}\")| 月份 | 销售额(万元) | 增长率 |
|---|---|---|
| 1月 | 120.5 | +15.2% |
| 2月 | 135.8 | +12.7% |
| 3月 | 148.3 | +9.2% |
标准差公式：σ = √(Σ(xi - μ)² / n)
\n总体来看，第一季度销售表现良好，呈现稳定增长趋势。
\n ", "groundtruth_llm_webkit_md": "# 数据分析报告\n\n以下是2024年第一季度的销售数据分析。\n\n## 数据处理代码\n\n```python\nimport pandas as pd\nimport numpy as np\n\n# 读取数据\ndf = pd.read_csv('sales_q1_2024.csv')\n\n# 计算统计信息\nmonthly_avg = df.groupby('month')['sales'].mean()\nprint(f\"平均销售额: {monthly_avg}\")\n```\n\n## 销售统计\n\n| 月份 | 销售额(万元) | 增长率 |\n|------|-------------|--------|\n| 1月 | 120.5 | +15.2% |\n| 2月 | 135.8 | +12.7% |\n| 3月 | 148.3 | +9.2% |\n\n标准差公式：$\\sigma = \\sqrt{\\frac{\\Sigma(x_i - \\mu)^2}{n}}$\n\n总体来看，第一季度销售表现良好，呈现稳定增长趋势。", "groundtruth_content_list": [{"type": "heading", "content": "数据分析报告", "level": 1}, {"type": "paragraph", "content": "以下是2024年第一季度的销售数据分析。"}, {"type": "heading", "content": "数据处理代码", "level": 2}, {"type": "code", "content": "import pandas as pd\nimport numpy as np\n\n# 读取数据\ndf = pd.read_csv('sales_q1_2024.csv')\n\n# 计算统计信息\nmonthly_avg = df.groupby('month')['sales'].mean()\nprint(f\"平均销售额: {monthly_avg}\")"}, {"type": "heading", "content": "销售统计", "level": 2}, {"type": "table", "content": "| 月份 | 销售额(万元) | 增长率 |\n|------|-------------|--------|\n| 1月 | 120.5 | +15.2% |\n| 2月 | 135.8 | +12.7% |\n| 3月 | 148.3 | +9.2% |"}, {"type": "paragraph", "content": "标准差公式：σ = √(Σ(xi - μ)² / n)"}, {"type": "paragraph", "content": "总体来看，第一季度销售表现良好，呈现稳定增长趋势。"}], "llm_webkit_md": "# 数据分析报告\n\n以下是2024年第一季度的销售数据分析。\n\n## 数据处理代码\n\n```python\nimport pandas as pd\nimport numpy as np\n\n# 读取数据\ndf = pd.read_csv('sales_q1_2024.csv')\n\n# 计算统计信息\nmonthly_avg = df.groupby('month')['sales'].mean()\nprint(f\"平均销售额: {monthly_avg}\")\n```\n\n## 销售统计\n\n| 月份 | 销售额(万元) | 增长率 |\n|------|-------------|--------|\n| 1月 | 120.5 | +15.2% |\n| 2月 | 135.8 | +12.7% |\n| 3月 | 148.3 | +9.2% |\n\n标准差公式：$\\sigma = \\sqrt{\\frac{\\Sigma(x_i - \\mu)^2}{n}}$\n\n总体来看，第一季度销售表现良好，呈现稳定增长趋势。", "content_list": [{"type": "heading", "content": "数据分析报告", "level": 1}, {"type": "paragraph", "content": "以下是2024年第一季度的销售数据分析。"}, {"type": "heading", "content": "数据处理代码", "level": 2}, {"type": "code", "content": "import pandas as pd\nimport numpy as np\n\n# 读取数据\ndf = pd.read_csv('sales_q1_2024.csv')\n\n# 计算统计信息\nmonthly_avg = df.groupby('month')['sales'].mean()\nprint(f\"平均销售额: {monthly_avg}\")"}, {"type": "heading", "content": "销售统计", "level": 2}, {"type": "table", "content": "| 月份 | 销售额(万元) | 增长率 |\n|------|-------------|--------|\n| 1月 | 120.5 | +15.2% |\n| 2月 | 135.8 | +12.7% |\n| 3月 | 148.3 | +9.2% |"}, {"type": "paragraph", "content": "标准差公式：σ = √(Σ(xi - μ)² / n)"}, {"type": "paragraph", "content": "总体来看，第一季度销售表现良好，呈现稳定增长趋势。"}], "url": "https://data-report.company.com/q1-2024-analysis", "domain": null, "language": "zh", "content_type": "business", "difficulty": null, "tags": null} +{"id": "sample-004-algorithm-explanation", "html": "\n这里介绍常见算法的时间复杂度。
\ndef quicksort(arr):\n if len(arr) <= 1:\n return arr\n \n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n \n return quicksort(left) + middle + quicksort(right)| 算法 | 最好情况 | 平均情况 | 最坏情况 |
|---|---|---|---|
| 快速排序 | O(n log n) | O(n log n) | O(n²) |
| 归并排序 | O(n log n) | O(n log n) | O(n log n) |
| 冒泡排序 | O(n) | O(n²) | O(n²) |
Master定理：T(n) = aT(n/b) + f(n)
\n其中 a ≥ 1, b > 1 是常数，f(n) 是正函数。
\n ", "groundtruth_llm_webkit_md": "# 算法复杂度分析\n\n这里介绍常见算法的时间复杂度。\n\n## 快速排序实现\n\n```python\ndef quicksort(arr):\n if len(arr) <= 1:\n return arr\n \n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n \n return quicksort(left) + middle + quicksort(right)\n```\n\n## 复杂度对比\n\n| 算法 | 最好情况 | 平均情况 | 最坏情况 |\n|------|----------|----------|----------|\n| 快速排序 | O(n log n) | O(n log n) | O(n²) |\n| 归并排序 | O(n log n) | O(n log n) | O(n log n) |\n| 冒泡排序 | O(n) | O(n²) | O(n²) |\n\nMaster定理：$T(n) = aT(n/b) + f(n)$\n\n其中 $a \\geq 1, b > 1$ 是常数，$f(n)$ 是正函数。", "groundtruth_content_list": [{"type": "heading", "content": "算法复杂度分析", "level": 1}, {"type": "paragraph", "content": "这里介绍常见算法的时间复杂度。"}, {"type": "heading", "content": "快速排序实现", "level": 2}, {"type": "code", "content": "def quicksort(arr):\n if len(arr) <= 1:\n return arr\n \n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n \n return quicksort(left) + middle + quicksort(right)"}, {"type": "heading", "content": "复杂度对比", "level": 2}, {"type": "table", "content": "| 算法 | 最好情况 | 平均情况 | 最坏情况 |\n|------|----------|----------|----------|\n| 快速排序 | O(n log n) | O(n log n) | O(n²) |\n| 归并排序 | O(n log n) | O(n log n) | O(n log n) |\n| 冒泡排序 | O(n) | O(n²) | O(n²) |"}, {"type": "equation-inline", "content": "T(n) = aT(n/b) + f(n)"}, {"type": "paragraph", "content": "其中 a ≥ 1, b > 1 是常数，f(n) 是正函数。"}], "llm_webkit_md": "# 算法复杂度分析\n\n这里介绍常见算法的时间复杂度。\n\n## 快速排序实现\n\n```python\ndef quicksort(arr):\n if len(arr) <= 1:\n return arr\n \n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n \n return quicksort(left) + middle + quicksort(right)\n```\n\n## 复杂度对比\n\n| 算法 | 最好情况 | 平均情况 | 最坏情况 |\n|------|----------|----------|----------|\n| 快速排序 | O(n log n) | O(n log n) | O(n²) |\n| 归并排序 | O(n log n) | O(n log n) | O(n log n) |\n| 冒泡排序 | O(n) | O(n²) | O(n²) |\n\nMaster定理：$T(n) = aT(n/b) + f(n)$\n\n其中 $a \\geq 1, b > 1$ 是常数，$f(n)$ 是正函数。", "content_list": [{"type": "heading", "content": "算法复杂度分析", "level": 1}, {"type": "paragraph", "content": "这里介绍常见算法的时间复杂度。"}, {"type": "heading", "content": "快速排序实现", "level": 2}, {"type": "code", "content": "def quicksort(arr):\n if len(arr) <= 1:\n return arr\n \n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n \n return quicksort(left) + middle + quicksort(right)"}, {"type": "heading", "content": "复杂度对比", "level": 2}, {"type": "table", "content": "| 算法 | 最好情况 | 平均情况 | 最坏情况 |\n|------|----------|----------|----------|\n| 快速排序 | O(n log n) | O(n log n) | O(n²) |\n| 归并排序 | O(n log n) | O(n log n) | O(n log n) |\n| 冒泡排序 | O(n) | O(n²) | O(n²) |"}, {"type": "equation-inline", "content": "T(n) = aT(n/b) + f(n)"}, {"type": "paragraph", "content": "其中 a ≥ 1, b > 1 是常数，f(n) 是正函数。"}], "url": "https://algorithm-guide.cs.edu/complexity-analysis", "domain": null, "language": "zh", "content_type": "computer_science", "difficulty": null, "tags": null} diff --git a/examples/test_model.py b/examples/test_model.py new file mode 100644 index 0000000..59b88aa --- /dev/null +++ b/examples/test_model.py @@ -0,0 +1,16 @@ +from webmainbench import DataLoader, Evaluator, ExtractorFactory + +# 1. 加载评测数据集 +dataset = DataLoader.load_jsonl("WebMainBench/data/WebMainBench_llm-webkit_v1_WebMainBench_dataset_merge_2549_llm_webkit.jsonl") + +# 2. 创建抽取器 +extractor = ExtractorFactory.create("test-model") + +# 3. 运行评测 +evaluator = Evaluator() +result = evaluator.evaluate(dataset, extractor) + +# 4. 查看结果 +print(f"Overall Score: {result.overall_metrics}") +print(f"Category Metrics: {result.category_metrics}") +print(f"Error Analysis: {result.error_analysis}") diff --git a/tests/test_test_model_extractor.py b/tests/test_test_model_extractor.py new file mode 100644 index 0000000..aca17d2 --- /dev/null +++ b/tests/test_test_model_extractor.py @@ -0,0 +1,41 @@ +import unittest +from webmainbench.extractors.test_model_extractor import TestModelExtractor + +class TestTestModelExtractor(unittest.TestCase): + """测试 TestModelExtractor 的基本功能""" + + def setUp(self): + """初始化测试用的抽取器实例""" + self.extractor = TestModelExtractor("test-model") + + # 使用 data 目录下的 test_model.jsonl 作为测试数据 + import json + from pathlib import Path + + # 读取第一个样本作为测试用例 + data_path = Path(__file__).parent.parent / "data" / "test_model.jsonl" + with open(data_path, "r", encoding="utf-8") as f: + first_line = f.readline() + sample_dict = json.loads(first_line) + + # 由于 TestModelExtractor 期望 sample 支持属性访问，这里用 SimpleNamespace 包装 + from types import SimpleNamespace + self.sample_data = SimpleNamespace(**sample_dict) + + def test_extract_from_sample(self): + """测试extract_from_sample方法""" + result = self.extractor.extract_from_sample(self.sample_data) + self.assertTrue(result.success) + self.assertEqual(result.content, self.sample_data.llm_webkit_md) + self.assertEqual(result.content_list, self.sample_data.content_list) + self.assertEqual(result.language, self.sample_data.language) + self.assertEqual(result.confidence_score, 1.0) + + def test_extract_with_empty_html(self): + """测试extract方法遇到空html的情况""" + result = self.extractor.extract("") + self.assertFalse(result.success) + self.assertIn("Empty HTML input", result.error_message) + +if __name__ == "__main__": + unittest.main() diff --git a/webmainbench/data/dataset.py b/webmainbench/data/dataset.py index 78ddb48..806e3a5 100644 --- a/webmainbench/data/dataset.py +++ b/webmainbench/data/dataset.py @@ -18,7 +18,8 @@ class DataSample: html: str # HTML with cc-select=true annotations groundtruth_content: str # Groundtruth markdown content groundtruth_content_list: List[Dict[str, Any]] # Groundtruth content_list from llm-webkit - + content_list: List[Dict[str, Any]] = None # Content_list from llm-webkit + content: str = None # Content from llm-webkit # Optional metadata url: Optional[str] = None domain: Optional[str] = None @@ -26,6 +27,7 @@ class DataSample: content_type: Optional[str] = None # article, forum, blog, etc. difficulty: Optional[str] = None # easy, medium, hard tags: Optional[List[str]] = None + llm_webkit_md: Optional[str] = None # Extracted results (populated during evaluation) extracted_results: Optional[Dict[str, Any]] = None @@ -37,6 +39,9 @@ def to_dict(self) -> Dict[str, Any]: "html": self.html, "groundtruth_content": self.groundtruth_content, "groundtruth_content_list": self.groundtruth_content_list, + "content_list": self.content_list, + "content": self.content, + "llm_webkit_md": self.llm_webkit_md, "url": self.url, "domain": self.domain, "language": self.language, diff --git a/webmainbench/evaluator/evaluator.py b/webmainbench/evaluator/evaluator.py index b9b6887..c11e7ca 100644 --- a/webmainbench/evaluator/evaluator.py +++ b/webmainbench/evaluator/evaluator.py @@ -307,8 +307,11 @@ def _process_batch(self, batch_samples: List[DataSample], extractor: BaseExtract def _evaluate_sample(self, sample: DataSample, extractor: BaseExtractor) -> Dict[str, Any]: """Evaluate a single sample.""" - # Extract content - extraction_result = extractor.extract(sample.html, sample.url) + if extractor.__class__.__name__ == 'TestModelExtractor': + extraction_result = extractor.extract_from_sample(sample) + else: + # Extract content + extraction_result = extractor.extract(sample.html, sample.url) # Prepare result sample_result = { diff --git a/webmainbench/extractors/__init__.py b/webmainbench/extractors/__init__.py index e94c225..0208627 100644 --- a/webmainbench/extractors/__init__.py +++ b/webmainbench/extractors/__init__.py @@ -8,7 +8,7 @@ from .factory import ExtractorFactory from .llm_webkit_extractor import LlmWebkitExtractor from .jina_extractor import JinaExtractor - +from .test_model_extractor import TestModelExtractor __all__ = [ @@ -17,4 +17,5 @@ "ExtractorFactory", "LlmWebkitExtractor", "JinaExtractor", + "TestModelExtractor", ] \ No newline at end of file diff --git a/webmainbench/extractors/base.py b/webmainbench/extractors/base.py index 5e408b1..e25e3c2 100644 --- a/webmainbench/extractors/base.py +++ b/webmainbench/extractors/base.py @@ -103,7 +103,33 @@ def _extract_content(self, html: str, url: str = None) -> ExtractionResult: Returns: ExtractionResult instance """ - pass + + def extract_from_sample(self, sample: Dict[str, Any]) -> ExtractionResult: + """ + 直接从数据样本（如评测数据集的dict）中读取groundtruth内容，返回ExtractionResult。 + 适用于评测流程的基线测试或简单抽取器。 + + 参数: + sample: 包含groundtruth内容的数据样本dict + + 返回: + ExtractionResult实例，内容直接取自sample + """ + # 兼容常见字段 + # 这里直接从sample中获取'llm-webkit_md'字段内容，注意字段名有'-'，不能用点操作符，需要用[]方式 + content = sample.llm_webkit_md + content_list = sample.content_list + language = sample.language + # 置信度直接设为1.0，表示“完美抽取” + confidence_score = 1.0 + + return ExtractionResult( + content=content, + content_list=content_list, + language=language, + confidence_score=confidence_score, + success=True + ) def extract(self, html: str, url: str = None) -> ExtractionResult: """ diff --git a/webmainbench/extractors/test_model_extractor.py b/webmainbench/extractors/test_model_extractor.py new file mode 100644 index 0000000..ed3d0d7 --- /dev/null +++ b/webmainbench/extractors/test_model_extractor.py @@ -0,0 +1,27 @@ +""" +Test Model Extractor for WebMainBench +""" + +from typing import Dict, Any, Optional +from .base import BaseExtractor, ExtractionResult +from .factory import extractor + +@extractor("test-model") +class TestModelExtractor(BaseExtractor): + """ + 一个用于测试的抽取器，不做实际抽取，直接返回数据集中的content和content_list字段。 + 适用于评估流程的验证和基线测试。 + """ + + version = "1.0.0" + description = "Test extractor that returns groundtruth content/content_list for evaluation baseline" + + def _setup(self) -> None: + """测试模型无需特殊初始化。""" + pass + + def _extract_content(self, html: str, url: str = None) -> ExtractionResult: + """ + 直接从输入的html参数（假定为数据集样本的dict或json字符串）中读取content和content_list字段。 + """ + pass