diff --git a/README.md b/README.md index ee195473..03c2d475 100644 --- a/README.md +++ b/README.md @@ -13,19 +13,19 @@ **OmniParser** is a comprehensive method for parsing user interface screenshots into structured and easy-to-understand elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. ## News -- [2025/3] We support local logging of trajecotry so that you can use OmniParser+OmniTool to build training data pipeline for your favorate agent in your domain. [Documentation WIP] -- [2025/3] We are gradually adding multi agents orchstration and improving user interface in OmniTool for better experience. +- [2025/3] We support local logging of trajectory so that you can use OmniParser+OmniTool to build a training data pipeline for your favorite agent in your domain. [Documentation WIP] +- [2025/3] We are gradually adding multi-agent orchestration and improving the user interface in OmniTool for a better experience. - [2025/2] We release OmniParser V2 [checkpoints](https://huggingface.co/microsoft/OmniParser-v2.0). [Watch Video](https://1drv.ms/v/c/650b027c18d5a573/EWXbVESKWo9Buu6OYCwg06wBeoM97C6EOTG6RjvWLEN1Qg?e=alnHGC) - [2025/2] We introduce OmniTool: Control a Windows 11 VM with OmniParser + your vision model of choice. OmniTool supports out of the box the following large language models - OpenAI (4o/o1/o3-mini), DeepSeek (R1), Qwen (2.5VL) or Anthropic Computer Use. [Watch Video](https://1drv.ms/v/c/650b027c18d5a573/EehZ7RzY69ZHn-MeQHrnnR4BCj3by-cLLpUVlxMjF4O65Q?e=8LxMgX) -- [2025/1] V2 is coming. We achieve new state of the art results 39.5% on the new grounding benchmark [Screen Spot Pro](https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding/tree/main) with OmniParser v2 (will be released soon)! Read more details [here](https://github.com/microsoft/OmniParser/tree/master/docs/Evaluation.md). -- [2024/11] We release an updated version, OmniParser V1.5 which features 1) more fine grained/small icon detection, 2) prediction of whether each screen element is interactable or not. Examples in the demo.ipynb. -- [2024/10] OmniParser was the #1 trending model on huggingface model hub (starting 10/29/2024). -- [2024/10] Feel free to checkout our demo on [huggingface space](https://huggingface.co/spaces/microsoft/OmniParser)! (stay tuned for OmniParser + Claude Computer Use) -- [2024/10] Both Interactive Region Detection Model and Icon functional description model are released! [Hugginface models](https://huggingface.co/microsoft/OmniParser) +- [2025/1] V2 is coming. We achieve new state-of-the-art results 39.5% on the new grounding benchmark [Screen Spot Pro](https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding/tree/main) with OmniParser v2 (will be released soon)! Read more details [here](https://github.com/microsoft/OmniParser/tree/master/docs/Evaluation.md). +- [2024/11] We release an updated version, OmniParser V1.5 which features 1) more fine-grained/small icon detection, 2) prediction of whether each screen element is interactable or not. Examples in the demo.ipynb. +- [2024/10] OmniParser was the #1 trending model on Huggingface model hub (starting 10/29/2024). +- [2024/10] Feel free to check out our demo on [Huggingface space](https://huggingface.co/spaces/microsoft/OmniParser)! (stay tuned for OmniParser + Claude Computer Use) +- [2024/10] Both Interactive Region Detection Model and Icon functional description model are released! [Huggingface models](https://huggingface.co/microsoft/OmniParser) - [2024/09] OmniParser achieves the best performance on [Windows Agent Arena](https://microsoft.github.io/WindowsAgentArena/)! ## Install -First clone the repo, and then install environment: +First clone the repo, and then install the environment: ```python cd OmniParser conda create -n "omni" python==3.12 @@ -33,7 +33,7 @@ conda activate omni pip install -r requirements.txt ``` -Ensure you have the V2 weights downloaded in weights folder (ensure caption weights folder is called icon_caption_florence). If not download them with: +Ensure you have the V2 weights downloaded in the weights folder (ensure the caption weights folder is called icon_caption_florence). If not, download them with: ``` # download the model checkpoints to local directory OmniParser/weights/ for f in icon_detect/{train_args.yaml,model.pt,model.yaml} icon_caption/{config.json,generation_config.json,model.safetensors}; do huggingface-cli download microsoft/OmniParser-v2.0 "$f" --local-dir weights; done @@ -41,7 +41,7 @@ Ensure you have the V2 weights downloaded in weights folder (ensure caption weig ``` ## Examples: -We put together a few simple examples in the demo.ipynb. +We have included a few simple examples in demo.ipynb. ## Gradio Demo -To run gradio demo, simply run: +To run the Gradio demo, run: ```python python gradio_demo.py ``` ## Model Weights License -For the model checkpoints on huggingface model hub, please note that icon_detect model is under AGPL license since it is a license inherited from the original yolo model. And icon_caption_blip2 & icon_caption_florence is under MIT license. Please refer to the LICENSE file in the folder of each model: https://huggingface.co/microsoft/OmniParser. +For the model checkpoints on the Huggingface model hub, please note that the icon_detect model is under the AGPL license, inherited from the original YOLO model. The icon_caption_blip2 & icon_caption_florence models are under the MIT license. Please refer to the LICENSE file in each model's folder: https://huggingface.co/microsoft/OmniParser. ## 📚 Citation Our technical report can be found [here](https://arxiv.org/abs/2408.00203). -If you find our work useful, please consider citing our work: +If you find our work useful, please consider citing it: ``` @misc{lu2024omniparserpurevisionbased, title={OmniParser for Pure Vision Based GUI Agent},