Total runs:
5.4M
Run Growth:
4.8M
Growth Rate:
90.17%
Sponsored by
PoYo.AI
zai-org / cogagent-9b-20241220
Model's Last Updated: December 25 2024
image-text-to-text
Introduction of cogagent-9b-20241220
Model Details of cogagent-9b-20241220
CogAgent
🌐 Github | 🤗 Huggingface Space | 📄 Technical Report | 📜 arxiv paper
About the Model
The
CogAgent-9B-20241220
model is based on
GLM-4V-9B
, a bilingual
open-source VLM base model. Through data collection and optimization, multi-stage training, and strategy improvements,
CogAgent-9B-20241220
achieves significant advancements in GUI perception, inference prediction accuracy, action space
completeness, and task generalizability. The model supports bilingual (Chinese and English) interaction with both
screenshots and language input.
This version of the CogAgent model has already been applied in ZhipuAI's GLM-PC product . We hope this release will assist researchers and developers in advancing the research and applications of GUI agents based on vision-language models.
Running the Model
Please refer to our GitHub for specific examples of running the model.
Input and Output
cogagent-9b-20241220
is an agent execution model rather than a conversational model. It does not support continuous
conversations but does support continuous execution history. Below are guidelines on how users should format their input
for the model and interpret the formatted output.
Run the Model
Please visit our github for specific running examples, as well as the part for prompt concatenation (this directly affects whether the model runs correctly) .
In particular, pay attention to the prompt concatenation process. You can refer to app/client.py#L115 for concatenating user input prompts.
current_platform = identify_os() # "Mac" or "WIN" or "Mobile"，注意大小写
platform_str = f"(Platform: {current_platform})\n"
format_str = "(Answer in Action-Operation-Sensitive format.)\n" # You can use other format to replace "Action-Operation-Sensitive"
history_str = "\nHistory steps: "
for index, (grounded_op_func, action) in enumerate(zip(history_grounded_op_funcs, history_actions)):
history_str += f"\n{index}. {grounded_op_func}\t{action}" # start from 0.
query = f"Task: {task}{history_str}\n{platform_str}{format_str}" # Be careful about the \n
A minimal user input concatenation code is as follows:
"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
The concatenated Python string will look like:
"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
Due to the length, if you would like to understand the meaning and representation of each field in detail, please refer to the GitHub .
Previous Work
In November 2023, we released the first generation of CogAgent. You can find related code and weights in the CogVLM & CogAgent Official Repository .
CogVLM📖 Paper: CogVLM: Visual Expert for Pretrained Language Models CogVLM is a powerful open-source vision-language model (VLM). CogVLM-17B has 10 billion vision parameters and 7 billion language parameters, supporting 490x490 resolution image understanding and multi-turn conversations. CogVLM-17B achieved state-of-the-art performance on 10 classic cross-modal benchmarks including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+, RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA, and TDIUC benchmarks. |
CogAgent📖 Paper: CogAgent: A Visual Language Model for GUI Agents CogAgent is an improved open-source vision-language model based on CogVLM. CogAgent-18B has 11 billion vision parameters and 7 billion language parameters, supporting image understanding at 1120x1120 resolution. Beyond CogVLM's capabilities, it also incorporates GUI agent capabilities. CogAgent-18B achieved state-of-the-art performance on 9 classic cross-modal benchmarks, including VQAv2, OK-VQ, TextVQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE benchmarks. It significantly outperformed existing models on GUI operation datasets like AITW and Mind2Web. |
License
Please follow the Model License for using the model weights.
Runs of zai-org cogagent-9b-20241220 on huggingface.co
496
Total runs
2
24-hour runs
160
3-day runs
208
7-day runs
206
30-day runs
More Information About cogagent-9b-20241220 huggingface.co Model
Url of cogagent-9b-20241220
cogagent-9b-20241220 huggingface.co Url
Provider of cogagent-9b-20241220 huggingface.co
Other API from zai-org
Updated:September 07 2026
Total runs:
1.8M
Run Growth:
-104.0K
Growth Rate:
-5.84%
Total runs:
705.8K
Run Growth:
-474.8K
Growth Rate:
-66.84%
Total runs:
475.1K
Run Growth:
-840.0K
Growth Rate:
-168.02%
Total runs:
380.7K
Run Growth:
210.9K
Growth Rate:
65.25%
Total runs:
175.3K
Run Growth:
-6.5K
Growth Rate:
-3.69%
Total runs:
103.7K
Run Growth:
20.1K
Growth Rate:
19.60%
Total runs:
68.7K
Run Growth:
-16.9K
Growth Rate:
-23.93%
Total runs:
67.7K
Run Growth:
-11.8K
Growth Rate:
-17.43%
Total runs:
62.5K
Run Growth:
51.3K
Growth Rate:
83.84%
Total runs:
60.8K
Run Growth:
-325
Growth Rate:
-0.53%
Total runs:
55.1K
Run Growth:
-24.3K
Growth Rate:
-43.24%
Total runs:
35.1K
Run Growth:
33.9K
Growth Rate:
96.53%
Total runs:
33.1K
Run Growth:
12.1K
Growth Rate:
35.35%
Total runs:
32.8K
Run Growth:
27.8K
Growth Rate:
85.57%
Total runs:
26.7K
Run Growth:
-351.9K
Growth Rate:
-1242.25%
Total runs:
24.7K
Run Growth:
-20.9K
Growth Rate:
-78.17%
Total runs:
21.0K
Run Growth:
-723.4K
Growth Rate:
-3437.18%
Total runs:
18.0K
Run Growth:
4.4K
Growth Rate:
24.42%
Total runs:
17.9K
Run Growth:
3.6K
Growth Rate:
18.76%
Total runs:
17.8K
Run Growth:
2.2K
Growth Rate:
12.08%
Total runs:
16.5K
Run Growth:
-3.5K
Growth Rate:
-20.19%
Total runs:
16.0K
Run Growth:
-12.6K
Growth Rate:
-78.55%
Total runs:
15.1K
Run Growth:
-13.3K
Growth Rate:
-87.97%
Total runs:
14.8K
Run Growth:
3.7K
Growth Rate:
24.88%
Total runs:
14.3K
Run Growth:
-5.5K
Growth Rate:
-38.53%
Total runs:
12.2K
Run Growth:
7.3K
Growth Rate:
60.25%
Total runs:
10.0K
Run Growth:
290
Growth Rate:
2.84%
Total runs:
6.7K
Run Growth:
-493
Growth Rate:
-4.30%
Total runs:
6.6K
Run Growth:
-1.4K
Growth Rate:
-20.66%
Total runs:
6.4K
Run Growth:
1.0K
Growth Rate:
14.34%
Total runs:
4.3K
Run Growth:
578
Growth Rate:
13.35%
Total runs:
2.9K
Run Growth:
2.0K
Growth Rate:
68.05%
Total runs:
2.1K
Run Growth:
-8.4K
Growth Rate:
-395.14%
Total runs:
2.1K
Run Growth:
-114
Growth Rate:
-5.49%
Total runs:
2.0K
Run Growth:
-1.9K
Growth Rate:
-95.26%
Total runs:
1.3K
Run Growth:
-449
Growth Rate:
-34.94%
Total runs:
1.3K
Run Growth:
-327
Growth Rate:
-25.47%
Total runs:
1.3K
Run Growth:
581
Growth Rate:
45.60%
Total runs:
1.0K
Run Growth:
-664
Growth Rate:
-63.91%
Total runs:
1.0K
Run Growth:
509
Growth Rate:
49.42%
Total runs:
889
Run Growth:
420
Growth Rate:
47.24%
Total runs:
861
Run Growth:
-3.5K
Growth Rate:
-472.53%
Total runs:
836
Run Growth:
505
Growth Rate:
60.62%
Total runs:
836
Run Growth:
507
Growth Rate:
58.95%
Total runs:
779
Run Growth:
24
Growth Rate:
2.93%
Total runs:
778
Run Growth:
-77
Growth Rate:
-10.81%
Total runs:
680
Run Growth:
342
Growth Rate:
49.49%
Total runs:
631
Run Growth:
111
Growth Rate:
17.82%
Total runs:
616
Run Growth:
-841
Growth Rate:
-136.53%
Total runs:
467
Run Growth:
-30
Growth Rate:
-6.24%
Total runs:
466
Run Growth:
-576
Growth Rate:
-122.81%
Total runs:
450
Run Growth:
-166
Growth Rate:
-35.47%
Total runs:
443
Run Growth:
-353
Growth Rate:
-79.68%
Total runs:
426
Run Growth:
-75
Growth Rate:
-16.89%
Total runs:
398
Run Growth:
-1.0K
Growth Rate:
-256.46%
Total runs:
337
Run Growth:
115
Growth Rate:
34.23%
Total runs:
315
Run Growth:
44
Growth Rate:
14.29%
Total runs:
307
Run Growth:
277
Growth Rate:
90.23%
Total runs:
295
Run Growth:
81
Growth Rate:
27.55%
Total runs:
291
Run Growth:
69
Growth Rate:
23.96%
Total runs:
289
Run Growth:
-51
Growth Rate:
-17.65%
Total runs:
288
Run Growth:
55
Growth Rate:
19.30%
Total runs:
273
Run Growth:
-105
Growth Rate:
-39.18%
Total runs:
256
Run Growth:
37
Growth Rate:
14.51%
Total runs:
252
Run Growth:
121
Growth Rate:
48.59%
Total runs:
247
Run Growth:
69
Growth Rate:
28.16%
Total runs:
238
Run Growth:
59
Growth Rate:
25.21%
Total runs:
229
Run Growth:
56
Growth Rate:
24.78%