Sponsored by
Vmake Labs Video Enhancer
zai-org / cogvlm2-video-llama3-base
Model's Last Updated: July 24 2024
text-generation
Introduction of cogvlm2-video-llama3-base
Model Details of cogvlm2-video-llama3-base
CogVLM2-Video-Llama3-Base
Introduction
CogVLM2-Video achieves state-of-the-art performance on multiple video question answering tasks. It can achieve video understanding within one minute. We provide two example videos to demonstrate CogVLM2-Video's video understanding and video temporal grounding capabilities.
BenchMark
The following diagram shows the performance of CogVLM2-Video on the MVBench , VideoChatGPT-Bench and Zero-shot VideoQA datasets (MSVD-QA, MSRVTT-QA, ActivityNet-QA). Where VCG-* refers to the VideoChatGPTBench, ZS-* refers to Zero-Shot VideoQA datasets and MV-* refers to main categories in the MVBench.
Performance on VideoChatGPT-Bench and Zero-shot VideoQA dataset:
| Models | VCG-AVG | VCG-CI | VCG-DO | VCG-CU | VCG-TU | VCG-CO | ZS-AVG |
|---|---|---|---|---|---|---|---|
| IG-VLM GPT4V | 3.17 | 3.40 | 2.80 | 3.61 | 2.89 | 3.13 | 65.70 |
| ST-LLM | 3.15 | 3.23 | 3.05 | 3.74 | 2.93 | 2.81 | 62.90 |
| ShareGPT4Video | N/A | N/A | N/A | N/A | N/A | N/A | 46.50 |
| VideoGPT+ | 3.28 | 3.27 | 3.18 | 3.74 | 2.83 | 3.39 | 61.20 |
| VideoChat2_HD_mistral | 3.10 | 3.40 | 2.91 | 3.72 | 2.65 | 2.84 | 57.70 |
| PLLaVA-34B | 3.32 | 3.60 | 3.20 | 3.90 | 2.67 | 3.25 | 68.10 |
| CogVLM2-Video | 3.41 | 3.49 | 3.46 | 3.87 | 2.98 | 3.23 | 66.60 |
Performance on MVBench dataset:
| Models | AVG | AA | AC | AL | AP | AS | CO | CI | EN | ER | FA | FP | MA | MC | MD | OE | OI | OS | ST | SC | UA |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IG-VLM GPT4V | 43.7 | 72.0 | 39.0 | 40.5 | 63.5 | 55.5 | 52.0 | 11.0 | 31.0 | 59.0 | 46.5 | 47.5 | 22.5 | 12.0 | 12.0 | 18.5 | 59.0 | 29.5 | 83.5 | 45.0 | 73.5 |
| ST-LLM | 54.9 | 84.0 | 36.5 | 31.0 | 53.5 | 66.0 | 46.5 | 58.5 | 34.5 | 41.5 | 44.0 | 44.5 | 78.5 | 56.5 | 42.5 | 80.5 | 73.5 | 38.5 | 86.5 | 43.0 | 58.5 |
| ShareGPT4Video | 51.2 | 79.5 | 35.5 | 41.5 | 39.5 | 49.5 | 46.5 | 51.5 | 28.5 | 39.0 | 40.0 | 25.5 | 75.0 | 62.5 | 50.5 | 82.5 | 54.5 | 32.5 | 84.5 | 51.0 | 54.5 |
| VideoGPT+ | 58.7 | 83.0 | 39.5 | 34.0 | 60.0 | 69.0 | 50.0 | 60.0 | 29.5 | 44.0 | 48.5 | 53.0 | 90.5 | 71.0 | 44.0 | 85.5 | 75.5 | 36.0 | 89.5 | 45.0 | 66.5 |
| VideoChat2_HD_mistral | 62.3 | 79.5 | 60.0 | 87.5 | 50.0 | 68.5 | 93.5 | 71.5 | 36.5 | 45.0 | 49.5 | 87.0 | 40.0 | 76.0 | 92.0 | 53.0 | 62.0 | 45.5 | 36.0 | 44.0 | 69.5 |
| PLLaVA-34B | 58.1 | 82.0 | 40.5 | 49.5 | 53.0 | 67.5 | 66.5 | 59.0 | 39.5 | 63.5 | 47.0 | 50.0 | 70.0 | 43.0 | 37.5 | 68.5 | 67.5 | 36.5 | 91.0 | 51.5 | 79.0 |
| CogVLM2-Video | 62.3 | 85.5 | 41.5 | 31.5 | 65.5 | 79.5 | 58.5 | 77.0 | 28.5 | 42.5 | 54.0 | 57.0 | 91.5 | 73.0 | 48.0 | 91.0 | 78.0 | 36.0 | 91.5 | 47.0 | 68.5 |
Evaluation details
We follow the previous works to evaluate the performance of our model. In different benchmarks, we craft task-specific prompts for each benchmark:
# For MVBench
prompt = f"Carefully watch the video and pay attention to the cause and sequence of events, the detail and movement of objects, and the action and pose of persons. Based on your observations, select the best option that accurately addresses the question.\n " + f"{prompt.replace('Short Answer.', '')}\n" + "Short Answer:"
# For VideoChatGPT-Bench
prompt = f"Carefully watch the video and pay attention to the cause and sequence of events, the detail and movement of objects, and the action and pose of persons. Based on your observations, comprehensively answer the following question. Your answer should be long and cover all the related aspects\n " + f"{prompt.replace('Short Answer.', '')}\n" + "Answer:"
# For Zero-shot VideoQA
prompt = f"The input consists of a sequence of key frames from a video. Answer the question comprehensively including all the possible verbs and nouns that can discribe the events, followed by significant events, characters, or objects that appear throughout the frames.\n " + f"{prompt.replace('Short Answer.', '')}\n" + "Answer:"
For evaluation codes, please refer to the evaluation script in PLLaVA.
Using This Model
This repository is a
base
version model and does not support chat.
You can quickly install the Python package dependencies and run model inference in our github .
License
This model is released under the CogVLM2 LICENSE . For models built with Meta Llama 3, please also adhere to the LLAMA3_LICENSE .
Training details
Pleaser refer to our technical report for training formula and hyperparameters.
Runs of zai-org cogvlm2-video-llama3-base on huggingface.co
234
Total runs
-3
24-hour runs
4
3-day runs
5
7-day runs
27
30-day runs
More Information About cogvlm2-video-llama3-base huggingface.co Model
Url of cogvlm2-video-llama3-base
cogvlm2-video-llama3-base huggingface.co Url
Provider of cogvlm2-video-llama3-base huggingface.co
Other API from zai-org
Total runs:
1.2M
Run Growth:
-612.2K
Growth Rate:
-48.56%
Total runs:
751.5K
Run Growth:
621.2K
Growth Rate:
85.65%
Total runs:
454.6K
Run Growth:
-508.3K
Growth Rate:
-105.86%
Total runs:
201.6K
Run Growth:
-1.1M
Growth Rate:
-551.58%
Total runs:
150.5K
Run Growth:
119.2K
Growth Rate:
86.54%
Total runs:
131.6K
Run Growth:
-39.2K
Growth Rate:
-28.96%
Total runs:
102.2K
Run Growth:
27.4K
Growth Rate:
25.93%
Total runs:
50.8K
Run Growth:
-15.7K
Growth Rate:
-30.19%
Total runs:
43.6K
Run Growth:
-46.5K
Growth Rate:
-102.04%
Total runs:
42.0K
Run Growth:
26.0K
Growth Rate:
62.62%
Total runs:
35.7K
Run Growth:
34.1K
Growth Rate:
95.49%
Total runs:
25.8K
Run Growth:
-43.1K
Growth Rate:
-167.27%
Total runs:
24.9K
Run Growth:
-174.9K
Growth Rate:
-687.71%
Total runs:
16.0K
Run Growth:
-6.1K
Growth Rate:
-37.92%
Total runs:
15.5K
Run Growth:
-28.9K
Growth Rate:
-184.65%
Total runs:
15.2K
Run Growth:
-6.1K
Growth Rate:
-41.73%
Total runs:
14.3K
Run Growth:
-225
Growth Rate:
-1.56%
Total runs:
12.2K
Run Growth:
-7.0K
Growth Rate:
-58.07%
Total runs:
11.2K
Run Growth:
-3.3K
Growth Rate:
-29.93%
Total runs:
10.0K
Run Growth:
-5.5K
Growth Rate:
-54.36%
Total runs:
6.3K
Run Growth:
1.0K
Growth Rate:
15.83%
Total runs:
6.0K
Run Growth:
-11.7K
Growth Rate:
-193.30%
Total runs:
5.1K
Run Growth:
-6.9K
Growth Rate:
-127.40%
Total runs:
4.9K
Run Growth:
48
Growth Rate:
0.98%
Total runs:
3.5K
Run Growth:
-49.3K
Growth Rate:
-1417.29%
Total runs:
2.1K
Run Growth:
252
Growth Rate:
11.80%
Total runs:
1.8K
Run Growth:
-11.4K
Growth Rate:
-642.66%
Total runs:
1.7K
Run Growth:
-152
Growth Rate:
-10.19%
Total runs:
1.4K
Run Growth:
-1.3K
Growth Rate:
-96.15%
Total runs:
1.3K
Run Growth:
-8.0K
Growth Rate:
-604.99%
Total runs:
1.2K
Run Growth:
-1.2K
Growth Rate:
-97.32%
Total runs:
1.2K
Run Growth:
529
Growth Rate:
45.33%
Total runs:
1.0K
Run Growth:
225
Growth Rate:
22.57%
Total runs:
991
Run Growth:
-2.0K
Growth Rate:
-201.62%
Total runs:
977
Run Growth:
-149
Growth Rate:
-15.14%
Total runs:
935
Run Growth:
372
Growth Rate:
39.96%
Total runs:
773
Run Growth:
-494
Growth Rate:
-64.66%
Total runs:
685
Run Growth:
24
Growth Rate:
3.56%
Total runs:
671
Run Growth:
-345
Growth Rate:
-52.59%
Total runs:
662
Run Growth:
66
Growth Rate:
10.06%
Total runs:
569
Run Growth:
-14
Growth Rate:
-2.47%
Total runs:
561
Run Growth:
33
Growth Rate:
5.95%
Total runs:
552
Run Growth:
39
Growth Rate:
7.09%
Total runs:
528
Run Growth:
231
Growth Rate:
44.85%
Total runs:
443
Run Growth:
-353
Growth Rate:
-79.68%
Total runs:
430
Run Growth:
-680
Growth Rate:
-160.76%
Total runs:
398
Run Growth:
-299
Growth Rate:
-76.86%
Total runs:
347
Run Growth:
-232
Growth Rate:
-68.44%
Total runs:
336
Run Growth:
81
Growth Rate:
24.47%
Total runs:
317
Run Growth:
-2
Growth Rate:
-0.64%
Total runs:
306
Run Growth:
18
Growth Rate:
6.02%
Total runs:
294
Run Growth:
-7
Growth Rate:
-2.47%
Total runs:
294
Run Growth:
-22
Growth Rate:
-7.75%
Total runs:
291
Run Growth:
-12
Growth Rate:
-4.27%
Total runs:
289
Run Growth:
-51
Growth Rate:
-17.65%
Total runs:
266
Run Growth:
5
Growth Rate:
1.95%
Total runs:
265
Run Growth:
-24
Growth Rate:
-9.56%
Total runs:
253
Run Growth:
33
Growth Rate:
13.31%