Dongfu Jiang (姜东甫)

2024

1. arXiv

   VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation

   Xuan He^*, Dongfu Jiang^*, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, and 11 more authors

   ArXiv, 2024

   

   @article{he2024videoscore,
     title = {VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation},
     author = {He, Xuan and Jiang, Dongfu and Zhang, Ge and Ku, Max and Soni, Achint and Siu, Sherman and Chen, Haonan and Chandra, Abhranil and Jiang, Ziyan and Arulraj, Aaran and Wang, Kai and Do, Quy Duc and Ni, Yuansheng and Lyu, Bohan and Narsupalli, Yaswanth and Fan, Rongqi and Lyu, Zhiheng and Lin, Yuchen and Chen, Wenhu},
     journal = {ArXiv},
     booltitle = {ArXiv},
     year = {2024},
     eprint = {2406.15252},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
     url = {https://arxiv.org/abs/2406.15252},
     github = {TIGER-AI-Lab/VideoScore},
     huggingface = {https://huggingface.co/collections/TIGER-Lab/videoscore-6678c9451192e834e91cc0bf},
     twitter = {https://twitter.com/DongfuJiang/status/1805438506137010326},
     selected = true,
     num_co_first_author = {2},
   }

   

2. arXiv

   WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

   Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin

   ArXiv, 2024

   

   @article{Lu2024WildVisionEV,
     title = {WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences},
     author = {Lu, Yujie and Jiang, Dongfu and Chen, Wenhu and Wang, William Yang and Choi, Yejin and Lin, Bill Yuchen},
     journal = {ArXiv},
     booltitle = {ArXiv},
     year = {2024},
     eprint = {2406.11069},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
     url = {https://arxiv.org/abs/2406.11069},
     twitter = {https://twitter.com/billyuchenlin/status/1755207605537120513},
     huggingface = {https://huggingface.co/spaces/WildVision/vision-arena},
     selected = true,
   }

   

3. arXiv

   GenAI Arena: An Open Evaluation Platform for Generative Models

   Dongfu Jiang^*, Max W.F. Ku^*, Tianle Li^*, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen

   ArXiv, 2024

   

   @article{Jiang2024GenAIAA,
     title = {GenAI Arena: An Open Evaluation Platform for Generative Models},
     author = {Jiang, Dongfu and Ku, Max W.F. and Li, Tianle and Ni, Yuansheng and Sun, Shizhuo and Fan, Rongqi and Chen, Wenhu},
     journal = {ArXiv},
     booltitle = {ArXiv},
     year = {2024},
     eprint = {2406.04485},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
     url = {https://arxiv.org/abs/2406.04485},
     github = {TIGER-AI-Lab/GenAI-Arena},
     huggingface = {https://huggingface.co/spaces/TIGER-Lab/GenAI-Arena},
     selected = true,
     num_co_first_author = {3}
   }

   

4. arXiv

   MANTIS: Interleaved Multi-Image Instruction Tuning

   Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen

   ArXiv, 2024

   

   @article{jiang2024mantis,
     title = {MANTIS: Interleaved Multi-Image Instruction Tuning},
     author = {Jiang, Dongfu and He, Xuan and Zeng, Huaye and Wei, Cong and Ku, Max and Liu, Qian and Chen, Wenhu},
     journal = {ArXiv},
     booltitle = {ArXiv},
     year = {2024},
     eprint = {2405.01483},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
     url = {https://arxiv.org/abs/2405.01483},
     github = {TIGER-AI-Lab/Mantis},
     twitter = {https://twitter.com/DongfuJiang/status/1786552974598078677},
     huggingface = {https://huggingface.co/collections/TIGER-Lab/mantis-6619b0834594c878cdb1d6e4},
     selected = true,
   }

   

5. ACL 2024

   VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation

   Max W.F. Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen

   In ACL, 2024

   

   @inproceedings{Ku2023VIEScoreTE,
     title = {VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation},
     author = {Ku, Max W.F. and Jiang, Dongfu and Wei, Cong and Yue, Xiang and Chen, Wenhu},
     booktitle = {ACL},
     year = {2024},
     volume = {abs/2312.14867},
     url = {https://arxiv.org/abs/2312.14867},
     github = {TIGER-AI-Lab/VIEScore},
     twitter = {https://twitter.com/DongfuJiang/status/1742043191732302076},
     selected = true,
   }

   

6. TMLR 2024

   TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks

   Dongfu Jiang^*, Yishan Li^*, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen

   Transactions on Machine Learning Research, 2024

   

   We present TIGERScore, a \textbfTrained metric that follows \textbfInstruction \textbfGuidance to perform \textbfExplainable, and \textbfReference-free evaluation over a wide spectrum of text generation tasks. Different from other automatic evaluation methods that only provide arcane scores, TIGERScore is guided by natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA-2, trained on our meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. The dataset consists of 42K quadruple in the form of (instruction, input, system output error analysis). We collected the ‘system outputs’ through from a large variety of models to cover different types of errors. To quantitatively assess our metric, we evaluate its correlation with human ratings on 5 held-in datasets, 2 held-out datasets and show that \metricname can achieve the open-source SoTA correlation with human ratings across these datasets and almost approaches GPT-4 evaluator. As a reference-free metric, its correlation can even surpass the best existing reference-based metrics. To further qualitatively assess the rationale generated by our metric, we conduct human evaluation on the generated explanations and found that the explanations are 70.8% accurate. Through these experimental results, we believe \metricname demonstrates the possibility of building universal explainable metrics to evaluate any text generation task.

   @article{jiang2024tigerscore,
     title = {{TIGERS}core: Towards Building Explainable Metric for All Text Generation Tasks},
     author = {Jiang, Dongfu and Li, Yishan and Zhang, Ge and Huang, Wenhao and Lin, Bill Yuchen and Chen, Wenhu},
     journal = {Transactions on Machine Learning Research},
     year = {2024},
     selected = true,
     github = {TIGER-AI-Lab/TIGERScore},
     twitter = {https://twitter.com/DongfuJiang/status/1735508082510168425},
     huggingface = {https://huggingface.co/collections/TIGER-Lab/tigerscore-657020bfae61260b6131f1ca},
     issn = {2835-8856},
     url = {https://openreview.net/forum?id=EE1CBKC0SZ},
     num_co_first_author = {2}
   }

   

2023

1. CVPR 2024

   MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

   Xiang Yue^*, Yuansheng Ni^*, Kai Zhang^*, Tianyu Zheng^*, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, and 14 more authors

   In CVPR oral, 2023

   

   @inproceedings{Yue2023MMMUAM,
     title = {MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI},
     author = {Yue, Xiang and Ni, Yuansheng and Zhang, Kai and Zheng, Tianyu and Liu, Ruoqi and Zhang, Ge and Stevens, Samuel and Jiang, Dongfu and Ren, Weiming and Sun, Yuxuan and Wei, Cong and Yu, Botao and Yuan, Ruibin and Sun, Renliang and Yin, Ming and Zheng, Boyuan and Yang, Zhenzhu and Liu, Yibo and Huang, Wenhao and Sun, Huan and Su, Yu and Chen, Wenhu},
     year = {2023},
     booktitle = {CVPR <span style="color: red; font-weight: bold;">oral</span>},
     url = {https://arxiv.org/abs/2311.16502},
     github = {MMMU-Benchmark/MMMU},
     twitter = {https://twitter.com/xiangyue96/status/1729698316554801358},
     huggingface = {https://huggingface.co/datasets/MMMU/MMMU},
     selected = true,
     num_co_first_author = {4}
   }

   

2. ACL 2023

   LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

   Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin

   In ACL, Jul 2023

   

   We present LLM-Blender, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs). Our framework consists of two modules: PairRanker and GenFuser, addressing the observation that optimal LLMs for different examples can significantly vary. PairRanker employs a specialized pairwise comparison method to distinguish subtle differences between candidate outputs. It jointly encodes the input text and a pair of candidates, using cross-attention encoders to determine the superior one. Our results demonstrate that PairRanker exhibits the highest correlation with ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates, generating an improved output by capitalizing on their strengths and mitigating their weaknesses. To facilitate large-scale evaluation, we introduce a benchmark dataset, MixInstruct, which is a mixture of multiple instruction datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly outperform individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.

   @inproceedings{jiang-etal-2023-llm,
     title = {{LLM}-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion},
     author = {Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen},
     booktitle = {ACL},
     month = jul,
     year = {2023},
     address = {Toronto, Canada},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.acl-long.792},
     doi = {10.18653/v1/2023.acl-long.792},
     pages = {14165--14178},
     selected = true,
     github = {yuchenlin/LLM-Blender},
     twitter = {https://twitter.com/billyuchenlin/status/1668666357058277377},
     huggingface = {https://huggingface.co/llm-blender},
   }