Dongfu Jiang (姜东甫)

2023

1. arXiv

   MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

   Xiang Yue, Yuansheng Ni, Kai Zhang, and 19 more authors

   In , 2023

   arXiv Bib Code Website

   @inproceedings{Yue2023MMMUAM,
     title = {MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI},
     author = {Yue, Xiang and Ni, Yuansheng and Zhang, Kai and Zheng, Tianyu and Liu, Ruoqi and Zhang, Ge and Stevens, Samuel and Jiang, Dongfu and Ren, Weiming and Sun, Yuxuan and Wei, Cong and Yu, Botao and Yuan, Ruibin and Sun, Renliang and Yin, Ming and Zheng, Boyuan and Yang, Zhenzhu and Liu, Yibo and Huang, Wenhao and Sun, Huan and Su, Yu and Chen, Wenhu},
     year = {2023},
     journal = {arXiv preprint arXiv:2311.16502},
     url = {https://api.semanticscholar.org/CorpusID:265466525},
     selected = true,
   }

   

2. arXiv

   TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks

   Dongfu Jiang, Yishan Li, Ge Zhang, and 3 more authors

   arXiv preprint arXiv:2310.00752, 2023

   Abs arXiv Bib Code Website

   We present TIGERScore, a \textbfTrained metric that follows \textbfInstruction \textbfGuidance to perform \textbfExplainable, and \textbfReference-free evaluation over a wide spectrum of text generation tasks. Different from other automatic evaluation methods that only provide arcane scores, TIGERScore is guided by the natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA, trained on our meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. The dataset consists of 48K quadruple in the form of (instruction, input, system output →error analysis). We collected the ‘system outputs’ through diverse channels to cover different types of errors. To quantitatively assess our metric, we evaluate its correlation with human ratings on 5 held-in datasets, 2 held-out datasets and show that TIGERScore can achieve the highest overall Spearman’s correlation with human ratings across these datasets and outperforms other metrics significantly. As a reference-free metric, its correlation can even surpass the best existing reference-based metrics. To further qualitatively assess the rationale generated by our metric, we conduct human evaluation on the generated explanations and found that the explanations are 70.8% accurate. Through these experimental results, we believe TIGERScore demonstrates the possibility of building universal explainable metrics to evaluate any text generation task.

   @article{jiang2023TIGERScore,
     title = {TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks},
     author = {Jiang, Dongfu and Li, Yishan and Zhang, Ge and Huang, Wenhao and Lin, Bill Yuchen and Chen, Wenhu},
     journal = {arXiv preprint arXiv:2310.00752},
     year = {2023},
     selected = true,
     url = {https://arxiv.org/abs/2310.00752},
   }

   

3. ACL

   LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

   Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin

   In ACL 2023, Jul 2023

   Abs arXiv Bib Blog Code Website

   We present LLM-Blender, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs). Our framework consists of two modules: PairRanker and GenFuser, addressing the observation that optimal LLMs for different examples can significantly vary. PairRanker employs a specialized pairwise comparison method to distinguish subtle differences between candidate outputs. It jointly encodes the input text and a pair of candidates, using cross-attention encoders to determine the superior one. Our results demonstrate that PairRanker exhibits the highest correlation with ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates, generating an improved output by capitalizing on their strengths and mitigating their weaknesses. To facilitate large-scale evaluation, we introduce a benchmark dataset, MixInstruct, which is a mixture of multiple instruction datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly outperform individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.

   @inproceedings{jiang-etal-2023-llm,
     title = {{LLM}-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion},
     author = {Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen},
     booktitle = {ACL 2023},
     month = jul,
     year = {2023},
     address = {Toronto, Canada},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.acl-long.792},
     doi = {10.18653/v1/2023.acl-long.792},
     pages = {14165--14178},
     selected = true,
   }