Publications | Dongfu Jiang (姜东甫)

2024

MANTIS: Interleaved Multi-Image Instruction Tuning

Dongfu Jiang, Xuan He, Huaye Zeng, and 4 more authors

In , 2024

@inproceedings{jiang2024mantis,
  title = {MANTIS: Interleaved Multi-Image Instruction Tuning},
  author = {Jiang, Dongfu and He, Xuan and Zeng, Huaye and Wei, Cong and Ku, Max and Liu, Qian and Chen, Wenhu},
  year = {2024},
  eprint = {2405.01483},
  archiveprefix = {arXiv},
  primaryclass = {cs.CV},
  journal = {ArXiv},
  url = {https://arxiv.org/abs/2405.01483},
  selected = true,
}

TMLR 2024
TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks

Dongfu Jiang, Yishan Li, Ge Zhang, and 3 more authors

ArXiv, 2024

Abs arXiv Bib Code Website

We present TIGERScore, a \textbfTrained metric that follows \textbfInstruction \textbfGuidance to perform \textbfExplainable, and \textbfReference-free evaluation over a wide spectrum of text generation tasks. Different from other automatic evaluation methods that only provide arcane scores, TIGERScore is guided by natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA-2, trained on our meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. The dataset consists of 42K quadruple in the form of (instruction, input, system output →error analysis). We collected the ‘system outputs’ through from a large variety of models to cover different types of errors. To quantitatively assess our metric, we evaluate its correlation with human ratings on 5 held-in datasets, 2 held-out datasets and show that TIGERScore can achieve the open-source SoTA correlation with human ratings across these datasets and almost approaches GPT-4 evaluator. As a reference-free metric, its correlation can even surpass the best existing reference-based metrics. To further qualitatively assess the rationale generated by our metric, we conduct human evaluation on the generated explanations and found that the explanations are 70.8% accurate. Through these experimental results, we believe TIGERScore demonstrates the possibility of building universal explainable metrics to evaluate any text generation task. All the resourced are released in our project website: \urlhttps://tiger-ai-lab.github.io/TIGERScore/.
@article{jiang2023TIGERScore, title = {TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks}, author = {Jiang, Dongfu and Li, Yishan and Zhang, Ge and Huang, Wenhao and Lin, Bill Yuchen and Chen, Wenhu}, journal = {ArXiv}, volume = {abs/2310.00752}, year = {2024}, selected = true, url = {https://arxiv.org/abs/2310.00752}, }

2023

arXiv

VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation

Max W.F. Ku, Dongfu Jiang, Cong Wei, and 2 more authors

In , 2023

arXiv Bib Code Website

@inproceedings{Ku2023VIEScoreTE,
  title = {VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation},
  author = {Ku, Max W.F. and Jiang, Dongfu and Wei, Cong and Yue, Xiang and Chen, Wenhu},
  journal = {ArXiv},
  year = {2023},
  volume = {abs/2312.14867},
  url = {https://arxiv.org/abs/2312.14867},
  selected = true,
}

CVPR 2024 oral

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Xiang Yue, Yuansheng Ni, Kai Zhang, and 19 more authors

In , 2023

arXiv Bib Code Website

@inproceedings{Yue2023MMMUAM,
  title = {MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI},
  author = {Yue, Xiang and Ni, Yuansheng and Zhang, Kai and Zheng, Tianyu and Liu, Ruoqi and Zhang, Ge and Stevens, Samuel and Jiang, Dongfu and Ren, Weiming and Sun, Yuxuan and Wei, Cong and Yu, Botao and Yuan, Ruibin and Sun, Renliang and Yin, Ming and Zheng, Boyuan and Yang, Zhenzhu and Liu, Yibo and Huang, Wenhao and Sun, Huan and Su, Yu and Chen, Wenhu},
  year = {2023},
  journal = {arXiv preprint arXiv:2311.16502},
  url = {https://arxiv.org/abs/2311.16502},
  selected = true,
}

ACL 2023
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin

In ACL 2023, Jul 2023

Abs arXiv Bib Blog Code Website

We present LLM-Blender, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs). Our framework consists of two modules: PairRanker and GenFuser, addressing the observation that optimal LLMs for different examples can significantly vary. PairRanker employs a specialized pairwise comparison method to distinguish subtle differences between candidate outputs. It jointly encodes the input text and a pair of candidates, using cross-attention encoders to determine the superior one. Our results demonstrate that PairRanker exhibits the highest correlation with ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates, generating an improved output by capitalizing on their strengths and mitigating their weaknesses. To facilitate large-scale evaluation, we introduce a benchmark dataset, MixInstruct, which is a mixture of multiple instruction datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly outperform individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.
@inproceedings{jiang-etal-2023-llm, title = {{LLM}-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion}, author = {Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen}, booktitle = {ACL 2023}, month = jul, year = {2023}, address = {Toronto, Canada}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.acl-long.792}, doi = {10.18653/v1/2023.acl-long.792}, pages = {14165--14178}, selected = true, }