You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is the repository for our ANAH series of papers, containing [ANAH: Analytical Annotation of Hallucinations in Large Language Models](https://arxiv.org/abs/2405.20315) and [ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models](https://arxiv.org/abs/2407.04693).
5
+
This is the repository for our ANAH series of papers, which aims to reduce hallucinations in LLM through research involving benchmarking, detection, and mitigation of hallucinations:
6
+
7
+
-**[Benchmark]**[ANAH: Analytical Annotation of Hallucinations in Large Language Models](https://arxiv.org/abs/2405.20315)
8
+
-**[Detection]**[ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models](https://arxiv.org/abs/2407.04693)
9
+
-**[Mitigation]**[Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs](http://arxiv.org/abs/2503.02846)
6
10
7
11
The repo contains:
8
12
9
13
+ The [data](#huggingface-dataset) for training and evaluating the LLM which consists of sentence-level hallucination annotations.
10
14
+ The [model](#huggingface-model) for annotating the hallucination.
11
15
+ The [code](#evaluation) for evaluating the hallucinations level of LLM-generated content and the LLMs' ability to annotate hallucination.
12
-
16
+
+ The [code](#maskdpo-training) for performing fine-grained factuality alignment.
13
17
14
18
## 🚀 What's New
19
+
-**[2025.01.23]** Mask-DPO has been accepted by the ICLR 2025. 🎉🎉🎉
15
20
-**[2024.09.26]** ANAH-v2 has been accepted by the NeurIPS 2024. 🎉🎉🎉
16
21
-**[2024.07.12]** ANAH-v2 [Annotator](https://huggingface.co/opencompass/anah-v2) has been open-sourced. 🔥🔥🔥
17
22
-**[2024.07.03]** ANAH [Annotator-7B](https://huggingface.co/opencompass/anah-7b) & [20B](https://huggingface.co/opencompass/anah-20b) have been open-sourced. 🔥🔥🔥
@@ -46,10 +51,24 @@ Through iterative self-training, we simultaneously and progressively scale up th
46
51
47
52
The final dataset encompasses both over ∼3k topics, ∼196k model responses, and ∼822k annotated sentences, in English and Chinese.
48
53
54
+
The final hallucination annotator (detector) with only 7B parameters surpasses the performance of GPT-4 and obtains new state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference.
55
+
49
56
<palign="center">
50
57
<imgsrc="docs/figure/teaser-v2.jpg"height="500">
51
58
</p>
52
59
60
+
## Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs
Mask-DPO is a fine-grained factuality alignment method based on Direct Preference Optimization (DPO).
65
+
66
+
Incorporating sentence-level factuality as mask signals, Mask-DPO only learns from factually correct sentences in the preferred samples and prevents the penalty on factual contents in the not preferred samples, which resolves the ambiguity in the preference learning.
where `chosem_item` is the fine-grained factual information about `chosem`, `sents` is a sentence-level slice of the `content` in `chosen`, and `type` is the hallucination situation of the corresponding sentence.
201
+
`rejected_item` is the same.
202
+
203
+
We recommend you to use [ANAH-v2](https://huggingface.co/opencompass/anah-v2) for fine-grained hallucination annotation of your data. Of course you can use other methods as well.
204
+
205
+
206
+
### 3. Training
207
+
208
+
After putting the data and initial model paths to the corresponding locations in config, you can use the following command to train the model.
209
+
210
+
```bash
211
+
python -m torch.distributed.run \
212
+
--nproc_per_node=4 \
213
+
--nnodes=1 \
214
+
--node_rank=0 \
215
+
--rdzv_id=1234 \
216
+
--rdzv_backend=c10d \
217
+
--rdzv_endpoint=127.0.0.1:1234 \
218
+
./maskdpo/train.py \
219
+
./maskdpo/example_config.py \
220
+
--deepspeed deepspeed_zero3 \
221
+
--launcher pytorch
222
+
```
223
+
150
224
## ❤️ Acknowledgements
151
225
152
-
ANAH is built with [InternLM](https://github.com/InternLM/InternLM) and [LMDeploy](https://github.com/InternLM/lagent). Thanks for their awesome work!
226
+
ANAH is built with [InternLM](https://github.com/InternLM/InternLM), [Xtuner](https://github.com/InternLM/xtuner) and [LMDeploy](https://github.com/InternLM/lagent). Thanks for their awesome work!
153
227
154
228
## 🖊️ Citation
155
229
@@ -168,6 +242,12 @@ If you find this project useful in your research, please consider citing:
168
242
journal={arXiv preprint arXiv:2407.04693},
169
243
year={2024}
170
244
}
245
+
246
+
@inproceedings{gumask,
247
+
title={Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs},
248
+
author={Gu, Yuzhe and Zhang, Wenwei and Lyu, Chengqi and Lin, Dahua and Chen, Kai},
249
+
booktitle={The Thirteenth International Conference on Learning Representations}
0 commit comments