Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrieval
This repo contains the annotated datasets and expriments implementation introduced in our resource paper in SIGIR2022 Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrieval. [Paper].
📢 What's New
- 🌟 2023-01: Multiple models fine-tuned with Multi-CPR dataset are open source on the ModelScope platform. Released Models 开源模型
Introduction
Multi-CPR is a multi-domain Chinese dataset for passage retrieval. The dataset is collected from three different domains, including E-commerce, Entertainment video and Medical. Each dataset contains millions of passages and a certain amount of human annotated query-passage related pairs.
Examples of annotated query-passage related pairs in three different domains:
| Domain | Query | Passage |
|---|---|---|
| E-commerce | 尼康z62 (Nikon z62) | Nikon/尼康二代全画幅微单机身Z62 Z72 24-70mm套机 (Nikon/Nikon II, full-frame micro-single camera, body Z62 Z72 24-70mm set) |
| Entertainment video | 海神妈祖 (Ma-tsu, Goddess of the Sea) | 海上女神妈祖 (Ma-tsu, Goddess of the Sea) |
| Medical | 大人能把手放在睡觉婴儿胸口吗 (Can adults put their hands on the chest of a sleeping baby?) |
大人不能把手放在睡觉婴儿胸口，对孩子呼吸不好，要注意 (Adults should not put their hands on the chest of a sleeping baby as this is not good for the baby's breathing.) |
Data Format
Datasets of each domain share a uniform format, more details can be found in our paper:
- qid: A unique id for each query that is used in evaluation
- pid: A unique id for each passaage that is used in evaluation
| File name | number of record | format |
|---|---|---|
| corpus.tsv | 1002822 | pid, passage content |
| train.query.txt | 100000 | qid, query content |
| dev.query.txt | 1000 | qid, query content |
| qrels.train.tsv | 100000 | qid, '0', pid, '1' |
| qrels.dev.tsv | 1000 | qid, '0', pid, '1' |
Experiments
The retrieval and rerank folders contain how to train a BERT-base dense passage retrieval and reranking model based on Multi-CPR dataset. This code is based on the previous work tevatron and reranker produced by luyug. Many thanks to luyug.
Dense Retrieval Resutls
| Models | Datasets | Encoder | E-commerce | Entertainment video | Medical | |||
|---|---|---|---|---|---|---|---|---|
| MRR@10 | Recall@1000 | MRR@10 | Recall@1000 | MRR@10 | Recall@1000 | |||
| DPR | General | BERT | 0.2106 | 0.7750 | 0.1950 | 0.7710 | 0.2133 | 0.5220 |
| DPR-1 | In-domain | BERT | 0.2704 | 0.9210 | 0.2537 | 0.9340 | 0.3270 | 0.7470 |
| DPR-2 | In-domain | BERT-CT | 0.2894 | 0.9260 | 0.2627 | 0.9350 | 0.3388 | 0.7690 |
BERT-reranking results
| Retrieval | Reranker | E-commerce | Entertainment video | Medical |
|---|---|---|---|---|
| MRR@10 | MRR@10 | MRR@10 | ||
| DPR-1 | - | 0.2704 | 0.2537 | 0.3270 |
| DPR-1 | BERT | 0.3624 | 0.3772 | 0.3885 |
Requirements
python=3.8
transformers==4.18.0
tqdm==4.49.0
datasets==1.11.0
torch==1.11.0
faiss==1.7.0
Released Models
We have uploaded some checkpoints finetuned with Multi-CPR to ModelScope Model hub. It should be noted that the open-source models on ModelScope are fine-tuned based on the ROM or CoROM model rather than the original BERT model. ROM is a pre-trained language model specially designed for dense passage retrieval task. More details about the ROM model, please refer to paper ROM
| Model Type | Domain | Description | Link |
|---|---|---|---|
| Retrieval | General | - | nlp_corom_sentence-embedding_chinese-base |
| Retrieval | E-commerce | - | nlp_corom_sentence-embedding_chinese-base-ecom |
| Retrieval | Medical | - | nlp_corom_sentence-embedding_chinese-base-medical |
| ReRanking | General | - | nlp_rom_passage-ranking_chinese-base |
| ReRanking | E-commerce | - | nlp_corom_passage-ranking_chinese-base-ecom |
| ReRanking | Medical | - | nlp_corom_passage-ranking_chinese-base-medical |
开源模型
基于Multi-CPR数据集训练的预训练语言模型文本表示(召回)模型、语义相关性(精排)模型已逐步通过ModelScope平台开源，欢迎大家下载体验。在ModelScope上开源的模型都是基于ROM或者CoROM模型为底座训练的而不是原始的BERT模型，ROM是一个专门针对文本召回任务设计的预训练语言模型，更多关于ROM模型细节可以参考论文ROM
| 模型类别 | 领域 | 模型描述 | 下载链接 |
|---|---|---|---|
| Retrieval | General | 中文通用领域文本表示模型(召回阶段) | nlp_corom_sentence-embedding_chinese-base |
| Retrieval | E-commerce | 中文电商领域文本表示模型(召回阶段) | nlp_corom_sentence-embedding_chinese-base-ecom |
| Retrieval | Medical | 中文医疗领域文本表示模型(召回阶段) | nlp_corom_sentence-embedding_chinese-base-medical |
| ReRanking | General | 中文通用领域语义相关性模型(精排阶段) | nlp_rom_passage-ranking_chinese-base |
| ReRanking | E-commerce | 中文电商领域语义相关性模型(精排阶段) | nlp_corom_passage-ranking_chinese-base-ecom |
| ReRanking | Medical | 中文医疗领域语义相关性模型(精排阶段) | nlp_corom_passage-ranking_chinese-base-medical |
Citing us
If you feel the datasets helpful, please cite:
@inproceedings{Long2022MultiCPRAM,
author = {Dingkun Long and Qiong Gao and Kuan Zou and Guangwei Xu and Pengjun Xie and Ruijie Guo and Jian Xu and Guanjun Jiang and Luxi Xing and Ping Yang},
title = {Multi-CPR: {A} Multi Domain Chinese Dataset for Passage Retrieval},
booktitle = {{SIGIR}},
pages = {3046--3056},
publisher = {{ACM}},
year = {2022}
}