0. Papers
There are two solutions based on this architecture.
- BSNLP 2019 ACL workshop: solution and paper on multilingual shared task.
- The second place solution of Dialogue AGRR-2019 task and paper.
Description
This repository contains solution of NER task based on PyTorch reimplementation of Google's TensorFlow repository for the BERT model that was released together with the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
This implementation can load any pre-trained TensorFlow checkpoint for BERT (in particular Google's pre-trained models).
Old version is in "old" branch.
2. Usage
2.1 Create data
from modules.data import bert_data
data = bert_data.LearnData.create(
train_df_path=train_df_path,
valid_df_path=valid_df_path,
idx2labels_path="/path/to/vocab",
clear_cache=True
)
2.2 Create model
from modules.models.bert_models import BERTBiLSTMAttnCRF
model = BERTBiLSTMAttnCRF.create(len(data.train_ds.idx2label))
2.3 Create Learner
from modules.train.train import NerLearner
num_epochs = 100
learner = NerLearner(
model, data, "/path/for/save/best/model", t_total=num_epochs * len(data.train_dl))
2.4 Predict
from modules.data.bert_data import get_data_loader_for_predict
learner.load_model()
dl = get_data_loader_for_predict(data, df_path="/path/to/df/for/predict")
preds = learner.predict(dl)
2.5 Evaluate
from sklearn_crfsuite.metrics import flat_classification_report
from modules.analyze_utils.utils import bert_labels2tokens, voting_choicer
from modules.analyze_utils.plot_metrics import get_bert_span_report
from modules.analyze_utils.main_metrics import precision_recall_f1
pred_tokens, pred_labels = bert_labels2tokens(dl, preds)
true_tokens, true_labels = bert_labels2tokens(dl, [x.bert_labels for x in dl.dataset])
tokens_report = flat_classification_report(true_labels, pred_labels, digits=4)
print(tokens_report)
results = precision_recall_f1(true_labels, pred_labels)
3. Results
We didn't search best parametres and obtained the following results.
| Model | Data set | Dev F1 tok | Dev F1 span | Test F1 tok | Test F1 span |
|---|---|---|---|---|---|
| OURS | |||||
| M-BERTCRF-IO | FactRuEval | - | - | 0.8543 | 0.8409 |
| M-BERTNCRF-IO | FactRuEval | - | - | 0.8637 | 0.8516 |
| M-BERTBiLSTMCRF-IO | FactRuEval | - | - | 0.8835 | 0.8718 |
| M-BERTBiLSTMNCRF-IO | FactRuEval | - | - | 0.8632 | 0.8510 |
| M-BERTAttnCRF-IO | FactRuEval | - | - | 0.8503 | 0.8346 |
| M-BERTBiLSTMAttnCRF-IO | FactRuEval | - | - | 0.8839 | 0.8716 |
| M-BERTBiLSTMAttnNCRF-IO | FactRuEval | - | - | 0.8807 | 0.8680 |
| M-BERTBiLSTMAttnCRF-fit_BERT-IO | FactRuEval | - | - | 0.8823 | 0.8709 |
| M-BERTBiLSTMAttnNCRF-fit_BERT-IO | FactRuEval | - | - | 0.8583 | 0.8456 |
| - | - | - | - | - | - |
| BERTBiLSTMCRF-IO | CoNLL-2003 | 0.9629 | - | 0.9221 | - |
| B-BERTBiLSTMCRF-IO | CoNLL-2003 | 0.9635 | - | 0.9229 | - |
| B-BERTBiLSTMAttnCRF-IO | CoNLL-2003 | 0.9614 | - | 0.9237 | - |
| B-BERTBiLSTMAttnNCRF-IO | CoNLL-2003 | 0.9631 | - | 0.9249 | - |
| Current SOTA | |||||
| DeepPavlov-RuBERT-NER | FactRuEval | - | - | - | 0.8266 |
| CSE | CoNLL-2003 | - | - | 0.931 | - |
| BERT-LARGE | CoNLL-2003 | 0.966 | - | 0.928 | - |
| BERT-BASE | CoNLL-2003 | 0.964 | - | 0.924 | - |