language-evaluation (Experimental)
Collection of evaluation code for natural language generation.
Note: API could change frequently without notice
Metrics
CocoEvaluator: coco-caption (BLEU1-4, METEOR, ROUGE, CIDEr, SPICE)RougeEvaluator: sentence-level rouge (ROUGE-1, ROUGE-2, ROUGE-L with f-measure)Rouge155Evaluator: summary-level rouge (ROUGE-1, ROUGE-2, ROUGE-L with f-measure)
Requirements
- Java 1.8.0+ (used by coco-caption evaluator)
- Python 3.6+
libxml-parser-perl(used by ROUGE.1.5.5.pl)
Installation and Usage
Install external dependencies (e.g. Java 1.8.0+, libxml-parser-perl):
# Oracle Java
sudo add-apt-repository ppa:webupd8team/java
sudo apt upadte
apt-get install oracle-java8-installer
# libxml-parser-perl
sudo apt install libxml-parser-perlThen run:
pip install git+https://github.com/bckim92/language-evaluation.git
python -c "import language_evaluation; language_evaluation.download('coco')"Python API (or see language_evaluation_test.py):
import language_evaluation
from pprint import PrettyPrinter
pprint = PrettyPrinter().pprint
predicts = ['i am a boy', 'she is a girl']
answers = ['am i a boy ?', 'is she a girl ?']
evaluator = language_evaluation.CocoEvaluator()
results = evaluator.run_evaluation(predicts, answers)
pprint(results)
# {'Bleu_1': 0.9999999997500004,
# 'Bleu_2': 0.5773502690332603,
# 'Bleu_3': 4.3679023223468616e-06,
# 'Bleu_4': 1.4287202142987477e-08,
# 'CIDEr': 3.333333333333333,
# 'METEOR': 0.43354749322305886,
# 'ROUGE_L': 0.75,
# 'SPICE': 0.6666666666666666}
evaluator = language_evaluation.RougeEvaluator(num_parallel_calls=5)
results = evaluator.run_evaluation(predicts, answers)
pprint(results)
# {'rouge1': 1.0,
# 'rouge2': 0.3333333333333333,
# 'rougeL': 0.75}
evaluator = language_evaluation.Rouge155Evaluator(num_parallel_calls=5)
results = evaluator.run_evaluation(predicts, answers)
pprint(results)
# {'rouge1': 1.0,
# 'rouge2': 0.3333333333333333,
# 'rougeL': 0.75}Notes
- TODOs
- Support more metrics (e.g. embedding-based)
- Support command-line interface
- Support full functionality and configuration for rouge
- Implement summary-level rouge scorer in pure python
- Add tests & CI
Related Projects
- tylin/coco-caption
- bckim92/coco-caption-py3
- Maluuba/nlg-eval
- google-research/google-research/rouge
- bheinzerling/pyrouge
License
See LICENSE.md.