• Stars
    star
    5,573
  • Rank 7,341 (Top 0.2 %)
  • Language
    Python
  • License
    MIT License
  • Created over 2 years ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Official Implementation of OCR-free Document Understanding Transformer (Donut) and Synthetic Document Generator (SynthDoG), ECCV 2022

Donut 🍩 : Document Understanding Transformer

Paper Conference Demo Demo PyPI Downloads

Official Implementation of Donut and SynthDoG | Paper | Slide | Poster

Introduction

Donut 🍩, Document understanding transformer, is a new method of document understanding that utilizes an OCR-free end-to-end Transformer model. Donut does not require off-the-shelf OCR engines/APIs, yet it shows state-of-the-art performances on various visual document understanding tasks, such as visual document classification or information extraction (a.k.a. document parsing). In addition, we present SynthDoG 🐢, Synthetic Document Generator, that helps the model pre-training to be flexible on various languages and domains.

Our academic paper, which describes our method in detail and provides full experimental results and analyses, can be found here:

OCR-free Document Understanding Transformer.
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park. In ECCV 2022.

image

Pre-trained Models and Web Demos

Gradio web demos are available! Demo Demo
image
  • You can run the demo with ./app.py file.
  • Sample images are available at ./misc and more receipt images are available at CORD dataset link.
  • Web demos are available from the links in the following table.
  • Note: We have updated the Google Colab demo (as of June 15, 2023) to ensure its proper working.
Task Sec/Img Score Trained Model
Demo
CORD (Document Parsing) 0.7 /
0.7 /
1.2
91.3 /
91.1 /
90.9
donut-base-finetuned-cord-v2 (1280) /
donut-base-finetuned-cord-v1 (1280) /
donut-base-finetuned-cord-v1-2560
gradio space web demo,
google colab demo (updated at 23.06.15)
Train Ticket (Document Parsing) 0.6 98.7 donut-base-finetuned-zhtrainticket google colab demo (updated at 23.06.15)
RVL-CDIP (Document Classification) 0.75 95.3 donut-base-finetuned-rvlcdip gradio space web demo,
google colab demo (updated at 23.06.15)
DocVQA Task1 (Document VQA) 0.78 67.5 donut-base-finetuned-docvqa gradio space web demo,
google colab demo (updated at 23.06.15)

The links to the pre-trained backbones are here:

  • donut-base: trained with 64 A100 GPUs (~2.5 days), number of layers (encoder: {2,2,14,2}, decoder: 4), input size 2560x1920, swin window size 10, IIT-CDIP (11M) and SynthDoG (English, Chinese, Japanese, Korean, 0.5M x 4).
  • donut-proto: (preliminary model) trained with 8 V100 GPUs (~5 days), number of layers (encoder: {2,2,18,2}, decoder: 4), input size 2048x1536, swin window size 8, and SynthDoG (English, Japanese, Korean, 0.4M x 3).

Please see our paper for more details.

SynthDoG datasets

image

The links to the SynthDoG-generated datasets are here:

To generate synthetic datasets with our SynthDoG, please see ./synthdog/README.md and our paper for details.

Updates

2023-06-15 We have updated all Google Colab demos to ensure its proper working.
2022-11-14 New version 1.0.9 is released (pip install donut-python --upgrade). See 1.0.9 Release Notes.
2022-08-12 Donut 🍩 is also available at huggingface/transformers πŸ€— (contributed by @NielsRogge). donut-python loads the pre-trained weights from the official branch of the model repositories. See 1.0.5 Release Notes.
2022-08-05 A well-executed hands-on tutorial on donut 🍩 is published at Towards Data Science (written by @estaudere).
2022-07-20 First Commit, We release our code, model weights, synthetic data and generator.

Software installation

PyPI Downloads

pip install donut-python

or clone this repository and install the dependencies:

git clone https://github.com/clovaai/donut.git
cd donut/
conda create -n donut_official python=3.7
conda activate donut_official
pip install .

We tested donut-python == 1.0.1 with:

Note: From several reported issues, we have noticed increased challenges in configuring the testing environment for donut-python due to recent updates in key dependency libraries. While we are actively working on a solution, we have updated the Google Colab demo (as of June 15, 2023) to ensure its proper working. For assistance, we encourage you to refer to the following demo links: CORD Colab Demo, Train Ticket Colab Demo, RVL-CDIP Colab Demo, DocVQA Colab Demo.

Getting Started

Data

This repository assumes the following structure of dataset:

> tree dataset_name
dataset_name
β”œβ”€β”€ test
β”‚   β”œβ”€β”€ metadata.jsonl
β”‚   β”œβ”€β”€ {image_path0}
β”‚   β”œβ”€β”€ {image_path1}
β”‚             .
β”‚             .
β”œβ”€β”€ train
β”‚   β”œβ”€β”€ metadata.jsonl
β”‚   β”œβ”€β”€ {image_path0}
β”‚   β”œβ”€β”€ {image_path1}
β”‚             .
β”‚             .
└── validation
    β”œβ”€β”€ metadata.jsonl
    β”œβ”€β”€ {image_path0}
    β”œβ”€β”€ {image_path1}
              .
              .

> cat dataset_name/test/metadata.jsonl
{"file_name": {image_path0}, "ground_truth": "{\"gt_parse\": {ground_truth_parse}, ... {other_metadata_not_used} ... }"}
{"file_name": {image_path1}, "ground_truth": "{\"gt_parse\": {ground_truth_parse}, ... {other_metadata_not_used} ... }"}
     .
     .
  • The structure of metadata.jsonl file is in JSON Lines text format, i.e., .jsonl. Each line consists of
    • file_name : relative path to the image file.
    • ground_truth : string format (json dumped), the dictionary contains either gt_parse or gt_parses. Other fields (metadata) can be added to the dictionary but will not be used.
  • donut interprets all tasks as a JSON prediction problem. As a result, all donut model training share a same pipeline. For training and inference, the only thing to do is preparing gt_parse or gt_parses for the task in format described below.

For Document Classification

The gt_parse follows the format of {"class" : {class_name}}, for example, {"class" : "scientific_report"} or {"class" : "presentation"}.

  • Google colab demo is available here.
  • Gradio web demo is available here.

For Document Information Extraction

The gt_parse is a JSON object that contains full information of the document image, for example, the JSON object for a receipt may look like {"menu" : [{"nm": "ICE BLACKCOFFEE", "cnt": "2", ...}, ...], ...}.

  • More examples are available at CORD dataset.
  • Google colab demo is available here.
  • Gradio web demo is available here.

For Document Visual Question Answering

The gt_parses follows the format of [{"question" : {question_sentence}, "answer" : {answer_candidate_1}}, {"question" : {question_sentence}, "answer" : {answer_candidate_2}}, ...], for example, [{"question" : "what is the model name?", "answer" : "donut"}, {"question" : "what is the model name?", "answer" : "document understanding transformer"}].

  • DocVQA Task1 has multiple answers, hence gt_parses should be a list of dictionary that contains a pair of question and answer.
  • Google colab demo is available here.
  • Gradio web demo is available here.

For (Pseudo) Text Reading Task

The gt_parse looks like {"text_sequence" : "word1 word2 word3 ... "}

  • This task is also a pre-training task of Donut model.
  • You can use our SynthDoG 🐢 to generate synthetic images for the text reading task with proper gt_parse. See ./synthdog/README.md for details.

Training

This is the configuration of Donut model training on CORD dataset used in our experiment. We ran this with a single NVIDIA A100 GPU.

python train.py --config config/train_cord.yaml \
                --pretrained_model_name_or_path "naver-clova-ix/donut-base" \
                --dataset_name_or_paths '["naver-clova-ix/cord-v2"]' \
                --exp_version "test_experiment"    
  .
  .                                                                                                                                                                                                                                         
Prediction: <s_menu><s_nm>Lemon Tea (L)</s_nm><s_cnt>1</s_cnt><s_price>25.000</s_price></s_menu><s_total><s_total_price>25.000</s_total_price><s_cashprice>30.000</s_cashprice><s_changeprice>5.000</s_changeprice></s_total>
Answer: <s_menu><s_nm>Lemon Tea (L)</s_nm><s_cnt>1</s_cnt><s_price>25.000</s_price></s_menu><s_total><s_total_price>25.000</s_total_price><s_cashprice>30.000</s_cashprice><s_changeprice>5.000</s_changeprice></s_total>
Normed ED: 0.0
Prediction: <s_menu><s_nm>Hulk Topper Package</s_nm><s_cnt>1</s_cnt><s_price>100.000</s_price></s_menu><s_total><s_total_price>100.000</s_total_price><s_cashprice>100.000</s_cashprice><s_changeprice>0</s_changeprice></s_total>
Answer: <s_menu><s_nm>Hulk Topper Package</s_nm><s_cnt>1</s_cnt><s_price>100.000</s_price></s_menu><s_total><s_total_price>100.000</s_total_price><s_cashprice>100.000</s_cashprice><s_changeprice>0</s_changeprice></s_total>
Normed ED: 0.0
Prediction: <s_menu><s_nm>Giant Squid</s_nm><s_cnt>x 1</s_cnt><s_price>Rp. 39.000</s_price><s_sub><s_nm>C.Finishing - Cut</s_nm><s_price>Rp. 0</s_price><sep/><s_nm>B.Spicy Level - Extreme Hot Rp. 0</s_price></s_sub><sep/><s_nm>A.Flavour - Salt & Pepper</s_nm><s_price>Rp. 0</s_price></s_sub></s_menu><s_sub_total><s_subtotal_price>Rp. 39.000</s_subtotal_price></s_sub_total><s_total><s_total_price>Rp. 39.000</s_total_price><s_cashprice>Rp. 50.000</s_cashprice><s_changeprice>Rp. 11.000</s_changeprice></s_total>
Answer: <s_menu><s_nm>Giant Squid</s_nm><s_cnt>x1</s_cnt><s_price>Rp. 39.000</s_price><s_sub><s_nm>C.Finishing - Cut</s_nm><s_price>Rp. 0</s_price><sep/><s_nm>B.Spicy Level - Extreme Hot</s_nm><s_price>Rp. 0</s_price><sep/><s_nm>A.Flavour- Salt & Pepper</s_nm><s_price>Rp. 0</s_price></s_sub></s_menu><s_sub_total><s_subtotal_price>Rp. 39.000</s_subtotal_price></s_sub_total><s_total><s_total_price>Rp. 39.000</s_total_price><s_cashprice>Rp. 50.000</s_cashprice><s_changeprice>Rp. 11.000</s_changeprice></s_total>
Normed ED: 0.039603960396039604                                                                                                                                  
Epoch 29: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 200/200 [01:49<00:00,  1.82it/s, loss=0.00327, exp_name=train_cord, exp_version=test_experiment]

Some important arguments:

  • --config : config file path for model training.
  • --pretrained_model_name_or_path : string format, model name in Hugging Face modelhub or local path.
  • --dataset_name_or_paths : string format (json dumped), list of dataset names in Hugging Face datasets or local paths.
  • --result_path : file path to save model outputs/artifacts.
  • --exp_version : used for experiment versioning. The output files are saved at {result_path}/{exp_version}/*

Test

With the trained model, test images and ground truth parses, you can get inference results and accuracy scores.

python test.py --dataset_name_or_path naver-clova-ix/cord-v2 --pretrained_model_name_or_path ./result/train_cord/test_experiment --save_path ./result/output.json
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 100/100 [00:35<00:00,  2.80it/s]
Total number of samples: 100, Tree Edit Distance (TED) based accuracy score: 0.9129639764131697, F1 accuracy score: 0.8406020841373987

Some important arguments:

  • --dataset_name_or_path : string format, the target dataset name in Hugging Face datasets or local path.
  • --pretrained_model_name_or_path : string format, the model name in Hugging Face modelhub or local path.
  • --save_path: file path to save predictions and scores.

How to Cite

If you find this work useful to you, please cite:

@inproceedings{kim2022donut,
  title     = {OCR-Free Document Understanding Transformer},
  author    = {Kim, Geewook and Hong, Teakgyu and Yim, Moonbin and Nam, JeongYeon and Park, Jinyoung and Yim, Jinyeong and Hwang, Wonseok and Yun, Sangdoo and Han, Dongyoon and Park, Seunghyun},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2022}
}

License

MIT license

Copyright (c) 2022-present NAVER Corp.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

More Repositories

1

deep-text-recognition-benchmark

Text recognition (optical character recognition) with deep learning methods, ICCV 2019
Jupyter Notebook
3,692
star
2

stargan-v2

StarGAN v2 - Official PyTorch Implementation (CVPR 2020)
Python
3,478
star
3

CRAFT-pytorch

Official implementation of Character Region Awareness for Text Detection (CRAFT)
Python
3,024
star
4

CutMix-PyTorch

Official Pytorch implementation of CutMix regularizer
Python
1,211
star
5

voxceleb_trainer

In defence of metric learning for speaker recognition
Python
1,029
star
6

WCT2

Software that can perform photorealistic style transfer without the need of any post-processing steps.
Python
869
star
7

synthtiger

Official Implementation of SynthTIGER (Synthetic Text Image Generator), ICDAR 2021
Python
482
star
8

tunit

Rethinking the Truly Unsupervised Image-to-Image Translation - Official PyTorch Implementation (ICCV 2021)
Python
452
star
9

rexnet

Official Pytorch implementation of ReXNet (Rank eXpansion Network) with pretrained models
Python
451
star
10

AdamP

AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (ICLR 2021)
Python
411
star
11

overhaul-distillation

Official PyTorch implementation of "A Comprehensive Overhaul of Feature Distillation" (ICCV 2019)
Python
409
star
12

cord

CORD: A Consolidated Receipt Dataset for Post-OCR Parsing
384
star
13

cutblur

Rethinking Data Augmentation for Image Super-resolution (CVPR 2020)
Jupyter Notebook
379
star
14

wsolevaluation

Evaluating Weakly Supervised Object Localization Methods Right (CVPR 2020)
Python
331
star
15

assembled-cnn

Tensorflow implementation of "Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network"
Python
329
star
16

generative-evaluation-prdc

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.
Python
239
star
17

ext_portrait_segmentation

Python
238
star
18

ClovaCall

ClovaCall dataset and Pytorch LAS baseline code (Interspeech 2020)
Python
218
star
19

fewshot-font-generation

The unified repository for few-shot font generation methods. This repository includes FUNIT (ICCV'19), DM-Font (ECCV'20), LF-Font (AAAI'21) and MX-Font (ICCV'21).
Python
203
star
20

stargan-v2-tensorflow

StarGAN v2 - Official Tensorflow Implementation (CVPR 2020)
Python
187
star
21

EXTD_Pytorch

Official EXTD Pytorch code
Python
187
star
22

CLEval

CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks
Python
185
star
23

TedEval

TedEval: A Fair Evaluation Metric for Scene Text Detectors
Python
176
star
24

rebias

Official Pytorch implementation of ReBias (Learning De-biased Representations with Biased Representations), ICML 2020
Python
168
star
25

aasist

Official PyTorch implementation of "AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks"
Python
167
star
26

SATRN

Official Tensorflow Implementation of SATRN (CVPR Workshop WTDDLE 2020)
Python
162
star
27

lffont

Official PyTorch implementation of LF-Font (Few-shot Font Generation with Localized Style Representations and Factorization) AAAI 2021
Python
156
star
28

bros

Python
156
star
29

som-dst

SOM-DST: Efficient Dialogue State Tracking by Selectively Overwriting Memory (ACL 2020)
Python
150
star
30

mxfont

Official PyTorch implementation of MX-Font (Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts) ICCV 2021
Python
148
star
31

dmfont

Official PyTorch implementation of DM-Font (ECCV 2020)
Python
133
star
32

rainbow-memory

Official pytorch implementation of Rainbow Memory (CVPR 2021)
Python
119
star
33

FocusSeq2Seq

[EMNLP 2019] Mixture Content Selection for Diverse Sequence Generation (Question Generation / Abstractive Summarization)
Python
113
star
34

attention-feature-distillation

Official implementation for (Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching, AAAI-2021)
Python
111
star
35

frostnet

FrostNet: Towards Quantization-Aware Network Architecture Search
Python
106
star
36

webvicob

Official Implementation of Web-based Visual Corpus Builder (Webvicob), ICDAR 2023
Python
101
star
37

length-adaptive-transformer

Official Pytorch Implementation of Length-Adaptive Transformer (ACL 2021)
Python
99
star
38

spade

Python
81
star
39

embedding-expansion

Official MXNet implementation of "Embedding Expansion: Augmentation in Embedding Space for Deep Metric Learning" (CVPR 2020)
Python
76
star
40

symmetrical-synthesis

Official Tensorflow implementation of "Symmetrical Synthesis for Deep Metric Learning" (AAAI 2020)
Python
71
star
41

units

Python
70
star
42

lookwhostalking

Look Who’s Talking: Active Speaker Detection in the Wild
Python
70
star
43

subword-qac

Subword Language Model for Query Auto-Completion
Python
67
star
44

ssmix

Official PyTorch Implementation of SSMix (Findings of ACL 2021)
Python
60
star
45

SSUL

[NeurIPS 2021] SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning
Python
59
star
46

BESTIE

[CVPR 2022] Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement
Python
55
star
47

PointWSSIS

[CVPR2023] The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation
Python
55
star
48

c3_sinet

Python
52
star
49

puridiver

Official PyTorch Implementation of PuriDivER CVPR 2022.
Python
45
star
50

EResFD

Lightweight Face Detector from CLOVA
Python
44
star
51

minimal-rnr-qa

[NAACL 2021] Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering
Python
36
star
52

ECLIPSE

(CVPR 2024) ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning
Python
34
star
53

group-transformer

Official code for Group-Transformer (Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model, COLING-2020).
Python
25
star
54

ProxyDet

Official implementation of the paper "ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection"
Python
22
star
55

GeNAS

Official pytorch implementation for GeNAS: Neural Architecture Search with Better Generalization
Python
15
star
56

meev

Python
12
star
57

pkm-transformers

Official implementation of PKM-augmented language models (Findings of EMNLP 2020)
9
star
58

DCutMix

DCutMix official repo
Python
8
star
59

TVQ-VAE

Official pytorch implementation for TVQ-VAE
Jupyter Notebook
8
star
60

textual-kd-slu

Official Implementation of Textual KD SLU (ICASSP 2021)
Python
6
star
61

vat-d

Official Implementation of VAT-D
Python
5
star
62

ActiveASR_AugCR

Repositoty for Efficient Active Learning for Automatic Speech Recognition via Augmented Consistency Regularization
3
star
63

WSSS-BED

Rethinking Saliency-Guided Weakly-Supervised Semantic Segmentation
Python
1
star