HOI-Learning-List

Some recent (2015-now) Human-Object Interaction Learning studies. If you find any errors or problems, please don't hesitate to comment.

A list of Transfomer-based vision works: https://github.com/DirtyHarryLYL/Transformer-in-Vision.

Image Dataset/Benchmark

• SynHOI (arXiv 2023.5), synthetic HOI data [Paper]

• HICO-DET-SG, V-COCO-SG (new splits of HICO-DET and V-COCO) [Paper], [Code]

• Bongard-HOI [Paper], [Code]

• SWiG-HOI, [Paper], [Website]

• New Metric: mPD, [Paper], [Code]

• DIABOLO [Paper], [Website]

• HOI-COCO (CVPR2021) [Website]

• PaStaNet-HOI (TPAMI2021) [Benchmark]

• HAKE (CVPR2020) [YouTube] [bilibili] [Website] [Paper] [HAKE-Action-Torch] [HAKE-Action-TF]

• Ambiguous-HOI (CVPR2020) [Website] [Paper]

• HICO-DET (WACV2018) [Website] [Paper]

• HCVRD (AAAI2018) [Website] [Paper]

• V-COCO (May 2015) [Website] [Paper]

• HICO (ICCV2015) [Website] [Paper]

• OpenImage [Website] [Paper]

• PIC [Website]

More...

Video HOI Datasets

• MPHOI-72 RGB-D, [Paper], [Project]

• VidHOI [Paper], [Project]

• AVA [Website], HOIs (human-object, human-human), and pose (body motion) actions

• Action Genome [Website], action verbs and spatial relationships

• CAD120 [Paper], [Website]

• Sth-else [Paper], [Website]

3D HOI Datasets

• BEHAVE, [Paper], [Project]

Suevry

• Human object interaction detection: Design and survey (Image and Vision Computing 2022), [Paper]

Method

HOI Image Generation

• Exploiting Relationship for Complex-scene Image Generation (arXiv 2021.04) [Paper]

• Specifying Object Attributes and Relations in Interactive Scene Generation (arXiv 2019.11) [Paper]

HOI Recognition: Image-based, to recognize all the HOIs in one image.

• OpenTAP (ECCV'22) [Paper], [Code]

• RelViT (ICLR'22) [Paper], [Code]

• DEFR (arXiv 2021.12) [Paper]

• Interaction Compass (ICCV 2021) [Paper]

• DEFR-CLIP (arXiv 2021.07) [Paper]

• PaStaNet: Toward Human Activity Knowledge Engine (CVPR2020) [Code] [Data] [Paper] [YouTube] [bilibili]

• Pairwise (ECCV2018) [Paper]

• Attentional Pooling for Action Recognition (NIPS2017) [Code] [Paper]

• Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering (ECCV2016) [Code] [Paper]

• Contextual Action Recognition with R*CNN (ICCV2015) [Code] [Paper]

• HOCNN (ICCV2015) [Code] [Paper]

• SGAP-Net (AAAI2020) [Paper]

More...

Unseen or zero-shot learning (image-level recognition).

• ICompass (ICCV2021) [Paper], [Code]

• Compositional Learning for Human Object Interaction (ECCV2018) [Paper]

• Zero-Shot Human-Object Interaction Recognition via Affordance Graphs (Sep. 2020) [Paper]

More...

HOI Detection: Instance-based, to detect the human-object pairs and classify the interactions.

• PSN (arXiv 2023), [Paper]

• SOV-STG (arXiv 2023), [Paper], [Code]

• Shikra (arXiv 2023), [Paper], [Code]

• HOKEM (arXiv 2023), [Paper]

• SQA (ICASSP 2023), [Paper], [Code]

• OpenCat (CVPR 2023), [Paper]

• DiffHOI (arXiv 2023.5), [Paper]

• ViPLO (CVPR 2023), [Paper], [Code]

• MUREN (CVPR 2023), [Paper], [Project]

• HOICLIP (CVPR 2023), [Paper], [Code]

• CQL (CVPR 2023), [Paper], [Code]

• UniVRD (arXiv 2023), [Paper]

• SKGHOI (arXiv 2023), [Paper]

• Weakly-HOI-CLIP (ICLR 2023), [Paper], [Code]

• FGAHOI (arXiv 2023), [Paper], [Code]

• PR-Net (arXiv 2023), [Paper]

• PQNet (MMAsia 2022), [Paper]

• MHOI (TCSVT 2022), [Paper]

• RLIP (NeurIPS 2022), [Paper], [Code]

• PartMap (ECCV2022), [Paper], [Code]

• K-BAN (arXiv 2022), [Paper]

• SGCN4HOI (IEEE SMC 2022), [Paper]

• HQM (ECCV 2022), [Paper], [Code]

• ODM (ECCV 2022), [Paper]

• SDT (arXiv 2022), [Paper]

• DOQ (CVPR 2022), [Paper], [Code]

• STIP (CVPR 2022), [Paper]

• DT (CVPR 2022), [Paper]

• IF (CVPR 2022), [Paper], [Code]

• CPC (CVPR 2022), [Paper], [Code]

• CATN (CVPR 2022), [Paper]

• SSRT (CVPR 2022), [Paper]

• GEN-VLKT (CVPR 2022), [Paper], [Code]

• MSTR (CVPR 2022), [Paper]

• Iwin (ECCV 2022), [Paper]

• RGBM (arXiv 2022.2), [Paper]

• GPV-2 (ECCV 2022), [Paper], [Project]

• OC-Immunity (AAAI 2022) [Paper], [Code]

• OCN (AAAI 2022) [Paper], [Code]

• QAHOI (arXiv 2021) [Paper], [Code]

• PhraseHOI (AAAI 2022) [Paper]

• DEFR (arXiv 2021.12) [Paper]

• UPT (CVPR 2022) [Paper], [Code]

• HRNet (TIP 2021) [Paper]

• ACP++ (TIP 2021) [Paper], [Code]

• SG2HOI (ICCV 2021) [Paper]

• CDN (NeurIPS 2021) [Paper], [Code]

• GTNet (arXiv 2021.8) [Paper], [Code]

• HOI-MO-Net (IVC 2021) [Paper]

• IPGN (TIP 2021.7) [Paper]

• SCG (ICCV 2021, SAG, v2) [Paper], [Code]

• Human Object Interaction Detection using Two-Direction Spatial Enhancement and Exclusive Object Prior (arXiv) [Paper]

• PST (ICCV2021) [Paper]

• RR-Net (arXiv 2021.5) [Paper]

• HOTR (CVPR2021) [Paper], [Code]

• GGNet (CVPR2021) [Paper], [Code]

• ATL (CVPR2021) [Paper], [Code]

• FCL (CVPR2021) [Paper], [Code]

• AS-Net (CVPR2021) [Paper], [Code]

• End-to-End Human Object Interaction Detection with HOI Transformer (CVPR2021), [Paper], [Code]

• QPIC (CVPR2021) [Paper], [Code]

• TIN (TPAMI2021) [Paper], [Code]

• IDN (NeurIPS2020) [Paper] [Code]

• DIRV (AAAI2021) [Paper]

• DecAug (AAAI2021) [Paper]

• OSGNet (IEEE Access) [Paper]

• PFNet (CVM) [Paper]

• UniDet (ECCV2020) [Paper]

• DRG (ECCV2020) [Paper] [Code]

• FCMNet (ECCV2020) [Paper]

• Contextual Heterogeneous Graph Network for Human-Object Interaction Detection (ECCV2020) [Paper]

• PD-Net (ECCV2020) [Paper-1] [Paper-2] [Code]

• VCL (ECCV2020) [Paper] [Code]

• ACP (ECCV2020) [Paper] [Code]

• ConsNet (ACMMM2020) [Paper] [Code], HICO-DET Python API: A general Python toolkit for the HICO-DET dataset, including APIs for data loading & processing, human-object pair IoU & NMS calculation, and standard evaluation. [Code] [Documentation]

• Action-Guided Attention Mining and Relation Reasoning Network for Human-Object Interaction Detection (IJCAI2020) [Paper]

• PaStaNet (CVPR2020) [Code] [Data] [Paper] [YouTube] [bilibili]

• DJ-RN (CVPR2020) [Code] [Paper]

• Cascaded Human-Object Interaction Recognition (CVPR2020) [Code] [Paper]

• PPDM (CVPR2020) [Code] [Paper]

• IP-Net (CVPR2020) [Code] [Paper]

• VSGNet (CVPR2020) [Code] [Paper]

• HOID (CVPR2020) [Code] [Paper]

• Diagnosing Rarity in Human-Object Interaction Detection (CVPRW2020) [Paper]

• MLCNet (ICMR2020) [Paper]

• SIGN (ICME2020) [Paper]

• In-GraphNet (IJCAI-PRICAI 2020) [Paper]

• PMFNet(ICCV2019) [Code] [Paper]

• No-Frills (ICCV2019) [Code] [Paper]

• Analogy (ICCV2019) [Code] [Paper]

• RPNN (ICCV2019) [Paper]

• Deep Contextual Attention for Human-Object Interaction Detection (ICCV2019) [Paper]

• Interactiveness (CVPR2019) [Code] [Paper]

• Turbo (AAAI2019) [Paper]

• GPNN (ECCV2018) [Code] [Paper]

• iCAN (BMVC2018) [Code] [Paper]

• InteractNet (CVPR2018) [Paper]

• Scaling Human-Object Interaction Recognition through Zero-Shot Learning (WACV2018) [Paper]

• HO-RCNN (WACV2018) [Code] [Paper]

• VS-GATs (Mar. 2020) [Paper]

• Classifying All Interacting Pairs in a Single Shot (Jan. 2020) [Paper]

• Novel Human-Object Interaction Detection via Adversarial Domain Generalization (May. 2020) [Paper]

• PMN (Jul. 2020) [Paper] [Code]

• SAG (Dec 2020) [Paper] [Code]

• SABRA (Dec 2020) [Paper]

More...

Unseen or zero/low-shot or weakly-supervised learning (instance-level detection).

• Unal et.al. (arXiv 2023), [Paper]

• RLIP (NeurIPS 2022), [Paper], [Code]

• THID (CVPR 2022), [Paper], [Code]

• EoID (arXiv 2022), [Paper], [Code]

• SCL (arXiv 2022), [Paper], [Code]

• GEN-VLKT (CVPR 2022), [Paper], [Code]

• OC-Immunity (AAAI 2022), [Paper], [Code]

• CHOID (CVPR 2021), [Paper], [Code]

• Align-Former (BMVC 2021), [Paper]

• Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection (ICCV2021) [Paper], [Code]

• DGIG-Net (TOC2021) [Paper]

• ATL (CVPR2021) [Paper], [Code]

• FCL (CVPR2021) [Paper], [Code]

• Detecting Human-Object Interaction with Mixed Supervision (WACV 2021) [Paper]

• ConsNet (ACMMM2020) [Paper] [Code]

• Zero-Shot Human-Object Interaction Recognition via Affordance Graphs (Sep. 2020) [Paper]

• VCL (ECCV2020) [Paper] [Code]

• HOID (CVPR2020) [Code] [Paper]

• Novel Human-Object Interaction Detection via Adversarial Domain Generalization (May. 2020) [Paper]

• Analogy (ICCV2019) [Code] [Paper]

• Functional (AAAI2020) [Paper]

• Scaling Human-Object Interaction Recognition through Zero-Shot Learning (WACV2018) [Paper]

More...

Video HOI methods

• SPDTP (arXiv, Jun 2022), [Paper]

• V-HOI (arXiv, Jun 2022), [Paper]

• Detecting Human-Object Relationships in Videos (ICCV2021) [Paper]

• STIGPN (Aug 2021), [Paper], [Code]

• VidHOI (May 2021), [Paper]

• LIGHTEN (ACMMM2020) [Paper] [Code]

• Generating Videos of Zero-Shot Compositions of Actions and Objects (Jul 2020), HOI GAN, [Paper]

• Grounded Human-Object Interaction Hotspots from Video (ICCV2019) [Code] [Paper]

• GPNN (ECCV2018) [Code] [Paper]

More...

3D HOI Reconstruction/Generation

• HUMANISE (arXiv2022) [Paper], [Project]

• CHAIRS (arXiv2022) [Paper], [Project]

• NeuralDome (arXiv2022) [Paper]

• ARCTIC (arXiv2022) [Paper], [Project]

• IMoS (EUROGRAPHICS2023) [Paper]

• BEHAVE (CVPR2022) [Paper], [Project]

• CHORE (ECCV2022) [Project], [Paper]

• D3D-HOI (arXiv2021) [Paper], [Project]

• DJ-RN (CVPR2020) [Paper], [Code]

• GRAB (ECCV2020) [Paper], [Project]

• PROX (ICCV2019) [Paper], [Project]

Result

PaStaNet-HOI:

Proposed by TIN (TPAMI version, Transferable Interactiveness Network). It is built on HAKE data, includes 110K+ images and 520 HOIs (without the 80 "no_interaction" HOIs of HICO-DET to avoid the incomplete labeling). It has a more severe long-tailed data distribution thus is more difficult.

Detector: COCO pre-trained

HICO-DET:

1) Detector: COCO pre-trained

2) Detector: pre-trained on COCO, fine-tuned on HICO-DET train set (with GT human-object pair boxes) or one-stage detector (point-based, transformer-based)

The finetuned detector would learn to only detect the interactive humans and objects (with interactiveness), thus suppressing many wrong pairings (non-interactive human-object pairs) and boosting the performance.

3) Ground Truth human-object pair boxes (only evaluating HOI recognition)

4) Interactiveness detection (interactive or not + pair box detection):

5) Enhanced with HAKE:

6) Zero-Shot HOI detection:

Unseen action-object combination scenario (UC)

• * indicates large Visual-Language model pretraining, \eg, CLIP.
• For the details of the setting, please refer to corresponding publications. This is not officially published and might miss some publications. Please find the corresponding publications.

Zero-shot* HOI detection without fine-tuning (NF)

• * indicates a formulation that assesses the generalization of a pre-training model to unseen distributions, proposed in RLIP. zero-shot follows the terminology from CLIP.

Unseen object scenario (UO)

Unseen action scenario (UA)

Unseen action scenario (UV), results from EoID

Another setting

7) Few-Shot HOI detection:

1% HICO-Det Data used in fine-tuning

10% HICO-Det Data used in fine-tuning

8) Weakly-supervised HOI detection:

Ambiguous-HOI

Detector: COCO pre-trained

SWiG-HOI

V-COCO: Scenario1

1) Detector: COCO pre-trained or one-stage detector

2) Enhanced with HAKE:

3) Weakly-supervised HOI detection:

HOI-COCO:

based on V-COCO

HICO

1) Default

2) Enhanced with HAKE: