danfenghong/IEEE_TGRS_SpectralFormer danfenghong/IEEE_TGRS_SpectralFormer

  • Stars
    star
    213
  • Rank 185,410 (Top 4 %)
  • Language
    Python
  • Created almost 3 years ago
  • Updated over 2 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
danfenghong/IEEE_TGRS_SpectralFormer
github

danfenghong

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot. Spectralformer: Rethinking hyperspectral image classification with transformers, IEEE Transactions on Geoscience and Remote Sensing (TGRS), 2021

Spectralformer: Rethinking hyperspectral image classification with transformers

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot

The code in this toolbox implements the "Spectralformer: Rethinking hyperspectral image classification with transformers". More specifically, it is detailed as follow.

alt text

Citation

Please kindly cite the papers if this code is useful and helpful for your research.

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot. Spectralformer: Rethinking hyperspectral image classification with transformers, IEEE Transactions on Geoscience and Remote Sensing (TGRS), 2022, vol. 60, pp. 1-15, Art no. 5518615, DOI: 10.1109/TGRS.2021.3130716.

@article{hong2022spectralformer,
  title={Spectralformer: Rethinking hyperspectral image classification with transformers},
  author={Hong, Danfeng and Han, Zhu and Yao, Jing and Gao, Lianru and Zhang, Bing and Plaza, Antonio and Chanussot, Jocelyn},
  journal={IEEE Trans. Geosci. Remote Sens.},
  year={2022},
  volume={60},
  pages={1-15},
  note = {DOI: 10.1109/TGRS.2021.3130716}
}

System-specific notes

The data were generated by Matlab R2016a or higher versions, and the codes of networks were tested using PyTorch 1.6 version (CUDA 10.1) in Python 3.7 on Ubuntu system.

How to use it?

This toolbox consists of two proposed modules, i.e., group-wise spectral embedding (GSE: by setting band_patches larger than 1) and cross-layer adaptive fusion (CAF: by setting mode to CAF), that can be plug-and-played into both pixel-wise and patch-wise hyperspectral image classification. For more details, please refer to the paper.

Here an example experiment is given by using Indian Pines hyperspectral data. Directly run demo.py functions with different network parameter settings to produce the results. Please note that due to the randomness of the parameter initialization, the experimental results might have slightly different from those reported in the paper.

❗ Please kindly be careful on assigning arguments such as epoches and weight_decay. For re-performing of the last three columns in TABLE VIII of our paper, i.e., the ViT, Pixel-wise and Patch-wise SpectralFormer on the Indian Pines dataset, you can either re-train by following:

ViT: python demo.py --dataset='Indian' --epoch=1400 --patches=1 --band_patches=1 --mode='ViT' --weight_decay=0

Pixel-wise SpectralFormer: python demo.py --dataset='Indian' --epoches=290 --patches=1 --band_patches=3 --mode='CAF' --weight_decay=0

Patch-wise SpectralFormer: python demo.py --dataset='Indian' --epoches=300 --patches=7 --band_patches=3 --mode='CAF' --weight_decay=5e-3

or directly test by loading the network parameters in the log file we have tuned locally,

ViT: python demo.py --dataset='Indian' --flag_test=test --patches=1 --band_patches=1 --mode='ViT'

Pixel-wise SpectralFormer: python demo.py --dataset='Indian' --flag_test=test --patches=1 --band_patches=3 --mode='CAF'

Patch-wise SpectralFormer: python demo.py --dataset='Indian' --flag_test=test --patches=7 --band_patches=3 --mode='CAF'

For the datasets of Pavia University and Houston, you can download the data we use from the following links of google drive or baiduyun:

Google drive: https://drive.google.com/drive/folders/1nRphkwDZ74p-Al_O_X3feR24aRyEaJDY?usp=sharing

Baiduyun: https://pan.baidu.com/s/1ZRmPQYahqvbkMoH_B6v1xw (access code: 29qw)

Also, the suggestions for the re-training are,

Pavia University:

ViT: python demo.py --dataset='Pavia' --epoch=1000 --patches=1 --band_patches=1 --mode='ViT' --weight_decay=0

Pixel-wise SpectralFormer: python demo.py --dataset='Pavia' --epoches=500 --patches=1 --band_patches=3 --mode='CAF' --weight_decay=5e-3

Patch-wise SpectralFormer: python demo.py --dataset='Pavia' --epoches=480 --patches=7 --band_patches=7 --mode='CAF' --weight_decay=5e-3

Houston:

ViT: python demo.py --dataset='Houston' --epoch=900 --patches=1 --band_patches=1 --mode='ViT' --weight_decay=0

Pixel-wise SpectralFormer: python demo.py --dataset='Houston' --epoches=520 --patches=1 --band_patches=3 --mode='CAF' --weight_decay=5e-3

Patch-wise SpectralFormer: python demo.py --dataset='Houston' --epoches=600 --patches=7 --band_patches=3 --mode='CAF' --weight_decay=5e-3

The suggestions for the testing are,

Pavia:

ViT: python demo.py --dataset='Pavia' --flag_test=test --patches=1 --band_patches=1 --mode='ViT'

Pixel-wise SpectralFormer: python demo.py --dataset='Pavia' --flag_test=test --patches=1 --band_patches=3 --mode='CAF'

Patch-wise SpectralFormer: python demo.py --dataset='Pavia' --flag_test=test --patches=7 --band_patches=7 --mode='CAF'

Houston:

ViT: python demo.py --dataset='Houston' --flag_test=test --patches=1 --band_patches=1 --mode='ViT'

Pixel-wise SpectralFormer: python demo.py --dataset='Houston' --flag_test=test --patches=1 --band_patches=3 --mode='CAF'

Patch-wise SpectralFormer: python demo.py --dataset='Houston' --flag_test=test --patches=7 --band_patches=3 --mode='CAF'

If you encounter memory error when testing the patch-wise SpectralFormer on the Pavia and Houston datasets, commenting trainloader and some other unnecessary part of code or separting the whole image into several subimages both work.

If you want to run the code in your own data, you can accordingly change the input (e.g., data, labels) and tune the parameters.

If you encounter the bugs while using this code, please do not hesitate to contact us.

Licensing

Copyright (C) 2021 Danfeng Hong

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 of the License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program.

Contact Information:

Danfeng Hong: [email protected]
Danfeng Hong is with the Key Laboratory of Digital Earth Science, Aerospace Information Research Institute, Chinese Academy of Sciences, 100094 Beijing, China.

Zhu Han: [email protected]
Zhu Han is with the Key Laboratory of Digital Earth Science, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China, and also with the College of Resources and Environment, University of Chinese Academy of Sciences, Beijing 100049, China.

More Repositories

1

IEEE_TGRS_GCN

Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot. Graph Convolutional Networks for Hyperspectral Image Classification, IEEE Trans. Geosci. Remote Sens., 2021, 59(7): 5966-5978.
Python
231
star
2

IEEE_TPAMI_SpectralGPT

Hong, D., Zhang, B., Li, X., Li, Y., Li, C., Yao, J., Yokoya, N., Li, H., Ghamisi, P., Jia, X., Plaza, A. and Gamba, P., Benediktsson, J., Chanussot, J. (2024). SpectralGPT: Spectral remote sensing foundation model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. DOI:10.1109/TPAMI.2024.3362475.
Python
160
star
3

IEEE_TIP_UIU-Net

Xin Wu, Danfeng Hong, Jocelyn Chanussot. UIU-Net: U-Net in U-Net for Infrared Small Object Detection, IEEE Trans. Image. Process., 2022, 32: 364-376.
Python
129
star
4

IEEE_TGRS_MDL-RS

Danfeng Hong, Lianru Gao, Naoto Yokoya, Jing Yao, Jocelyn Chanussot, Qian Du, Bing Zhang. More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification, IEEE TGRS, 2021, 59(5): 4340-4354.
Python
101
star
5

ISPRS_S2FL

Danfeng Hong, JIngliang Hu, Jing Yao, Jocelyn Chanussot, Xiao Xiang Zhu. Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model, ISPRS JP&RS, 2021.
MATLAB
97
star
6

ECCV2020_CUCaNet

Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution, ECCV, 2020. (PyTorch)
Python
90
star
7

IEEE_TNNLS_EGU-Net

Danfeng Hong, Lianru Gao, Jing Yao, Naoto Yokoya, Jocelyn Chanussot, Uta Heiden, Bing Zhang. Endmember-Guided Unmixing Network (EGU-Net): A General Deep Learning Framework for Self-Supervised Hyperspectral Unmixing, IEEE TNNLS, 2021.
Python
55
star
8

SpectralMamba

Yao, Jing, Danfeng Hong, Chenyu Li, and Jocelyn Chanussot. "SpectralMamba: Efficient Mamba for Hyperspectral Image Classification." arXiv preprint arXiv:2404.08489, 2024.
51
star
9

RSE_Cross-city

Danfeng Hong, Bing Zhang, Hao Li, Yuxuan Li, Jing Yao, Chenyu Li, Martin Werner, Jocelyn Chanussot, Alexander Zipf, Xiao Xiang Zhu. Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adapation Networks, Remote Sensing of Enviroment, 2023.
Python
46
star
10

IEEE_GRSL_EndNet

Danfeng Hong, Lianru Gao, Renlong Hang, Bing Zhang, Jocelyn Chanussot. Deep Encoder-Decoder Networks for Classification of Hyperspectral and LiDAR Data, IEEE GRSL, 2020.
Python
45
star
11

IEEE_TGRS_CCR-Net

Xin Wu, Danfeng Hong, Jocelyn Chanussot. Convolutional Neural Networks for Multimodal Remote Sensing Data Classification, IEEE Transactions on Geoscience and Remote Sensing, 2021.
Python
43
star
12

ISPRS_HD-Net

Yuxuan Li, Danfeng Hong, Chenyu Li, Jing Yao, Jocelyn Chanussot. HD-Net: High-resolution decoupled network for building footprint extraction via deeply supervised body and boundary decomposition, ISPRS Journal of Photogrammetry and Remote Sensing, 2024.
Python
23
star
13

IEEE_TGRS_J-SLoL

Lianru Gao, Danfeng Hong, Jing Yao, Bing Zhang, Paolo Gamba, Jocelyn Chanussot. Spectral Superresolution of Multispectral Imagery with Joint Sparse and Low-Rank Learning, IEEE TGRS, 2020.
MATLAB
20
star
14

IEEE_TGRS_SeCoDe

Matlab code of the TGRS paper entitled "Sparsity-Enhanced Convolutional Decomposition: A Novel Tensor-Based Paradigm for Blind Hyperspectral Unmixing".
MATLAB
18
star
15

Information_Fusion_CasFormer

Li, Chenyu, Bing Zhang, Danfeng Hong, Jun Zhou, Gemine Vivone, Shutao Li, and Jocelyn Chanussot. "CasFormer: Cascaded transformers for fusion-aware computational hyperspectral imaging." Information Fusion, 2024, 102408.
Python
16
star
16

Outcome-of-the-2023-IEEE-WHISPERS-C2Seg-Challenge

Python
16
star
17

ALMM_TIP

Danfeng Hong, Naoto Yokoya, Jocelyn Chanussot, Xiaoxiang Zhu. An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing, IEEE TIP, 2019.
MATLAB
13
star
18

ISPRS_LeMA

Danfeng Hong, Naoto Yokoya, Nan Ge, Jocelyn Chanussot, Xiaoxiang Zhu. Learnable Manifold Alignment (LeMA): A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification, ISPRS JP&RS, 2019.
MATLAB
10
star
19

ECCV2018_J-Play

Danfeng Hong, Naoto Yokoya, Jian Xu, Xiaoxiang Zhu. Joint & Progressive Learning from High-Dimensional Data for Multi-Label Classification, ECCV, 2018.
MATLAB
9
star
20

IEEE_JSTSP_SULoRA

SULoRA: Danfeng Hong, Xiaoxiang Zhu. Subspace Unmixing with Low-Rank Attribute Embedding for Hyperspectral Data Analysis, IEEE JSTSP, 2018.
MATLAB
9
star
21

IEEE_JSTARS_RLML

Danfeng Hong, Naoto Yokoya, Xiaoxiang Zhu. Learning a Robust Local Manifold Representation for Hyperspectral Dimensionality Reduction, IEEE JSTARS, 2017.
MATLAB
7
star
22

IEEE_GRSL_LCSC

Danfeng Hong, Xin Wu, Lianru Gao, Bing Zhang, Jocelyn Chanussot. Learning Locality-Constrained Sparse Coding for Spectral Enhancement of Multispectral Imagery, IEEE GRSL, 2020.
7
star
23

IEEE_TGRS_LRR-Net

7
star
24

IEEE_TGRS_CoSpace

Danfeng Hong, Naoto Yokoya, Jocelyn Chanussot, Xiaoxiang Zhu. CoSpace: Common Subspace Learning From Hyperspectral-Multispectral Correspondences, IEEE TGRS, 2019.
MATLAB
6
star
25

IEEE_TGRS_IAPs

Danfeng Hong, Xin Wu, Pedram Ghamisi, Jocelyn Chanussot, Naoto Yokoya, Xiaoxiang Zhu. Invariant Attribute Profiles: A Spatial-Frequency Joint Feature Extractor for Hyperspectral Image Classification, IEEE TGRS, 2020, 58(6): 3791-3808.
C++
6
star