Awesome Data Fusion

List of reference,algorithms, applications in RS data fusions (contribution are welcome)

Table of Contents

• Overview of Data Fusion
  • Trends
  • Basic Concept
  • Sensors
• Focus, Taxonomy
  • Research Focus
    • 1. Resolution enhancement:
    • 2. Feature/object detection:
    • 3. Data Alignment.
  • Taxonomy
• Current Challengings
  • 1. Issues Related to Radiation
  • 2. Issues related to Topology
  • 3. Issues Related to Data
  • 4. Issues Related to Method and Process
  • 5. Issues related to Physics
• Algorithms
  • 1.Spatial Resolution Enhancement Algorithms
  • 2. Spatiotemporal Fusion by Traditional Methods
  • 3. Spatio-Temporal Fusion by Neural Network
  • 4. SAR to Optical
  • 5.Registration
  • 6. Super Resolution Enhancement
  • 7. Classification
  • 8.Change Detction
• Quality Assessment
  • 1.Assessment Index With Reference Images.
  • 2.Assessment Index Without Reference Images.
• Community
  • IEEE GRSS data fusion contest
  • Discussion
• Special Issue "Multisensor Data Fusion in Remote Sensing, 2018
  • Software and Open Source Tools
    • Cloud Services
    • Useful Library
  • Database
• TERMS

Overview of Data Fusion

Trends

There are popular topics and review literatures in different periods.

1992~2000: Data fusion, spatial resolution

• Data Fusion Subpanel of the Joint Directors of Laboratories (JDL. 1991)

• Fusion of satellite images of different spatial resolutions(Linas et al.1997)

• Multisensor Image Fusion in Remote Sensing: Concepts, Methods and Applications(Pohl et al.1997)

• Some terms of reference in data fusion(Wald,1999)

2001~2010: classification, unsupervision, change detection, model, multi-resolution, quality, feature extraction Wavelet transform

• #Handbook of Multisensor Data Fusion(Hall et al.2001)
• Mathematical Techniques in Multisensor Data Fusion (Hall et al. 2004)
• Multi-Sensor Data Fusion: An Introduction (Mitchell 2007)
• Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics(Thomas et al.2008)
• Decision Fusion for the Classification of Hyperspectral Data: Outcome of the 2008 GRS-S Data Fusion Contest (Licciardi et al.2009)
• Advances in Multi-Sensor Data Fusion: Algorithms and Applications (Dong et al.2009)
• Multi-source remote sensing data fusion: status and trends (Zhang et al.2010)

2011~2021:hyperspectum, deep learning, Heterogeneous, fusion framework, sparse expressions.

• A review of remote sensing image fusion methods(Ghassemian et al.2016)
• Data Fusion and Remote Sensing: An ever-growing relationship(Schmitt et al.2016)
• Spatiotemporal Fusion of Multisource Remote Sensing Data: Literature Survey, Taxonomy, Principles, Applications, and Future Directions(Zhu et al.2018)
• Spatiotemporal Image Fusion in Remote Sensing (Belgiu et al.2019)
• Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources(Zhu et al.2019)
• Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities

Basic Concept

DOD: Data fusion is a multilevel, multifaceted process dealing with the automatic detection, association, correlation, estimation, and combination of data and information from multiple source. Community: Data fusion is a formal framework in which are expressed means and tools for the alliance of data originating from different sources. It aims at obtaining information of greater quality; the exact definition of ‘greater quality’ will depend upon the application

Sensors

Focus, Taxonomy

Research Focus

The application of data fusion in remote sensing is mainly divided in two scenarios:

1. Resolution enhancement:

It aims to provide higher resolution by combining multi modal data. The results are more like to be transition data, base map, or continuous time series for applications need high temporal and spatial resolution.

They are mostly:

• Based on pixel level.
• Focus on Robustness, temporal continuous

For examples:

Typical applications: Plant detection, weather detection, ecology, change detection, land cover, etc.

2. Feature/object detection:

It aims to improve accuracy and precision of specific task, based on complementary property of heterogeneous information in multimodality. Often, temporal continuity can be sacrificed.

They are mostly:

• End to End
• based on pixel level or decision level.
• Focus on accuracy of task (instead of fused images)

For examples:

Typical Applications: change detection, land change coverage, land classification, disaster monitoring, building recognition, vehicle recognition, etc.

3. Data Alignment.

It aims to find and adjust the unlinear connections between different sensors, It could be divided into matching and co-registration, Furthermore, the problems need to be resolved could be regarded as topological and radiation issues. They are mostly:

• Necessary for all multi-modal applications
• part of preprocess
• can be selectively ignored or modified(such as adaptive registration)

For examples:

Taxonomy

Depending on the main problem solved by the model/algorithm, the development of data fusion can be divided into many areas

• According to process levels: Pixel, Attribute, Decision
• According to Principle: Weighted function based, Decomposition based, Learning based (including sparse dictionary, DL), Bayesian based, Hybrid based
• According to modal: Homogeneous fusion, Heterogeneous fusion (NIR,-VIS, Optical-SAR, Optical-Thermal, hyper-Multi etc.,), Remote sensing site fusion, Remote sensing non-observation fusion (Data assimilation)
• According to mechanisms: Competitive integration, Complementary integration(like temporal and spatial), Cooperative integration(like 3D reconstruction)
• According to Dimension: Spatial Dimension Enhancement, Spectral Dimension Enhancement, Time dimension enhancement, End to End.
• According to Applications: spatial resolution enhancement, Matching and co-registration of multisource data, Change detection, Object recognition, Agriculture, Ecology etc.,
• According to procedure: Matching, Co-registration, Process, Quality Assessment.

Current Challengings

In almost every small direction (as shown in the previous subsection). There are a number of issues so authors only list some most important challenges to demonstrate.

As mentioned before, spatial enhancement and fusion of complementary information are main scientific focus. Apart from these, there are:

1. Issues Related to Radiation

2. Issues related to Topology

Figure1. Error due to deviation in DEM, Relationship between error and Angle

3. Issues Related to Data

Noise:

Sample:

4. Issues Related to Method and Process

5. Issues related to Physics

Algorithms

There are currently more than 200 spatio-temporal models, so only part of baseline models or popular papers are included.

1.Spatial Resolution Enhancement Algorithms

Multisensor image fusion for spatial resolution enhancement such as pan-sharpening, multi/hyperspectral image fusion, and downscaling of multiresolution imagery

2. Spatiotemporal Fusion by Traditional Methods

(Normally Prediction by fusing two high temporal and high spatial resolution sensors with correction,Multisensor and multimodal data fusion using a variety of sensors such as optical imaging, SAR, and LiDAR)

3. Spatio-Temporal Fusion by Neural Network

4. SAR to Optical

5.Registration

6. Super Resolution Enhancement

7. Classification

8.Change Detction

Quality Assessment

1.Assessment Index With Reference Images.

2.Assessment Index Without Reference Images.

Community

IEEE GRSS data fusion contest(Link)

Year: 2020 Title: Global Land Cover Mapping with Weak Supervision Data: MSS, SENTINEL MSS

Track1:

Year: 2019 Title: Reconstruct both a 3D geometric model and a segmentation of semantic classes for an urban scene Data: WorldView-3, VIR, NIR, LiDAR

Track 1: Single-view semantic 3D

Track 3: Multi-view semantic stereo

Year: 2018 Title: urban land use and land cover classification Data: Hyper, Multi, Lidar, RGB(5cm) updating

2017,2016,2015 are updating!

Discussion

• Special Issue "Multisensor Data Fusion in Remote Sensing, 2018"

Software and Open Source Tools

We will focus on cloud computing and some important machine learning libraries

Cloud Services

• Google Earth Engine
• AWS
• Updating

Useful Library

• Updating

Database

• CAVE dataset
• Harvard dataset
• iCVL dataset
• NUS datase
• NTIRE18 dataset
• Chikusei dataset
• Indian Pines, Salinas, KSC et al.
• Sen1-2 Dataset
• SEN12MS

TERMS

There are terms which are slightly different from those in other areas.

Measurements（SIGNAL/image）: Primarily the outputs of a sensor，represent the raw information, normally in format of singal, images. The elementary support of the measurement is a pixel in the case of an image, and is called a sample in the general case

Object: It is defined by its properties, e.g., its color, its materials, its shapes, its neighborhood, etc. It can be a field, a building, the edge of a road, a cloud, an oceanic eddy, etc.

attribute(Feature): It is a property of an object. Mathematical attribute: aggregation of measurements made for each of the elements of the object Modality: It refers to the raw input used by the sensors.

Spatial context of a pixel, computed by local variance, or structure function or any spatial operator. This operation can be extended to time context in the case of time-series of measurements. Equivalent terms are local variability, local fluctuations, spatial or time texture, or pattern.