royalosyin/Python-Practical-Application-on-Climate-Variability-Studies royalosyin/Python-Practical-Application-on-Climate-Variability-Studies

  • Stars
    star
    231
  • Rank 173,434 (Top 4 %)
  • Language
    Jupyter Notebook
  • License
    MIT License
  • Created about 7 years ago
  • Updated 4 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
royalosyin/Python-Practical-Application-on-Climate-Variability-Studies
github

royalosyin

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

This tutorial is a companion volume of Matlab versionm but add more. Main objective is the transference of know-how in practical applications and management of statistical tools commonly used to explore meteorological time series, focusing on applications to study issues related with the climate variability and climate change. This tutorial starts with some basic statistic for time series analysis as estimation of means, anomalies, standard deviation, correlations, arriving the estimation of particular climate indexes (Niño 3), detrending single time series and decomposition of time series, filtering, interpolation of climate variables on regular or irregular grids, leading modes of climate variability (EOF or HHT), signal processing in the climate system (spectral and wavelet analysis). In addition, this tutorial also deals with different data formats such as CSV, NetCDF, Binary, and matlab'mat, etc. It is assumed that you have basic knowledge and understanding of statistics and Python.

Python & Practical Application on Climate Variability Studies

Main objective of this tutorial is the transference of know-how in practical applications and management of statistical tools commonly used to explore meteorological time series, focusing on applications to study issues related with the climate variability and climate change. This tutorial starts with some basic statistic for time series analysis as estimation of means, anomalies, standard deviation, correlations, arriving the estimation of particular climate indexes (Niño 3), detrending single time series and decomposition of time series, filtering, interpolation of climate variables on regular or irregular grids, leading modes of climate variability (EOF or HHT), signal processing in the climate system (spectral and wavelet analysis). In addition, this tutorial also deals with different data formats such as CSV, NetCDF, Binary, and matlab'mat, etc. It is assumed that you have basic knowledge and understanding of statistics and Python.

Generic libraries for scientific analysis

The default Python library for dealing with large arrays of numeric data (e.g. four dimensional latitude/longitude/altitude/time data arrays) is numpy, while the netCDF data format is commonly used in Atmospheric science and Oceanography as it is convinient to store various variables of many dimensions. The default for reading and writing netCDF files is netCDF4 (the capability to read and write text files, including .csv, is built into numpy).

What is NetCDF?

NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. NetCDF was developed and is maintained at Unidata. Unidata provides data and software tools for use in geoscience education and research. The NetCDF homepage may be found at http://www.unidata.ucar.edu/software/netcdf/. The NetCDF source-code is hosted at GitHub, and may be found directly at http://github.com/Unidata/netcdf-c.

How to deal with NetCDF and other data with Python?

we mainly use netCDF4-python, NumPy and SciPy to process NetCDF and other data formats.

  • netCDF4-python

netcdf4-python is a Python interface to the netCDF C library. netCDF version 4 has many features not found in earlier versions of the library and is implemented on top of HDF5. This module can read and write files in both the new netCDF 4 and the old netCDF 3 format, and can create files that are readable by HDF5 clients. The API modelled after Scientific.IO.NetCDF, and should be familiar to users of that module (see more http://unidata.github.io/netcdf4-python/).

  • NumPy

NumPy is the fundamental package for scientific computing with Python (see more http://www.numpy.org/). It contains among other things: a powerful N-dimensional array object sophisticated (broadcasting) functions tools for integrating C/C++ and Fortran code useful linear algebra, Fourier transform, and random number capabilities Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases.

  • SciPy

SciPy is a collection of mathematical algorithms and convenience functions built on the Numpy extension of Python (https://www.scipy.org/index.html). It adds significant power to the interactive Python session by providing the user with high-level commands and classes for manipulating and visualizing data. With SciPy an interactive Python session becomes a data-processing and system-prototyping environment rivaling systems such as MATLAB, IDL, Octave, R-Lab, and SciLab. The additional benefit of basing SciPy on Python is that this also makes a powerful programming language available for use in developing sophisticated programs and specialized applications. Scientific applications using SciPy benefit from the development of additional modules in numerous niches of the software landscape by developers across the world. Everything from parallel programming to web and data-base subroutines and classes have been made available to the Python programmer. All of this power is available in addition to the mathematical libraries in SciPy.

Data Sources

We will use the data publicly available as possible. The data are mainly downloaded from https://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.derived.surfaceflux.html

Visualization

Matplotlib

Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. Matplotlib can be used in Python scripts, the Python and IPython shell, the jupyter notebook, web application servers, and four graphical user interface toolkits. Matplotlib tries to make easy things easy and hard things possible. You can generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc., with just a few lines of code. For a sampling, see the screenshots, thumbnail gallery, and examples directory For simple plotting the pyplot module provides a MATLAB-like interface, particularly when combined with IPython. For the power user, you have full control of line styles, font properties, axes properties, etc, via an object oriented interface or via a set of functions familiar to MATLAB users. See more from https://matplotlib.org/

Basemap

Basemap is a great tool for creating maps using python in a simple way. It’s a matplotlib extension, so it has got all its features to create data visualizations, and adds the geographical projections and some datasets to be able to plot coast lines, countries, and so on directly from the library. Basemap has got some documentation, but some things are a bit more difficult to find. I started this documentation to extend a little the original documentation and examples, but it grew a little, and now covers many of the basemap possibilities. See more from https://basemaptutorial.readthedocs.io/en/latest/

More

We will mainly apply these mostly generic libraries to carry out data analysis step by step. The procedures or steps are common to the atmospheric and ocean sciences. Although other advanced libraries will simpilfy the procedures, the underlying ideas should be the same in essense. We will also introduce more other libraries such as xarray and iris in the following parts.

More Repositories

1

Practice-SQL-with-SQLite-and-Jupyter-Notebook

Practice basic SQL syntax with Jupyter notebook. SQL is particularly useful in handling structured data where there are relations between different entities/variables of the data. SQL is a very important tool for data scientists to have in their repertoire.
Jupyter Notebook
130
star
2

A-Beginner-Guide-to-Carry-out-Extreme-Value-Analysis-with-Codes-in-Python

A beginner's guide to carry out extreme value analysis, which consists of basic steps, multiple distribution fitting, confidential intervals, IDF/DDF, and a simple application of IDF information for roof drainage design. The guide mainly focuses on extreme rainfall analysis. However, the basic steps are also suitable for other climatic or hydrologic variables such as temperature, wind speed or runoff.
Jupyter Notebook
50
star
3

Work-with-DEM-data-using-Python-from-Simple-to-Complicated

Work with DEM data using Python from Simple to Complicated. Many python packages will be touched such as GDAL, numpy, xarray, rasterio, folium, cartopy, geopandas etc.
Jupyter Notebook
46
star
4

Calculate-Precipitation-based-Agricultural-Drought-Indices-with-Python

Precipitation-based indices are generally considered as the simplest indices because they are calculated solely based on long-term rainfall records that are often available. The mostly used precipitation-based indices consist of Decile Index (DI) Hutchinson Drought Severity Index (HDSI) Percen of Normal Index (PNI) Z-Score Index (ZSI) China-Z Index (CZI) Modified China-Z Index (MCZI) Rainfall Anomaly Index (RAI) Effective Drought Index (EDI) Standardized Precipitation Index (SPI).
Jupyter Notebook
30
star
5

Overlay-GeoTiff-Raster-with-nodata-On-Interactive-Map

Overlay a GeoTiff raster data on Interactive map created by folium python. This tutorial will finish three tasks: (1) Export a NetCDF data to a time series of geotiff images.; (2) Overlay a geotiff raster onto an interactive map created by folium python; (3) Create a custom OpenStreetMap tile server on a local machine with docker.
Jupyter Notebook
11
star
6

pySQLiteSWAT

This is a part of the training material of capacity building for hydrological modeling under climate change based on SWAT. In this part, SWAT simulations are post-processed by python-based tools such as Pandas, GeoPandas, PySAL, folium, etc.
Jupyter Notebook
7
star
7

pyFastAPI_PointsInPolygons

A Tutorial shows how to build simple Point-In-Polygon(PIP) Web API using FastAPI and other python packages.
Jupyter Notebook
5
star
8

FastAPI-Zarr-Xarray-Dask

A demo of FastAPI application to extract data at the specific points defined by [latitudes and longitudes] from a zarr dataset using Dask and Xarray. The key aim to speed up point data extraction from data in zarr formats using dask's local cluster.
Python
4
star
9

pyRaster2GeoTiff

A piece of Python code to convert rater data such as NetCDF to GeoTIFF format. The code was extracted from my LinkedIn article at https://www.linkedin.com/pulse/convert-netcdf4-file-geotiff-using-python-chonghua-yin/
Python
3
star
10

fortran-ls-svm

Support Vector Machine aplication written in fortran 90
Fortran
2
star
11

Fill-Missing-Values

Functions used to fill missing values in a 2D array or matrix. Mostly written by Fortran.
Fortran
2
star
12

Explain-100-Year-Weather-Events-with-Python

Jupyter Notebook
1
star
13

Create-Concise-Charts-for-Reporting

Some practical tutorials for creating concise charts for project summary reporting. At present, only focus on Line and Bar Charts. It might be extended in the future
1
star
14

DockerMet

Using Docker Compose to rapidly deploy a Jupyter Notebook environment for processing meteorological and climatic data. The Dockerfile was created based on the data-science stack image from Jupyter project. Besides default tools and packages, only added several met-clim specific packages such as xarray and dask for netCDF, sfgrib for Grib1/2, and MetPy for met.
Jupyter Notebook
1
star
15

myEvaPy

Using Numba.njit to speed up extreme value analysis (EVA)
Jupyter Notebook
1
star