Wetterdienst - Open weather data for humans
Warning
This library is a work in progress!
Breaking changes should be expected until a 1.0 release, so version pinning is recommended.
Note
Wetterdienst 0.57.0 switched from pandas to Polars, which may cause breaking changes
for certain user-space code heavily using pandas idioms, because Wetterdienst now
returns a Polars DataFrame. If you absolutely must use a pandas DataFrame, you can
cast the Polars DataFrame to pandas by using the .to_pandas()
method.
What our customers say:
"Our house is on fire. I am here to say, our house is on fire. I saw it with my own eyes using wetterdienst to get the data." - Greta Thunberg
โYou must be the change you wish to see in the world. And when it comes to climate I use wetterdienst.โ - Mahatma Gandhi
"Three things are (almost) infinite: the universe, human stupidity and the temperature time series of Hohenpeissenberg, Germany I got with the help of wetterdienst; and I'm not sure about the universe." - Albert Einstein
"We are the first generation to feel the effect of climate change and the last generation who can do something about it. I used wetterdienst to analyze the climate in my area and I can tell it's getting hot in here." - Barack Obama
Introduction
Overview
Welcome to Wetterdienst, your friendly weather service library for Python.
We are a group of like-minded people trying to make access to weather data in
Python feel like a warm summer breeze, similar to other projects like
rdwd for the R language, which originally drew our interest in this project.
Our long-term goal is to provide access to multiple weather services as well as other
related agencies such as river measurements. With wetterdienst
we try to use modern
Python technologies all over the place. The library is based on polars (we <3 pandas, it is still part of some
IO processes) across the board, uses Poetry for package administration and GitHub Actions for all things CI.
Our users are an important part of the development as we are not currently using the
data we are providing and only implement what we think would be the best. Therefore
contributions and feedback whether it be data related or library related are very
welcome! Just hand in a PR or Issue if you think we should include a new feature or data
source.
Data
For an overview of the data we have currently made available and under which license it is published take a look at the data section. Detailed information on datasets and parameters is given at the coverage subsection. Licenses and usage requirements may differ for each provider so check this out before including the data in your project to be sure that you fulfill copyright requirements!
Here is a short glimpse on the data that is included:
- DWD (Deutscher Wetterdienst / German Weather Service / Germany)
- Mosmix - statistical optimized scalar forecasts extracted from weather models
- Point forecast
- 5400 stations worldwide
- Both MOSMIX-L and MOSMIX-S is supported
- Up to 115 parameters
- Road Weather Observations
- Historical weather observations of German highway stations
- Radar
- 16 locations in Germany
- All of Composite, Radolan, Radvor, Sites and Radolan_CDC
- Radolan: calibrated radar precipitation
- Radvor: radar precipitation forecast
- ECCC (Environnement et Changement Climatique Canada / Environment and Climate Change Canada / Canada)
- Historical Weather Observations
- Historical (last ~180 years)
- Hourly, daily, monthly, (annual) resolution
- Time series of stations in Canada
- NOAA (National Oceanic And Atmospheric Administration / National Oceanic And Atmospheric Administration / United States Of America)
- Global Historical Climatology Network
- Historical, daily weather observations from around the globe
- more then 100k stations
- data for weather services which don't publish data themselves
- WSV (Wasserstraรen- und Schifffahrtsverwaltung des Bundes / Federal Waterways and Shipping Administration)
- Pegelonline
- data of river network of Germany
- coverage of last 30 days
- parameters like stage, runoff and more related to rivers
- EA (Environment Agency)
- Hydrology
- data of river network of UK
- parameters flow and ground water stage
- NWS (NOAA National Weather Service)
- Observation
- recent observations (last week) of US weather stations
- currently the list of stations is not completely right as we use a diverging source!
- Eaufrance
- Hubeau
- data of river network of France (continental)
- parameters flow and stage of rivers of last 30 days
- Geosphere (Geosphere Austria, formerly Central Institution for Meteorology and Geodynamics)
- Observation
- historical meteorological data of Austrian stations
- IMGW (Institute of Meteorology and Water Management)
- Meteorology
- meteorological data of polish weather stations
- daily and monthly summaries
- Hydrology
- hydrological data of polish river stations
- daily and monthly summaries
To get better insight on which data we have currently made available and under which license those are published take a look at the data section.
Features
- APIs for stations and values
- Get stations nearby a selected location
- Define your request by arguments such as parameter, period, resolution, start date, end date
- Define general settings in Settings context
- Command line interface
- Web-API via FastAPI
- Run SQL queries on the results
- Export results to databases and other data sinks
- Public Docker image
- Interpolation and Summary of station values
Setup
Native
Via PyPi (standard):
pip install wetterdienst
Via Github (most recent):
pip install git+https://github.com/earthobservations/wetterdienst
There are some extras available for wetterdienst
. Use them like:
pip install wetterdienst[http,sql]
- docs: Install the Sphinx documentation generator.
- ipython: Install iPython stack.
- export: Install openpyxl for Excel export and pyarrow for writing files in Feather- and Parquet-format.
- http: Install HTTP API prerequisites.
- sql: Install DuckDB for querying data using SQL.
- duckdb: Install support for DuckDB.
- influxdb: Install support for InfluxDB.
- cratedb: Install support for CrateDB.
- mysql: Install support for MySQL.
- postgresql: Install support for PostgreSQL.
- interpolation: Install support for station interpolation.
In order to check the installation, invoke:
wetterdienst --help
Docker
Docker images for each stable release will get pushed to GitHub Container Registry.
There are images in two variants, wetterdienst-standard
and wetterdienst-full
.
wetterdienst-standard
will contain a minimum set of 3rd-party packages,
while wetterdienst-full
will try to serve a full environment, including
all of the optional dependencies of Wetterdienst.
Pull the Docker image:
docker pull ghcr.io/earthobservations/wetterdienst-standard
Library
Use the latest stable version of wetterdienst
:
$ docker run -ti ghcr.io/earthobservations/wetterdienst-standard
Python 3.8.5 (default, Sep 10 2020, 16:58:22)
[GCC 8.3.0] on linux
import wetterdienst
wetterdienst.__version__
Command line script
The wetterdienst
command is also available:
# Make an alias to use it conveniently from your shell.
alias wetterdienst='docker run -ti ghcr.io/earthobservations/wetterdienst-standard wetterdienst'
wetterdienst --help
wetterdienst --version
wetterdienst info
Raspberry Pi / LINUX ARM
Running wetterdienst on Raspberry Pi, you need to install numpy and lxml prior to installing wetterdienst by running the following lines:
# not all installations may be required to get lxml running
sudo apt-get install gfortran
sudo apt-get install libopenblas-base
sudo apt-get install libopenblas-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install python3-lxml
Additionally expanding the Swap to 2048 mb may be required and can be done via swap-file:
sudo nano /etc/dphys-swapfile
Thanks chr-sto for reporting back to us!
Example
Task: Get historical climate summary for two German stations between 1990 and 2020
Library
>>> import polars as pl
>>> _ = pl.Config.set_tbl_hide_dataframe_shape(True)
>>> from wetterdienst import Settings
>>> from wetterdienst.provider.dwd.observation import DwdObservationRequest
>>> settings = Settings( # default
... ts_shape="long", # tidy data
... ts_humanize=True, # humanized parameters
... ts_si_units=True # convert values to SI units
... )
>>> request = DwdObservationRequest(
... parameter=["climate_summary"],
... resolution="daily",
... start_date="1990-01-01", # if not given timezone defaulted to UTC
... end_date="2020-01-01", # if not given timezone defaulted to UTC
... settings=settings
... ).filter_by_station_id(station_id=(1048, 4411))
>>> stations = request.df
>>> stations.head()
โโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโโฌโโโโโโโโโโโโโโฌโโโโโโโโโโ
โ station_id โ from_date โ to_date โ height โ latitude โ longitude โ name โ state โ
โ --- โ --- โ --- โ --- โ --- โ --- โ --- โ --- โ
โ str โ datetime[ฮผs, โ datetime[ฮผs, โ f64 โ f64 โ f64 โ str โ str โ
โ โ UTC] โ UTC] โ โ โ โ โ โ
โโโโโโโโโโโโโโชโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโชโโโโโโโโโชโโโโโโโโโโโชโโโโโโโโโโโโชโโโโโโโโโโโโโโชโโโโโโโโโโก
โ 01048 โ 1934-01-01 โ ... โ 228.0 โ 51.1278 โ 13.7543 โ Dresden-Klo โ Sachsen โ
โ โ 00:00:00 UTC โ 00:00:00 UTC โ โ โ โ tzsche โ โ
โ 04411 โ 1979-12-01 โ ... โ 155.0 โ 49.9195 โ 8.9671 โ Schaafheim- โ Hessen โ
โ โ 00:00:00 UTC โ 00:00:00 UTC โ โ โ โ Schlierbach โ โ
โโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโ
>>> values = request.values.all().df
>>> values.head()
โโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโโ
โ station_id โ dataset โ parameter โ date โ value โ quality โ
โ --- โ --- โ --- โ --- โ --- โ --- โ
โ str โ str โ str โ datetime[ฮผs, UTC] โ f64 โ f64 โ
โโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโชโโโโโโโโโโก
โ 01048 โ climate_summary โ wind_gust_max โ 1990-01-01 00:00:00 UTC โ null โ null โ
โ 01048 โ climate_summary โ wind_gust_max โ 1990-01-02 00:00:00 UTC โ null โ null โ
โ 01048 โ climate_summary โ wind_gust_max โ 1990-01-03 00:00:00 UTC โ 5.0 โ 10.0 โ
โ 01048 โ climate_summary โ wind_gust_max โ 1990-01-04 00:00:00 UTC โ 9.0 โ 10.0 โ
โ 01048 โ climate_summary โ wind_gust_max โ 1990-01-05 00:00:00 UTC โ 7.0 โ 10.0 โ
โโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโ
values.to_pandas() # to get a pandas DataFrame and e.g. create some matplotlib plots
Client
# Get list of all stations for daily climate summary data in JSON format
wetterdienst stations --provider=dwd --network=observations --parameter=kl --resolution=daily
# Get daily climate summary data for specific stations
wetterdienst values --provider=dwd --network=observations --station=1048,4411 --parameter=kl --resolution=daily
Further examples (code samples) can be found in the examples folder.
Acknowledgements
We want to acknowledge all environmental agencies which provide their data open and free of charge first and foremost for the sake of endless research possibilities.
We want to acknowledge Jetbrains and the Jetbrains OSS Team for providing us with licenses for Pycharm Pro, which we are using for the development.
We want to acknowledge all contributors for being part of the improvements to this library that make it better and better every day.
Important Links
- Full documentation: https://wetterdienst.readthedocs.io/
- Usage: https://wetterdienst.readthedocs.io/en/latest/usage/
- Contribution: https://wetterdienst.readthedocs.io/en/latest/contribution/
- Known Issues: https://wetterdienst.readthedocs.io/en/latest/known_issues/
- Changelog: https://wetterdienst.readthedocs.io/en/latest/changelog.html
- Examples (runnable scripts): https://github.com/earthobservations/wetterdienst/tree/main/example
- Benchmarks: https://github.com/earthobservations/wetterdienst/tree/main/benchmarks