• Stars
    star
    132
  • Rank 265,534 (Top 6 %)
  • Language
    Python
  • License
    Other
  • Created over 5 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Python package for retrieving water data from USGS or the multi-agency Water Quality Portal

dataretrieval: Download hydrologic data

⚠️ USGS data availability and format are changing on Water Quality Portal (WQP). Beginning in February 2024 data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html

What is dataretrieval?

dataretrieval was created to simplify the process of loading hydrologic data into the Python environment. Like the original R version dataRetrieval, it is designed to retrieve the major data types of U.S. Geological Survey (USGS) hydrology data that are available on the Web, as well as data from the Water Quality Portal (WQP), which currently houses water quality data from the Environmental Protection Agency (EPA), U.S. Department of Agriculture (USDA), and USGS. Direct USGS data is obtained from a service called the National Water Information System (NWIS).

Note that the python version is not a direct port of the original: it attempts to reproduce the functionality of the R package, though its organization and interface often differ.

If there's a hydrologic or environmental data portal that you'd like dataretrieval to work with, raise it as an issue.

Here's an example using dataretrieval to retrieve data from the National Water Information System (NWIS).

# first import the functions for downloading data from NWIS
import dataretrieval.nwis as nwis

# specify the USGS site code for which we want data.
site = '03339000'


# get instantaneous values (iv)
df = nwis.get_record(sites=site, service='iv', start='2017-12-31', end='2018-01-01')

# get water quality samples (qwdata)
df2 = nwis.get_record(sites=site, service='qwdata', start='2017-12-31', end='2018-01-01')

# get basic info about the site
df3 = nwis.get_record(sites=site, service='site')

Services available from NWIS include:

  • instantaneous values (iv)
  • daily values (dv)
  • statistics (stat)
  • site info (site)
  • discharge peaks (peaks)
  • discharge measurements (measurements)
  • water quality samples (qwdata)

To access the full functionality available from NWIS web services, nwis.get record appends any additional kwargs into the REST request. For example

nwis.get_record(sites='03339000', service='dv', start='2017-12-31', parameterCd='00060')

will download daily data with the parameter code 00060 (discharge).

Accessing the "Internal" NWIS

If you're connected to the USGS network, dataretrieval call pull from the internal (non-public) NWIS interface. Most dataretrieval functions pass kwargs directly to NWIS's REST API, which provides simple access to internal data; simply specify "access='3'". For example

nwis.get_record(sites='05404147',service='iv', start='2021-01-01', end='2021-3-01', access='3')

More services and documentation to come!

Quick start

dataretrieval can be installed using pip:

$ python3 -m pip install -U dataretrieval

or conda:

$ conda install -c conda-forge dataretrieval

More examples of use are include in demos.

Issue tracker

Please report any bugs and enhancement ideas using the dataretrieval issue tracker:

https://github.com/USGS-python/dataretrieval/issues

Feel free to also ask questions on the tracker.

Contributing

Any help in testing, development, documentation and other tasks is welcome. For more details, see the file CONTRIBUTING.md.

Coverage Status

Package Support

The Water Mission Area of the USGS supports the development and maintenance of dataretrieval and most likely further into the future. Resources are available primarily for maintenance and responding to user questions. Priorities on the development of new features are determined by the dataretrieval development team.

Acknowledgments

This material is partially based upon work supported by the National Science Foundation (NSF) under award 1931297. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.

Disclaimer

This software is preliminary or provisional and is subject to revision. It is being provided to meet the need for timely best science. The software has not received final approval by the U.S. Geological Survey (USGS). No warranty, expressed or implied, is made by the USGS or the U.S. Government as to the functionality of the software and related material nor shall the fact of release constitute any such warranty. The software is provided on the condition that neither the USGS nor the U.S. Government shall be held liable for any damages resulting from the authorized or unauthorized use of the software.

Citation

Hodson, T.O., Hariharan, J.A., Black, S., and Horsburgh, J.S., 2023, dataretrieval (Python): a Python package for discovering and retrieving water data available from U.S. federal hydrologic web services: U.S. Geological Survey software release, https://doi.org/10.5066/P94I5TX3.

More Repositories

1

dataRetrieval

This R package is designed to obtain USGS or EPA water quality sample data, streamflow data, and metadata directly from web services. See: https://rconnect.usgs.gov/dataRetrieval/
R
232
star
2

ISIS3

Integrated Software for Imagers and Spectrometers v3. ISIS3 is a digital image processing software package to manipulate imagery collected by current and past NASA and International planetary missions.
C++
180
star
3

COAWST

COAWST modeling system git repository
Fortran
94
star
4

nhdplusTools

See official repository at: https://code.usgs.gov/water/nhdplusTools
R
82
star
5

geobipy

Geophysical Bayesian Inference in Python. Docs:
Python
76
star
6

EGRET

An R-package for the analysis of long-term changes in water quality and streamflow, including the water-quality method Weighted Regressions on Time, Discharge, and Season (WRTDS). https://doi-usgs.github.io/EGRET/
R
73
star
7

geoknife

R tools for geo-web processing of gridded data via the Geo Data Portal. geoknife slices up gridded data according to overlap with irregular features, such as watersheds, lakes, points, etc.
R
69
star
8

CIDA-Viz

Datasets and docs for CIDA's data visualization challenge
R
61
star
9

fort-pymdwizard

The MetadataWizard is a useful tool designed to facilitate FGDC metadata creation for spatial and non-spatial data sets. It is a cross-platform desktop application built using an open-source Python architecture.
Python
60
star
10

hurricane-irma

R
55
star
11

chart-challenge-22

#30DayChartChallenge 2022, USGS edition
Jupyter Notebook
44
star
12

modflow-setup

Python package for automating the setup of MODFLOW groundwater flow models
Python
40
star
13

gems-tools-arcmap

Tools for working with the GeMS geologic map database schema in ArcGIS
Python
39
star
14

gems-tools-pro

GeMS Tools for ArcGIS Pro
Python
38
star
15

flow-tiles

A tile map showing a month of streamflow conditions across the U.S.
R
35
star
16

sfrmaker

Rapid construction of MODFLOW SFR Package input from hydrography data.
Python
34
star
17

EflowStats

Calculates Hydrologic Indicator stats and fundamental properties of daily streamflow for a given set of data
R
33
star
18

sciencebasepy

Python
33
star
19

streamMetabolizer

streamMetabolizer uses inverse modeling to estimate aquatic metabolism (photosynthesis and respiration) from time series data on dissolved oxygen, water temperature, depth, and light.
Stan
31
star
20

usgscsm

This repository stores USGS Community Sensor Model (CSM) camera models
C++
24
star
21

vizstorm-GIF

R
22
star
22

hurricane-harvey

Visualizing the impacts of hurricane Harvey through precipitation and streamflow data.
R
22
star
23

sbtools

Tools for interfacing R with ScienceBase data services.
R
21
star
24

toxEval

The toxEval R-package includes a set of functions to analyze, visualize, and organize measured concentration data as it relates to chosen biological effects benchmarks. See https://doi-usgs.github.io/toxEval/ for more details
R
19
star
25

hydroloom

hydrologic geospatial fabric creation tools. See official repository here: https://code.usgs.gov/water/hydroloom
R
16
star
26

HASP

Hydrologic AnalySis Package. See official source code: https://doi-usgs.github.io/HASP/
R
16
star
27

gage-conditions-gif

R
15
star
28

plio

Planetary I/O Module
Python
15
star
29

water-use

Visualization using USGS water use data.
JavaScript
14
star
30

ncdfgeom

NetCDF-CF Geometry and Timeseries Tools for R: https://code.usgs.gov/water/ncdfgeom
R
14
star
31

OWDI-Lower-Colorado-Drought-Vis

Lower Colorado Drought Visualization
HTML
13
star
32

ale

Abstraction Layer for Ephemerides (ALE)
XC
13
star
33

loadflex

Models and Tools for Watershed Flux Estimates
R
12
star
34

water-use-15

USGS water use data visualization emphasizing the newly added 2015 dataset.
JavaScript
11
star
35

volcano-avog2s

Fortran
10
star
36

snow-to-flow

What does changing snowmelt mean for flow in the western US?
Vue
10
star
37

modelcatalog-tam

tools and metadata supporting the usgs model catalog
Python
9
star
38

scipiper

R
9
star
39

vizlab-chart-challenge-23

Jupyter Notebook
9
star
40

EGRETci

A bootstrap method for estimating uncertainty of water quality trends. https://doi-usgs.github.io/EGRETci/
R
8
star
41

python-for-hydrology

python and flopy tutorials for hydrologic applications and MODFLOW modeling
Jupyter Notebook
7
star
42

intersectr

See official repository here: https://code.usgs.gov/water/intersectr
R
7
star
43

temperature-prediction

Vue
7
star
44

lake-temperature-model-prep

Pipeline #1
R
6
star
45

hurricane-maria

visualization for hurricane Maria 2017
R
6
star
46

drought-timeline

A history of hydrological drought in the U.S.
Vue
6
star
47

hurricane-matthew

Visualizing the impacts of hurricane Matthew in the Southeastern US through precipitation and streamflow data.
R
6
star
48

ds-pipelines-targets-1-course

Intro to Data Analysis Pipelines using `targets` for USGS Data Science
R
5
star
49

delaware-basin-story

A JavaScript Project Using the Vue Framework.
Vue
5
star
50

climate-fish-habitat

Shifts in fish habitat under climate change
JavaScript
5
star
51

hyRefactor

HTML
5
star
52

gages-through-the-ages

A JavaScript project visualizing the changes in number of monitoring locations over time.
Vue
5
star
53

vizlab-home

Landing page for the USGS Vizlab
Vue
4
star
54

lsforce

A Python-based single-force seismic inversion framework for massive landslides
Jupyter Notebook
4
star
55

exampleRproj

Example repo of a very basic RStudio project setup. Used for Intro to R classes.
R
4
star
56

FoundationalDataProducts

A repository hosting the data contained in a living catalog of available foundational data products. See URL for the interactive site.
JavaScript
4
star
57

linesink-maker

Rapidly build GFLOW models from GIS hydrography data
Python
4
star
58

gw-general-models

Jupyter Notebook
4
star
59

ghsc-esi-shakemap

ShakeMap
Python
4
star
60

gw-conditions

Similar to gage-conditions-gif but for groundwater!
R
4
star
61

pleasant-lake-flopy-example

Repository for the worked example in the 2022 Groundwater Technology Spotlight on Flopy.
Jupyter Notebook
3
star
62

xai-workflows

This repository is an exploratory space for developing XAI workflows for PGDL models.
Jupyter Notebook
3
star
63

knoten

Python Geospatial Sensor Exploitation Library
Python
3
star
64

gw-res-time

Jupyter Notebook
3
star
65

ghsc-esi-groundmotion-processing

Parsing and processing ground motion data
Python
3
star
66

nawqa_wqp

Scripts/workflow for Water Quality Portal pulls for NAWQA trends and networks analyses.
HTML
3
star
67

fire-hydro

A single-page visual website about the impacts of wildfires in the Western US on water supply and water quality.
Vue
3
star
68

neversink_workflow

C++
3
star
69

mda.streams

backend tools for powstreams
R
3
star
70

loadflexBatch

Demo script and data for loadflex batch mode development
R
3
star
71

ds-pipelines-targets-3-course

Many-task pipelines using targets
R
2
star
72

lake-temp-timeseries

A gif of surface temperatures for 185,549 lakes
R
2
star
73

water-cycle

Vue
2
star
74

pools-and-fluxes

An interactive chart of the major pools and fluxes in the water cycle.
Vue
2
star
75

planetary-sdi

JavaScript
2
star
76

cloud-ht2c

2
star
77

national-flow-observations

This repository pulls national flow data from NWIS
R
2
star
78

normal_mode_GSN

Codes for the GSN normal mode paper
Python
2
star
79

powstreams

Tools for Powell Center working group on stream metabolism
HTML
2
star
80

DataRetrieval.jl

Julia port of the R and Python data retrieval packages
Julia
2
star
81

lake-temperature-out

outputs and summaries from lake modeling pipelines
R
1
star
82

lake-temperature-process-models

R
1
star
83

ds-pipelines-4-course

Shared cache pipelines
1
star
84

swigcsm

Wrappers for the CSM
C++
1
star
85

great-lakes-microplastics

Microplastics in the Great Lakes
JavaScript
1
star
86

ds-pipeline-demo

R
1
star
87

OWDI-drought

CSS
1
star
88

hyswap

hyswap: HYdrologic Surface Water Analysis Package
Python
1
star
89

habs-proxies-forecast-chl

Collection of code from the HABs Proxies group to forecast chl-a for Ecological Forecasting Challenge 2022
R
1
star
90

pgmtl-data-release

A repository for data release scripts and workflows for releasing process-guided meta-transfer learning predictions
R
1
star
91

rt-quic-db

A database application for RT-QuIC data with tools for visualization and analysis
CSS
1
star
92

vizlab-bottled-water

R
1
star