• Stars
    star
    44
  • Rank 620,141 (Top 13 %)
  • Language
    R
  • License
    Other
  • Created about 10 years ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

R interface to the Data Retriever

rdataretriever

Build Status Build statuscran version Documentation Status Downloads + Downloads (old package name) DOI DOI

R interface to the Data Retriever.

The rdataretriever provides access to cleaned versions of hundreds of commonly used public datasets with a single line of code.

These datasets come from many different sources and most of them require some cleaning and restructuring prior to analysis. The rdataretriever uses a set of actively maintained recipes for downloading, cleaning, and restructing these datasets using a combination of the Frictionless Data Specification and custom data cleaning scripts.

The rdataretriever also facilitates the automatic storage of these datasets in a choice of database management systems (PostgreSQL, SQLite, MySQL, MariaDB) or flat file formats (CSV, XML, JSON) for later use and integration with large data analysis pipelines.

The rdatretriever also facilitates reproducibile science by providing tools to archive and rerun the precise version of a dataset and associated cleaning steps that was used for a specific analysis.

The rdataretriever handles the work of cleaning, storing, and archiving data so that you can focus on analysis, inference and visualization.

Table of Contents

Installation

The rdataretriever is an R wrapper for the Python package, Data Retriever. This means that Python and the retriever Python package need to be installed first.

Basic Installation

If you just want to use the Data Retriever from within R follow these instuctions run the following commands in R. This will create a local Python installation that will only be used by R and install the needed Python package for you.

install.packages('reticulate') # Install R package for interacting with Python
reticulate::install_miniconda() # Install Python
reticulate::py_install('retriever') # Install the Python retriever package
install.packages('rdataretriever') # Install the R package for running the retriever
rdataretriever::get_updates() # Update the available datasets

After running these commands restart R.

Advanced Installation for Python Users

If you are using Python for other tasks you can use rdataretriever with your existing Python installation (though the basic installation above will also work in this case by creating a separate miniconda install and Python environment).

Install the retriever Python package

Install the retriever Python package into your prefered Python environment using either conda (64-bit conda is required):

conda install -c conda-forge retriever

or pip:

pip install retriever

Select the Python environment to use in R

rdataretriever will try to find Python environments with retriever (see the reticulate documentation on order of discovery for more details) installed. Alternatively you can select a Python environment to use when working with rdataretriever (and other packages using reticulate).

The most robust way to do this is to set the RETICULATE_PYTHON environment variable to point to the preferred Python executable:

Sys.setenv(RETICULATE_PYTHON = "/path/to/python")

This command can be run interactively or placed in .Renviron in your home directory.

Alternatively you can do select the Python environment through the reticulate package for either conda:

library(reticulate)
use_conda('name_of_conda_environment')

or virtualenv:

library(reticulate)
use_virtualenv("path_to_virtualenv_environment")

You can check to see which Python environment is being used with:

py_config()

Install the rdataretriever R package

install.packages("rdataretriever") # latest release from CRAN
remotes::install_github("ropensci/rdataretriever") # development version from GitHub

Installing Tabular Datasets

library(rdataretriever)

# List the datasets available via the Retriever
rdataretriever::datasets()

# Install the portal into csv files in your working directory
rdataretriever::install_csv('portal')

# Download the raw portal dataset files without any processing to the
# subdirectory named data
rdataretriever::download('portal', './data/')

# Install and load a dataset as a list
portal = rdataretriever::fetch('portal')
names(portal)
head(portal$species)

Installing Spatial Datasets

Set-up and Requirements

Tools

  • PostgreSQL with PostGis, psql(client), raster2pgsql, shp2pgsql, gdal,

The rdataretriever supports installation of spatial data into Postgres DBMS.

  1. Install PostgreSQL and PostGis

    To install PostgreSQL with PostGis for use with spatial data please refer to the OSGeo Postgres installation instructions.

    We recommend storing your PostgreSQL login information in a .pgpass file to avoid supplying the password every time. See the .pgpass documentation for more details.

    After installation, Make sure you have the paths to these tools added to your system's PATHS. Please consult an operating system expert for help on how to change or add the PATH variables.

    For example, this could be a sample of paths exported on Mac:

    #~/.bash_profile file, Postgres PATHS and tools.
    export PATH="/Applications/Postgres.app/Contents/MacOS/bin:${PATH}"
    export PATH="$PATH:/Applications/Postgres.app/Contents/Versions/10/bin"
    
  2. Enable PostGIS extensions

    If you have Postgres set up, enable PostGIS extensions. This is done by using either Postgres CLI or GUI(PgAdmin) and run

    For psql CLI

    psql -d yourdatabase -c "CREATE EXTENSION postgis;"
    psql -d yourdatabase -c "CREATE EXTENSION postgis_topology;"

    For GUI(PgAdmin)

    CREATE EXTENSION postgis;
    CREATE EXTENSION postgis_topology

    For more details refer to the PostGIS docs.

Sample commands

rdataretriever::install_postgres('harvard-forest') # Vector data
rdataretriever::install_postgres('bioclim') # Raster data

# Install only the data of USGS elevation in the given extent
rdataretriever::install_postgres('usgs-elevation', list(-94.98704597353938, 39.027001800158615, -94.3599408119917, 40.69577051867074))

Provenance

To ensure reproducibility the rdataretriever supports creating snapshots of the data and the script in time.

Use the commit function to create and store the snapshot image of the data in time. Provide a descriptive message for the created commit. This is comparable to a git commit, however the function bundles the data and scripts used as a backup.

With provenace, you will be able to reproduce the same analysis in the future.

Commit a dataset

By default commits will be stored in the provenance directory .retriever_provenance, but this directory can be changed by setting the environment variable PROVENANCE_DIR.

rdataretriever::commit('abalone-age',
                       commit_message='A snapshot of Abalone Dataset as of 2020-02-26')

You can also set the path for an individual commit:

rdataretriever::commit('abalone-age',
                       commit_message='Data and recipe archive for Abalone Data on 2020-02-26',
					   path='.')

View a log of committed datasets in the provenance directory

rdataretriever::commit_log('abalone-age')

Install a committed dataset

To reanalyze a committed dataset, rdataretriever will obtain the data and script from the history and rdataretriever will install this particular data into the given back-end. For example, SQLite:

rdataretriever::install_sqlite('abalone-age-a76e77.zip') 

Datasets stored in provenance directory can be installed directly using hash value

rdataretriever::install_sqlite('abalone-age', hash_value='a76e77')

Using Docker Containers

To run the image interactively

docker-compose run --service-ports rdata /bin/bash

To run tests

docker-compose run rdata Rscript load_and_test.R

Release

Make sure you have tests passing on R-oldrelease, current R-release and R-devel

To check the package

R CMD Build #build the package
R CMD check  --as-cran --no-manual rdataretriever_[version]tar.gz

To Test

setwd("./rdataretriever") # Set working directory
# install all deps
# install.packages("reticulate")
library(DBI)
library(RPostgreSQL)
library(RSQLite)
library(reticulate)
library(RMariaDB)
install.packages(".", repos = NULL, type="source")
roxygen2::roxygenise()
devtools::test()

To get citation information for the rdataretriever in R use citation(package = 'rdataretriever')

Acknowledgements

A big thanks to Ben Morris for helping to develop the Data Retriever. Thanks to the rOpenSci team with special thanks to Gavin Simpson, Scott Chamberlain, and Karthik Ram who gave helpful advice and fostered the development of this R package. Development of this software was funded by the National Science Foundation as part of a CAREER award to Ethan White.


ropensci footer

More Repositories

1

drake

An R-focused pipeline toolkit for reproducibility and high-performance computing
R
1,331
star
2

skimr

A frictionless, pipeable approach to dealing with summary statistics
HTML
1,095
star
3

targets

Function-oriented Make-like declarative workflows for R
R
880
star
4

rtweet

🐦 R client for interacting with Twitter's [stream and REST] APIs
R
785
star
5

tabulizer

Bindings for Tabula PDF Table Extractor Library
R
518
star
6

pdftools

Text Extraction, Rendering and Converting of PDF Documents
C++
489
star
7

magick

Magic, madness, heaven, sin
R
440
star
8

visdat

Preliminary Exploratory Visualisation of Data
R
439
star
9

stplanr

Sustainable transport planning with R
R
412
star
10

RSelenium

An R client for Selenium Remote WebDriver
R
332
star
11

rnoaa

R interface to many NOAA data APIs
R
326
star
12

osmdata

R package for downloading OpenStreetMap data
C++
307
star
13

charlatan

Create fake data in R
R
289
star
14

software-review

rOpenSci Software Peer Review.
R
279
star
15

iheatmapr

Complex, interactive heatmaps in R
R
259
star
16

taxize

A taxonomic toolbelt for R
R
250
star
17

rrrpkg

Use of an R package to facilitate reproducible research
248
star
18

elastic

R client for the Elasticsearch HTTP API
R
244
star
19

tesseract

Bindings to Tesseract OCR engine for R
R
236
star
20

qualtRics

Download ⬇️ Qualtrics survey data directly into R!
R
214
star
21

git2r

R bindings to the libgit2 library
R
212
star
22

biomartr

Genomic Data Retrieval with R
R
207
star
23

writexl

Portable, light-weight data frame to xlsx exporter for R
C
202
star
24

rnaturalearth

An R package to hold and facilitate interaction with natural earth map data 🌍
R
191
star
25

googleLanguageR

R client for the Google Translation API, Google Cloud Natural Language API and Google Cloud Speech API
HTML
190
star
26

textreuse

Detect text reuse and document similarity
R
188
star
27

tokenizers

Fast, Consistent Tokenization of Natural Language Text
R
179
star
28

rentrez

talk with NCBI entrez using R
R
178
star
29

piggyback

πŸ“¦ for using large(r) data files on GitHub
R
172
star
30

rcrossref

R client for various CrossRef APIs
R
165
star
31

osmextract

Download and import OpenStreetMap data from Geofabrik and other providers
R
163
star
32

dataspice

🌢️ Create lightweight schema.org descriptions of your datasets
R
156
star
33

tic

Tasks Integrating Continuously: CI-Agnostic Workflow Definitions
R
153
star
34

rgbif

Interface to the Global Biodiversity Information Facility API
R
151
star
35

webchem

Chemical Information from the Web
R
149
star
36

geojsonio

Convert many data formats to & from GeoJSON & TopoJSON
R
148
star
37

MODIStsp

An "R" package for automatic download and preprocessing of MODIS Land Products Time Series
R
147
star
38

tsbox

tsbox: Class-Agnostic Time Series in R
R
146
star
39

DataPackageR

An R package to enable reproducible data processing, packaging and sharing.
R
145
star
40

dev_guide

rOpenSci Packages: Development, Maintenance, and Peer Review
R
141
star
41

ghql

GraphQL R client
R
141
star
42

jqr

R interface to jq
R
139
star
43

osfr

R interface to the Open Science Framework (OSF)
R
139
star
44

osmplotr

Data visualisation using OpenStreetMap objects
R
130
star
45

opencv

R bindings for OpenCV
C++
130
star
46

ssh

Native SSH client in R based on libssh
C
126
star
47

tarchetypes

Archetypes for targets and pipelines
R
120
star
48

RefManageR

R package RefManageR
R
114
star
49

ezknitr

Avoid the typical working directory pain when using 'knitr'
R
112
star
50

spocc

Species occurrence data toolkit for R
R
109
star
51

hunspell

High-Performance Stemmer, Tokenizer, and Spell Checker for R
C++
106
star
52

crul

R6 based http client for R (for developers)
R
102
star
53

gistr

Interact with GitHub gists from R
R
101
star
54

spelling

Tools for Spell Checking in R
R
101
star
55

weathercan

R package for downloading weather data from Environment and Climate Change Canada
R
100
star
56

rfishbase

R interface to the fishbase.org database
R
100
star
57

UCSCXenaTools

πŸ“¦ An R package for accessing genomics data from UCSC Xena platform, from cancer multi-omics to single-cell RNA-seq https://cran.r-project.org/web/packages/UCSCXenaTools/
R
99
star
58

gutenbergr

Search and download public domain texts from Project Gutenberg
R
98
star
59

bib2df

Parse a BibTeX file to a tibble
R
97
star
60

ckanr

R client for the CKAN API
R
97
star
61

git2rdata

An R package for storing and retrieving data.frames in git repositories.
R
97
star
62

rsvg

SVG renderer for R based on librsvg2
C
95
star
63

EML

Ecological Metadata Language interface for R: synthesis and integration of heterogenous data
R
94
star
64

nasapower

API Client for NASA POWER Global Meteorology, Surface Solar Energy and Climatology in R
R
93
star
65

FedData

Functions to Automate Downloading Geospatial Data Available from Several Federated Data Sources
R
93
star
66

cyphr

:shipit: Humane encryption
R
92
star
67

av

Working with Video in R
C
88
star
68

mapscanner

R package to print maps, draw on them, and scan them back in
R
87
star
69

opencage

🌐 R package for the OpenCage API -- both forward and reverse geocoding 🌐
R
87
star
70

GSODR

API Client for Global Surface Summary of the Day ('GSOD') Weather Data Client in R
R
85
star
71

tidync

NetCDF exploration and data extraction
R
85
star
72

openalexR

Getting bibliographic records from OpenAlex
R
85
star
73

rzmq

R package for ZMQ
C++
82
star
74

gittargets

Data version control for reproducible analysis pipelines in R with {targets}.
R
81
star
75

rebird

Wrapper to the eBird API
R
79
star
76

bikedata

🚲 Extract data from public hire bicycle systems
R
79
star
77

vcr

Record HTTP calls and replay them
R
78
star
78

historydata

Datasets for Historians
R
78
star
79

dittodb

dittodb: A Test Environment for DB Queries in R
R
78
star
80

arkdb

Archive and unarchive databases as flat text files
R
78
star
81

fingertipsR

R package to interact with Public Health England’s Fingertips data tool
R
78
star
82

CoordinateCleaner

Automated flagging of common spatial and temporal errors in biological and palaeontological collection data, for the use in conservation, ecology and palaeontology.
HTML
77
star
83

smapr

An R package for acquisition and processing of NASA SMAP data
R
77
star
84

nodbi

Document DBI connector for R
R
76
star
85

opentripplanner

An R package to set up and use OpenTripPlanner (OTP) as a local or remote multimodal trip planner.
R
73
star
86

nlrx

nlrx NetLogo R
R
71
star
87

rb3

A bunch of downloaders and parsers for data delivered from B3
R
69
star
88

tidyhydat

An R package to import Water Survey of Canada hydrometric data and make it tidy
R
69
star
89

robotstxt

robots.txt file parsing and checking for R
R
68
star
90

slopes

Package to calculate slopes of roads, rivers and trajectories
R
65
star
91

tradestatistics

R package to access Open Trade Statistics API
R
65
star
92

unconf17

Website for 2017 rOpenSci Unconf
JavaScript
64
star
93

roadoi

Use Unpaywall with R
R
64
star
94

codemetar

an R package for generating and working with codemeta
R
64
star
95

terrainr

Get DEMs and orthoimagery from the USGS National Map, georeference your images and merge rasters, and visualize with Unity 3D
R
64
star
96

NLMR

πŸ“¦ R package to simulate neutral landscape models πŸ”
R
63
star
97

parzer

Parse geographic coordinates
R
63
star
98

tiler

Generate geographic and non-geographic map tiles from R
R
63
star
99

rWBclimate

R interface for the World Bank climate data
R
62
star
100

comtradr

Functions for Interacting with the UN Comtrade API
R
60
star