RSocrata

⚠️ Important Notice (Temporary -- August 24, 2023)

RSocrata is temporarily unavailable on CRAN due to a recent removal related to complications when testing external URLs. We are actively working to address the issue and we may move to reduce the amount of testing that we perform until we can find a more robust solution for error handling. The team is committed to resolving this as quickly as possible, and we anticipate having RSocrata restored on CRAN within the next week or so.

In the meantime, please use devtools to install RSocrata directly from GitHub:

devtools::install_github("Chicago/RSocrata")

For detailed information on the issue, please refer to #213 Fail gracefully when Socrata is down and to #214 Remove tests that rely on external URLs for CRAN tests (to address CRAN removal).

We appreciate your understanding and patience as we work through this matter.

Master

Stable beta branch. Test about-to-be-released features in a stable pre-release build before it is submitted to CRAN.

Dev

"Nightly" alpha branch. Test the latest features and bug fixes -- enjoy at your own risk.

A tool for downloading and uploading Socrata datasets

Provided with a URL to a dataset resource published on a Socrata webserver, or a Socrata SoDA (Socrata Open Data Application Program Interface) web API query, or a Socrata "human-friendly" URL, read.socrata() returns an R data frame. Converts dates to POSIX format. Supports CSV and JSON download file formats from Socrata. Manages the throttling of data returned from Socrata and allows users to provide an application token. Supports SoDA query parameters in the URL string for further filtering, sorting, and queries. Upload data to Socrata data portals using "upsert" and "replace" methods.

Use ls.socrata() to list all datasets available on a Socrata webserver.

testthat test coverage.

Installation

To get the current released version from CRAN:

install.packages("RSocrata")

The most recent beta with soon-to-be-released changes can be installed from GitHub:

# install.packages("devtools")
devtools::install_github("Chicago/RSocrata")

The "nightly" version with the most recent bug fixes and features is also available. This version is always an alpha and may contain significant bugs. You can install it from the dev branch from GitHub:

# install.packages("devtools")
devtools::install_github("Chicago/RSocrata", ref="dev")

Examples

Reading SoDA valid URLs

earthquakesDataFrame <- read.socrata("https://soda.demo.socrata.com/resource/4334-bgaj.csv")
nrow(earthquakesDataFrame) # 1007 (two "pages")
class(earthquakesDataFrame$Datetime[1]) # POSIXlt

Reading "human-readable" URLs

earthquakesDataFrame <- read.socrata("https://soda.demo.socrata.com/dataset/USGS-Earthquakes-for-2012-11-01-API-School-Demo/4334-bgaj")
nrow(earthquakesDataFrame) # 1007 (two "pages")
class(earthquakesDataFrame$Datetime[1]) # POSIXlt

Using API key to read datasets

token <- "ew2rEMuESuzWPqMkyPfOSGJgE"
earthquakesDataFrame <- read.socrata("https://soda.demo.socrata.com/resource/4334-bgaj.csv", app_token = token)
nrow(earthquakesDataFrame)

Download private datasets from portal

# Store user email and password
socrataEmail <- Sys.getenv("SOCRATA_EMAIL", "[email protected]")
socrataPassword <- Sys.getenv("SOCRATA_PASSWORD", "7vFDsGFDUG")

privateResourceToReadCsvUrl <- "https://soda.demo.socrata.com/resource/a9g2-feh2.csv" # dataset

read.socrata(url = privateResourceToReadCsvUrl, email = socrataEmail, password = socrataPassword)

List all datasets on portal

allSitesDataFrame <- ls.socrata("https://soda.demo.socrata.com")
nrow(allSitesDataFrame) # Number of datasets
allSitesDataFrame$title # Names of each dataset

Upload data to portal

# Store user email and password
socrataEmail <- Sys.getenv("SOCRATA_EMAIL", "[email protected]")
socrataPassword <- Sys.getenv("SOCRATA_PASSWORD", "7vFDsGFDUG")

datasetToAddToUrl <- "https://soda.demo.socrata.com/resource/xh6g-yugi.json" # dataset
 
# Generate some data
x <- sample(-1000:1000, 1)
y <- sample(-1000:1000, 1)
df_in <- data.frame(x,y)
 
# Upload to Socrata
write.socrata(df_in,datasetToAddToUrl,"UPSERT",socrataEmail,socrataPassword)

Issues

Please report issues, request enhancements or fork us at the City of Chicago github.

Contributing

If you would like to contribute to this project, please see the contributing documentation and the product roadmap.