sqlite-http
A SQLite extension for making HTTP requests purely in SQL.
- Create GET, POST, and other HTTP requests, like
curl,wget, andfetch - Download response bodies, header, status codes, timing info
- Set rate limits, timeouts
Usage
.load ./http0
select http_get_body('https://text.npr.org/');
/*
<!DOCTYPE html>
<html lang="en">
<head>
<title>NPR : National Public Radio</title>
....
*/Query for all custom headers in an endpoint.
select name, value
from http_headers_each(
http_get_headers('https://api.github.com/')
)
where name like 'X-%';
/*
ββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
β name β value β
ββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββ€
β X-Ratelimit-Limit β 60 β
β X-Ratelimit-Used β 8 β
β X-Content-Type-Options β nosniff β
β X-Github-Media-Type β github.v3; format=json β
β X-Github-Request-Id β CCCA:5FDF:1014BC2:10965F9:62F3DE4E β
β X-Ratelimit-Remaining β 52 β
β X-Ratelimit-Resource β core β
β X-Frame-Options β deny β
β X-Ratelimit-Reset β 1660152798 β
β X-Xss-Protection β 0 β
ββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββ
*/Scrape data from a JSON endpoint.
select http_get_body('https://api.github.com/repos/sqlite/sqlite')
->> '$.description' as description;
/*
βββββββββββββββββββββββββββββββββββββββββββββββββ
β description β
βββββββββββββββββββββββββββββββββββββββββββββββββ€
β Official Git mirror of the SQLite source tree β
βββββββββββββββββββββββββββββββββββββββββββββββββ
*/Pass in specific headers in a request.
select
value
from json_each(
http_get_body(
'https://api.github.com/issues',
http_headers(
'Authorization', 'token ghp_16C7e42F292c6912E7710c8'
)
)
);
Documentation
See docs.md for a full API reference.
Installing
| Language | Install | |
|---|---|---|
| Python | pip install sqlite-http |
 |
| Datasette | datasette install datasette-sqlite-http |
 |
| Node.js | npm install sqlite-http |
 |
| Deno | deno.land/x/sqlite_http |
 |
| Ruby | gem install sqlite-http |
 |
| Github Release |  |
The Releases page contains pre-built binaries for Linux amd64, MacOS amd64 (no arm), and Windows.
As a loadable extension
If you want to use sqlite-http as a Runtime-loadable extension, Download the http0.dylib (for MacOS), http0.so (Linux), or http0.dll (Windows) file from a release and load it into your SQLite environment.
Note: The
0in the filename (http0.dylib/http0.so/http0.dll) denotes the major version ofsqlite-http. Currentlysqlite-httpis pre v1, so expect breaking changes in future versions.
For example, if you are using the SQLite CLI, you can load the library like so:
.load ./http0
select http_version();
-- v0.0.1Or in Python, using the builtin sqlite3 module:
import sqlite3
con = sqlite3.connect(":memory:")
con.enable_load_extension(True)
con.load_extension("./http0")
print(con.execute("select http_version()").fetchone())
# ('v0.0.1',)Or in Node.js using better-sqlite3:
const Database = require("better-sqlite3");
const db = new Database(":memory:");
db.loadExtension("./http0");
console.log(db.prepare("select http_version()").get());
// { 'http_version()': 'v0.0.1' }Or with Datasette, with the "no network" option to limit DDoS attacks:
datasette data.db --load-extension ./http0-no-net
See also
- sqlite-html, for parsing and querying HTML using CSS selectors in SQLite (pairs great with this tool)
- pgsql-http, a similar yet very different HTTP library for POstgreSQL (didn't know about this before I started this, but interestingly enough came up with a very similar API)
- riyaz-ali/sqlite, the brilliant Go library that this library depends on
- nalgeon/sqlean, several pre-compiled handy SQLite functions, in C