• Stars
    star
    237
  • Rank 169,885 (Top 4 %)
  • Language
    HTML
  • License
    Apache License 2.0
  • Created over 2 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Datasette running in your browser using WebAssembly and Pyodide

Datasette Lite

Datasette running in your browser using WebAssembly and Pyodide

Live tool: https://lite.datasette.io/

More about this project:

How this works

Datasette Lite runs the full server-side Datasette Python web application directly in your browser, using the Pyodide build of Python compiled to WebAssembly.

When you launch the demo, your browser will download and start executing a full Python interpreter, install the datasette package (and its dependencies), download one or more SQLite database files and start the application running in a browser window (actually a Web Worker attached to that window).

Loading CSV data

You can load data from a CSV file hosted online (provided it allows access-control-allow-origin: *) by passing that URL as a ?csv= parameter - or by clicking the "Load CSV by URL" button and pasting in a URL.

This example loads a CSV of college fight songs from the fivethirtyeight/data GitHub repository:

You can pass ?csv= multiple times to load more than one CSV file. You can then execute SQL joins to combine that data.

This example loads the latest Covid-19 per-county data from the NY Times, the 2019 county populations data from the US Census, joins them on FIPS code and runs a query that calculates cases per million across that data:

https://lite.datasette.io/?csv=https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties-recent.csv&csv=https://raw.githubusercontent.com/simonw/covid-19-datasette/main/us_census_county_populations_2019.csv#/data?sql=select%0A++%5Bus-counties-recent%5D.%2C%0A++us_census_county_populations_2019.population%2C%0A++1.0++%5Bus-counties-recent%5D.cases+%2F+us_census_county_populations_2019.population+*+1000000+as+cases_per_million%0Afrom%0A++%5Bus-counties-recent%5D%0A++join+us_census_county_populations_2019+on+us_census_county_populations_2019.fips+%3D+%5Bus-counties-recent%5D.fips%0Awhere%0A++population+%3E+10000%0Aorder+by%0A++cases_per_million+desc

Loading JSON data

If you have data in a JSON file that looks something like this you can load it directly into Datasette Lite using the ?json=URL parameter:

[
  {
    "id": 1,
    "name": "Item 1"
  },
  {
    "id": 2,
    "name": "Item 2"
  }
]

This also works with JSON documents where one of the keys is a list of objects, such as this one:

{
  "rows": [
    {
      "id": 1,
      "name": "Item 1"
    },
    {
      "id": 2,
      "name": "Item 2"
    }
  ]
}

In this case it will search for the first key that contains a list of objects.

If a document is a JSON object where every value is a JSON object, like this:

{
  "anchor-positioning": {
    "spec": "https://drafts.csswg.org/css-anchor-position-1/#anchoring"
  },
  "array-at": {
    "spec": "https://tc39.es/ecma262/multipage/indexed-collections.html#sec-array.prototype.at"
  },
  "array-flat": {
    "caniuse": "array-flat",
    "spec": "https://tc39.es/ecma262/multipage/indexed-collections.html#sec-array.prototype.flat"
  }
}

Each of those objects will be loaded as a separate row, with a _key primary key column containing the object key.

This example loads scraped data from this repo.

Newline-delimited JSON works too - for example a file that looks like this:

{"id": 1, "name": "Item 1"}
{"id": 2, "name": "Item 2"}

Loading SQLite databases

You can use this tool to open any SQLite database file that is hosted online and served with a access-control-allow-origin: * CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using datasette publish.

Copy the URL to the .db file and either paste it into the "Load SQLite DB by URL" prompt, or construct a URL like the following:

https://lite.datasette.io/?url=https://latest.datasette.io/fixtures.db

Some examples to try out:

Loading Parquet

To load a Parquet file, pass a URL to ?parquet=.

For example this file:

https://github.com/Teradata/kylo/blob/master/samples/sample-data/parquet/userdata1.parquet

Can be loaded like this:

https://lite.datasette.io/?parquet=https://github.com/Teradata/kylo/blob/master/samples/sample-data/parquet/userdata1.parquet

Initializing with SQL

You can also initialize the data.db database by passing the URL to a SQL file. The easiest way to do this is to create a GitHub Gist.

This example SQL file creates a table and populates it with three records. It's hosted in this Gist.

https://gist.githubusercontent.com/simonw/ac4e19920b4b360752ac0f3ce85ba238/raw/90d31cf93bf1d97bb496de78559798f849b17e85/demo.sql

You can paste this URL into the "Load SQL by URL" prompt, or you can pass it as the ?sql= parameter like this.

SQL will be executed before any CSV imports, so you can use initial SQL to create a table and then use ?csv= to import data into it.

Starting with just an in-memory database

To skip loading the default databases and just provide /_memory - useful for demonstrating plugins - pass ?memory=1, for example:

https://lite.datasette.io/?memory=1

Loading metadata

Datasette supports metadata, as a metadata.json or metadata.yml file.

You can load a metadata file in either of these formats by passing a URL to the ?metadata= query string option.

Special handling of GitHub URLs

A tricky thing about using Datasette Lite is that the files you load via URL need to be hosted somewhere that serves open CORS headers.

Both regular GitHub and GitHub Gists do this by default. This makes them excellent options to host data files that you want to load into Datasette Lite.

You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you.

Try the following to see this in action:

Installing plugins

Datasette has a number of plugins that enable new features.

You can install plugins into Datasette Lite by adding one or more ?install=name-of-plugin parameters to the URL.

Not all plugins are compatible with Datasette Lite at the moment, for example plugins that load their own JavaScript and CSS do not currently work, see issue #8.

Here's a list of plugins that have been tested with Datasette Lite, plus demo links to see them in action:

Analytics

By default, hits to https://lite.datasette.io/ are logged using Plausible.

Plausible is a privacy-focused, cookie-free, GDPR-compliant analytics system.

Each navigation within Datasette Lite is logged as a separate event to Plausible, capturing the fragment hash and the URL to the currently loaded file.

The site is hosted on GitHub Pages, which does not offer any analytics that are visible to the site owner. GitHub Pages can only log visits to the https://lite.datasette.io/ root page - it will not have visibility into any subsequent # fragment navigation.

To opt out of analytics, you can add ?analytics=off or &analytics=off to the URL. This will prevent any analytics being sent to Plausible.

More Repositories

1

datasette

An open source multi-tool for exploring and publishing data
Python
7,807
star
2

sqlite-utils

Python CLI utility and library for manipulating SQLite databases
Python
1,191
star
3

shot-scraper

A command-line utility for taking automated screenshots of websites
Python
1,006
star
4

csvs-to-sqlite

Convert CSV files into a SQLite database
Python
758
star
5

til

Today I Learned
HTML
719
star
6

django-sql-dashboard

Django app for building dashboards using raw SQL queries
Python
400
star
7

simonw

https://simonwillison.net/2020/Jul/10/self-updating-profile-readme/
Python
362
star
8

llm

Access large language models from the command-line
Python
309
star
9

db-to-sqlite

CLI tool for exporting tables or queries from any SQL database to a SQLite file
Python
302
star
10

djangode

Utilities functions for node.js that borrow some useful concepts from Django
JavaScript
256
star
11

csv-diff

Python CLI tool and library for diffing CSV and JSON files
Python
238
star
12

shot-scraper-template

Template repository for setting up shot-scraper
217
star
13

geocoders

Ultra simple API for geocoding a single string against various web services.
Python
184
star
14

ca-fires-history

Tracking fire data from www.fire.ca.gov
165
star
15

django-openid

A modern library for integrating OpenID with Django - incomplete, but really nearly there (promise)
Python
163
star
16

openai-to-sqlite

Save OpenAI API results to a SQLite database
Python
161
star
17

action-transcription

A tool for creating a repository of transcribed videos
Python
158
star
18

s3-credentials

A tool for creating credentials for accessing S3 buckets
Python
149
star
19

git-history

Tools for analyzing Git history using SQLite
Python
147
star
20

google-drive-to-sqlite

Create a SQLite database containing metadata from Google Drive
Python
142
star
21

django-queryset-transform

Experimental .transform(fn) method for Django QuerySets, for clever lazily evaluated optimisations.
Python
142
star
22

ratelimitcache

A memcached backed rate limiting decorator for Django.
Python
141
star
23

optfunc

Syntactic sugar for creating Python command line scripts by introspecting a function definition
Python
134
star
24

djng

Turtles all the way down
Python
129
star
25

cougar-or-not

An API for identifying cougars v.s. bobcats v.s. other USA cat species
Jupyter Notebook
119
star
26

simonwillisonblog

The source code behind my blog
JavaScript
118
star
27

advent-of-code-2022-in-rust

Copilot-assisted Advent of Code 2022 to learn Rust
Rust
114
star
28

djangopeople.net

A geographical community site for Django developers.
Python
111
star
29

scrape-chatgpt-plugin-prompts

Shell
107
star
30

s3-ocr

Tools for running OCR against files stored in S3
Python
103
star
31

datasette-app

The Datasette macOS application
JavaScript
100
star
32

django-redis-monitor

Request per second / SQLop per second monitoring for Django, using Redis for storage
Python
97
star
33

python-lib

Opinionated cookiecutter template for creating a new Python library
Python
97
star
34

ttok

Count and truncate text based on tokens
Python
96
star
35

mytweets

Script for saving a JSON archive of your tweets.
Python
81
star
36

airtable-export

Export Airtable data to YAML, JSON or SQLite files on disk
Python
79
star
37

datasette-graphql

Datasette plugin providing an automatic GraphQL API for your SQLite databases
Python
77
star
38

llm-mlc

LLM plugin for running models using MLC
Python
74
star
39

strip-tags

CLI tool for stripping tags from HTML
Python
73
star
40

django_cropper

Integration of jCrop with the Django admin
Python
71
star
41

click-app

Cookiecutter template for creating new Click command-line tools
Python
70
star
42

datasette-ripgrep

Web interface for searching your code using ripgrep, built as a Datasette plugin
Python
69
star
43

download-esm

Download ESM modules from npm and jsdelivr
Python
67
star
44

datasette.io

The official project website for Datasette
HTML
66
star
45

ftfy-web

Paste in some broken unicode text and FTFY will tell you how to fix it!
Python
63
star
46

markdown-to-sqlite

CLI tool for loading markdown files into a SQLite database
Python
63
star
47

sqlite-history

Track changes to SQLite tables using triggers
Python
62
star
48

yaml-to-sqlite

Utility for converting YAML files to SQLite
Python
62
star
49

sqlite-diffable

Tools for dumping/loading a SQLite database to diffable directory structure
Python
62
star
50

covid-19-datasette

Deploys a Datasette instance of COVID-19 data from Johns Hopkins CSSE and the New York Times
Python
61
star
51

dogproxy

Experimental HTTP proxy (using node.js) for avoiding the dog pile effect.
JavaScript
61
star
52

soupselect

CSS selector support for BeautifulSoup.
Python
60
star
53

laion-aesthetic-datasette

Use Datasette to explore LAION improved_aesthetics_6plus training data used by Stable DIffusion
Python
58
star
54

datasette-cluster-map

Datasette plugin that shows a map for any data with latitude/longitude columns
JavaScript
55
star
55

action-transcription-demo

A tool for creating a repository of transcribed videos
Python
53
star
56

datasette-vega

Datasette plugin for visualizing data using Vega
JavaScript
52
star
57

pge-outages-pre-2024

Tracking PG&E outages
Python
51
star
58

google-calendar-to-sqlite

Create a SQLite database containing your data from Google Calendar
Python
50
star
59

url-map

Use URL parameters to generate a map with markers, using Leaflet and OpenStreetMap
HTML
49
star
60

disaster-scrapers

Scrapers for disaster data - writes to https://github.com/simonw/disaster-data
Python
46
star
61

djp

A plugin system for Django
Python
46
star
62

geojson-to-sqlite

CLI tool for converting GeoJSON files to SQLite (with SpatiaLite)
Python
45
star
63

asgi-csrf

ASGI middleware for protecting against CSRF attacks
Python
44
star
64

datasette-chatgpt-plugin

A Datasette plugin that turns a Datasette instance into a ChatGPT plugin
Python
44
star
65

nodecast

A simple comet broadcast server, originally implemented as a demo for Full Frontal 2009.
JavaScript
44
star
66

bugle_project

Group collaboration tools for hackers in forts.
Python
42
star
67

django-html

A way of rendering django.forms widgets that differentiates between HTML and XHTML.
Python
42
star
68

datasette-auth-github

Datasette plugin that authenticates users against GitHub
Python
41
star
69

puppeteer-screenshot

Vercel app for taking screenshots of web pages using Puppeteer
JavaScript
40
star
70

llm-replicate

LLM plugin for models hosted on Replicate
Python
40
star
71

python-lib-template-repository

GitHub template repository for creating new Python libraries, using the simonw/python-lib cookiecutter template
39
star
72

django-signed

Signing utilities for Django, to try out an API which is being proposed for inclusion in Django core.
Python
37
star
73

museums

A website recommending niche museums to visit
JavaScript
36
star
74

pypi-rename

Cookiecutter template for creating renamed PyPI packages
Python
36
star
75

help-scraper

Record a history of --help for various commands
Python
35
star
76

dbf-to-sqlite

CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite
Python
35
star
77

disaster-data

Data scraped by https://github.com/simonw/disaster-scrapers
35
star
78

asyncinject

Run async workflows using pytest-fixtures-style dependency injection
Python
35
star
79

datasette-publish-vercel

Datasette plugin for publishing data using Vercel
Python
34
star
80

gzthermal-web

A web interface to gzthermal by caveman on encode.ru
Python
32
star
81

asgi-auth-github

ASGI middleware that authenticates users against GitHub
Python
31
star
82

json-head

JSON microservice for performing HEAD requests
Python
31
star
83

django-safeform

CSRF protection for Django forms.
Python
31
star
84

s3-image-proxy

A tiny proxy for serving and resizing images fetched from a private S3 bucket
Python
31
star
85

sqlite-transform

Tool for running transformations on columns in a SQLite database
Python
30
star
86

webhook-relay

A simple Node.js server for queueing and relaying webhook requests
JavaScript
30
star
87

datasette-tiddlywiki

Run TiddlyWiki in Datasette and save Tiddlers to a SQLite database
HTML
29
star
88

image-diff

CLI tool for comparing images
Python
29
star
89

getlatlon.com

Source code for getlatlon.com - a simple, single page, pure JavaScript Google Maps application.
29
star
90

sf-tree-history

Tracking the history of trees in San Francisco
29
star
91

scrape-hacker-news-by-domain

Scrape HN to track links from specific domains
JavaScript
28
star
92

timezones-api

A Datasette-powered API for finding the time zone for a latitude/longitude point
Python
26
star
93

owlsnearme

A website that tells you where your nearest owls are!
JavaScript
26
star
94

datasette-table

A Web Component for embedding a Datasette table on a page
JavaScript
26
star
95

xml-analyser

Simple command line tool for quickly analysing the structure of an arbitrary XML file
Python
26
star
96

shapefile-to-sqlite

Load shapefiles into a SQLite (optionally SpatiaLite) database
Python
26
star
97

cdc-vaccination-history

A git scraper recording the CDC's Covid Data Tracker numbers on number of vaccinations per state.
Python
24
star
98

json-flatten

Python functions for flattening a JSON object to a single dictionary of pairs, and unflattening that dictionary back to a JSON object
Python
24
star
99

datasette-json-html

Datasette plugin for rendering HTML based on JSON values
Python
24
star
100

djangocon-2022-productivity

Supporting links for my DjangoCon 2022 talk
23
star