• Stars
    star
    106
  • Rank 325,871 (Top 7 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 11 years ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

The code behind okfn.org

okfn.org

Run tests codecov

This is the Django/Django CMS project that runs http://okfn.org.

When do I need to modify this?

http://okfn.org/ runs on Django CMS.

Most of the contents are provided via plugins and our communication person is currently responsible to edit the content or add new stuff to the site. It is the idea, and the goal of the development that all content is dinamically handled by plugins. This means that we are aiming to have templates with placeholders instead of text. So, if you need to modify content, probably you can already do it using Django CMS built in capabilities.

If we need to make global styling or data model changes, then we need to edit the code in this repo.

Prerequisites and assumptions

You must have the following installed:

  • Python 3.10
  • Node JS 16

The /Dockerfile (used for staging/production) and the requirements file (built with pip-tools) in this repo shows you the application dependencies.

Development

Database

It is possible to run the app with no database content. If databaset settings are not set, we fall back to a blank SQLite database (defined in the .env.base file) so this step can be skipped for a minimal case.

To run with a database, you will need a local Postgres server.

  • Get a dump of the database (we store them as *.sql in Google Cloud Storage)
  • Create a local database sudo -u postgres createdb okfn
  • Populate it sudo -u postgres psql okfn < file=/path/to/dump.sql or sudo -u postgres pg_restore -d okfn path/to/dump.dump

Running the application

Cherry-pick environment variables from .env.example you'd like to set and add it to the .env file. If no env vars are set, we will fall back to default .env.base file.

If running with Postgres, you will need to set:

DB_ENGINE=django.db.backends.postgresql_psycopg2
DB_NAME=okfn
DB_USER=user
DB_PASSWORD=pass
DB_HOST=localhost
DB_PORT=5432

Prepare the app:

Make sure to have the correct node version:

nvm install 16
nvm use 16

Create a Python 3.10 local environment (e.g. python3.10 -m venv ~/okf-website-env)

pip install -r requirements.txt
pip install -r requirements.dev.txt
npm install
python manage.py migrate
python manage.py update_index

Start the server:

python manage.py runserver

Another option is to use Docker.

docker build -t okfn .
docker run -d -p 8888:80 okfn

File uploads

Because this is a CMS project, a lot of the site content is created via the web UI.
This means a lot of the images on the site are file uploads.
In staging and production, the uploaded files are hosted on Google Cloud Storage.

If you want to get all production/staging media files you can download them from Google Cloud Storage and put them in a local media directory (MEDIA_ROOT from settings).

Note: A dump of the production database will require all the media files of production. Same for staging.

Frontend

CSS

Currently we are using Tailwind, PostCSS and Vite as our CSS stack. Please check the documentation to get a sense of how it works: Installing Tailwind CSS as a PostCSS plugin

The css build is done by vite and the configuration files for it are vite.config.js, tailwind.config.cjs and postcss.config.cjs.

Running npm run build will compile our main styles.css file and place it in static/css/styles.css. (It then will be collected by Django when building the Dockerfile)

**Important: ** If you use a new tailwind class in some plugin, this class must be added somewhere on the file templates/styles.html, otherwise the class will not work when used on the plugin.

Javascript

  1. Clone the repository https://github.com/ishigami/okfn_front
  2. npm i
  3. npm run build
  4. copy ../dist/assets/main-[HASH].js to /static/js/scripts.js

Backend (Django and django-cms)

New plugins for blocks of content are located on foundation/okfplugins.

  1. Create a folder on foundation/okfplugins having:
  • admin.py: used on Django Admin interface
  • cms_plugin.py: configuration to appear on the Django CMS content editor sidebar
  • templates/: the django template files (access the plugin variable defined on models.py with instance.VARIABLE)
  • models.py: the model for the plugin object on the database

You can copy a simple plugin as heading or just_text and change the files above. If you do that don't forge to delete the migration files inside migrations/, except by init.py.

  1. Add the plugin to foundation/settings.py (INSTALLED_APPS setting).

  2. Run python manage.py makemigrations after you create the models.py.

  3. Run the migration to change the database with python manage.py migrate

  4. Start the server with python manage.py runserver

References for new design and components

Deployment

Production deployment is based on this Dockerfile. When we want to deploy, we just push to main branch (or develop branch for staging environment).

For more info read this doc.

Dependency Management

Dependencies are managed with pip-tools. Add new packages to requirements.in / requirements.dev.in and compile requirements.txt / requirements.dev.txt with pip-compile.

You can run pip list --outdated to see outdated packages.

Changelog

All changes must be documented at the CHANGELOG file

More Repositories

1

messytables

Tools for parsing messy tabular data. This is now superseded by https://github.com/frictionlessdata/tabulator-py
HTML
389
star
2

facetview

FacetView is a pure javascript frontend for ElasticSearch.
JavaScript
292
star
3

timemapper

Create and share elegant timelines and timemaps fast
JavaScript
262
star
4

opendatahandbook

Open Data Handbook v2
SCSS
202
star
5

handbook

Guides and introductions for participating in Labs and some of its projects.
CSS
169
star
6

opendataeditor

The Open Data Editor (ODE) is a no-code application to explore, validate and publish data in a simple way. Forever free and open source project powered by the Frictionless Framework.
TypeScript
158
star
7

bubbletree

Radial Bubble Tree Visualization
JavaScript
153
star
8

textus

A toolbox and web application for working with and presenting textual material from Shakespeare to Schopenhauer, and letters to literature.
JavaScript
148
star
9

dataexplorer

View, visualize, clean and process data in the browser.
JavaScript
146
star
10

opendatasurvey

The Open Data Survey application
JavaScript
137
star
11

bibserver

BibServer is open-source software what makes it easy to publish, manage and find bibliographies. BibServer is RESTful and web-friendly.
Python
126
star
12

labs-opd

HTML
123
star
13

dataportals.org

Open Data Portals and Sites around the world
HTML
116
star
14

reconcile-csv

A simple OpenRefine reconciliation service that runs on top of a CSV file
Clojure
113
star
15

opendefinition

Open Definition source
HTML
109
star
16

dataconverters

Python library and command line tool for converting data from one format to another
Python
99
star
17

timeliner

[DEPRECATED] See new TimeMapper repo https://github.com/okfn/timemapper
JavaScript
93
star
18

opendataday

Open Data Day website
HTML
86
star
19

docker-ckan

Docker images and Docker Compose setup for CKAN
Shell
82
star
20

okfn.github.com

Open Knowledge Labs website (and general issue tracker).
Jupyter Notebook
78
star
21

elasticsearch.js

A simple javascript library for working with ElasticSearch
JavaScript
78
star
22

wikipediajs

A small Javascript library for accessing information in Wikipedia articles such as dates, places, abstracts and more
JavaScript
77
star
23

dataproxy

Web application (targeted at appengine) to proxy data from certain data types into a JSON-P data type so that users can create mashups against remote data sets.
Python
74
star
24

opendatahandbook-old

Open Data Handbook
HTML
68
star
25

infra

Repository for OKF infrastructure and systems administration tools
Python
64
star
26

opd-product-browser-web

A repository of code, documentation and database scripts for the Open Product Data project.
PHP
64
star
27

licenses

Open source and open knowledge (data and content) licenses together with API and web service.
JavaScript
64
star
28

ckan.js

Javascript client library for CKAN with support for Node and the browser. It also provides a Recline compatible backend.
JavaScript
54
star
29

vdm

Versioned domain model. Python library for revisioning/versioning of databases.
Python
41
star
30

webstore

webstore is a web-api enabled datastore backed onto sql databases especially sqlite. It supports the RESTful JSON APIs standard to nosql dbs like CouchDB, MongoDB, Riak but with the power, reliability and familiarity of SQL and RDBMS.
Python
39
star
31

opendataindex

Open Data Index website
CSS
37
star
32

opd-product-browser-android

Android application for the Open Product Data project
Java
36
star
33

coding-standards

Open Knowledge coding standards and style guide.
35
star
34

ofs

A simple python-based abstraction library for the various blob storage out there including s3, google storage and local disk.
Python
32
star
35

listify

Take a Google Doc spreadsheet and turn it into a beautiful, searchable listing on your website - embed in seconds into Wordpress, simple HTML or ...
HTML
28
star
36

dashboard

Dashboard for communities: what's up, who's here and what are they working on. Designed for use at the Open Knowledge Foundation.
CoffeeScript
27
star
37

sqlaload

REPLACED BY dataset
Python
26
star
38

ckanclient-deprecated

DEPRECATED - please see https://github.com/ckan/ckanapi. [Python client library for CKAN]
Python
25
star
39

docker-rt

A full installation of RT in a docker image. To be used as a base for customisation.
Shell
22
star
40

bad-data

Examples of bad data, especially from government.
HTML
22
star
41

ddjbook

Data Driven Journalism Handbook
JavaScript
22
star
42

data-catalog-spec

Data Catalog Specification (Schema and Protocol)
Python
20
star
43

ckanjs-deprecated

DEPRECATED - see https://github.com/okfn/ckan.js [Javascript client library and app for CKAN data hub software]
JavaScript
20
star
44

gobetween

A simple app for proxying requests with CORS support. Map any domain to any URI as a base path, or, use a dedicated endpoint for proxying any URI.
JavaScript
20
star
45

helmut

A generic Google Refine Reconciliation API implementation
Python
19
star
46

oed

Extraction and Interface for Oxford English Dictionary (OED) 1st Edition
Python
19
star
47

datautil

Utilities for working with data.
Python
18
star
48

shakespeare-material

An open (free to use, reuse and redistribute) set of Shakespeare material including plays, poems, images and scripts.
Python
17
star
49

bibjson

BibJSON spec and website
HTML
17
star
50

measure

Measure is scripts and conventions to build KPI dashboards for projects.
Python
17
star
51

textus-viewer

A sophisticated HTML-based viewer for texts
JavaScript
17
star
52

opendatacommons.org

Open Data Commons licenses and website
SCSS
17
star
53

handbook-theme

Open Knowledge standard "nice" jekyll theme - used for OpenDataHandbook etc
CSS
16
star
54

datadigitizer

Crowd-sourcing digitization of data
JavaScript
15
star
55

pdcalc

Public Domain Calculators - determine what is public domain and what's not.
Python
14
star
56

recline.backend.gdocs

Javascript library for accessing Google Docs Spreadsheet - and also functions as a Recline backend!
JavaScript
14
star
57

rtei

Right to Education Index website
Python
14
star
58

ckanext-s3filestore

Use Amazon S3 as a filestore for CKAN
Python
14
star
59

yourtopia

Web app for crowdsourcing preferences about index weighting such as the Human Development Index. Powers the YourTopia site - Global Development beyond GDP.
JavaScript
14
star
60

ckanext-wordpresser

Wordpress integration for CKAN
Python
13
star
61

sphinx-theme-okfn

Standard Open Knowledge Foundation sphinx theme resembling that on http://okfn.org/
CSS
12
star
62

iatitools

Tools with working with IATI data from http://iatiregistry.org/
Python
12
star
63

a

A is for assets. This repository holds OKI web assets, pushed using CI to the S3 bucket behind:
SCSS
11
star
64

ckanext-datahub

Plugin for datahub
CSS
11
star
65

webstore-client

Python client library for webstore
Python
10
star
66

opd-brand-manager

A web interface for managing a directory of brands for the Open Product Data working group.
Python
10
star
67

ckanext-example

CKAN Extension that demonstrates some common patterns for customising a CKAN instance.
Python
10
star
68

gutenizer

Get texts from Project Gutenberg, extract and format
Python
10
star
69

moinmoin2mediawiki

Convert moinmoin data to mediawiki
Perl
9
star
70

ckan-drupal

Everything to do with ckan drupal integration.
PHP
9
star
71

docker-fiware-ckan

Docker and Docker compose configuration for the FIWARE CKAN instance
Python
9
star
72

ckanext-pdeu

Custom CKAN Extension for PublicData.eu
Python
9
star
73

beagle

Watch web resources for changes and email assigned users when changes happen to sites they're assigned to watch.
Python
9
star
74

recline.backend.couchdb

CouchDB backend for Recline
JavaScript
9
star
75

opendatasites

Dataset of open data websites (portals and catalogs) from around the world based on http://datacatalogs.org/
8
star
76

ibp-explorer

[ARCHIVED] Data Explorer for the Open Budget Survey, built in collaboration with the International Budget Partnership.
JavaScript
8
star
77

opd-product-browser-ios

iOS browser client for the Open Product Data project
8
star
78

dataexplorer-2011

This repo is deprecated and has been replaced by http://github.com/okfn/recline. Pure JS app for viewing and visualizing data
JavaScript
8
star
79

crime

Experimental crime dashboards
JavaScript
8
star
80

ckanext-lacounts

CKAN extension for the LA Counts project
HTML
8
star
81

okfn.org

Issue tracker for http://okfn.org/ and other Open Knowledge Foundation websites
PHP
8
star
82

wordpress-theme

OKFN corporate wordpress theme
PHP
7
star
83

oki-py

This is an example Python repository implementing OKI coding standards.
Python
7
star
84

tellme

TellMe is a toolkit to create *user-facing reports* from things happening in code.
Python
7
star
85

shakespeare

Open Shakespeare platform. Repository for managing and code and service for http://openshakespeare.org/ and related websites.
Python
7
star
86

ckan-google-docs

Google docs app script.
JavaScript
7
star
87

network

Main repository for the Open Knowledge Network.
HTML
7
star
88

opendatacommons

Legacy Open Data Commons website. Moved to https://github.com/okfn/opendatacommons.org
CSS
7
star
89

ckanext-vegaview

VegaJS' view extension for CKAN
JavaScript
7
star
90

ckanext-rdf

[DEPRECATED] - Please use https://github.com/ckan/ckanext-dcat
6
star
91

recline.backend.solr

A simple javascript library that wraps the SOLR JSON API and is usable by Recline
JavaScript
6
star
92

elastic-proxy

elastic search on the web
Python
6
star
93

ckanext-inspire

Python
6
star
94

data.okfn.org-new

Simple data package based data portal (and original site for frictionless data effort)
HTML
6
star
95

datapackage_pipelines_od4tj

Open Data for Tax Justice scraping pipelines and processors
Python
6
star
96

ckanext-mx

Theme for datos.gob.mx
CSS
6
star
97

climate-request

A new project at Open Knowledge Foundation allowing users to request climate data.
6
star
98

cop21

Negotiating texts for COP21, the 21st Conference of Parties (COP21) to the UNFCCC in Paris December 2015.
HTML
6
star
99

githubActivity.js

A simple javascript widget to provide a nice HTML list Github activity in a set of repositories
CoffeeScript
6
star
100

vendorjs

This is a common repo for storing javascript (and some css) vendor libraries for use as a submodule across a variety of Open Knowledge Foundation (or other) projects.
JavaScript
5
star