• Stars
    star
    122
  • Rank 292,031 (Top 6 %)
  • Language
    Elixir
  • Created about 7 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Rendre disponible, valoriser et améliorer les données transports

Transport

This is the repository of the french National Access Point (NAP) for mobility data.

This project brings a mobility focus on data hosted on data.gouv.fr, the french open data portal.

You will find user documentation at doc.transport.data.gouv.fr.

A status dashboard is available at https://stats.uptimerobot.com/q7nqyiO9yQ for a part of the project.

Glossary

A small glossary explaining the various terms can be found in this repo (glossary.md). Please feel free to add any term which appears initially foreign there.

Installation

You can install this 2 different ways:

  • manually, this is the best way to install it if you plan to work often on the project.
  • with docker, this is an easier installation process at the cost of a slightly more cumbersome development workflow.

Manual installation

  • Make sure you have Elixir, Node, Yarn and Docker installed and up-to-date
    • Elixir is often installed with asdf since it makes it easy to handle different Elixir versions accross projects. The project needs at least Elixir 1.8 and Erlang 21.0
  • Install Elixir dependencies with mix deps.get
  • Install Node.js dependencies with mix yarn install

If you wish to use asdf (recommended), make sure to install the correct plugins:

Installation can then be done with:

  • asdf install

Postgresql

You also need an up to date postgresql with postgis installed. Version 14+ is recommended.

For Mac users, you can use https://postgresapp.com/.

Dependencies

Download depencies using mix deps.get.

Reply "Yes" to the question "Shall I install Hex? (if running non-interactively, use "mix local.hex --force")".

Creating a database

Create the database with the command mix ecto.create.

Alternatively, you can create it manually. With the permission to create a database (on Debian based system, you need to be logged as postgres), type createdb transport_repo.

Applying the migrations

To have an up to date database schema run mix ecto.migrate.

Restoring the production database

The production database does not contains any sensitive data, you can retreive it for dev purpose.

  • You can retrieve the latest clever-cloud backup (you need some permissions to access it, if you don't have them, you can ask someone on the team to give you the database)
  • On the clever-cloud website, under transport-site-postgresql, there is a Backups section with download links.
  • restore the downloaded backup on you database: ./restore_db.sh <path_to_the_backup>

Binary CLI dependencies

The app uses a number of tools via transport-tools.

They are expected at ./transport-tools by default (but this can be configured via :transport_tools_folder in config.exs).

When working locally, you may want to have these tools readily available at times.

mkdir transport-tools
cd transport-tools

# jars are cross-platform, so we can copy them from the container (see `transport-tools` repository for exact naming)
# here we share the host folder `transport-site/transport-tools` with a folder inside the container named `/tmp-docker-folder`,
# in order to copy back the exact same jars we use in production
docker run --rm -v $(pwd):/tmp-docker-folder ghcr.io/etalab/transport-tools:latest /bin/sh -c "cp /usr/local/bin/*.jar /tmp-docker-folder"

For Rust binaries, you will have to compile them locally and copy them to the same folder.

Once this is done, make sure to configure your configuration via :transport_tools_folder.

Usage

Run the server with mix phx.server and you can visit 127.0.0.1:5000 on your browser.

Usage of the Elixir Proxy

apps/unlock is a sub-part of the "umbrella app", which is served on its own subdomain (https://proxy.transport.data.gouv.fr for production, https://proxy.prochainement.transport.data.gouv.fr/ for staging).

The proxy relies on this yaml configuration which is currently fetched at runtime once (but can be hot-reloaded via this backoffice page).

Each proxied "feed" (currently GTFS-RT data) has a private (target) url hidden from the general public, can be configured with an independent Time-To-Live (TTL), and is exposed as a credential-free public url to the public. When a query occurs, the incoming HTTP connection is kept on hold while the proxy issues a query to the target server, caching the response in RAM based on the configured TTL.

The backoffice implementation leverages LiveView to provide an automatically updated dashboard view with all the feeds, the size of the latest payload, the latest HTTP code returned by the target etc. Implementation is here.

When working in development, instead of fetching the configuration from GitHub, the configuration is taken from a local config file (config/proxy-config.yml, see config), in order to make it very easy to play with sample configurations locally.

For local work, you will have (for now at least) to add proxy.localhost 127.0.0.1 to your /etc/hosts file.

The app currently routes whatever starts with proxy. to the proxy (as implemented here), although in the future we will probably use a more explicit configuration.

Configuring OAuth to work with demo.data.gouv.fr

By default the development configuration is very simple and only allows the most basic scenarios.

If you need to login via demo.data.gouv.fr, follow these steps:

  • Create or edit config/dev.secret.exs
  • Add:
config :oauth2, Datagouvfr.Authentication,
  # go to CleverCloud staging site and pick `DATAGOUVFR_CLIENT_ID`
  client_id: "TODO-REPLACE",
  # same but use `DATAGOUVFR_CLIENT_SECRET`
  client_secret: "TODO-REPLACE"

Then make sure to restart the app.

The rest of the configuration is already set via dev.exs, with:

config :oauth2, Datagouvfr.Authentication,
  # SNIP
  site: "https://demo.data.gouv.fr",
  redirect_uri: "http://localhost:5000/login/callback"

Development

Testing

Running the tests

Run the tests with mix test

The application is an umbrella app. It means that it is split into several sub-projects (that you can see under /apps).

To run tests for a specific app, for example the transport or gbfs app, use this command:

# for apps/transport app
mix cmd --app transport mix test --color
# for apps/gbfs
mix cmd --app gbfs mix test --color

# or, for a single file, or single test
mix cmd --app transport mix test --color test/transport_web/integrations/backoffice_test.exs 
mix cmd --app transport mix test --color test/transport_web/integrations/backoffice_test.exs:8

The filenames must be relative to the app folder. This will be improved when we upgrade to a more modern Elixir version.

Measuring test coverage

We use excoveralls to measure which parts of the code are covered by testing (or not). This is useful to determine where we can improve the testing quality.

The following commands will launch the test and generate coverage:

# Display overall (whole app) coverage for all tests in the console
MIX_ENV=test mix coveralls --umbrella
# Same with a HTML report
MIX_ENV=test mix coveralls.html --umbrella

# Display coverage for each umbrella component, rather
MIX_ENV=test mix coveralls

The coverage is written on screen by default, or in the cover subfolders for HTML output.

Running in --umbrella mode will generate coverage report at the top-level cover folder, while running without it will generate reports under each umbrella sub-app (e.g. apps/transport/cover).

Linting

  • Run the elixir linter with mix credo --strict
  • Run the javascript linter with mix npm "run linter:ecma"
  • Run the sass linter with mix npm "run linter:sass"

Misc Elixir command

Translations

To extract all translations from the source, you can run mix gettext.extract --merge (and then edit the modified .po files).

Check all

To perform all the checks done on the CI with a single command, you can run mix check_all. It will run the different linters, credo, check the translations are up-to-date, and launch the tests.

DB migrations

To generate a new migration file: cd apps/transport && mix ecto.gen.migration <name of the migration> && cd ../..

The generated ecto migration file will be apps/transport/priv/repo/migrations/<timestamp>_<name of the migration>.exs

To apply all migrations on you database: mix ecto.migrate

One shot tasks

Some custom one shot tasks are available.

To run a custom task: mix <custom task>

  • Transport.ImportAom: import the aom data from the cerema
  • Transport.ImportEPCI: import the french EPCI from data.gouv
  • Transport.OpenApiSpec: generate an OpenAPI specification file

Docker installation

Development

If you don't plan to work a lot on this project, the docker installation is way easier.

You need a .env file, and can use .env.example to see which variables need to be set. (No need to setup the variable PG_URL, it is defined in the docker-compose.yml)

Then you only need to run: docker-compose up

And access it at http://localhost:5000

You can make changes in the repository and those will be applied with hot reload.

You can run any mix command with:

docker-compose run web mix <cmd>

For the tests you also need to add an environment variable:

docker-compose run -e web mix test

Production

The Dockerfile needed to run the continuous integration is in the project: https://github.com/etalab/transport-ops

Update it if needed (e.g. updating Elixir’s version) and then update .circleci/config.yml.

Domain names

The following domain names are currently in use by the deployed Elixir app:

These names are configured via a CNAME on CleverCloud.

The corresponding SSL certificates are auto-generated via Let's Encrypt and CleverCloud.

Blog

The project blog code and articles are hosted in the blog folder of the blog branch. A specific blog branch has been created with less restrictive merge rules, to allow publishing articles directly from the CMS without needing a github code review.

Technically, the blog is a hugo static website, enhanced with netlifyCMS that is automatically deployed using Netlify. NetlifyCMS allows github users who have write access to this repo to write and edit articles, without the need to use git nor github.

To write or edit an article, visit https://blog.transport.data.gouv.fr/admin/.

For developement purposes, you can run the blog locally. Install hugo, open a terminal, go the blog folder of the project and run hugo serve.

Troubleshootings

No usable OpenSSL found (during Erlang installation via ASDF)

MacOS come with a pre-installed version of LibreSSL which is a fork from OpenSSL. This could cause trouble since it's considered as a "no usable OpenSSL" by Erlang.

We can fix this error in 2 steps :

  1. Install OpenSSL 1.1 (via homebrew for example)
> brew install --prefix=openssl
  1. Force the use of the installed version when installing erlang by setting the --with-ssl option in the KERL_CONFIGURE_OPTIONS variable.
> export KERL_CONFIGURE_OPTIONS="--with-ssl=$(brew --prefix --installed [email protected])"
> asdf install erlang 24.0.4

See asdf-vm/asdf-erlang#82.

More Repositories

1

DVF-app

Exploration des données DVF
JavaScript
540
star
2

covid19-dashboard

Tableau de bord officiel de suivi de l'épidémie de COVID-19
JavaScript
162
star
3

sirene_as_api

Une API pour le fichier sirene
Ruby
99
star
4

piaf

Question Answering annotation platform - Plateforme d'annotation
Python
87
star
5

taxe-habitation

Code source de la taxe d'habitation 2017
C
84
star
6

data.gouv.fr

Ce dépôt rassemble les tickets techniques qui portent sur data.gouv.fr.
72
star
7

geo.data.gouv.fr

Trouvez facilement les données géographiques dont vous avez besoin
JavaScript
70
star
8

cadastre

Scripts de préparation des données cadastrales diffusées par Etalab
JavaScript
62
star
9

cra-envs

⚙️ Bundle env var in CRA at launch time!
TypeScript
55
star
10

dvf

Scripts permettant d'améliorer les données DVF
JavaScript
49
star
11

api-geo

Interroger les référentiels géographiques plus facilement
JavaScript
45
star
12

geozones

Simple spatial/administrative referential
Python
43
star
13

annuaire-entreprises-site

Vérifiez les informations légales publiques des entreprises, associations et services publics en France
TypeScript
43
star
14

decoupage-administratif

Données concernant le découpage administratif français, au format JSON
JavaScript
40
star
15

schema.data.gouv.fr

Schémas de données ouvertes sur des formats réglementaires ou non
Vue
39
star
16

transport-validator

GTFS validator
Rust
38
star
17

jours-feries-france

Librarie Python pour calculer les jours fériés en France
Python
37
star
18

csv-detective

CSV inspection
Python
36
star
19

cadastre.data.gouv.fr

Données cadastrales ouvertes - le site
JavaScript
35
star
20

taxe-fonciere

Code source des taxes foncières
COBOL
35
star
21

croquemort

A micro service to check dead links efficiently and asynchronously. In use at https://www.data.gouv.fr/
Python
34
star
22

geo.api.gouv.fr

Site Web de l'API Géo
JavaScript
32
star
23

csvapi

An instant JSON API for your CSV
Python
31
star
24

template.data.gouv.fr

Template CSS pour doc.data.gouv.fr, beta.gouv.fr et compagnie
HTML
28
star
25

geohisto

[UNMAINTAINED] Historic information for French regions, counties, overseas collectivities and towns based on INSEE and Wikipedia data, exported as (re)usable CSV files.
Python
28
star
26

data-codes-sources-fr

Métadonnées des dépôts de codes sources d'organismes publics français
Python
27
star
27

etalab

Livret de bienvenue destiné aux membres d’Etalab.
HTML
26
star
28

licence-ouverte

La licence recommandée pour la publication de données publiques, par Etalab.
25
star
29

udata-gouvfr

Skin and customization for the French opendata portal based on udata. This project is not maintained anymore. Consider using https://github.com/etalab/udata-front as an alternative.
Python
25
star
30

noms-de-domaine-organismes-secteur-public

Liste de noms de domaine d'organismes publics
Python
23
star
31

rncs_worker_api_entreprise

API pour récupérer et mettre à disposition les données du Répertoire National du Commerce et des Société
Ruby
22
star
32

algorithmes-publics

Guide sur les algorithmes publics à l'usage des administrations
22
star
33

guides.etalab.gouv.fr

Les guides d'Etalab : bonnes pratiques relatives aux données, algorithmes et codes sources
JavaScript
22
star
34

ban-data

Projet de suivi des données BAN et des scripts de traitements associés
Shell
20
star
35

transpo-rt

Simple API for public transport realtime data
Rust
20
star
36

calculette-impots-m-source-code

Code source du calcul de l'impôt sur les revenus
M
20
star
37

bureau-vote

Preparatory work for the publication of polling place shapes as open data.
Jupyter Notebook
18
star
38

cada.data.gouv.fr

A simple interface to search and display CADA advices
CSS
17
star
39

inventaire-codes-sources-organismes-publics

Inventaire des codes sources des organismes publics (archivé)
Shell
17
star
40

matchSIRET

A repo to combine initiatives to gather legal information about companies
Jupyter Notebook
16
star
41

codes-juridiques-francais

Les codes juridiques français, issus de la base LEGI de la DILA, au format Git et Markdown
16
star
42

csv-gg

Create forms from Table Schemas and let users create valid CSV rows
Vue
16
star
43

annuaire-entreprises-sirene-api

Shell
15
star
44

user-research

🎤 Entretiens utilisateurs pour la compréhension des usages 🤓 réduction des frictions entre utilisateurs et données.
14
star
45

calendrier.api.gouv.fr

API JSON pour les jours fériés en France
HTML
14
star
46

annuaire-entreprises-search-api

Python
14
star
47

catalogage-donnees

Outil de catalogage de données développé par Etalab (service en production sur catalogue.data.gouv.fr)
Python
14
star
48

programme10pourcent-site

Site du programme 10% d'Etalab
HTML
14
star
49

prix-carburants-data

Python
13
star
50

jours-feries-france-data

Exports aux formats CSV, ICS et JSON des jours fériés en France
Python
13
star
51

explore.data.gouv.fr

A simple frontend for https://github.com/etalab/csvapi
Vue
13
star
52

calculette-impots-m-language-parser

Calculette de l'impôt sur le revenu parsée
Jupyter Notebook
12
star
53

tiles.data.gouv.fr

Fonds de carte de qualité utilisables par tous
JavaScript
12
star
54

ouverture-des-codes-sources-publics

Guide pour l'ouverture des logiciels libres des organismes publics
12
star
55

amenagements-cyclables

Schema pour les données d'aménagements cyclables
12
star
56

transport-profil-netex-fr

Le contenu des normes des données de transport présentes sur le site https://normes.transport.data.gouv.fr.
11
star
57

rna_as_api

Une API pour le fichier RNA
Ruby
10
star
58

schema-irve

TableSchema pour les Infrastructures de Recharge de Véhicules Electriques (IRVE)
Elixir
10
star
59

programme10pourcent-socratext

Jupyter Notebook
10
star
60

youckan

Django-based SSO
Python
10
star
61

annuaire-entreprises-search-infra

Python
10
star
62

dashboard-aides-entreprises

Données et dashboard des aides publiques aux entreprises françaises durant la crise COVID-19
JavaScript
10
star
63

lois-non-codifiees-et-reglements-francais

Les lois non condifiées ainsi que les règlements français, issus de la base LEGI de la DILA, au format Git et Markdown
9
star
64

iris

Traitements et API permettant de travailler avec les IRIS
JavaScript
9
star
65

majic

Extraction des données MAJIC
JavaScript
9
star
66

apicarto

[Abandonné] Faire appel à l'information géographique pour simplifier les démarches
JavaScript
9
star
67

zammad-contact-form

A contact form that creates tickets in Zammad on behalf of the sender
Vue
9
star
68

admin_api_entreprise

Site vitrine / backoffice de API Entreprise
SCSS
9
star
69

annuaire-entreprises-api-proxy

Proxy d'API de l’Annuaire des Entreprises
TypeScript
8
star
70

etalab-support

Etalab product support
8
star
71

tchatcha

Captchacha: solution de vérification de formulaire
CSS
8
star
72

gpu-jpeg2k

C
8
star
73

sirene

🌬 Ease and speed up the kick off of the #OpenSirene hackathon, it contains tools to split/filter/reduce the huge CSV file.
Python
8
star
74

barometre-resultats

Le baromètre des résultats de l’action publique concrétise un engagement présidentiel fort et inédit sous la Ve République : la transparence des résultats de l’action publique. Il démontre également la détermination du Gouvernement à améliorer la vie quotidienne des citoyens dans chaque territoire, à accélérer la mise en œuvre des réformes et à renforcer l’évaluation de l’action publique, grâce à un pilotage par les résultats et par la donnée.
JavaScript
8
star
75

template-jekyll

Theme/Template beta/data.gouv.fr pour Jekyll
CSS
7
star
76

edigeo-parser

Blazing fast parser for EDIGEO files
JavaScript
7
star
77

entreprise.api.gouv.fr

Site publique d'API Entreprise
HTML
7
star
78

sorry.data.gouv.fr

Maintenance page
JavaScript
7
star
79

de-dpe-processing

Jupyter Notebook
7
star
80

udata-front

udata customizations for Etalab / data.gouv.fr.
JavaScript
7
star
81

covid19-dashboard-widgets

Vue
7
star
82

schema-catalogue-donnees

Un schéma pour les catalogues de données
7
star
83

datagouvfr-pages

Fichiers qui alimentent les pages statiques (notamment pages inventaire) du portail data.gouv.fr
HTML
7
star
84

wiki-data-gouv

Garagethon Wikidata-Datagouv
7
star
85

publier.etalab.studio

Vue
7
star
86

doc.data.gouv.fr

Documentation du site data.gouv.fr.
SCSS
7
star
87

tableschema-template

Template de départ pour les dépôts contenant un schéma Table Schema
7
star
88

HackFrancophonie

Wiki et code de l'open data camp #HackFrancophonie réunissant +8 pays de la Francophonie (19 février 2016, Paris)
Shell
7
star
89

cadastre-mvt

Scripts de génération de tuiles vectorielles pour le plan cadastral
JavaScript
6
star
90

sill-web

Web app of the SILL 2.0
TypeScript
6
star
91

sill.etalab.gouv.fr

🧢 Interface pour l'affichage du Socle Interministériel des Logiciels Libres
Clojure
6
star
92

vuepress-theme-gouv-fr

Un thème VuePress qui s'inspire de template.data.gouv.fr
Vue
6
star
93

documentation-libreoffice

Dépôt contenant la documentation pour LibreOffice rédigée par le ministère de la transition écologique et solidaire
6
star
94

GitLegistique.jl

Convert Git diffs of french law to legislative drafting
Julia
6
star
95

api-codes-sources-fr

API listant les dépôts de code d'organismes publics en France
Python
6
star
96

transport-topo

Tool to manipulate data in https://topo.transport.data.gouv.fr
Rust
6
star
97

ban-geocode

BAN in Elasticsearch
Python
6
star
98

geoids

🇬🇧 Consistent identifiers for geographical levels • 🇫🇷 Identifiants cohérents pour niveaux géographiques
6
star
99

contours-administratifs

Scripts de génération des contours administratifs Etalab
Shell
6
star
100

api-scpc

API par dessus le Service de Consultation du Plan Cadastral de la DGFiP
JavaScript
6
star