Data Science Template
This is a starter template for data science projects in Equinor, although it may also be useful for others. It contains many of the essential artifacts that you will need and presents a number of best practices including code setup, samples, MLOps using Azure, a standard document to guide and gather information relating to the data science process and more.
As it is impossible to create a single template that will meet every projects needs, this example should be considered a starting point and changed based upon the working and evolution of your project.
Before working with the contents of this template or Data Science projects in general it is recommended to familiarise yourself with the Equinor Data Science Technical Standards (Currently Equinor internal only)
Getting Started With This Template
This template is provided as a Cookiecutter template so you can quickly create an instance customised for your project. An assumption is that you have a working python installation.
To get running, first install the latest Cookiecutter if you haven't installed it yet (this requires Cookiecutter 1.4.0 or higher):
pip install -U cookiecutter
Create project
Then generate a new project for your own use based upon the template, answering the questions to customise the generated project:
cookiecutter https://github.com/equinor/data-science-template.git
The values you are prompted for are:
Value | Description |
---|---|
project_name | A name for your project. Used mostly within documentation |
project_description | A description to include in the README.md |
repo_name | The name of the github repository where the project will be held |
conda_name | The name of the conda environment to use |
package_name | A name for the generated python package. |
mlops_name | Default name for Azure ML. |
mlops_compute_name | Default Azure ML compute cluster name to use. |
author | The main author of the solution. Included in the setup.py file |
open_source_license | What type of open source license the project will be released under |
devops_organisation | An Azure DevOps organisation. Leave blank if you aren't using Azure DevOps |
If you are uncertain about what to enter for any value then just accept the defaults. You can always change the generated project later.
Getting problems? You can always download this repository using the download button above and reference the local copy e.g. cookiecutter c:\Downloads\data-science-template, however ideally fix any git proxy or other issues that are causing problems.
You are now ready to get started, however you should first create a new github repository for your new project and add your project using the following commands (substitute myproject with the name of your project and REMOTE-REPOSITORY-URL with the remote repository url).
cd myproject
git init
git add .
git commit -m "Initial commit"
git remote add origin REMOTE-REPOSITORY-URL
git remote -v
git push origin master
Continuous Integration
Continuous Integration (CI) increase quality by building, running tests and performing other validation whenever code is committed. The template contains a build pipeline for Azure DevOps, however requires a couple of manual steps to setup:
- Log in to http://dev.azure.com and browse to, or create an organisation & project. The project name should be the same as your github repository name.
- Under Pipelines -> Builds select New Pipeline
- Select github and then your repository. Login / grant any permissions as prompted
- In the review pane click run
You are now setup for CI and automated test / building. You should verify the badge link in this README corresponds with your DevOps project, and as a further step might setup any release pipelines for automated deployment.
At this stage the build pipeline doesn't include MLOps steps, although these can be added based uon your needs.
Finally
- Update the project readme file with additional project specific details including setup, configuration and usage.
- The docs\process_documentation.md file should be completed phase by phase, and each phase result shall be submitted for review and approval before the project moves on to the next phase. This is to assist with the gathering of essential information required to deliver a correct and robust solution. The git respoitory shall be added to the script that populates the knowledge repository to ease future knowledge sharing.
Generated Project Contents
Depending upon the selected options when creating the project, the generated structure will look similar to the below:
βββ .gitignore <- Files that should be ignored by git. Add seperate .gitignore files in sub folders if
β needed
βββ conda_env.yml <- Conda environment definition for ensuring consistent setup across environments
βββ LICENSE
βββ README.md <- The top-level README for developers using this project.
βββ requirements.txt <- The requirements file for reproducing the analysis environment, e.g.
β generated with `pip freeze > requirements.txt`. Might not be needed if using conda.
βββ setup.py <- Metadata about your project for easy distribution.
β
βββ data
β βββ interim_[desc] <- Interim files - give these folders whatever name makes sense.
β βββ processed <- The final, canonical data sets for modeling.
β βββ raw <- The original, immutable data dump.
β βββ temp <- Temporary files.
β βββ training <- Files relating to the training process
β
βββ docs <- Documentation
β βββ data_science_code_of_conduct.md <- Code of conduct.
β βββ process_documentation.md <- Standard template for documenting process and decisions.
β βββ writeup <- Sphinx project for project writeup including auto generated API.
β βββ conf.py <- Sphinx configurtation file.
β βββ index.rst <- Start page.
β βββ make.bat <- For generating documentation (Windows)
β βββ Makefikle <- For generating documentation (make)
β
βββ examples <- Add folders as needed e.g. examples, eda, use case
β
βββ extras <- Miscellaneous extras.
β βββ add_explorer_context_shortcuts.reg <- Adds additional Windows Explorer context menus for starting jupyter.
β
βββ notebooks <- Notebooks for analysis and testing
β βββ eda <- Notebooks for EDA
β β βββ example.ipynb <- Example python notebook
β βββ features <- Notebooks for generating and analysing features (1 per feature)
β βββ modelling <- Notebooks for modelling
β βββ preprocessing <- Notebooks for Preprocessing
β
βββ scripts <- Standalone scripts
β βββ deploy <- MLOps scripts for deployment (WIP)
β β βββ score.py <- Scoring script
β βββ train <- MLOps scripts for training
β β βββ submit-train.py <- Script for submitting a training run to Azure ML Service
β β βββ submit-train-local.py <- Script for local training using Azure ML
β β βββ train.py <- Example training script using the iris dataset
β βββ example.py <- Example sctipt
β βββ MLOps.ipynb <- End to end MLOps example (To be refactored into the above)
β
βββ src <- Code for use in this project.
β βββ examplepackage <- Example python package - place shared code in such a package
β βββ __init__.py <- Python package initialisation
β βββ examplemodule.py <- Example module with functions and naming / commenting best practices
β βββ features.py <- Feature engineering functionality
β βββ io.py <- IO functionality
β βββ pipeline.py <- Pipeline functionality
β
βββ tests <- Test cases (named after module)
βββ test_notebook.py <- Example testing that Jupyter notebooks run without errors
βββ examplepackage <- examplepackage tests
βββ examplemodule <- examplemodule tests (1 file per method tested)
βββ features <- features tests
βββ io <- io tests
βββ pipeline <- pipeline tests
Contributing to This Template
Contributions to this template are greatly appreciated and encouraged.
To contribute an update simply:
- Submit an issue describing your proposed change to the repo in question.
- The repo owner will respond to your issue promptly.
- Fork the desired repo, develop and test your code changes.
- Check that your code follows the PEP8 guidelines (line lengths up to 120 are ok) and other general conventions within this document.
- Ensure that your code adheres to the existing style. Refer to the Google Cloud Platform Samples Style Guide for the recommended coding standards for this organization.
- Ensure that as far as possible there are unit tests covering the functionality of any new code.
- Check that all existing unit tests still pass.
- Edit this document and the template README.md if needed to describe new files or other important information.
- Submit a pull request.
Template development environment
To develop this template further you might want to setup a virtual environment
Setup using
cd data-science-template
python -m venv dst-env
Activate environment
Max / Linux
source dst-env/bin/activate
Windows
dst-env\Scripts\activate
Install Dependencies
pip install -r requirements.txt
Testing
To run the template tests, install pytest using pip or conda and then from the repository root run
pytest tests
Linting
To verify that your code adheres to python standards run linting as shown below:
flake8 --max-line-length=120 *.py hooks/ tests/
Important Links
- https://wiki.statoil.no/wiki/index.php/Statoil_Data_Science_Technical_Standards - Data Science Technical Standards (Equinor Internal)
- https://dataplatformwiki.azurewebsites.net/doku.php - Data Platform wiki (Equinor internal)
- https://github.com/Statoil/data-science-shared - Shared Data Science Code Repository (Equinor internal)