• Stars
    star
    126
  • Rank 284,543 (Top 6 %)
  • Language
  • Created over 8 years ago
  • Updated over 7 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

This is a summary of QA practices Futurice uses and recommends to be used.

QA-best-practises

1. Purpose of this document

This is a summary of QA practices Futurice uses and recommends to be used. It is not supposed to be a detailed description and sometimes cannot fully be used of all tasks but as an overview of the most important QA processes and a list of good practices that should be used.

For the purpose of the clarity this document does not describe an error or incident process (i.e. a new bug is found from production).

2. QA practices

Everybody in Futurice team has QA responsibilities, even if there is a named QA manager or QA specialists. To Futurice QA means three things.

  1. The service runs as expected and is considered to be created with good practices.
  2. The service can be easily and cost efficiently maintained and operated.
  3. The service can be continuously improved and modified cost efficiently

On high level QA work and practices can be divided into two processes.

  1. QA during agile sprints
  2. QA during deployment process

Furthermore there are other QA actions.

Test-Driven Development

Futurice uses TDD and ATDD (Acceptance Test Driven Development) methods when applicable.

Method forces the implementation to follow architecture and consider modules that are used. Implementation starts by writing the automated test first and then implementing the functionality to pass the test.

Pair programming

Futurice uses Pair programming method when applicable. This is very convenient way to share knowledge and experience about the project and software under development.

Code review

Reviewing the code helps other team members to get information of certain functionality and gives a possibility to give feedback to responsible person and also ensures knowledge sharing between team members.

Manual functional testing

Manual testing is mostly done using Exploratory testing methodology and found errors are either fixed right away or prioritised and recorded to task/story/error management tool. Exploring the app or service can be started right after something functional is “ready”.

Exploratory testing is a very powerful tool in end to end testing where the whole system is covered by testing. In the method tester goes beyond what can be defined in a test case, applies user-like thinking as well as tries to break the system by various error scenarios and is never “done” with testing.

Non functional testing - localization, usability, performance/load testing

Around the functional requirements and testing there are usually non-functional requirements which can be tested applying localisation, usability, performance and load testing. The needs and the tools are projects specific. For usability testing Futurice has a mobile usability lab for use. For Performance tests Futurice has used web based services like BrowserMob to name one. For Load testing LoadUI and J meter are actively in use to validate service capabilities in high traffic and to find possible service bottlenecks.

Error management

Issues found are recorded to a specific tool or board with a information like priority, environment (software and device information), steps to reproduce, expected result, time and date and a screenshot. Tools like Jira, Trello, PivotalTracker, Request Tracker are actively used also for error tracking.

2.1 QA in Sprints (Definition of Done)

One of the principles of agile is that the master code branch should always be potentially shippable. That means it should be always production quality. This is achieved by the following means:

  1. Each user story (or feature) is developed individually in their own feature branch. The purpose of this is to ensure that each update to master branch is at the same time 1) as small as possible and 2) potentially shippable, complete story.

  2. Before the feature branch can be merged to the master branch, it must pass a list of actions, requirements and practices called Definition of Done (DoD). This is defined together with the customer PO and the development team and can be modified during project should project needs change.

2.1.1 DoD example for a project

  • Manual regression test case (usually UI test cases) have been updated for the smoke / sanity checks (Test manager)
  • Automated unit test cases have been written (Developer)
  • Feature development is completed. Acceptance criteria of the story have been fulfilled (Developer)
  • Code review (both feature and test cases) is done by another developer (developer)
  • Functional test cases have been passed in Demo environment (Automated & manual)
  • The user story (feature) has been approved (Product Owner) [1]
  • Integration tests done against stub/mockups, e2e when possible [2]
  • Concept documentation updated (Product Owner) [3]
  • All other documentation has been updated (Led by project manager)
  • All documentation changes have been reviewed and approved (customer / PO) [4]
  • Feature branch merged successfully to Master branch (Developer)
  • Automated tests passed in Test environment (Updated and run automatically by CI)
  • Functional tests passed in test environment / Exploratory tests have been done (Led by test manager)

Following items in proposed DoD have been bolded due

  1. Acceptance of the user story here means that the PO is happy with the design, UX and UI of a user story. In practice this means that once the user story is approved it can still have QA issues (performance, bugs etc). It should be noted that new POs, especially in an organization where agile methods have not been used before, have often found this responsibility quite heavy and challenging. Traditionally similar approvals are made during project steering groups.
  2. Usually it’s very difficult to run these in Demo environment due to security policies / high overhead of maintaining this. TEST environment should be integrated with appropriate test environment
  3. Similar to the first item. In traditional project methods this is usually done by Vendor’s PM after approvals have been made during project steering group. In agile concept documentation is normally not maintained at all. However as the concept documentation is most likely an acceptance criteria of this project it needs to be maintained to showcase the changes that we have made during the project.
  4. Approval of the documentation should be part of DoD especially initially. Each organization has their own documentation practices and the purpose of this is to ensure that we start doing it “right” from the very beginning.

2.2 Web / server specific QA during deployment process

The deployment process is a process how new release (or a version in master branch) is deployed to production. For this process there are three relevant environments.

  1. TEST. The test environment contains a single instance for the frontend. CI automatically updates the latest version of master branch here. The purpose of this environment is to test and validate that user stories are complete and systematically run regression testing (unit tests).
  2. QA. The QA environment should be as close as possible to the production environment. Therefore, it uses two instances for the frontend, connected to an Elastic Load Balancer (ELB). This is used prior to pushing updates to production environment, to run regression testing and acceptance testing by the customer.
  3. PROD. The production environment makes use of two or more frontend instances (behind an Elastic Load Balancer) to provide scalability and high availability of the public-facing website.

Continuous Integration (CI) server is used for automated tests and deploying code to environments. Automated tests are run always when any of these environments are updated.

2.2.1 Updating TEST environment

TEST environment is updated automatically by CI, whenever master branch has been updated. This means that automated test cases are also automatically run when master branch is updated.

TEST environment is the main testing server for Futurice. It is expected that any release that has been tested here is ready to be pushed to QA or Production (this depending of the terminology and integrations used in project).

It is not necessary to run manual regression test cases, when updating TEST. Quite often time spent on exploratory testing is more beneficial. However, it is a good practice to run these tests at least once per Sprint.

2.2.2 Updating QA environment

QA environment should be used as an environment for acceptance tests, demos or any 3rd party testing or audits (security, localization, load, usability etc.). This is NOT Futurice’s primary testing server (TEST is). Update to QA are always agreed between the customer’s and Futurice’s PMs.

The main reason to have two testing environments (TEST and QA) is to comply to security and integration requirements. TEST allows Futurice to develop the service using agile principles. QA allows time and control for the customer to run their current QA processes.

QA should not be updated unless the release has passed testing in TEST environment.

2.2.3 Updating PROD environment

PROD environment is the environment for the live site (production).

Good practice is to deploy and run automated tests in QA at least, prior to pushing update to PROD. However at PROD major functionality needs to be checked after every deployment.

It is recommended that the PO also have the right to push an update to PROD, without any testing in QA (if the version is passed in TEST). This is relevant where the update is very small or simple.

Ultimately in agile development, when the service is in continuous development, the goal is to have a lot of small updates also to PROD. I.e. it is more important that the deployment process is lean and quick than bulletproof. It is considered more important to be able to make fixes quickly than to have a bug free service (obviously with DoD and automated testing the quality should be also high). Futurice does NOT recommend starting with this approach immediately. However, this should be the direction where both organizations want to go.

2.3 Mobile application specific practices

Each mobile platform has its specific SDK, tool sets and Futurice’s best practices.

For Android https://github.com/futurice/android-best-practices

For iOS https://github.com/futurice/ios-good-practices

For Windows Phone https://github.com/futurice/windows-app-development-best-practices

2.3.1 Mobile native, web and hybrid app testing

Mobile platforms provide emulators within their SDK.

During the implementation phase developer can use simulated / emulated devices to validate changes but nothing can beat the real device testing where the whole system from the touch screen and integrated memory to mobile processors is considered.

Mobile browsers can also be tested by cloud based services like Browserstack.

Test devices

Futurice has a good variety of different devices and operating system versions in the device library, ranging from basic phones to advanced tablets.

Data traffic heavy services

For heavy data traffic application testing and performance validation Futurice has been using Elisa test lab (Elisa is a Finnish mobile service provider). Within the laboratory environment different loads and speeds can be tested in controlled / isolated environment without a need to arrange expensive and not that effective field testing sessions.

Mobile backend

Usually mobile app connects to a backend service via mobile network. To get the most out of testing, the tester needs to have a full control of the backend where the mobile application is connected to in order to prepare for and execute different end to end test scenarios.

2.3.2 Mobile app test releases

In mobile devices the test build installation can be done: Locally by cable using development SDK tools Controlled wirelessly via build distribution tools like TestFlight or HockeyApp. Releasing frameworks usually support crash log collection and version data. Dev, Alpha and Beta releases can also be provided in order to collect feedback before releasing the application to stores. Google Play store has option to set up alpha and beta test groups where the specific app is available for the group members only

2.3.3 Analytics testing

In modern mobile applications analytics and data collection represent critical functionality which also needs testing. Futurice considers this to be important part of functional testing.

Due to fact that the mobile app update process differs from the web apps one, the analytics setup needs to get right from the first try or then valuable data is missed. Update process is done via application store procedures, but whether the update will be done or not is up to the user settings and activity. So if the first version misses some critical analytics functionality and some users never update the application, that data will then be missed. The solution may be forced upgrade, where the app becomes unusable until the user updates to the newest version available but that might be too late.

More Repositories

1

android-best-practices

Do's and Don'ts for Android development, by Futurice developers
20,285
star
2

ios-good-practices

Good ideas for iOS development, by Futurice developers.
10,769
star
3

pepperoni-app-kit

Pepperoni - React Native App Starter Kit for Android and iOS
JavaScript
4,643
star
4

backend-best-practices

An evolving description of general best practices for backend development.
1,942
star
5

terraform-examples

Terraform samples for all the major clouds you can copy and paste. The future, co-created.
HCL
600
star
6

windows-app-development-best-practices

A collection of best practices for Windows App and C# developers
357
star
7

meeting-room-tablet

Google Apps compatible meeting room reservator
Java
179
star
8

futurice-ldap-user-manager

FUM is a user management system for LDAP (Lightweight Directory Access Protocol).
JavaScript
112
star
9

metalsmith-prismic-template

Template for static site generation with prismic.io and metalsmith
JavaScript
98
star
10

iot-service-kit

Internet of Things (IoT) Design Kit (cards, maps, 3D printable pieces) for rapid service concept iteration
93
star
11

field-ops-guide

A booklet to help you survive a software project
89
star
12

freesound-android

Unofficial Android client for the Freesound Project
Java
84
star
13

whereareyou

Passive indoor localization using WiFi signal strength
Python
80
star
14

power-ui

power.futurice.com new UI
JavaScript
57
star
15

project-handover-checklist

List of checkboxes for IT project handover
52
star
16

unity-good-practices

Good ideas for Unity3D development, by Futurice developers. http://www.futurice.com
49
star
17

vor

The new IoT Office Experience.
Java
46
star
18

android-jenkins-docker

Docker image with Jenkins that can build Android apps.
39
star
19

HoloLens-and-Leap-Motion

C#
37
star
20

alley-oop

Dynamic DNS server with an integrated Let's Encrypt proxy, enabling easy HTTPS and WSS for web servers on a local network (LAN)
Go
37
star
21

wappuapp-client

Android and iOS clients for the Futurice Whappu app!
JavaScript
33
star
22

projectnavigationgame

A quest for unknown unknowns, a board game to check the project readiness.
32
star
23

spice-hate_speech_detection

A SPICE-program funded project where the goal is to detect hate speech in social media.
Python
31
star
24

space-tyckiting

Futurice Space Tyckiting: Earth Domination Tour
JavaScript
30
star
25

clojure-workshop

Basic Clojure training material for a one day workshop
Clojure
27
star
26

symptomradar

Symptomradar (Oiretutka) crowdsources coronavirus symptoms from news media audience
TypeScript
25
star
27

spicehours

JavaScript
23
star
28

futurice-super

The purpose of this project is to promote transparency of the sales pipeline and help sales decide what cases to prioritize. This would be done by providing an interface for tribe-members to see upcoming cases and allow them to vote for the cases they find interesting.
JavaScript
23
star
29

file-sharing

Simple anonymous encrypting file service
JavaScript
22
star
30

spiceprogram

Spice Program website and source files for OSS docs
HTML
22
star
31

futuhours-next

A simple frontend for logging work hours
Elm
19
star
32

hannotaatio

Hannotaatio - A visual website feedback tool
JavaScript
19
star
33

festapp-server

JavaScript
18
star
34

minimal-play2

Play framework example application
Scala
18
star
35

rx-android-chat-exercise

Android client for a chat application RxJava exercise
Java
17
star
36

sns-push-notification-service

Simple proxy server for abstracting and simplifying SNS API
Python
17
star
37

futuswarm

Docker Swarm PaaS on AWS
Shell
17
star
38

reactivex-training

Some training material for a full-day ReactiveX training
JavaScript
16
star
39

cv-generator

Tool to generate single slide CVs
Python
16
star
40

vpn-management-server

Server for managing OpenVPN certificates
Python
15
star
41

wappuapp-backend

API for Wappu app client
JavaScript
14
star
42

gif-disco

GIF Disco is a virtual night club - a brilliant addition to any party. Take over the dance floor by recording your moves into an infinitely looping GIF animation.
Python
14
star
43

corona-simulations

Historical Estimates & Model Predictions for COVID-19 in Finland
Svelte
13
star
44

android-meetup-ui-challenge

Material Design UI challenge for the hackathon at Android Meet-up March 2015
Java
13
star
45

festapp-android

festapp Android client
Java
12
star
46

terraform-monitor-lambda

Monitors a Terraform repository and reports on configuration drift: changes that are in the repo, but not in the deployed infra, or vice versa. Hooks up to dashboards and alerts via CloudWatch or InfluxDB.
TypeScript
12
star
47

op-hackathon-templates

App templates demonstrating OP's API usage with Cycle.js
JavaScript
12
star
48

lean-service-creation-kit

Lean Service Creation (LSC) Kit
12
star
49

taxibutton

Futurice Taxi Button
JavaScript
11
star
50

spotify-hubot

Flowdock bot which adds linked Spotify tracks to a playlist
JavaScript
11
star
51

ansible-sentry

Ansible role for Sentry
Python
10
star
52

secret

For managing secrets.
Python
10
star
53

facegame

Facegame
JavaScript
10
star
54

haskell-aws-lambda-kleisli

Run Haskell on AWS Lambda. Deprecated: consider using https://github.com/phadej/aws-lambda-haskell-runtime
Haskell
10
star
55

intro-to-object-detection

instruction and sample codes to get started with object detection in dlib-python
Python
9
star
56

vpn-management-client

Java
9
star
57

festapp-ios

festapp iOS client
Objective-C
9
star
58

futulokki

Website and camera stream for Tampere office's neighbor - seagull named Steven
JavaScript
9
star
59

futuLog

A service to trace contacts in an office environment
TypeScript
9
star
60

Futility

Reusable code for iOS projects
Objective-C
9
star
61

process-arrow

Polymer element for showing a process arrow (progress diagram)
HTML
8
star
62

futurice-academy-homework-assignment

8
star
63

sentry

Dockerized Sentry installation
Python
7
star
64

myRetroGenerator

A static site generator that converts a markdown version of myRetro to an HTML page with the canvas
Haskell
7
star
65

event-app-client

Futurice internal event app
JavaScript
7
star
66

grunt-transifex-resjson

Grunt tasks for managing RESJSON resource files in Transifex translation service
JavaScript
7
star
67

android-rxmvvmdi

Project for introducing Android developers to how one can combine Rx, MVVM, DI, data-binding, and unit testing concepts
Java
7
star
68

jalapeno

CLI for creating, managing and sharing spiced up templates
Go
7
star
69

haskell-futurice-prelude

Futurice Haskell Prelude
Haskell
6
star
70

pepperoni-sample-app

Sample app to demonstrate how to create a React Native app with Pepperoni starter kit (http://getpepperoni.com)
JavaScript
6
star
71

react-native-best-practices

A slightly polemic repository with shared knowledge and best practices from React Native projects at Futurice.
6
star
72

prahapp-client

React Native application for Futuparty
Objective-C
6
star
73

haskell-futurice-site

Python
6
star
74

redux-training

JavaScript
6
star
75

futurice-principles-ethical-ai

Futurice Principles for Ethical AI
6
star
76

hackjunction-multitaction-demo

A demo project for HackJunction to demonstrate the Multitaction multitouch displays.
C#
6
star
77

haskell-flowdock-grep

Grep Flowdock logs
Haskell
6
star
78

heroku-cron

A simple cron-like process to run in worker dyno.
Haskell
5
star
79

fum2github

Compare Futurice FUM and GitHub users
Haskell
5
star
80

android-meetup-rxjava-challenge

RxJava challenge for the hackathon at Android Meet-up March 2015
Java
5
star
81

rxjava-android-exercise

Java
5
star
82

django-saber

Python
5
star
83

macsis

Access control based on Google account domain
JavaScript
5
star
84

rx-android-exercise

Java
5
star
85

futuit-hacks

Futurice IT miscellaneous scripts and tooling
Python
5
star
86

vaskapp

Vask App Client
Objective-C
5
star
87

haskell-flowdock-rest

Haskell bindings to Flowdock REST API
Haskell
5
star
88

sanko-game

Java
4
star
89

metalsmith-prismic-server

npm app package for running on Heroku or similar to compile and deploy content for sites built with metalsmith-prismic-template
JavaScript
4
star
90

ansible-cassandra

4
star
91

google-apps-contacts-copier

Python
4
star
92

docker-base-images

Base docker images for internal service
Dockerfile
4
star
93

machine-learning-best-practices

A collection of materials, pointers, and practices in approaching machine learning projects
4
star
94

jolt-wall

A small elm project to display jolts from Flowdock
Elm
4
star
95

hemmo-app

Mobiili applikaatio Pelastakaa Lapset ry:lle tukiperhevierailujen palautekyselyjä varten
JavaScript
4
star
96

django-history

Django History -- track changes over time for all changes
Python
4
star
97

wwweeklies.com

Repository for wwweeklies.com
HTML
4
star
98

PythonInBrowser

Chilicorn code club site
Python
4
star
99

uwp-reusables

A library of controls and logic that get reused frequently.
C#
4
star
100

djangomi

Python
4
star