This is a summary of QA practices Futurice uses and recommends to be used. It is not supposed to be a detailed description and sometimes cannot fully be used of all tasks but as an overview of the most important QA processes and a list of good practices that should be used.
For the purpose of the clarity this document does not describe an error or incident process (i.e. a new bug is found from production).
Everybody in Futurice team has QA responsibilities, even if there is a named QA manager or QA specialists. To Futurice QA means three things.
On high level QA work and practices can be divided into two processes.
Furthermore there are other QA actions.
Futurice uses TDD and ATDD (Acceptance Test Driven Development) methods when applicable.
Method forces the implementation to follow architecture and consider modules that are used. Implementation starts by writing the automated test first and then implementing the functionality to pass the test.
Futurice uses Pair programming method when applicable. This is very convenient way to share knowledge and experience about the project and software under development.
Reviewing the code helps other team members to get information of certain functionality and gives a possibility to give feedback to responsible person and also ensures knowledge sharing between team members.
Manual testing is mostly done using Exploratory testing methodology and found errors are either fixed right away or prioritised and recorded to task/story/error management tool. Exploring the app or service can be started right after something functional is “ready”.
Exploratory testing is a very powerful tool in end to end testing where the whole system is covered by testing. In the method tester goes beyond what can be defined in a test case, applies user-like thinking as well as tries to break the system by various error scenarios and is never “done” with testing.
Around the functional requirements and testing there are usually non-functional requirements which can be tested applying localisation, usability, performance and load testing. The needs and the tools are projects specific. For usability testing Futurice has a mobile usability lab for use. For Performance tests Futurice has used web based services like BrowserMob to name one. For Load testing LoadUI and J meter are actively in use to validate service capabilities in high traffic and to find possible service bottlenecks.
Issues found are recorded to a specific tool or board with a information like priority, environment (software and device information), steps to reproduce, expected result, time and date and a screenshot. Tools like Jira, Trello, PivotalTracker, Request Tracker are actively used also for error tracking.
One of the principles of agile is that the master code branch should always be potentially shippable. That means it should be always production quality. This is achieved by the following means:
Each user story (or feature) is developed individually in their own feature branch. The purpose of this is to ensure that each update to master branch is at the same time 1) as small as possible and 2) potentially shippable, complete story.
Before the feature branch can be merged to the master branch, it must pass a list of actions, requirements and practices called Definition of Done (DoD). This is defined together with the customer PO and the development team and can be modified during project should project needs change.
Following items in proposed DoD have been bolded due
The deployment process is a process how new release (or a version in master branch) is deployed to production. For this process there are three relevant environments.
Continuous Integration (CI) server is used for automated tests and deploying code to environments. Automated tests are run always when any of these environments are updated.
TEST environment is updated automatically by CI, whenever master branch has been updated. This means that automated test cases are also automatically run when master branch is updated.
TEST environment is the main testing server for Futurice. It is expected that any release that has been tested here is ready to be pushed to QA or Production (this depending of the terminology and integrations used in project).
It is not necessary to run manual regression test cases, when updating TEST. Quite often time spent on exploratory testing is more beneficial. However, it is a good practice to run these tests at least once per Sprint.
QA environment should be used as an environment for acceptance tests, demos or any 3rd party testing or audits (security, localization, load, usability etc.). This is NOT Futurice’s primary testing server (TEST is). Update to QA are always agreed between the customer’s and Futurice’s PMs.
The main reason to have two testing environments (TEST and QA) is to comply to security and integration requirements. TEST allows Futurice to develop the service using agile principles. QA allows time and control for the customer to run their current QA processes.
QA should not be updated unless the release has passed testing in TEST environment.
PROD environment is the environment for the live site (production).
Good practice is to deploy and run automated tests in QA at least, prior to pushing update to PROD. However at PROD major functionality needs to be checked after every deployment.
It is recommended that the PO also have the right to push an update to PROD, without any testing in QA (if the version is passed in TEST). This is relevant where the update is very small or simple.
Ultimately in agile development, when the service is in continuous development, the goal is to have a lot of small updates also to PROD. I.e. it is more important that the deployment process is lean and quick than bulletproof. It is considered more important to be able to make fixes quickly than to have a bug free service (obviously with DoD and automated testing the quality should be also high). Futurice does NOT recommend starting with this approach immediately. However, this should be the direction where both organizations want to go.
Each mobile platform has its specific SDK, tool sets and Futurice’s best practices.
Mobile platforms provide emulators within their SDK.
During the implementation phase developer can use simulated / emulated devices to validate changes but nothing can beat the real device testing where the whole system from the touch screen and integrated memory to mobile processors is considered.
Mobile browsers can also be tested by cloud based services like Browserstack.
Futurice has a good variety of different devices and operating system versions in the device library, ranging from basic phones to advanced tablets.
For heavy data traffic application testing and performance validation Futurice has been using Elisa test lab (Elisa is a Finnish mobile service provider). Within the laboratory environment different loads and speeds can be tested in controlled / isolated environment without a need to arrange expensive and not that effective field testing sessions.
Usually mobile app connects to a backend service via mobile network. To get the most out of testing, the tester needs to have a full control of the backend where the mobile application is connected to in order to prepare for and execute different end to end test scenarios.
In mobile devices the test build installation can be done: Locally by cable using development SDK tools Controlled wirelessly via build distribution tools like TestFlight or HockeyApp. Releasing frameworks usually support crash log collection and version data. Dev, Alpha and Beta releases can also be provided in order to collect feedback before releasing the application to stores. Google Play store has option to set up alpha and beta test groups where the specific app is available for the group members only
In modern mobile applications analytics and data collection represent critical functionality which also needs testing. Futurice considers this to be important part of functional testing.
Due to fact that the mobile app update process differs from the web apps one, the analytics setup needs to get right from the first try or then valuable data is missed. Update process is done via application store procedures, but whether the update will be done or not is up to the user settings and activity. So if the first version misses some critical analytics functionality and some users never update the application, that data will then be missed. The solution may be forced upgrade, where the app becomes unusable until the user updates to the newest version available but that might be too late.