Structured Acceptance Test

Structured Acceptance Test ("STAT") is a simple and extensible standard for acceptance testing processes. The target of the test can be any set of computer files, for example source code, images, audio files and documents. The process can be an automated computer program, manual execution of a test plan or even a committee review. The outcome of the process is findings and recommendations which can be used to improve the target and can indicate a pass or fail result.

The STAT Output Specification defines how a process can express the outcome, findings and recommendations

Who can use it?

This standard is applicable for any automated or manual process that accepts computer files as input and can express opinions about them. Examples of such processes include:

Static code analyzers
Code linters
Code style checkers
Minification checkers
Other static analyzers
Image metadata checkers
Image compression checkers
Spelling checkers
Grammar checkers
Digital signature verification
Dynamic code analysis
Unit tests (logic tests)
Automated UI tests
Continuous integration tests
Volatile testing
Link checkers
Package manager version checkers (requirement to use latest upstream versions)
Deployment testing (dry run)
Manual testing
Manual inspection
Manual walkthroughs
Unstructured customer feedback
Committee vote

The acceptance testing outcome, findings and recommendations can be used (consumed) by:

Interactive file editors (IDEs, text editors, word processors, image editors)
Command-line reporting tools (unit test reports)
Source code management / content management systems

Why should I use it? XKCD 927?

If your acceptance testing process uses a standardized output format then consumers can make better use of it.

Integrations are amazing. They allow clang compilers to show compile errors in your integrated development environment, they allow spelling errors to be underlined in your word processor and they show up as red flags when you review a pull request. But what if all of these validations can be shown everywhere they are relevant? Standardization allows this.

This is the first widely-applicable standardization of its type so XKCD 927 does not apply.

Specific features of this specification include:

It is simple, a few lines of Ruby can translate gcc, clang or aspell output into the required format.
The format is extensible, any acceptance testing process can use this format.
Validation output is streamable and available to the reporting tool incrementally.
Repeatability is specified.

Project Status

This standard is currently version 1.0.0. We follow Semantic Versioning.

[x] Defines the input format to select computer files
[x] Defines the output format for the outcome, findings and recommendations
[x] Follows the Google JSON Style Guide
[x] Supports real-time, incremental reporting
[x] An example computer program is produced that supports STAT
[x] An example computer program is produced that reads and reports the above output (Issue #6)
[x] Useful computer program is produced (or transformed) to support STAT
[x] Compatibility is established with Code Climate Engine specification

Copyright

Contributing

Development of this specification happens on GitHub. Please use issues and pull requests to help improve it.

Please help to raise awareness by opening an issue with your favorite static analysis tool to support this format.

Please open an issue with your favorite integrated development environment or source code sharing tool to request interoperability with this format.

Please add your own projects below!

Supporting acceptance testing programs:

Supporting consumers:

Interactive file editors (IDEs, text editors, word processors, image editors)
- YOUR NAME HERE
Command-line reporting tools (unit test reports)
- YOUR NAME HERE
Source code management / content management systems
- YOUR NAME HERE