Designed to run tests written in any language - because Suman runs tests as child processes - just write TAP to stdout via the runtime of choice (Golang, Java, Python, Node.js, Bash, whatever).
Suman is intended for unit testing in the browser, for backend and system testing, for end-to-end testing with Selenium, etc.
Originally designed for TypeScript and Babel transpilation, Suman was generified to support compiling from source to target for any language. Suman is written in TypeScript, and has strong support for TS. All in all, Suman is a batteries-included test runner which encourages testing best practices through its design. Starting with Suman is very easy, but you will find it has extremely powerful features, as your demands increase.
Perhaps most importantly, Suman makes testing significantly faster for both CPU bound and I/O bound test suites. The speed benefits are generally most useful during test development
As final points, Suman is designed to be universal and not just another JS framework - it's designed for the long-haul.
Elegance and utility of reporting output is paramount to the Suman philosophy. Reporters are the best place to contribute for people interested in creating beautiful test output. As they say (visual) design is what really separates contenders in a world of surplus.
I wrote Suman because I found test runners and test harnesses in the Node.js ecosystem to be lacking. I consider Mocha to be woeful, and AVA to be slow, under-featured and too tightly coupled with Babel. If your team is interested in speeding up your testing cycles, Suman is the absolute right place to look for answers. Suman is designed to be 'better all-around' than AVA, TapJS and Mocha, etc. Reading the issue tracker for Mocha made it very apparent that Mocha was never going to be vastly improved. If you look at the Karma codebase, you also find that it's not designed with Webpack, TypeScript or Babel in mind, etc.
If you can find a problem that Mocha or AVA has, which Suman does not solve, I will find some reward for you.
Suman was written so that it would suffer from none of the problems that existing test runners have.
Suman supports Node versions >= 6.0.0, since it makes heavy use of the Proxy class and Symbol primitives.
Windows support is on the roadmap, but will not be ready anytime soon. Currently, MacOS and *nix support only.
Browser will be supported, but not until ~Summer 2018.
Suman is not officially released yet - expected release date ~October 2017. Until then, expect bugs, missing docs, etc. Proceed at your own risk :D
The Suman docs sumanjs.org
The CLI can be used to run tests in any language; on the other hand, the test harness, only works with Node.js and browser based JS.
You do not need the suman CLI to run suman tests.
You do not need to run suman tests to use the CLI.
They are completely independent, while obviously being designed to work great together.
Suman is not just better than test runners in Node.js and JS land - it improves on test runners written for other languages as well. This is primarily because the Suman CLI is built so that you can have complete control over the parellization of your tests. You can select which tests should exclude each other and run in series, and which tests should run in parallel with each other. In the long run however, all tests should run in parallel in their own container with their own resources. In the long long run, each container should run on its own hardware, all in the name of speed. Suman has an experimental feature called "suman groups" which will allow you to group your tests together and assign a group to a container. Eventually the dream is support the assignment of each container to its own hardware, using different cloud platforms.
=> You can run Mocha tests and AVA tests using the Suman CLI, but not the other way around!
Easily containerize any test process in development, using
suman --ctrz x.js
Imagine you're a senior developer and a more junior developer joins the team, and they write a new test that brings down the whole CI/CD pipeline. With Suman, tests don't directly interact because they are run in separate processes; in Mocha, not so much. Avoid that fateful day. Using Suman, it will be clear which test process is causing the fatal problem; but with a single process it will not necessarily be clear that it is the junior developer's test which caused the issue.
Suman is much faster than AVA, because Suman does not require transpilation.
import * as suman from 'suman';
Suman can run a test in any language which exposes a script with a hashbang or a binary entrypoint file (e.g. Golang or C). To run Java tests, where Java does not compile to binary and where you cannot put a hashbang in a .class file, you would need to call Java from a shell script.
It is designed for maximum test performance, via careful parallelization at every juncture in the testing process/pipeline.
For command line tools:
$ npm install -g suman
=> Please do not use sudo to install suman globally; if you need to use sudo, then something is probably wrong => See: https://docs.npmjs.com/getting-started/fixing-npm-permissions => To avoid any problems with permissions, Suman recommends usage of NVM
For test suites in your project:
$ cd <project-root> && suman --init
=> to convert a Mocha test or whole directory(s) of Mocha tests to Suman tests use
$ suman --convert --src=<src-file/src-dir> --dest=<dest-dir>
=> to simply install Suman as dev-dependency in any project you can use
$ npm install -D suman,
$ suman --init is the much preferred way to initialized suman in a given project, because it will
add a directory to your project which you can move to a location of your choosing.
If you wish to avoid global NPM module installations, we commend you, see: http://sumanjs.org/tutorial-11-advanced-installation.html/
suman test/**/*.py # run python tests suman test/**/*.rb # run ruby tests suman test/**/*.sh test/**/*.go # run bash and golang tests suman test/src/*.spec.js --concurrency=6 # run the matching tests, no more than 6 Node.js processes at a time. suman -w project # run a set of tests when a project file changes suman -w tests # when developing a test, run it upon changes
The human side of this endeavor. (A link to docs will be here soon.);
Global installations of Suman simply look for local installations to run. So if you run the suman command installed via npm install -g suman, that CLI will just invoke the local installation of suman in any given project - this prevents any potential conflicts if there is a difference between the global/local module versions. Excellent!
If you use NVM and switch between Node.js versions, you can use bash functions (provided by SumanJS) which will denecessitate the need for any global installations of Suman at all.
Suman is designed to interop well with the most common libraries in the ecosystem for handling asynchronous code.
The purpose of the Suman library is to provide the most sophisticated test runner in the Node.js ecosystem, with a better developer experience, better features, higher performance, improved debuggability, and more expressiveness than AVA, Mocha, and Tape. Suman is a first-rate library and we hope you take the time to compare its capabilities with AVA, Mocha, Tape, TapJS, etc. One of the more exciting things about Suman is that it can run tests written in any language; all you have to do is output TAP (Test Anything Protocol).
The primary aims are:
|Node-able||Supports ES6/ES7 features||Supports test isolation using multiple Node.js processes||Concurrency within suites||Dependency Injection|
|AVA||No, requires transpilation first||Yes||Yes||Yes||No|
|Suman||Yep, you can run any given test suite with the plain old node executable||Yep, Suman will support all features||Yep||Yep||Yep|
|Implicit globals||Forces you to use their assertion library madness||Confusing JS contexts madness||Developer debugging / console.log output mixed with test output madness||no concurrency madness|
|Suman||Nope||Nope, Suman is completely assertion-lib agnostic||Nope, Suman greatly simplifies the context puzzle that Mocha provided||Nope, Suman runner uses silent option with child_process so your output doesn't mix with the test results||Nope|
Improved mechanics, syntax and semantics
Automatic test execution and/or test transpilation
suman --watchyou can execute test files or transpile test files as you write them
Use suman.once.js to run hooks before the test runner starts
Very simple but powerful dependency injection (DI/IoC)
Test runner tuning
Easy migration from Mocha
Optional but first-rate transpilation features
Log files help you debug
1. node x.js, 2. suman x.js and 3. suman --runner x.js
Freedom: Suman is not highly opinionated, but gives you powerful features
* we designed Suman with details in mind * fewer logical branches and contingencies than Mocha WRT rules for hooks * much better semantics, with new standard functions alongside Mocha's 'done' callback: 'ctn', 'pass', 'fail' and 'fatal' are new functions each with a unique purpose and meaning, and done is still in Suman's API with the same meaning as Mocha! * friendly error messages, that also get sent to suman-stderr.log for reference * when debugging, (the debug flag is set) timeouts will automatically be set to 'infinity'
=> as it gives the developer total control and access to a very large set of features, with the explicit goal of being bug-free first, full-featured second.
The Service Level Agreement is that Suman will constantly be up-to-date with the newest features available via the node executable. We will focus on what's in Node and not what's available in Babel or other transpilers. That being said, we will also work to ensure Babel features are also supported, but we will primarily focus on making Suman completely bug-free when it comes to the latest version of Node, not the latest abilities of Babel or the like. By the time any ES6/ES7/ES8 feature is available in Node, it will be supported by Suman. We want to emphasize the utility of the option of running things with the plain old Node executable, as opposed to adding the complexity of transpilation.
You can execute a test file with the plain
node executable, with
$ suman and with
$ suman --runner
Here are the differences between the 3 options:
|$ node a.test.js||$ suman a.test.js||$ suman --runner a.test.js|
|command line options||no||yes||yes|
|runs multiple processes||no||no||yes|
|easy to debug?||easy||medium||hard|
In order to run multiple files, you must use
$ suman --runner; the above table
only pertains to running a single test file (usually when developing a particular test.)
It should be abundantly clear why Suman is better than Mocha, but how is Suman better than AVA? (If it's not clear why Suman is better than Mocha then scroll back up). Suman borrows some excellent features from Mocha that AVA seems to ignore, including the ability to use nested describe blocks for more control and preventing the sharing of scope within tests. AVA basically co-opted Tape and added concurrency. Suman co-opted Mocha, added concurrency, better reporting, dependency injection and less confusing contexts for callbacks. Suman has more powerful facilities for asynchronous testing than AVA due to Mocha/Jasmine-style hooks and nested describes. Dependency injection also makes Suman extremely convenient and fun to use, compared to AVA. Suman is simply more powerful and richer in features than AVA.
AVA requires Babel transpilation, which adds unnecessary complexity for test environments, and is also much slower
AVA does not handle errors thrown in asynchronous code gracefully, Suman is much better in this regard.
AVA does not feature the nested describes of Mocha or Suman, which limits the expressiveness of the library tremendously!
AVA expects you to use its assertion library, whereas Suman will accept usage of any assertion library that you are already familiar with.
Furthermore, AVA does not prescribe solutions to common test problems -
* registering dynamic test cases (given a dynamic value acquired asynchronously) * starting up necessary services before running tests * injecting different dependencies into test files
Alternatively, with Suman:
*Suman has nested describe blocks, which are imperative for non-trivial tests; a simple but common example is a before hook that you want to run only for a certain subset of tests in your test file. The before hook and tests would go in a nested describe block. *Babel transpilation is totally optional - you can achieve the async/await pattern with generators and promises alone, you don't need ES7 for this *Suman uses domains to correctly map runtime/assertion errors to test cases and hooks, which provides a much more reliable and well-designed piece of software because it can handle any error that gets thrown, not just assertion errors.
(Please see contributing.md)