Make output report customizable

[mrazauskas]

In TSTyche repo I have 98 test files. If I run them locally in VS Code terminal, the output rendered by first test files cannot be reached any more. If there are any errors or failures, because the output is outside of scroll back buffer of the terminal.

The only way to see the output of a failing test, is to rerun a particular file.

Perhaps in a multi file run it would be better to emit only errors and file names. (Instead of printing out the whole test tree). Like this:

  ✔ tests/config-install.test.js › 9020.904792ms
  ✔ tests/config-target.test.js › 14780.683209ms
  ✔ tests/config-typescriptPath.test.js › 2426.265125ms
  ✔ tests/config-update.test.js › 2459.254291ms
  ✔ tests/feature-store.test.js › 16204.369208ms
  ✔ tests/feature-typescript-versions.test.js › 39605.810250ms
  ✔ tests/feature-watch.test.js › 70000.974792ms
  ✔ tests/validation-store.test.js › 5521.964250ms
  ✔ tests/validation-target.test.js › 5053.783042ms
  ✔ tests/validation-update.test.js › 231.104250ms
  ✔ tests/validation-watch.test.js › 197.263041ms

If a test file is passing, it isn’t important to see its structure. Keeping focus on errors and failures is most important.

(I would love to have dot reporter, of course. But that’s another story.)

[wellwelwel]

Hm, to do this, all we need to do is remove it:

poku/src/services/format.ts

Lines 100 to 109 in 7c852a5

	if (fileResults.success.size > 0) {
	Write.log(
	Array.from(fileResults.success)
	.map(
	([file, time]) =>
	`${indentation.test}${format('✔').success()} ${format(`${file} ${format(`› ${time}ms`).success()}`).dim()}`
	)
	.join('\n')
	);
	}

[mrazauskas]

To clarify more, output for a single file looks right. Very useful to see all the details. I try to point out that it becomes overwhelming in multi file runs.

[wellwelwel]

To clarify more, output for a single file looks right. Very useful to see all the details. I try to point out that it becomes overwhelming in multi file runs.

Do you think we could impose a limit to decide whether or not to display the successes?

[mrazauskas]

Hard to say how verbose a single file can be. So probably keep it verbose for a single file run and make it silent for a multi file run (this is how TSTyche works, by the way). Optionally, --verbose option could be added to enable / disable verbosity.

What you think?

[wellwelwel]

I don't know if a --verbose option would be confusing in relation to --debug.

I think it would be more interesting:

• A) remove success outputs and display when the debug option is used
• B) display only for 100% success or in case of failure, display only the failures

Personally, I think option B is more attractive.

[mrazauskas]

This is your project and you see the context better. All I would like to see is less output in case a test file is all passing.

[wellwelwel]

A --reporter option would be great. @mrazauskas, I've adapted the issue title 🙋🏻‍♂️

[mrazauskas]

A --reporter option would be great. @mrazauskas, I've adapted the issue title 🙋🏻‍♂️

By the way, that would be good solution. --reporter dot is all what I need ;D

[mrazauskas]

Just an idea, the --reporter option could allow adjusting verbosity of the reporter. For instance, --quiet can be replaced with --reporter=quiet.

I was running tests today and thinking that information like “Running tests in parallel” or “Fail fast is enabled” are useful for debugging. Something like --log=debug, --log=info, --log=error, --log=quiet could make sense.

[wellwelwel]

--quiet can be replaced with --reporter=quiet

About --quiet, I like the idea of having it as a direct flag due to familiar commands in CLI apps. The idea is that --quiet overrides any debug or reporter options.

For example:

{
  "test": "poku --reporter=dot"
}

npm run test -- --quiet

---

In addiction, I wondered how other test runners deal with multiple reporters, I intend to have this functionality in a way that makes it easy to implement new reporters 🙋🏻‍♂️

[mrazauskas]

I wondered how other test runners deal with multiple reporters

Jest loops through the list of reporters. A reporter must be an object with a set of callbacks like onRunComplete() (all are optional). There are several callbacks and they are call with various data during the test run. The object passed to onRunComplete() includes data of the whole test run.

In node:test custom reporters are based on a stream of events. That’s lovely pattern!

Inspired by that, I adopted an event emitter for TSTyche. Not only reporters are handle through events, but also the exit code or failFast logic (it handles a cancelation token passed to the runner).

[wellwelwel]

Reporters' Ideas:

• mini (probably the default)
• full
• dot (inspired by Mocha)

The names may not be exactly that.

Important

Both --debug and --quiet aren't reporters and should be complementary to all reporters.

---

Assert

Based on Deno's output, filter and highlight different lines when there is more than one line (regardless of whether it's a string, objects, etc.).

[mrazauskas]

The default reporter could be named poku. This came to my mind as a joke, but why not? ;D