Improve reporting for test scripts run by patchers
Any failure from ./script/test results results in a non-zero exit code and some kind of indication of the failure in the summary.
This does not mean we fail fast, we run all available target environments as we do currently.
The summary does report the number of environments tested (e.g. Tested with 6 out of 6 possible js targets) but gives no indication in summary of failures in those environments.
Also always exits with 0.
Because ./script/test produces a good amount of output and failures are buried in that output, it is too easy to assume success when something has failed.
- this work should be compatible with Git Bash on Windows
Cursory tests show that ./script/test-simple is not running, will need to add :main to compiler options.
Move duplicated code from ./script/test and ./script/test-simple to shared ./script/test-runner.
Produce 1 line of summary for each test target conveying status of pass, fail or skip.
Exit with 0 (success) when no failures and at least 1 target passed (consider target skips, if not all skipped, ok), otherwise exit with 1 (failure).
Colorize key outputs for at-a-glance parsing.
honor NO_COLOR (https://no-color.org/) for folks that do not like colorized output
To help the reviewer, I have included some screenshots of the new summary section.
The following will return exit status 0
The following will return exit status 1
Switched from !#/bin/sh to !#/bin/env bash to make life easier. I don't think this is controversial, but thought it worth raising.
Windows script/test.ps1 was not considered as part of this work.
Applied CLSJ-3098 fix for GraalVM on Windows.
Continuous Integration does not run ./script/test. As a patcher, it would be good to know that I am executing the same tests as CI, but this would be the subject of a separate ticket.
Scripted execution then manual verification of results on
linux via local Docker
Windows 10 via ssh to local Windows VM
Scripted tests and resulting logs are attached in cljs-3075...zip file. These scripts are specific to my setup but do include Dockerfile used for linux. Entry-point is verify.sh.
bin/cljsc and bin/classpath_conv were applied to patched versions in testing to support Windows runs. A side effect was that this also resolved test failures on my particular installation of linux due to file.encoding not being utf-8.
script/test did not work on Windows prior to this patch and it will still not work on Windows until CLSJ-3098 is applied.
CLJS-3075-2.patch added to Patch Tender
Ok, this one is ready for review. I thought visualizations were lacking, so I added some to the description to hopefully make review easier. Thanks, and let me know what you think.
Added CLSJ-3075-2.patch which:
includes fix for GraalVM on Windows
gets rid of usage of controversial bash eval
fixes an issue for grepping for failures. See CLSJ-3114
I will come back to this one. It needs a tweak to be compatible with Git Bash on Windows.
CLJS-3075.patch added to Patch Tender