Test recipe
Please download the pre-compiled binary from Latest Release. If you are using Windows, please download the zip file like tbx-xx.x.xxx-win.zip. Then, extract the archive and place tbx.exe on the Desktop folder.
The watermint toolbox can run from any path in the system if allowed by the system. But the instruction samples are using the Desktop folder. Please replace the path if you placed the binary other than the Desktop folder.
This document uses the Desktop folder for command example.
Windows:
cd $HOME\Desktop
.\tbx.exe dev test recipe
macOS, Linux:
$HOME/Desktop/tbx dev test recipe
Note for macOS Catalina 10.15 or above: macOS verifies Developer identity. Currently, tbx is not ready for it. Please select “Cancel” on the first dialogue. Then please proceed “System Preference”, then open “Security & Privacy”, select “General” tab.
You may find the message like:
“tbx” was blocked from use because it is not from an identified developer.
And you may find the button “Allow Anyway”. Please hit the button with your risk. At second run, please hit button “Open” on the dialogue.
| Option | Description | Default |
|---|---|---|
-all |
Test all recipes | false |
-no-timeout |
Do not timeout running recipe tests | false |
-single |
Recipe name to test | |
-verbose |
Verbose output for testing | false |
| Option | Description | Default |
|---|---|---|
-auto-open |
Auto open URL or artifact folder | false |
-bandwidth-kb |
Bandwidth limit in K bytes per sec for upload/download content. 0 for unlimited | 0 |
-budget-memory |
Memory budget (limits some feature to reduce memory footprint) | normal |
-budget-storage |
Storage budget (limits logs or some feature to reduce storage usage) | normal |
-concurrency |
Maximum concurrency for running operation | Number of processors |
-debug |
Enable debug mode | false |
-experiment |
Enable experimental feature(s). | |
-extra |
Extra parameter file path | |
-lang |
Display language | auto |
-output |
Output format (none/text/markdown/json) | text |
-proxy |
HTTP/HTTPS proxy (hostname:port). Please specify DIRECT if you want skip setting proxy. |
|
-quiet |
Suppress non-error messages, and make output readable by a machine (JSON format) | false |
-retain-job-data |
Job data retain policy | default |
-secure |
Do not store tokens into a file | false |
-verbose |
Show current operations for more detail. | false |
-workspace |
Workspace path |
Report file path will be displayed last line of the command line output. If you missed command line output, please see path below. [job-id] will be the date/time of the run. Please see the latest job-id.
| OS | Path pattern | Example |
|---|---|---|
| Windows | %HOMEPATH%\.toolbox\jobs\[job-id]\reports |
C:\Users\bob.toolbox\jobs\20190909-115959.597\reports |
| macOS | $HOME/.toolbox/jobs/[job-id]/reports |
/Users/bob/.toolbox/jobs/20190909-115959.597/reports |
| Linux | $HOME/.toolbox/jobs/[job-id]/reports |
/home/bob/.toolbox/jobs/20190909-115959.597/reports |
Recipe test result
The command will generate a report in three different formats. result.csv, result.json, and result.xlsx.
| Column | Description |
|---|---|
| path | Path to the recipe |
| name | Name of the recipe |
| skip | True if the test skipped |
| timeout_enabled | True if timeout mode enabled |
| use_mock | True if use mock mode enabled |
| timeout | True if the test timeout |
| duration | Test duration in milliseconds |
| no_error | True if no error received |
| error | Error |
If you run with -budget-memory low option, the command will generate only JSON format report.
In case of a report become large, a report in .xlsx format will be split into several chunks like follows; result_0000.xlsx, result_0001.xlsx, result_0002.xlsx, …
The executable automatically detects your proxy configuration from the environment. However, if you got an error or you want to specify explicitly, please add -proxy option, like -proxy hostname:port. Currently, the executable doesn’t support proxies which require authentication.