367 lines
11 KiB
Markdown
367 lines
11 KiB
Markdown
# Running Tests
|
||
|
||
## Use --test Option
|
||
|
||
Hurl is run by default as an HTTP client, returning the body of the last HTTP response.
|
||
|
||
```shell
|
||
$ hurl hello.hurl
|
||
Hello World!
|
||
```
|
||
|
||
When multiple input files are provided, Hurl outputs the body of the last HTTP response of each file.
|
||
|
||
```shell
|
||
$ hurl hello.hurl assert_json.hurl
|
||
Hello World![
|
||
{ "id": 1, "name": "Bob"},
|
||
{ "id": 2, "name": "Bill"}
|
||
]
|
||
```
|
||
|
||
For testing, we are only interested in the asserts results, we don't need the HTTP body response. To use Hurl as a
|
||
test tool with an adapted output, you can use [`--test` option]:
|
||
|
||
```shell
|
||
$ hurl --test hello.hurl assert_json.hurl
|
||
[1mhello.hurl[0m: [1;32mSuccess[0m (6 request(s) in 245 ms)
|
||
[1massert_json.hurl[0m: [1;32mSuccess[0m (8 request(s) in 308 ms)
|
||
--------------------------------------------------------------------------------
|
||
Executed files: 2
|
||
Executed requests: 10 (17.82/s)
|
||
Succeeded files: 2 (100.0%)
|
||
Failed files: 0 (0.0%)
|
||
Duration: 561 ms
|
||
```
|
||
|
||
Or, in case of errors:
|
||
|
||
```shell
|
||
$ hurl --test hello.hurl error_assert_status.hurl
|
||
[1mhello.hurl[0m: [1;32mSuccess[0m (4 request(s) in 5 ms)
|
||
[1;31merror[0m: [1mAssert status code[0m
|
||
[1;34m-->[0m error_assert_status.hurl:9:6
|
||
[1;34m |[0m
|
||
[1;34m |[0m [90mGET http://localhost:8000/not_found[0m
|
||
[1;34m |[0m[90m ...[0m
|
||
[1;34m 9 |[0m HTTP 200
|
||
[1;34m |[0m[1;31m ^^^ actual value is <404>[0m
|
||
[1;34m |[0m
|
||
|
||
[1merror_assert_status.hurl[0m: [1;31mFailure[0m (1 request(s) in 2 ms)
|
||
--------------------------------------------------------------------------------
|
||
Executed files: 2
|
||
Executed requests: 5 (500.0/s)
|
||
Succeeded files: 1 (50.0%)
|
||
Failed files: 1 (50.0%)
|
||
Duration: 10 ms
|
||
```
|
||
|
||
> With or without `--test`, all asserts are always executed. `--test` adds a run recap and disables the output of the
|
||
> last response. To ignore asserts execution, you can use [`--ignore-asserts`].
|
||
|
||
In test mode, files are executed in parallel to speed-ud the execution. If a sequential run is needed, you can use
|
||
[`--jobs 1`] option to execute tests one by one.
|
||
|
||
```shell
|
||
$ hurl --test --jobs 1 *.hurl
|
||
```
|
||
|
||
[`--repeat` option] can be used to repeat run files and do [performance check]. For instance, this call will run 1000 tests
|
||
in parallel:
|
||
|
||
```shell
|
||
$ hurl --test --repeat 1000 stress.hurl
|
||
```
|
||
|
||
|
||
### Selecting Tests
|
||
|
||
Hurl can take multiple files into inputs:
|
||
|
||
```shell
|
||
$ hurl --test test/integration/a.hurl test/integration/b.hurl test/integration/c.hurl
|
||
```
|
||
|
||
```shell
|
||
$ hurl --test test/integration/*.hurl
|
||
```
|
||
|
||
Or you can simply give a directory and Hurl will find files with `.hurl` extension recursively:
|
||
|
||
```shell
|
||
$ hurl --test test/integration/
|
||
```
|
||
|
||
Finally, you can use [`--glob` option] to test files that match a given pattern:
|
||
|
||
```shell
|
||
$ hurl --test --glob "test/integration/**/*.hurl"
|
||
```
|
||
|
||
## Debugging
|
||
|
||
### Debug Logs
|
||
|
||
If you need more error context, you can use [`--error-format long` option] to print HTTP bodies for failed asserts:
|
||
|
||
```shell
|
||
$ hurl --test --error-format long hello.hurl error_assert_status.hurl
|
||
[1mhello.hurl[0m: [1;32mSuccess[0m (4 request(s) in 6 ms)
|
||
[1;32mHTTP/1.1 404
|
||
[0m[1;36mServer[0m: Werkzeug/3.0.3 Python/3.12.4
|
||
[1;36mDate[0m: Wed, 10 Jul 2024 15:42:41 GMT
|
||
[1;36mContent-Type[0m: text/html; charset=utf-8
|
||
[1;36mContent-Length[0m: 207
|
||
[1;36mServer[0m: Flask Server
|
||
[1;36mConnection[0m: close
|
||
|
||
<!doctype html>
|
||
<html lang=en>
|
||
<title>404 Not Found</title>
|
||
<h1>Not Found</h1>
|
||
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
|
||
|
||
|
||
[1;31merror[0m: [1mAssert status code[0m
|
||
[1;34m-->[0m error_assert_status.hurl:9:6
|
||
[1;34m |[0m
|
||
[1;34m |[0m [90mGET http://localhost:8000/not_found[0m
|
||
[1;34m |[0m[90m ...[0m
|
||
[1;34m 9 |[0m HTTP 200
|
||
[1;34m |[0m[1;31m ^^^ actual value is <404>[0m
|
||
[1;34m |[0m
|
||
|
||
[1merror_assert_status.hurl[0m: [1;31mFailure[0m (1 request(s) in 2 ms)
|
||
--------------------------------------------------------------------------------
|
||
Executed files: 2
|
||
Executed requests: 5 (454.5/s)
|
||
Succeeded files: 1 (50.0%)
|
||
Failed files: 1 (50.0%)
|
||
Duration: 11 ms
|
||
```
|
||
|
||
Individual requests can be modified with [`[Options]` section][options] to turn on logs for a particular request, using
|
||
[`verbose`] and [`very-verbose`] option.
|
||
|
||
With this Hurl file:
|
||
|
||
```hurl
|
||
GET https://foo.com
|
||
HTTP 200
|
||
|
||
GET https://bar.com
|
||
[Options]
|
||
very-verbose: true
|
||
HTTP 200
|
||
|
||
GET https://baz.com
|
||
HTTP 200
|
||
```
|
||
|
||
Running `hurl --test .` will output debug logs for the request to `https://bar.com`.
|
||
|
||
[`--verbose`] / [`--very-verbose`] can also be enabled globally, for every requests of every tested files:
|
||
|
||
```shell
|
||
$ hurl --test --very-verbose .
|
||
```
|
||
|
||
### HTTP Responses
|
||
|
||
In test mode, HTTP responses are not displayed. One way to get HTTP responses even in test mode is to use
|
||
[`--output` option] of `[Options]` section: `--output file` allows to save a particular response to a file, while
|
||
`--output -` allows to redirect HTTP responses to standard output.
|
||
|
||
```hurl
|
||
GET http://foo.com
|
||
HTTP 200
|
||
|
||
GET https://bar.com
|
||
[Options]
|
||
output: -
|
||
HTTP 200
|
||
```
|
||
|
||
```shell
|
||
$ hurl --test .
|
||
<html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">
|
||
<title>301 Moved</TITLE></head><body>
|
||
<h1>301 Moved</h1>
|
||
The document has moved
|
||
<a HREF="https://www.bar.com/">here</a>.
|
||
</body></html>
|
||
[1m/tmp/test.hurl[0m: [1;32mSuccess[0m (2 request(s) in 184 ms)
|
||
--------------------------------------------------------------------------------
|
||
Executed files: 1
|
||
Executed requests: 2 (10.7/s)
|
||
Succeeded files: 1 (100.0%)
|
||
Failed files: 0 (0.0%)
|
||
Duration: 187 ms
|
||
```
|
||
|
||
## Stress and Performance Tests
|
||
|
||
Hurl can be used to perform stress tests:
|
||
|
||
- with [query duration]:
|
||
|
||
```hurl
|
||
GET https://example.org/foo
|
||
HTTP 200
|
||
[Asserts]
|
||
duration < 1200
|
||
```
|
||
|
||
- [response metrics] with [`--very-verbose`] or [`--json`] to get structured timing data:
|
||
|
||
```shell
|
||
$ hurl --json foo.hurl | jq
|
||
...
|
||
"timings": {
|
||
"app_connect": 0,
|
||
"begin_call": "2025-10-06T07:00:57.127794Z",
|
||
"connect": 120915,
|
||
"end_call": "2025-10-06T07:00:57.280724Z",
|
||
"name_lookup": 92987,
|
||
"pre_transfer": 121014,
|
||
"start_transfer": 152721,
|
||
"total": 152807
|
||
}
|
||
...
|
||
```
|
||
|
||
In performance uses-cases, it's important to know how Hurl is working to get the best request per second load. Each Hurl file
|
||
is processed by its own libcurl instance (roughly speaking a living HTTP connection). Let's say we want to stress our
|
||
application with 100,000 HTTP requests and see how it's behaving. The simplest idea would be to make a single Hurl file
|
||
repeating 100,000 requests:
|
||
|
||
```hurl
|
||
GET https://my-website/health
|
||
[Options]
|
||
repeat: 100000
|
||
```
|
||
|
||
and run it:
|
||
|
||
```shell
|
||
$ hurl --test perf.hurl
|
||
```
|
||
|
||
With a single Hurl file, we won't benefit from any parallel runs (files are run in parallel, requests _within_ a file are
|
||
run sequentially). To benefit from parallel run, we can use [`--repeat`] as a CLI argument to repeat file execution. It's
|
||
as if we've written 100,000 identical Hurl files and run them:
|
||
|
||
|
||
```hurl
|
||
GET https://my-website/health
|
||
```
|
||
|
||
and run it:
|
||
|
||
```shell
|
||
$ hurl --test --repeat 100000 perf.hurl
|
||
```
|
||
|
||
In this case, we're running 100,000 Hurl files in parallel. Now, we have another problem: as each file is basically a
|
||
HTTP connection, we can run out of TCP port, a phenomenon known as [ephemeral ports exhaustion]. So, to recap:
|
||
|
||
- running 100,000 requests in a single file won't benefit from parallel run
|
||
- running 100,000 one request files can lead to TCP port resources issues
|
||
|
||
The solution is to combine the two approaches: running 10 files of 10,000 HTTP requests. With 10 files of 10,000
|
||
requests, we're going to execute those files in parallel in their own worker. Each worker will be working sequentially,
|
||
maximizing the usage and not running into TCP ports exhaustion. We can force jobs count to be exactly 10 so
|
||
all workers start at the same time and no one is waiting to get a job done:
|
||
|
||
```hurl
|
||
GET https://my-website/health
|
||
[Options]
|
||
repeat: 10000
|
||
```
|
||
|
||
```shell
|
||
$ hurl --test --repeat 10 --jobs 10 perf.hurl
|
||
```
|
||
|
||
|
||
|
||
## Generating Report
|
||
|
||
In the different reports, files are always referenced in the input order (which, as tests are executed in parallel, can
|
||
be different from the execution order).
|
||
|
||
### HTML Report
|
||
|
||
Hurl can generate an HTML report by using the [`--report-html DIR`] option.
|
||
|
||
If the HTML report already exists, the test results will be appended to it.
|
||
|
||
<div class="picture">
|
||
<img class="u-drop-shadow u-border u-max-width-100" src="/docs/assets/img/hurl-html-report.png" width="670" alt="Hurl HTML Report">
|
||
</div>
|
||
|
||
The input Hurl files (HTML version) are also included and are easily accessed from the main page.
|
||
|
||
<div class="picture">
|
||
<img class="u-drop-shadow u-border u-max-width-100" src="/docs/assets/img/hurl-html-file.png" width="380" alt="Hurl HTML file">
|
||
</div>
|
||
|
||
### JSON Report
|
||
|
||
A JSON report can be produced by using the [`--report-json DIR`]. The report directory will contain a `report.json`
|
||
file, including each test file executed with [`--json`] option and a reference to each HTTP response of the run dumped
|
||
to disk.
|
||
|
||
If the JSON report already exists, it will be updated with the new test results.
|
||
|
||
### JUnit Report
|
||
|
||
A JUnit report can be produced by using the [`--report-junit FILE`] option.
|
||
|
||
If the JUnit report already exists, it will be updated with the new test results.
|
||
|
||
### TAP Report
|
||
|
||
A TAP report ([Test Anything Protocol]) can be produced by using the [`--report-tap FILE`] option.
|
||
|
||
If the TAP report already exists, it will be updated with the new test results.
|
||
|
||
## Use Variables in Tests
|
||
|
||
To use variables in your tests, you can:
|
||
|
||
- use [`--variable` option]
|
||
- use [`--variables-file` option]
|
||
- define environment variables, for instance `HURL_VARIABLE_foo=bar`
|
||
|
||
You will find a detailed description in the [Injecting Variables] section of the docs.
|
||
|
||
[`--output /dev/null`]: /docs/manual.md#output
|
||
[`--test`]: /docs/manual.md#test
|
||
[`--report-html DIR`]: /docs/manual.md#report-html
|
||
[`--report-json DIR`]: /docs/manual.md#report-json
|
||
[`--report-junit FILE`]: /docs/manual.md#report-junit
|
||
[`--report-tap FILE`]: /docs/manual.md#report-tap
|
||
[`--test` option]: /docs/manual.md#test
|
||
[`--glob` option]: /docs/manual.md#glob
|
||
[`--variable` option]: /docs/manual.md#variable
|
||
[`--variables-file` option]: /docs/manual.md#variables-file
|
||
[Injecting Variables]: /docs/templates.md#injecting-variables
|
||
[Test Anything Protocol]: https://testanything.org
|
||
[`--jobs 1`]: /docs/manual.md#jobs
|
||
[`--json`]: /docs/manual.md#json
|
||
[`--error-format long` option]: /docs/manual.md#error-format
|
||
[options]: /docs/request.md#options
|
||
[`--verbose`]: /docs/manual.md#verbose
|
||
[`--very-verbose`]: /docs/manual.md#very-verbose
|
||
[`verbose`]: /docs/manual.md#verbose
|
||
[`very-verbose`]: /docs/manual.md#very-verbose
|
||
[`--output` option]: /docs/manual.md#output
|
||
[`--repeat` option]: /docs/manual.md#repeat
|
||
[query duration]: /docs/asserting-response.md#duration-assert
|
||
[response metrics]: /docs/response.md#timings
|
||
[`--ignore-asserts`]: /docs/manual.md#ignore-asserts
|
||
[performance check]: /docs/running-tests.md#stress-and-performance-tests
|
||
[ephemeral ports exhaustion]: https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/
|
||
[`--repeat`]: /docs/manual.md#repeat |