mirror of https://github.com/mongodb/mongo
SERVER-107550: Seed readme docs for resmoke-related tooling (#38540)
GitOrigin-RevId: a155fefece1624a3b9790bfa3b138d9fed29b43b
This commit is contained in:
Steve McClure
MongoDB Bot
parent
19567cc64e
commit
787ec5b7e8
|
|
@ -135,6 +135,7 @@ WORKSPACE.bazel @10gen/devprod-build @svc-auto-approve-bot
|
||||||
|
|
||||||
# The following patterns are parsed from ./buildscripts/resmokeconfig/suites/OWNERS.yml
|
# The following patterns are parsed from ./buildscripts/resmokeconfig/suites/OWNERS.yml
|
||||||
/buildscripts/resmokeconfig/suites/**/* @10gen/mongo-default-approvers @svc-auto-approve-bot
|
/buildscripts/resmokeconfig/suites/**/* @10gen/mongo-default-approvers @svc-auto-approve-bot
|
||||||
|
/buildscripts/resmokeconfig/suites/**/README.md @10gen/devprod-correctness @svc-auto-approve-bot
|
||||||
/buildscripts/resmokeconfig/suites/**/aggregation* @10gen/query @svc-auto-approve-bot
|
/buildscripts/resmokeconfig/suites/**/aggregation* @10gen/query @svc-auto-approve-bot
|
||||||
/buildscripts/resmokeconfig/suites/**/aggregation_disabled_optimization.yml @10gen/query-optimization @svc-auto-approve-bot
|
/buildscripts/resmokeconfig/suites/**/aggregation_disabled_optimization.yml @10gen/query-optimization @svc-auto-approve-bot
|
||||||
/buildscripts/resmokeconfig/suites/**/aggregation_repeat_queries_multiplan_single_solutions.yml @10gen/query-optimization @svc-auto-approve-bot
|
/buildscripts/resmokeconfig/suites/**/aggregation_repeat_queries_multiplan_single_solutions.yml @10gen/query-optimization @svc-auto-approve-bot
|
||||||
|
|
@ -193,6 +194,7 @@ WORKSPACE.bazel @10gen/devprod-build @svc-auto-approve-bot
|
||||||
|
|
||||||
# The following patterns are parsed from ./buildscripts/resmokelib/testing/hooks/OWNERS.yml
|
# The following patterns are parsed from ./buildscripts/resmokelib/testing/hooks/OWNERS.yml
|
||||||
/buildscripts/resmokelib/testing/hooks/**/* @10gen/mongo-default-approvers @svc-auto-approve-bot
|
/buildscripts/resmokelib/testing/hooks/**/* @10gen/mongo-default-approvers @svc-auto-approve-bot
|
||||||
|
/buildscripts/resmokelib/testing/hooks/**/README.md @10gen/devprod-correctness @svc-auto-approve-bot
|
||||||
/buildscripts/resmokelib/testing/hooks/**/analyze_shard_key.py @10gen/server-cluster-scalability @svc-auto-approve-bot
|
/buildscripts/resmokelib/testing/hooks/**/analyze_shard_key.py @10gen/server-cluster-scalability @svc-auto-approve-bot
|
||||||
/buildscripts/resmokelib/testing/hooks/**/queryable_server_hook.py @10gen/server-backup-restore @svc-auto-approve-bot
|
/buildscripts/resmokelib/testing/hooks/**/queryable_server_hook.py @10gen/server-backup-restore @svc-auto-approve-bot
|
||||||
/buildscripts/resmokelib/testing/hooks/**/change_streams.py @10gen/query-execution-change-streams @svc-auto-approve-bot
|
/buildscripts/resmokelib/testing/hooks/**/change_streams.py @10gen/query-execution-change-streams @svc-auto-approve-bot
|
||||||
|
|
@ -260,6 +262,9 @@ WORKSPACE.bazel @10gen/devprod-build @svc-auto-approve-bot
|
||||||
/docs/owners/** @10gen/devprod-correctness @svc-auto-approve-bot
|
/docs/owners/** @10gen/devprod-correctness @svc-auto-approve-bot
|
||||||
/docs/**/golden_data_test_framework.md @10gen/query-optimization @svc-auto-approve-bot
|
/docs/**/golden_data_test_framework.md @10gen/query-optimization @svc-auto-approve-bot
|
||||||
|
|
||||||
|
# The following patterns are parsed from ./docs/testing/OWNERS.yml
|
||||||
|
/docs/testing/ @10gen/devprod-correctness @svc-auto-approve-bot
|
||||||
|
|
||||||
# The following patterns are parsed from ./etc/OWNERS.yml
|
# The following patterns are parsed from ./etc/OWNERS.yml
|
||||||
/etc/backports_required_for_multiversion_tests.yml @10gen/devprod-correctness @svc-auto-approve-bot
|
/etc/backports_required_for_multiversion_tests.yml @10gen/devprod-correctness @svc-auto-approve-bot
|
||||||
/etc/burn_in_tests.yml @10gen/devprod-correctness @svc-auto-approve-bot
|
/etc/burn_in_tests.yml @10gen/devprod-correctness @svc-auto-approve-bot
|
||||||
|
|
|
||||||
|
|
@ -1,183 +1 @@
|
||||||
# Resmoke Test Suites
|
> Content moved to [buildscripts/resmokeconfig/suites/README.md](../../buildscripts/resmokeconfig/suites/README.md).
|
||||||
|
|
||||||
Resmoke stores test suites represented as `.yml` files in the `buildscripts/resmokeconfig/suites`
|
|
||||||
directory. These `.yml` files allow users to spin up a variety of configurations to run tests
|
|
||||||
against.
|
|
||||||
|
|
||||||
# Suite Fields
|
|
||||||
|
|
||||||
## test_kind - [Root Level]
|
|
||||||
|
|
||||||
This represents the type of tests that are running in this suite. Some examples include: _js_test,
|
|
||||||
cpp_unit_test, cpp_integration_tests, benchmark_test, fsm_workload_test, etc._ You can see all
|
|
||||||
available options in the `_SELECTOR_REGISTRY` at `mongo/buildscripts/resmokelib/selector.py`.
|
|
||||||
|
|
||||||
Ex:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
test_kind: js_test
|
|
||||||
```
|
|
||||||
|
|
||||||
## selector - [Root Level]
|
|
||||||
|
|
||||||
The selector determines test files to include/exclude in the suite.
|
|
||||||
|
|
||||||
Ex:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
selector:
|
|
||||||
roots:
|
|
||||||
- jstests/aggregation/**/*.js
|
|
||||||
exclude_files:
|
|
||||||
- jstests/aggregation/extras/*.js
|
|
||||||
- jstests/aggregation/data/*.js
|
|
||||||
exclude_with_any_tags:
|
|
||||||
- requires_pipeline_optimization
|
|
||||||
```
|
|
||||||
|
|
||||||
### selector.roots
|
|
||||||
|
|
||||||
File path(s) of test files to include. If a path without a glob is provided, it must exist.
|
|
||||||
|
|
||||||
### selector.exclude_files
|
|
||||||
|
|
||||||
File path(s) of test files to exclude. If a path without a glob is provided, it must exist.
|
|
||||||
|
|
||||||
### selector.exclude_with_any_tags
|
|
||||||
|
|
||||||
Exclude test files by tag name(s). To see all available tags, run
|
|
||||||
`./buildscripts/resmoke.py list-tags`.
|
|
||||||
|
|
||||||
## executor - [Root Level]
|
|
||||||
|
|
||||||
Configuration for the test execution framework.
|
|
||||||
|
|
||||||
Ex:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
executor:
|
|
||||||
archive:
|
|
||||||
---
|
|
||||||
config:
|
|
||||||
---
|
|
||||||
hooks:
|
|
||||||
---
|
|
||||||
fixture:
|
|
||||||
```
|
|
||||||
|
|
||||||
### executor.archive
|
|
||||||
|
|
||||||
Upon failure, data files can be uploaded to s3. A failure is when a `hook` or `test` throws an
|
|
||||||
exception. Data files will be archived in the following situations:
|
|
||||||
|
|
||||||
1. Any `hook` included in this section throws an exception.
|
|
||||||
2. If `tests: true` and any `test` in the suite throws an exception.
|
|
||||||
|
|
||||||
Ex:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
archive:
|
|
||||||
hooks:
|
|
||||||
- Hook1
|
|
||||||
- Hook2
|
|
||||||
---
|
|
||||||
tests: true
|
|
||||||
```
|
|
||||||
|
|
||||||
### executor.config
|
|
||||||
|
|
||||||
This section contains additional configuration for each test. The structure of this can vary
|
|
||||||
significantly based on the `test_kind`. For specific information, you can look at the
|
|
||||||
implementation of the `test_kind` of concern in the `buildscripts/resmokelib/testing/testcases`
|
|
||||||
directory.
|
|
||||||
|
|
||||||
Ex:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
config:
|
|
||||||
shell_options:
|
|
||||||
global_vars:
|
|
||||||
TestData:
|
|
||||||
defaultReadConcernLevel: null
|
|
||||||
nodb: ""
|
|
||||||
gssapiServiceName: "mockservice"
|
|
||||||
eval: >-
|
|
||||||
var testingReplication = true;
|
|
||||||
load('jstests/libs/override_methods/set_read_and_write_concerns.js');
|
|
||||||
load('jstests/libs/override_methods/enable_causal_consistency_without_read_pref.js');
|
|
||||||
```
|
|
||||||
|
|
||||||
Above is an example of the most common `test_kind` -- `js_test`. `js_test` uses `shell_options` to
|
|
||||||
customize the mongo shell when running tests.
|
|
||||||
|
|
||||||
`global_vars` allows for setting global variables. A `TestData` object is a special global variable
|
|
||||||
that is used to hold testing data. Parts of `TestData` can be updated via `resmoke` command-line
|
|
||||||
invocation, via `.yml` (as shown above), and during runtime. The global `TestData` object is merged
|
|
||||||
intelligently and made available to the `js_test` running. Behavior can vary on key collision, but
|
|
||||||
in general this is the order of precedence: (1) resmoke command-line (2) [suite].yml (3)
|
|
||||||
runtime/default.
|
|
||||||
|
|
||||||
The mongo shell can also be invoked with flags &
|
|
||||||
named arguments. Flags must have the `''` value, such as in the case for `nodb` above.
|
|
||||||
|
|
||||||
`eval` can also be used to run generic javascript code in the shell. You can directly include
|
|
||||||
javascript code, or you can put it in a separate script & `load` it.
|
|
||||||
|
|
||||||
### executor.hooks
|
|
||||||
|
|
||||||
All hooks inherit from the `buildscripts.resmokelib.testing.hooks.interface.Hook` parent class and
|
|
||||||
can override any subset of the following empty base methods: `before_suite`, `after_suite`,
|
|
||||||
`before_test`, `after_test`. At least 1 base method must be overridden, otherwise the hook will
|
|
||||||
not do anything at all. During test suite execution, each hook runs its custom logic in the
|
|
||||||
respective scenarios. Some customizable tasks that hooks can perform include: _validating data,
|
|
||||||
deleting data, performing cleanup_, etc. You can see all existing hooks in the
|
|
||||||
`buildscripts/resmokelib/testing/hooks` directory.
|
|
||||||
|
|
||||||
Ex:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
hooks:
|
|
||||||
- class: CheckReplOplogs
|
|
||||||
- class: CheckReplDBHash
|
|
||||||
- class: ValidateCollections
|
|
||||||
- class: CleanEveryN
|
|
||||||
n: 20
|
|
||||||
- class: MyHook
|
|
||||||
param1: something
|
|
||||||
param2: somethingelse
|
|
||||||
```
|
|
||||||
|
|
||||||
The hook name in the `.yml` must match its Python class name in the
|
|
||||||
`buildscripts/resmokelib/testing/hooks` directory. Parameters can also be included in the `.yml`
|
|
||||||
and will be passed to the hook's constructor (the `hook_logger` & `fixture` parameters are
|
|
||||||
automatically included, so those should not be included in the `.yml`).
|
|
||||||
|
|
||||||
### executor.fixture
|
|
||||||
|
|
||||||
This represents the test fixture to run tests against. The `class` sub-field corresponds to the
|
|
||||||
Python class name of a fixture in the `buildscripts/resmokelib/testing/fixtures` directory. All
|
|
||||||
other sub-fields are passed into the constructor of the fixture. These sub-fields will vary based
|
|
||||||
on the fixture used.
|
|
||||||
|
|
||||||
Ex:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
fixture:
|
|
||||||
class: ShardedClusterFixture
|
|
||||||
num_shards: 2
|
|
||||||
mongos_options:
|
|
||||||
bind_ip_all: ""
|
|
||||||
set_parameters:
|
|
||||||
enableTestCommands: 1
|
|
||||||
mongod_options:
|
|
||||||
bind_ip_all: ""
|
|
||||||
set_parameters:
|
|
||||||
enableTestCommands: 1
|
|
||||||
periodicNoopIntervalSecs: 1
|
|
||||||
writePeriodicNoops: true
|
|
||||||
```
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
For inspiration on creating a new test suite, you can check out a variety of examples in the
|
|
||||||
`buildscripts/resmokeconfig/suites` directory.
|
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,9 @@ filters:
|
||||||
- "*":
|
- "*":
|
||||||
approvers:
|
approvers:
|
||||||
- 10gen/mongo-default-approvers
|
- 10gen/mongo-default-approvers
|
||||||
|
- "README.md":
|
||||||
|
approvers:
|
||||||
|
- 10gen/devprod-correctness
|
||||||
- "aggregation*":
|
- "aggregation*":
|
||||||
approvers:
|
approvers:
|
||||||
- 10gen/query
|
- 10gen/query
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,187 @@
|
||||||
|
# Resmoke Test Suites
|
||||||
|
|
||||||
|
Test "suites" are configuration files that group which tests to run, and how.
|
||||||
|
|
||||||
|
Yaml files enumerate the test files that the suite encompasses, as well as any test fixtures and their configuations to leverage, options for the shell, hooks, and more.
|
||||||
|
|
||||||
|
## Minimal Example
|
||||||
|
|
||||||
|
```
|
||||||
|
test_kind: js_test
|
||||||
|
selector:
|
||||||
|
roots:
|
||||||
|
- jstests/mytests/**/*.js
|
||||||
|
executor:
|
||||||
|
config:
|
||||||
|
shell_options:
|
||||||
|
nodb: ""
|
||||||
|
```
|
||||||
|
|
||||||
|
# Suite Fields
|
||||||
|
|
||||||
|
## `test_kind`
|
||||||
|
|
||||||
|
This represents the type of tests that are running in this suite. Some examples include: _js_test,
|
||||||
|
cpp_unit_test, cpp_integration_tests, benchmark_test, fsm_workload_test_, etc. You can see all
|
||||||
|
available options in the `_SELECTOR_REGISTRY` at [buildscripts/resmokelib/selector.py](../../../buildscripts/resmokelib/selector.py).
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
test_kind: js_test
|
||||||
|
```
|
||||||
|
|
||||||
|
## `selector`
|
||||||
|
|
||||||
|
The selector determines test files to include/exclude in the suite.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
selector:
|
||||||
|
roots:
|
||||||
|
- jstests/aggregation/**/*.js
|
||||||
|
exclude_files:
|
||||||
|
- jstests/aggregation/extras/*.js
|
||||||
|
- jstests/aggregation/data/*.js
|
||||||
|
exclude_with_any_tags:
|
||||||
|
- requires_pipeline_optimization
|
||||||
|
```
|
||||||
|
|
||||||
|
### `selector.roots`
|
||||||
|
|
||||||
|
File path(s) of test files to include. If a path without a glob is provided, it must exist.
|
||||||
|
|
||||||
|
### `selector.exclude_files`
|
||||||
|
|
||||||
|
File path(s) of test files to exclude. If a path without a glob is provided, it must exist.
|
||||||
|
|
||||||
|
### `selector.exclude_with_any_tags`
|
||||||
|
|
||||||
|
Exclude test files by tag name(s). To see all available tags, run
|
||||||
|
`./buildscripts/resmoke.py list-tags`.
|
||||||
|
|
||||||
|
## `executor`
|
||||||
|
|
||||||
|
Configuration for the test execution framework.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
executor:
|
||||||
|
archive:
|
||||||
|
---
|
||||||
|
config:
|
||||||
|
---
|
||||||
|
hooks:
|
||||||
|
---
|
||||||
|
fixture:
|
||||||
|
```
|
||||||
|
|
||||||
|
### `executor.archive`
|
||||||
|
|
||||||
|
Upon failure, data files can be uploaded to s3. A failure is when a `hook` or `test` throws an
|
||||||
|
exception. Data files will be archived in the following situations:
|
||||||
|
|
||||||
|
1. Any `hook` included in this section throws an exception.
|
||||||
|
2. If `tests: true` and any `test` in the suite throws an exception.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
archive:
|
||||||
|
hooks:
|
||||||
|
- Hook1
|
||||||
|
- Hook2
|
||||||
|
---
|
||||||
|
tests: true
|
||||||
|
```
|
||||||
|
|
||||||
|
### `executor.config`
|
||||||
|
|
||||||
|
This section contains additional configuration for each test. The structure of this can vary
|
||||||
|
significantly based on the `test_kind`. For specific information, you can look at the
|
||||||
|
implementation of the `test_kind` of concern in the `buildscripts/resmokelib/testing/testcases`
|
||||||
|
directory.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
config:
|
||||||
|
shell_options:
|
||||||
|
global_vars:
|
||||||
|
TestData:
|
||||||
|
defaultReadConcernLevel: null
|
||||||
|
nodb: ""
|
||||||
|
gssapiServiceName: "mockservice"
|
||||||
|
eval: >-
|
||||||
|
var testingReplication = true;
|
||||||
|
load('jstests/libs/override_methods/set_read_and_write_concerns.js');
|
||||||
|
load('jstests/libs/override_methods/enable_causal_consistency_without_read_pref.js');
|
||||||
|
```
|
||||||
|
|
||||||
|
Above is an example of the most common `test_kind` -- `js_test`. `js_test` uses `shell_options` to
|
||||||
|
customize the mongo shell when running tests.
|
||||||
|
|
||||||
|
`global_vars` allows for setting global variables. A `TestData` object is a special global variable
|
||||||
|
that is used to hold testing data. Parts of `TestData` can be updated via `resmoke` command-line
|
||||||
|
invocation, via `.yml` (as shown above), and during runtime. The global `TestData` object is merged
|
||||||
|
intelligently and made available to the `js_test` running. Behavior can vary on key collision, but
|
||||||
|
in general this is the order of precedence: (1) resmoke command-line (2) [suite].yml (3)
|
||||||
|
runtime/default.
|
||||||
|
|
||||||
|
The mongo shell can also be invoked with flags &
|
||||||
|
named arguments. Flags must have the `''` value, such as in the case for `nodb` above.
|
||||||
|
|
||||||
|
`eval` can also be used to run generic javascript code in the shell. You can directly include
|
||||||
|
javascript code, or you can put it in a separate script & `load` it.
|
||||||
|
|
||||||
|
### `executor.hooks`
|
||||||
|
|
||||||
|
Specify hooks can run before, after, and between individual tests to execute specified logic.
|
||||||
|
|
||||||
|
The hook name in the `.yml` must match its Python class name in the
|
||||||
|
`buildscripts/resmokelib/testing/hooks` directory. Parameters can also be included in the `.yml`
|
||||||
|
and will be passed to the hook's constructor (the `hook_logger` & `fixture` parameters are
|
||||||
|
automatically included, so those should not be included in the `.yml`).
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
hooks:
|
||||||
|
- class: CheckReplOplogs
|
||||||
|
- class: CheckReplDBHash
|
||||||
|
- class: ValidateCollections
|
||||||
|
- class: CleanEveryN
|
||||||
|
n: 20
|
||||||
|
- class: MyHook
|
||||||
|
param1: something
|
||||||
|
param2: somethingelse
|
||||||
|
```
|
||||||
|
|
||||||
|
> Read more about implementing hooks in [buildscripts/resmokelib/testing/hooks/README.md](../../../buildscripts/resmokelib/testing/hooks/README.md)
|
||||||
|
|
||||||
|
### `executor.fixture`
|
||||||
|
|
||||||
|
This represents the test fixture to run tests against. The `class` sub-field corresponds to the
|
||||||
|
Python class name of a fixture in the [buildscripts/resmokelib/testing/fixtures](../../..buildscripts/resmokelib/testing/fixtures)` directory. All
|
||||||
|
other sub-fields are passed into the constructor of the fixture. These sub-fields will vary based
|
||||||
|
on the fixture used.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
fixture:
|
||||||
|
class: ShardedClusterFixture
|
||||||
|
num_shards: 2
|
||||||
|
mongos_options:
|
||||||
|
bind_ip_all: ""
|
||||||
|
set_parameters:
|
||||||
|
enableTestCommands: 1
|
||||||
|
mongod_options:
|
||||||
|
bind_ip_all: ""
|
||||||
|
set_parameters:
|
||||||
|
enableTestCommands: 1
|
||||||
|
periodicNoopIntervalSecs: 1
|
||||||
|
writePeriodicNoops: true
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,99 @@
|
||||||
|
# Resmoke
|
||||||
|
|
||||||
|
Resmoke is MongoDB's integration test runner.
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
Though the source is built with bazel, resmoke is not yet integrated. This means that the source has to be built prior to using resmoke, eg:
|
||||||
|
|
||||||
|
```
|
||||||
|
bazel build install-dist-test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Actions
|
||||||
|
|
||||||
|
<!-- the following is a copy-paste from `resmoke --help` -->
|
||||||
|
|
||||||
|
```
|
||||||
|
run Runs the specified tests.
|
||||||
|
list-suites Lists the names of the suites available to execute.
|
||||||
|
generate-matrix-suites
|
||||||
|
Generate matrix suite config files from the mapping files.
|
||||||
|
find-suites Lists the names of the suites that will execute the specified tests.
|
||||||
|
list-tags Lists the tags and their documentation available in the suites.
|
||||||
|
generate-multiversion-exclude-tags
|
||||||
|
Create a tag file associating multiversion tests to tags for exclusion. Compares the BACKPORTS_REQUIRED_FILE on the current branch with the same file on the last-lts and/or last-continuous branch to determine which tests should be
|
||||||
|
denylisted.
|
||||||
|
core-analyzer Analyzes the core dumps from the specified input files.
|
||||||
|
hang-analyzer Hang Analyzer module. A prototype hang analyzer for Evergreen integration to help investigate test timeouts 1. Script supports taking dumps, and/or dumping a summary of useful information about a process 2. Script will iterate
|
||||||
|
through a list of interesting processes, and run the tools from step 1. The list of processes can be provided as an option. 3. Java processes will be dumped using jstack, if available. Supports Linux, MacOS X, and Windows.
|
||||||
|
powercycle Powercycle test. Tests robustness of mongod to survive multiple powercycle events. Client & server side powercycle test script. This script is used in conjunction with certain Evergreen hosts created with the `evergreen host
|
||||||
|
create` command.
|
||||||
|
generate-fcv-constants
|
||||||
|
==SUPPRESS==
|
||||||
|
test-discovery Discover what tests are run by a suite.
|
||||||
|
suiteconfig Display configuration of a test suite.
|
||||||
|
multiversion-config
|
||||||
|
Display configuration for multiversion testing
|
||||||
|
generate-fuzz-config
|
||||||
|
Generate a mongod.conf and mongos.conf using config fuzzer.
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: `bisect`, `setup-multiversion`, and `symbolize` commands have been moved to [`db-contrib-tool`](https://github.com/10gen/db-contrib-tool#readme).
|
||||||
|
|
||||||
|
## Suites
|
||||||
|
|
||||||
|
Many of the above commands use the concept of a "suite". Loosely, suites group which tests run, and how.
|
||||||
|
|
||||||
|
Read more about suites [here](../../buildscripts/resmokeconfig/suites/README.md).
|
||||||
|
|
||||||
|
## run
|
||||||
|
|
||||||
|
The `run` command is the most used feature of resmoke.
|
||||||
|
|
||||||
|
The most typical approach is to run a particular JS test file given a suite, eg:
|
||||||
|
|
||||||
|
```
|
||||||
|
buildscripts/resmoke.py run --suites=no_passthrough jstests/noPassthrough/shell/js/string.js
|
||||||
|
```
|
||||||
|
|
||||||
|
That executes the content of that file, using the suite configuration as a fixture setup. The suite "no_passthrough" is associated with the file [buildscripts/resmokeconfig/suites/no_passthrough.yml](../../buildscripts/resmokeconfig/suites/no_passthrough.yml).
|
||||||
|
|
||||||
|
Run has **100+ flags**! Use `resmoke run --help` to inspect them. To avoid risk of multiple sources of truth that can drift and become stale, **we do not attempt to document them all here** - they should each be self-descriptive and documented within the CLI help.
|
||||||
|
|
||||||
|
Below are very high-level descriptions for high-usage flags.
|
||||||
|
|
||||||
|
### Suites (`--suites`)
|
||||||
|
|
||||||
|
The run subcommand can run suites (list of tests and the MongoDB topology and
|
||||||
|
configuration to run them against), and explicitly named test files.
|
||||||
|
|
||||||
|
A single suite can be specified using the `--suite` flag, and multiple suites
|
||||||
|
can be specified by providing a comma separated list to the `--suites` flag.
|
||||||
|
|
||||||
|
Additional documentation on our suite configuration can be found in
|
||||||
|
[buildscripts/resmokeconfig/suites/README.md](../../buildscripts/resmokeconfig/suites/README.md).
|
||||||
|
|
||||||
|
### Testable Installations (`--installDir`)
|
||||||
|
|
||||||
|
resmoke can run tests against any testable installation of MongoDB (such
|
||||||
|
as ASAN, Debug, Release). When possible, resmoke will automatically locate and
|
||||||
|
run with a locally built copy of MongoDB Server, so long as that build was
|
||||||
|
installed to a subdirectory of the root of the git repository, and there is
|
||||||
|
exactly one build. In other situations, the `--installDir` flag, passed to run
|
||||||
|
subcommand, can be used to indicate the location of the mongod/mongos binaries.
|
||||||
|
|
||||||
|
As an alternative, you may instead prefer to use the resmoke.py wrapper script
|
||||||
|
located in the same directory as the mongod binary, which will automatically
|
||||||
|
set `installDir` for you.
|
||||||
|
|
||||||
|
Note that this wrapper is unavailable in packaged installations of MongoDB
|
||||||
|
Server, such as those provided by Homebrew, and other package managers. If you
|
||||||
|
would like to run tests against a packaged installation, you must explicitly
|
||||||
|
pass `--installDir` to resmoke.py
|
||||||
|
|
||||||
|
### Resmoke test telemetry
|
||||||
|
|
||||||
|
We capture telemetry from resmoke using open telemetry.
|
||||||
|
|
||||||
|
Using open telemetry (OTel) we capture more specific information about the internals of resmoke. This data is used for improvements specifically when running in evergreen. This data is captured on every resmoke invocation but only sent to honeycomb when running in evergreen. More info about how we use OTel in resmoke can be found [here](otel_resmoke.md).
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
# Fixtures
|
||||||
|
|
||||||
|
Fixtures define a specific topology that tests run against.
|
||||||
|
|
@ -3,6 +3,9 @@ filters:
|
||||||
- "*":
|
- "*":
|
||||||
approvers:
|
approvers:
|
||||||
- 10gen/mongo-default-approvers
|
- 10gen/mongo-default-approvers
|
||||||
|
- "README.md":
|
||||||
|
approvers:
|
||||||
|
- 10gen/devprod-correctness
|
||||||
- "analyze_shard_key.py":
|
- "analyze_shard_key.py":
|
||||||
approvers:
|
approvers:
|
||||||
- 10gen/server-cluster-scalability
|
- 10gen/server-cluster-scalability
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
# Hooks
|
||||||
|
|
||||||
|
All hooks inherit from the `buildscripts.resmokelib.testing.hooks.interface.Hook` parent class and
|
||||||
|
can override any subset of the following empty base methods: `before_suite`, `after_suite`,
|
||||||
|
`before_test`, `after_test`. At least 1 base method must be overridden, otherwise the hook will
|
||||||
|
not do anything at all. During test suite execution, each hook runs its custom logic in the
|
||||||
|
respective scenarios. Some customizable tasks that hooks can perform include: _validating data,
|
||||||
|
deleting data, performing cleanup_, etc.
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
version: 2.0.0
|
||||||
|
filters:
|
||||||
|
- "*":
|
||||||
|
approvers:
|
||||||
|
- 10gen/devprod-correctness
|
||||||
|
|
@ -3,40 +3,35 @@
|
||||||
Most tests for MongoDB are run through resmoke, our test runner and orchestration tool.
|
Most tests for MongoDB are run through resmoke, our test runner and orchestration tool.
|
||||||
The entry point for resmoke can be found at `buildscripts/resmoke.py`
|
The entry point for resmoke can be found at `buildscripts/resmoke.py`
|
||||||
|
|
||||||
## run
|
## Concepts
|
||||||
|
|
||||||
The run subcommand can run suites (list of tests and the MongoDB topology and
|
Learn more about related topics using their own targeted documentation:
|
||||||
configuration to run them against), and explicitly named test files.
|
|
||||||
|
|
||||||
A single suite can be specified using the `--suite` flag, and multiple suites
|
- [resmoke](../../buildscripts/resmokelib/README.md), the test runner
|
||||||
can be specified by providing a comma separated list to the `--suites` flag.
|
- [suites](../../buildscripts/resmokeconfig/suites/README.md), how tests are grouped and configured
|
||||||
|
- [fixtures](../../buildscripts/resmokelib/testing/fixtures/README.md), specify the server topology that tests run against
|
||||||
|
- [hooks](../../buildscripts/resmokelib/testing/hooks/README.md), logic to run before, after and/or between individual tests
|
||||||
|
|
||||||
Additional parameters for the run subcommand can be found on the help page,
|
## Basic Example
|
||||||
accessible by running `buildscripts/resmoke.py run --help`
|
|
||||||
|
|
||||||
Additional documentation on our suite configuration can be found on the
|
First, ensure that your python `venv` is active and up to date:
|
||||||
[Suites configuration file page](../../buildscripts/docs/suites.md)
|
|
||||||
|
|
||||||
### Testable Installations (`--installDir`)
|
```
|
||||||
|
python3 -m venv python3-venv
|
||||||
|
source python3-venv/bin/activate
|
||||||
|
buildscripts/poetry_sync.sh
|
||||||
|
```
|
||||||
|
|
||||||
resmoke can run tests against any testable installation of MongoDB (such
|
and you've built the source binaries to run against, eg:
|
||||||
as ASAN, Debug, Release). When possible, resmoke will automatically locate and
|
|
||||||
run with a locally built copy of MongoDB Server, so long as that build was
|
|
||||||
installed to a subdirectory of the root of the git repository, and there is
|
|
||||||
exactly one build. In other situations, the `--installDir` flag, passed to run
|
|
||||||
subcommand, can be used to indicate the location of the mongod/mongos binaries.
|
|
||||||
|
|
||||||
As an alternative, you may instead prefer to use the resmoke.py wrapper script
|
```
|
||||||
located in the same directory as the mongod binary, which will automatically
|
bazel build install-dist-test
|
||||||
set `installDir` for you.
|
```
|
||||||
|
|
||||||
Note that this wrapper is unavailable in packaged installations of MongoDB
|
Now, **run the test content** from one test file:
|
||||||
Server, such as those provided by Homebrew, and other package managers. If you
|
|
||||||
would like to run tests against a packaged installation, you must explicitly
|
|
||||||
pass `--installDir` to resmoke.py
|
|
||||||
|
|
||||||
### Resmoke test telemetry
|
```
|
||||||
|
buildscripts/resmoke.py run --suites=no_passthrough jstests/noPassthrough/shell/js/string.js
|
||||||
|
```
|
||||||
|
|
||||||
We capture telemetry from resmoke using open telemetry.
|
The suite defined in [buildscripts/resmokeconfig/suites/no_passthrough.yml](../../buildscripts/resmokeconfig/suites/no_passthrough.yml) includes that `string.js` file via glob selections, specifies no fixtures, no hooks, and a minimal config for the executor.
|
||||||
|
|
||||||
Using open telemetry (OTel) we capture more specific information about the internals of resmoke. This data is used for improvements specifically when running in evergreen. This data is captured on every resmoke invocation but only sent to honeycomb when running in evergreen. More info about how we use OTel in resmoke can be found [here](otel_resmoke.md).
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue