mirror of https://github.com/mongodb/mongo
SERVER-107798: Build out Resmoke-related Readme's (#38778)
GitOrigin-RevId: dc9f369f152887f02ab9b96bf0ab3dcd3e5e053f
This commit is contained in:
Steve McClure
MongoDB Bot
parent
28e9ecb9c8
commit
76f1fb6f34
|
|
@ -88,7 +88,8 @@ WORKSPACE.bazel @10gen/devprod-build @svc-auto-approve-bot
|
|||
/buildscripts/resmokeconfig/ @10gen/devprod-correctness @svc-auto-approve-bot
|
||||
|
||||
# The following patterns are parsed from ./buildscripts/resmokeconfig/matrix_suites/OWNERS.yml
|
||||
/buildscripts/resmokeconfig/matrix_suites/**/* @10gen/mongo-default-approvers @svc-auto-approve-bot
|
||||
/buildscripts/resmokeconfig/matrix_suites/ @10gen/mongo-default-approvers @svc-auto-approve-bot
|
||||
/buildscripts/resmokeconfig/matrix_suites/README.md @10gen/devprod-correctness @svc-auto-approve-bot
|
||||
/buildscripts/resmokeconfig/matrix_suites/generated_suites/*viewless_timeseries.yml @10gen/server-catalog-and-routing @svc-auto-approve-bot
|
||||
/buildscripts/resmokeconfig/matrix_suites/generated_suites/*injected_catalog_metadata.yml @10gen/server-catalog-and-routing @svc-auto-approve-bot
|
||||
/buildscripts/resmokeconfig/matrix_suites/generated_suites/sharding_task_executor_pool_size* @10gen/server-networking-and-observability @svc-auto-approve-bot
|
||||
|
|
@ -301,6 +302,7 @@ WORKSPACE.bazel @10gen/devprod-build @svc-auto-approve-bot
|
|||
/evergreen/perf-submission.sh @10gen/devprod-performance-analysis @svc-auto-approve-bot
|
||||
|
||||
# The following patterns are parsed from ./jstests/OWNERS.yml
|
||||
/jstests/README.md @10gen/devprod-correctness @svc-auto-approve-bot
|
||||
/jstests/**/*analyze_shard_key* @10gen/server-cluster-scalability @svc-auto-approve-bot
|
||||
/jstests/**/*move*chunk* @10gen/server-cluster-scalability @svc-auto-approve-bot
|
||||
/jstests/**/*moveChunk* @10gen/server-cluster-scalability @svc-auto-approve-bot
|
||||
|
|
|
|||
|
|
@ -1,23 +1,26 @@
|
|||
version: 1.0.0
|
||||
version: 2.0.0
|
||||
filters:
|
||||
- "*":
|
||||
approvers:
|
||||
- 10gen/mongo-default-approvers
|
||||
- "generated_suites/*viewless_timeseries.yml":
|
||||
- "README.md":
|
||||
approvers:
|
||||
- 10gen/devprod-correctness
|
||||
- "/generated_suites/*viewless_timeseries.yml":
|
||||
approvers:
|
||||
- 10gen/server-catalog-and-routing
|
||||
- "generated_suites/*injected_catalog_metadata.yml":
|
||||
- "/generated_suites/*injected_catalog_metadata.yml":
|
||||
approvers:
|
||||
- 10gen/server-catalog-and-routing
|
||||
- "generated_suites/sharding_task_executor_pool_size*":
|
||||
- "/generated_suites/sharding_task_executor_pool_size*":
|
||||
approvers:
|
||||
- 10gen/server-networking-and-observability
|
||||
- "generated_suites/change_streams_*":
|
||||
- "/generated_suites/change_streams_*":
|
||||
approvers:
|
||||
- 10gen/query-execution-change-streams
|
||||
- "generated_suites/*ingress_request_rate_limiter*":
|
||||
- "/generated_suites/*ingress_request_rate_limiter*":
|
||||
approvers:
|
||||
- 10gen/server-workload-scheduling
|
||||
- "generated_suites/bulk_write_*":
|
||||
- "/generated_suites/bulk_write_*":
|
||||
approvers:
|
||||
- 10gen/query-execution-router
|
||||
|
|
|
|||
|
|
@ -3,12 +3,11 @@
|
|||
## Summary
|
||||
|
||||
Matrix Suites are defined as a combination of explict
|
||||
suite files (in `buildscripts/resmokeconfig/suites` by default)
|
||||
[suite files](../../../buildscripts/resmokeconfig/suites/README.md)
|
||||
and a set of "overrides" for specific keys. The intention is
|
||||
to avoid duplication of suite definitions as much as
|
||||
possible with the eventual goal of having most suites be
|
||||
fully composed of reusable sections, similar to how Genny's
|
||||
workloads are defined as a set of parameterized `PhaseConfig`s.
|
||||
fully composed of reusable sections.
|
||||
|
||||
## Usage
|
||||
|
||||
|
|
@ -70,9 +69,9 @@ modifiers), the key being extended must already exist and also be a list.
|
|||
The generated matrix suites live in the `buildscripts/resmokeconfig/matrix_suites/generated_suites`
|
||||
directory. These files may be edited for local testing but must remain consistent with the mapping
|
||||
files. There is a task in the commit queue that enforces this. To generate a new version of these
|
||||
matrix suites, you may run `python3 ./buildscripts/resmoke.py generate-matrix-suites`. This commands
|
||||
matrix suites, you may run `python3 ./buildscripts/resmoke.py generate-matrix-suites`. This command
|
||||
will overwrite the current generated matrix suites on disk so make sure you do not have any unsaved
|
||||
changes to these files. Finally, run `bazel run //:format` to ensure consistent formatting.
|
||||
changes to these files.
|
||||
|
||||
## Validating matrix suites
|
||||
|
||||
|
|
|
|||
|
|
@ -2,11 +2,13 @@
|
|||
|
||||
Test "suites" are configuration files that group which tests to run, and how.
|
||||
|
||||
Yaml files enumerate the test files that the suite encompasses, as well as any test fixtures and their configuations to leverage, options for the shell, hooks, and more.
|
||||
Yaml files enumerate the test files that the suite encompasses, as well as any test fixtures and their configurations to leverage, options for the shell, hooks, and more.
|
||||
|
||||
## Minimal Example
|
||||
|
||||
```
|
||||
`my_suite.yml`:
|
||||
|
||||
```yaml
|
||||
test_kind: js_test
|
||||
selector:
|
||||
roots:
|
||||
|
|
@ -17,13 +19,44 @@ executor:
|
|||
nodb: ""
|
||||
```
|
||||
|
||||
This relays the following:
|
||||
|
||||
- The suite name is the filename, "my_suite"
|
||||
- The suite includes all JS files in the `jstests/mytests` directory
|
||||
- Those tests are run against a shell that is passed the `nodb: ""` options
|
||||
|
||||
The following is one with placeholders that illustrate the overall structure:
|
||||
|
||||
```yaml
|
||||
test_kind: js_test
|
||||
selector:
|
||||
roots:
|
||||
- jstests/mytests/**/*.js
|
||||
executor:
|
||||
config:
|
||||
shell_options:
|
||||
nodb: ""
|
||||
global_vars:
|
||||
TestData:
|
||||
defaultReadConcernLevel: null
|
||||
hooks:
|
||||
- class: ValidateCollections
|
||||
- class: CleanEveryN
|
||||
n: 20
|
||||
fixture:
|
||||
class: ShardedClusterFixture
|
||||
num_shards: 2
|
||||
archive:
|
||||
tests: true
|
||||
hooks:
|
||||
- ValidateCollections
|
||||
```
|
||||
|
||||
# Suite Fields
|
||||
|
||||
## `test_kind`
|
||||
|
||||
This represents the type of tests that are running in this suite. Some examples include: _js_test,
|
||||
cpp_unit_test, cpp_integration_tests, benchmark_test, fsm_workload_test_, etc. You can see all
|
||||
available options in the `_SELECTOR_REGISTRY` at [buildscripts/resmokelib/selector.py](../../../buildscripts/resmokelib/selector.py).
|
||||
This represents the type of tests that are running in this suite.
|
||||
|
||||
Example:
|
||||
|
||||
|
|
@ -31,6 +64,30 @@ Example:
|
|||
test_kind: js_test
|
||||
```
|
||||
|
||||
The following kinds are supported (see [`_SELECTOR_REGISTRY`](../../../buildscripts/resmokelib/selector.py)):
|
||||
|
||||
- `all_versions_js_test`
|
||||
- `benchmark_test`
|
||||
- `bulk_write_cluster_js_test`
|
||||
- `cpp_integration_test`
|
||||
- `cpp_libfuzzer_test`
|
||||
- `cpp_unit_test`
|
||||
- `db_test`
|
||||
- `fsm_workload_test`
|
||||
- `js_test`
|
||||
- `json_schema_test`
|
||||
- `multi_stmt_txn_passthrough`
|
||||
- `magic_restore_js_test`
|
||||
- `parallel_fsm_workload_test`
|
||||
- `pretty_printer_test`
|
||||
- `py_test`
|
||||
- `query_tester_self_test`
|
||||
- `query_tester_server_test`
|
||||
- `sdam_json_test`
|
||||
- `server_selection_json_test`
|
||||
- `sleep_test`
|
||||
- `tla_plus_test`
|
||||
|
||||
## `selector`
|
||||
|
||||
The selector determines test files to include/exclude in the suite.
|
||||
|
|
@ -52,50 +109,33 @@ selector:
|
|||
|
||||
File path(s) of test files to include. If a path without a glob is provided, it must exist.
|
||||
|
||||
### `selector.root`
|
||||
|
||||
A file containing glob patterns, one per line, typically used by test_kind cpp_unit_test (usually build/unittests.txt). Specifies which tests to consider for including into the suite. If no other options are specified, these are the tests that will be run. Glob patterns are supported (and common) here.
|
||||
|
||||
### `selector.include_files`
|
||||
|
||||
A list of strings representing glob patterns. Includes only this subset of tests in the suite. These files will be included even if they would otherwise be excluded by tags. Will error if a test specified here was not included in the roots.
|
||||
|
||||
### `selector.exclude_files`
|
||||
|
||||
File path(s) of test files to exclude. If a path without a glob is provided, it must exist.
|
||||
A list of strings representing glob patterns. Excludes this list of tests from the suite. These files will be excluded even if they would otherwise be included by tags. Will error if a test specified here was not included in the roots.
|
||||
|
||||
### `selector.include_with_any_tags`
|
||||
|
||||
A list of strings. Only jstests which define a list of tags which includes any of these tags will be included in the suite, unless otherwise excluded by filename.
|
||||
|
||||
To see all available tags, run `./buildscripts/resmoke.py list-tags`.
|
||||
|
||||
### `selector.exclude_with_any_tags`
|
||||
|
||||
Exclude test files by tag name(s). To see all available tags, run
|
||||
`./buildscripts/resmoke.py list-tags`.
|
||||
A list of strings. Any jstest which defines a list of tags which includes any of these tags will be excluded from the suite, unless otherwise included by filename.
|
||||
|
||||
To see all available tags, run `./buildscripts/resmoke.py list-tags`.
|
||||
|
||||
## `executor`
|
||||
|
||||
Configuration for the test execution framework.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
executor:
|
||||
archive:
|
||||
---
|
||||
config:
|
||||
---
|
||||
hooks:
|
||||
---
|
||||
fixture:
|
||||
```
|
||||
|
||||
### `executor.archive`
|
||||
|
||||
Upon failure, data files can be uploaded to s3. A failure is when a `hook` or `test` throws an
|
||||
exception. Data files will be archived in the following situations:
|
||||
|
||||
1. Any `hook` included in this section throws an exception.
|
||||
2. If `tests: true` and any `test` in the suite throws an exception.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
archive:
|
||||
hooks:
|
||||
- Hook1
|
||||
- Hook2
|
||||
---
|
||||
tests: true
|
||||
```
|
||||
Defines how the tests will be executed.
|
||||
|
||||
### `executor.config`
|
||||
|
||||
|
|
@ -123,6 +163,14 @@ config:
|
|||
Above is an example of the most common `test_kind` -- `js_test`. `js_test` uses `shell_options` to
|
||||
customize the mongo shell when running tests.
|
||||
|
||||
#### `executor.config.shell_options`
|
||||
|
||||
Any parameters (besides `global_vars`) will directly be passed to the mongo shell executable.
|
||||
|
||||
##### `executor.config.shell_options.global_vars`
|
||||
|
||||
Will use this as the base for the string passed to `--eval`. Anything specified in `shell_options.eval` will be appended after these. Formats any objects so that they will evaluate properly as a string.
|
||||
|
||||
`global_vars` allows for setting global variables. A `TestData` object is a special global variable
|
||||
that is used to hold testing data. Parts of `TestData` can be updated via `resmoke` command-line
|
||||
invocation, via `.yml` (as shown above), and during runtime. The global `TestData` object is merged
|
||||
|
|
@ -138,10 +186,11 @@ javascript code, or you can put it in a separate script & `load` it.
|
|||
|
||||
### `executor.hooks`
|
||||
|
||||
Specify hooks can run before, after, and between individual tests to execute specified logic.
|
||||
Specify hooks to run before, after, and between individual tests to execute specified logic.
|
||||
|
||||
The hook name in the `.yml` must match its Python class name in the
|
||||
`buildscripts/resmokelib/testing/hooks` directory. Parameters can also be included in the `.yml`
|
||||
> Read more about hooks in [buildscripts/resmokelib/testing/hooks/README.md](../../../buildscripts/resmokelib/testing/hooks/README.md)
|
||||
|
||||
The hook name in the `.yml` must match its Python class name of the hook. Parameters can also be included in the `.yml`
|
||||
and will be passed to the hook's constructor (the `hook_logger` & `fixture` parameters are
|
||||
automatically included, so those should not be included in the `.yml`).
|
||||
|
||||
|
|
@ -159,14 +208,13 @@ hooks:
|
|||
param2: somethingelse
|
||||
```
|
||||
|
||||
> Read more about implementing hooks in [buildscripts/resmokelib/testing/hooks/README.md](../../../buildscripts/resmokelib/testing/hooks/README.md)
|
||||
|
||||
### `executor.fixture`
|
||||
|
||||
This represents the test fixture to run tests against. The `class` sub-field corresponds to the
|
||||
Python class name of a fixture in the [buildscripts/resmokelib/testing/fixtures](../../..buildscripts/resmokelib/testing/fixtures)` directory. All
|
||||
other sub-fields are passed into the constructor of the fixture. These sub-fields will vary based
|
||||
on the fixture used.
|
||||
Specify a test fixture to run around the tests.
|
||||
|
||||
> Read more about fixtures in [buildscripts/resmokelib/testing/fixtures/README.md](../../../buildscripts/resmokelib/testing/fixtures/README.md).
|
||||
|
||||
The `class` sub-field corresponds to the Python class name of a fixture. All other sub-fields are passed into the constructor of the fixture. These sub-fields will vary based on the fixture used.
|
||||
|
||||
Example:
|
||||
|
||||
|
|
@ -185,3 +233,31 @@ fixture:
|
|||
periodicNoopIntervalSecs: 1
|
||||
writePeriodicNoops: true
|
||||
```
|
||||
|
||||
### `executor.archive`
|
||||
|
||||
Upon failure, data files can be uploaded to s3. A failure is when a `hook` or `test` throws an
|
||||
exception. Data files will be archived in the following situations:
|
||||
|
||||
1. Any `hook` included in this section throws an exception.
|
||||
2. If `tests: true` and any `test` in the suite throws an exception.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
archive:
|
||||
hooks:
|
||||
- Hook1
|
||||
- Hook2
|
||||
tests: true
|
||||
```
|
||||
|
||||
#### `executor.archive.hooks`
|
||||
|
||||
Specify a list of hook class names to archive on failure. Set to `true` to archive _all_ hooks.
|
||||
|
||||
Read more about [hooks](../../../buildscripts/resmokelib/testing/hooks/README.md).
|
||||
|
||||
#### `executor.archive.tests`
|
||||
|
||||
Specify a list of test files to archive on failure. Wildcard selection a valid. Set to `true` to archive _all_ tests.
|
||||
|
|
|
|||
|
|
@ -2,6 +2,8 @@
|
|||
|
||||
Resmoke is MongoDB's integration test runner.
|
||||
|
||||
The JS Tests it can run live in the `jstests/` directory - reference its [README](../../jstests/README.md) to learn about their content.
|
||||
|
||||
## Build
|
||||
|
||||
Though the source is built with bazel, resmoke is not yet integrated. This means that the source has to be built prior to using resmoke, eg:
|
||||
|
|
|
|||
|
|
@ -1,3 +1,27 @@
|
|||
# Fixtures
|
||||
|
||||
Fixtures define a specific topology that tests run against.
|
||||
|
||||
## Supported Fixtures
|
||||
|
||||
Specify any these in your [Suite](../../../../buildscripts/resmokeconfig/suites/README.md) config.
|
||||
|
||||
- [`BulkWriteFixture`](./bulk_write.py) - Fixture which provides JSTests with a set of clusters to run tests against.
|
||||
- [`ExternalFixture`](./external.py) - Fixture which provides JSTests capability to connect to external (non-resmoke) cluster.
|
||||
- [`ExternalShardedClusterFixture`](./shardedcluster.py) - Fixture to interact with external sharded cluster fixture.
|
||||
- [`MongoDFixture`](./standalone.py) - Fixture which provides JSTests with a standalone mongod to run against.
|
||||
- [`MongoTFixture`](./mongot.py) - Fixture which provides JSTests with a mongot to run alongside a mongod.
|
||||
- [`MultiReplicaSetFixture`](./multi_replica_set.py) - Fixture which provides JSTests with a set of replica sets to run against.
|
||||
- [`MultiShardedClusterFixture`](./multi_sharded_cluster.py) - Fixture which provides JSTests with a set of sharded clusters to run against.
|
||||
- [`ReplicaSetFixture`](./replicaset.py) - Fixture which provides JSTests with a replica set to run against.
|
||||
- [`ShardedClusterFixture`](./shardedcluster.py) - Fixture which provides JSTests with a sharded cluster to run against.
|
||||
- Used when the MongoDB deployment is started by the JavaScript test itself with `MongoRunner`, `ReplSetTest`, or `ShardingTest`.
|
||||
- [`YesFixture`](./yesfixture.py) - Fixture which spawns several `yes` executables to generate lots of log messages.
|
||||
|
||||
## Interfaces
|
||||
|
||||
- [`Fixture`](./interface.py) - Base class for all fixtures.
|
||||
- [`MultiClusterFixture`](./interface.py) - Base class for fixtures that may consist of multiple independent participant clusters.
|
||||
- The participant clusters can function independently without coordination, but are bound together only for some duration as they participate in some process such as a migration. The participant clusters are fixtures themselves.
|
||||
- [`NoOpFixture`](./interface.py) - A Fixture implementation that does not start any servers.
|
||||
- [`ReplFixture`](./interface.py) - Base class for all fixtures that support replication.
|
||||
|
|
|
|||
|
|
@ -1,8 +1,87 @@
|
|||
# Hooks
|
||||
|
||||
All hooks inherit from the `buildscripts.resmokelib.testing.hooks.interface.Hook` parent class and
|
||||
can override any subset of the following empty base methods: `before_suite`, `after_suite`,
|
||||
`before_test`, `after_test`. At least 1 base method must be overridden, otherwise the hook will
|
||||
not do anything at all. During test suite execution, each hook runs its custom logic in the
|
||||
respective scenarios. Some customizable tasks that hooks can perform include: _validating data,
|
||||
deleting data, performing cleanup_, etc.
|
||||
Hooks are a mechanism to run routines _around_ the tests, at the test content boundaries.
|
||||
|
||||
## Supported hooks
|
||||
|
||||
Specify any these in your [Suite](../../../../buildscripts/resmokeconfig/suites/README.md) config.
|
||||
|
||||
- [`AnalyzeShardKeysInBackground`](./analyze_shard_key.py) - A hook for running `analyzeShardKey` commands while a test is running.
|
||||
- [`AntithesisLogging`](./antithesis_logging.py) - Prints antithesis commands before & after test run.
|
||||
- [`BackgroundInitialSync`](./initialsync.py) - Background Initial Sync
|
||||
- After every test, this hook checks if a background node has finished initial sync and if so validates it, tears it down, and restarts it.
|
||||
- This test accepts a parameter `n` that specifies a number of tests after which it will wait for replication to finish before validating and restarting the initial sync node.
|
||||
- This requires the ReplicaSetFixture to be started with `start_initial_sync_node=True`. If used at the same time as `CleanEveryN`, the `n` value passed to this hook should be equal to the `n` value for `CleanEveryN`.
|
||||
- [`CheckClusterIndexConsistency`](./cluster_index_consistency.py) - Checks that indexes are the same across chunks for the same collections.
|
||||
- [`CheckMetadataConsistencyInBackground`](./metadata_consistency) - Check the metadata consistency of a sharded cluster.
|
||||
- [`CheckOrphansDeleted`](./orphans.py) - Check if the range deleter failed to delete any orphan documents.
|
||||
- [`CheckReplChangeCollectionConsistency`](./change_collection_consistency.py) - Check that config.system.change_collection is consistent between the primary and secondaries for all tenants.
|
||||
- [`CheckReplDBHashInBackground`](./dbhash_background.py) - A hook for comparing the dbhashes of all replica set members while a test is running.
|
||||
- [`CheckReplDBHash`](./dbhash.py) - Check if the dbhashes match.
|
||||
- [`CheckReplOplogs`](./oplog.py) - Check that `local.oplog.rs` matches on the primary and secondaries.
|
||||
- [`CheckReplPreImagesConsistency`](./preimages_consistency.py) - Check that `config.system.preimages` is consistent between the primary and secondaries.
|
||||
- [`CheckRoutingTableConsistency`](./routing_table_consistency.py) - Verifies the absence of corrupted entries in config.chunks and config.collections.
|
||||
- [`CheckShardFilteringMetadata`](./shard_filtering_metadata.py) - Inspect filtering metadata on shards
|
||||
- [`CleanEveryN`](./cleanup.py) - Restart the fixture after it has ran `n` tests.
|
||||
- [`CleanupConcurrencyWorkloads`](./cleanup_concurrency_workloads.py) - Drop all databases, except those that have been excluded.
|
||||
- For concurrency tests that run on different DBs, drop all databases except ones in `exclude_dbs`. For tests that run on the same DB, drop all databases except ones in `exclude_dbs` and the DB used by the test/workloads. For tests that run on the same collection, drop all collections in all databases except for `exclude_dbs` and the collection used by the test/workloads.
|
||||
- On mongod-related fixtures, this will clear the dbpath
|
||||
- [`ClusterParameter`](./cluster_parameter.py) - Sets the specified cluster server parameter.
|
||||
- [`ContinuousAddRemoveShard`](./add_remove_shards.py) - Continuously adds and removes shards at regular intervals. If running with `configsvr` transitions, will transition in/out of config shard mode.
|
||||
- [`ContinuousInitialSync`](./continuous_initial_sync.py) - Periodically initial sync nodes then step them up.
|
||||
- [`ContinuousStepdown`](./stepdown.py) - regularly connect to replica sets and send a `replSetStepDown` command.
|
||||
- [`DoReconfigInBackground`](./reconfig_background.py) - A hook for running a safe reconfig against a replica set while a test is running.
|
||||
- [`DropUserCollections`](./drop_user_collections.py) - Drops all user collections.
|
||||
- [`EnableChangeStream`](./enable_change_stream.py) - Enable change stream hook class.
|
||||
- Enables change stream in the multi-tenant environment for the replica set and the sharded
|
||||
cluster.
|
||||
- [`EnableSpuriousWriteConflicts`](./enable_spurious_write_conflicts.py) - Toggles write conflicts.
|
||||
- [`FCVUpgradeDowngradeInBackground`](./fcv_upgrade_downgrade.py) - A hook to run background FCV upgrade and downgrade against test servers while a test is running.
|
||||
- [`FuzzRuntimeParameters`](./fuzz_runtime_parameters.py) - Regularly connect to nodes and sends them a `setParameter` command.
|
||||
- [`FuzzRuntimeStress`](./fuzz_runtime_stress.py) - Test hook that periodically changes the amount of stress the system is experiencing.
|
||||
- [`FuzzerRestoreSettings`](./fuzzer_restore_settings.py) - Cleans up unwanted changes from fuzzer.
|
||||
- [`GenerateAndCheckPerfResults`](./generate_and_check_perf_results.py) - Combine JSON results from individual benchmarks and check their reported values against any thresholds set for them.
|
||||
- Combines test results from individual benchmark files to a single file. This is useful for generating the json file to feed into the Evergreen performance visualization plugin.
|
||||
- [`HelloDelays`](./hello_failures.py) - Sets Hello fault injections.
|
||||
- [`IntermediateInitialSync`](./initialsync.py) - Intermediate Initial Sync
|
||||
- This hook accepts a parameter `n` that specifies a number of tests after which it will start up a node to initial sync, wait for replication to finish, and then validate the data.
|
||||
- This requires the ReplicaSetFixture to be started with 'start_initial_sync_node=True'.
|
||||
- [`LagOplogApplicationInBackground`](./secondary_lag.py) - Toggles secondary oplog application lag.
|
||||
- [`LibfuzzerHook`](./cpp_libfuzzer.py) - Merges inputs after a fuzzer run.
|
||||
- [`MagicRestoreEveryN`](./magic_restore.py) - Open a backup cursor and run magic restore process after `n` tests have run.
|
||||
- Requires the use of `MagicRestoreFixture`.
|
||||
- [`PeriodicKillSecondaries`](./periodic_kill_secondaries.py) - Periodically kills the secondaries in a replica set.
|
||||
- Also verifies that the secondaries can reach the SECONDARY state without having connectivity to the primary after an unclean shutdown.
|
||||
- [`PeriodicStackTrace`](./periodic_stack_trace.py) - Test hook that sends the stacktracing signal to mongo processes at randomized intervals.
|
||||
- [`QueryableServerHook`](./queryable_server_hook.py) - Starts the queryable server before each test for queryable restores. Restarts the queryable server between tests.
|
||||
- [`RunChangeStreamsInBackground`](./change_streams.py) - Run in the background full cluster change streams while a test is running. Open and close the change stream every `1..10` tests (random using `config.RANDOM_SEED`).
|
||||
- [`RunDBCheckInBackground`](./dbcheck_background.py) - A hook for running `dbCheck` on a replica set while a test is running.
|
||||
- This includes dbhashes for all non-local databases and non-replicated system collections that match on the primary and secondaries.
|
||||
- It also will check the performance results against any thresholds that are set for each benchmark. If no thresholds are set for a test, this hook should always pass.
|
||||
- [`RunQueryStats`](./run_query_stats.py) - Runs `$queryStats` after every test, and clears the query stats store before every test.
|
||||
- [`SimulateCrash`](./simulate_crash.py) - A hook to simulate crashes.
|
||||
- [`ValidateCollections`](./validate.py) - Run full validation.
|
||||
- [`ValidateCollectionsInBackground`](./validate_background.py) - A hook to run background collection validation against test servers while a test is running.
|
||||
- This will run on all collections in all databases on every stand-alone node, primary replica-set node, or primary shard node.
|
||||
- [`ValidateDirectSecondaryReads`](./validate_direct_secondary_reads.py) - Only supported in suites that use `ReplicaSetFixture`.
|
||||
- To be used with `set_read_preference_secondary.js` and `implicit_enable_profiler.js` in suites that read directly from secondaries in a replica set. Check the profiler collections of all databases at the end of the suite to verify that each secondary only ran the read commands it got directly from the shell.
|
||||
- [`WaitForReplication`](./wait_for_replication.py) - Wait for replication to complete.
|
||||
|
||||
## Interfaces
|
||||
|
||||
All hooks inherit from the [`buildscripts.resmokelib.testing.hooks.interface.Hook`](./interface.py) parent class and can override any subset of the following empty base methods:
|
||||
|
||||
- `before_suite`
|
||||
- `before_test`
|
||||
- `after_test`
|
||||
- `after_suite`
|
||||
|
||||
At least 1 base method must be overridden, otherwise the hook will not do anything at all. During test suite execution, each hook runs its custom logic in the respective scenarios. Some customizable tasks that hooks can perform include: _validating data, deleting data, performing cleanup_, etc.
|
||||
|
||||
- [`BGHook`](./bghook.py) - A hook that repeatedly calls `run_action()` in a background thread for the duration of the test suite.
|
||||
- [`DataConsistencyHook`](./jsfile.py) - A hook for running a static JavaScript file that checks data consistency of the server.
|
||||
- If the mongo shell process running the JavaScript file exits with a non-zero return code, then an `errors.ServerFailure` exception is raised to cause resmoke.py's test execution to stop.
|
||||
- [`Hook`](./interface.py) - Common interface all Hooks will inherit from.
|
||||
- [`JSHook`](./jsfile.py) - A hook interface with a static JavaScript file to execute.
|
||||
- [`PerClusterDataConsistencyHook`](./jsfile.py) - A hook that runs on each independent cluster of the fixture.
|
||||
- The independent cluster itself may be another fixture.
|
||||
|
|
|
|||
|
|
@ -1,32 +1,35 @@
|
|||
version: 1.0.0
|
||||
version: 2.0.0
|
||||
filters:
|
||||
- "*analyze_shard_key*":
|
||||
- "README.md":
|
||||
approvers:
|
||||
- 10gen/devprod-correctness
|
||||
- "**/*analyze_shard_key*":
|
||||
approvers:
|
||||
- 10gen/server-cluster-scalability
|
||||
- "*move*chunk*":
|
||||
- "**/*move*chunk*":
|
||||
approvers:
|
||||
- 10gen/server-cluster-scalability
|
||||
- "*moveChunk*":
|
||||
- "**/*moveChunk*":
|
||||
approvers:
|
||||
- 10gen/server-cluster-scalability
|
||||
- "*move*collection*":
|
||||
- "**/*move*collection*":
|
||||
approvers:
|
||||
- 10gen/server-cluster-scalability
|
||||
- "*move*range*":
|
||||
- "**/*move*range*":
|
||||
approvers:
|
||||
- 10gen/server-cluster-scalability
|
||||
- "*reshard*":
|
||||
- "**/*reshard*":
|
||||
approvers:
|
||||
- 10gen/server-cluster-scalability
|
||||
- "*unshard*collection*":
|
||||
- "**/*unshard*collection*":
|
||||
approvers:
|
||||
- 10gen/server-cluster-scalability
|
||||
- "*move*primary*":
|
||||
- "**/*move*primary*":
|
||||
approvers:
|
||||
- 10gen/server-catalog-and-routing
|
||||
- "*refine_collection_shard_key*":
|
||||
- "**/*refine_collection_shard_key*":
|
||||
approvers:
|
||||
- 10gen/server-catalog-and-routing
|
||||
- "*dbcheck*":
|
||||
- "**/*dbcheck*":
|
||||
approvers:
|
||||
- 10gen/server-dbcheck
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ It is best to use variable names that attempt to describe what a value is used f
|
|||
|
||||
All assertions in a test should attempt to verify the most specific property possible. For example, if you are trying to test that a certain collection exists, it is better to assert that the collection’s exact name exists in the list of collections, as opposed to verifying that the collection count is equal to 1. The desired collection’s existence is sufficient for the collection count to be 1, but not necessary (a different collection could exist in its place). Be wary of adding these kind of indirect assertions in a test.
|
||||
|
||||
## Modules in Practice
|
||||
## Modern JS: Modules in Practice
|
||||
|
||||
We have fully migrated to the modularized JavaScript world so any new test should use modules and adapt the new style.
|
||||
|
||||
|
|
@ -103,3 +103,49 @@ import * as MyModule from "/path/to/my_module.js";
|
|||
```
|
||||
|
||||
This can help the language server to discover the methods and provide code navigation for it.
|
||||
|
||||
### Use Mocha-style Constructs
|
||||
|
||||
The [mochalite.js](../jstests/libs/mochalite.js) library ports over a subset of [MochaJS](https://mochajs.org/) functionality for the shell, including:
|
||||
|
||||
- `it` test contruction
|
||||
- `describe` suite structures
|
||||
- `before` and `after` hooks, to run _once_ around _all_ `it` tests
|
||||
- `beforeEach` and `afterEach` hooks, to run around _each_ `it` test
|
||||
- The above (excluding `describe`) support `async` functions
|
||||
- Resmoke test filtering using the `--mochagrep` flag, which mirrors the [`grep`](https://mochajs.org/#-grep-regexp-g-regexp) flag from MochaJS
|
||||
|
||||
Example using all APIs:
|
||||
|
||||
```
|
||||
import {after, afterEach, before, beforeEach, describe, it} from "jstests/libs/mochalite.js";
|
||||
|
||||
describe("simple inserts and finds", () => {
|
||||
before(() => {
|
||||
this.fixtureDB = startupNewDB();
|
||||
});
|
||||
beforeEach(() => {
|
||||
this.fixtureDB.seed();
|
||||
});
|
||||
afterEach(async () => {
|
||||
await this.fixtureDB.clear();
|
||||
});
|
||||
after(() => {
|
||||
this.fixtureDB.shutdown();
|
||||
});
|
||||
it("should do something", () => {
|
||||
this.fixtureDB.insert({ name: "test" });
|
||||
assert.eq(this.fixtureDB.find({ name: "test" }).count(), 1);
|
||||
});
|
||||
it("should error on invalid data", () => {
|
||||
const e = assert.throws(() => this.fixtureDB.insert({ "notafield": undefined }));
|
||||
assert.eq(e.message, "Field 'notafield' not found");
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
Use the filter on resmoke to run just the one test:
|
||||
|
||||
```
|
||||
buildscripts/resmoke.py run --suites=no_passthrough --mochagrep "do something" jstests/noPassthrough/mytest.js
|
||||
```
|
||||
|
|
|
|||
Loading…
Reference in New Issue