SERVER-112569: Document fixture lifecycles better (#42763)

GitOrigin-RevId: 21e309ba0911ea3a94ef3d52714545367114ed6b

.github/CODEOWNERS

@@ -374,6 +374,7 @@ WORKSPACE.bazel @10gen/devprod-build @svc-auto-approve-bot
 
 # The following patterns are parsed from ./jstests/OWNERS.yml
 /jstests/README.md @10gen/devprod-correctness @svc-auto-approve-bot
+/jstests/tags.md @10gen/devprod-correctness @svc-auto-approve-bot
 /jstests/**/*analyze_shard_key* @10gen/server-cluster-scalability @svc-auto-approve-bot
 /jstests/**/*move*chunk* @10gen/server-cluster-scalability @svc-auto-approve-bot
 /jstests/**/*moveChunk* @10gen/server-cluster-scalability @svc-auto-approve-bot

buildscripts/resmokeconfig/suites/README.md

@@ -103,13 +103,13 @@ A list of strings representing glob patterns. Excludes this list of tests from t
 
 A list of strings. Only jstests which define a list of tags which includes any of these tags will be included in the suite, unless otherwise excluded by filename.
 
-To see all available tags, run `./buildscripts/resmoke.py list-tags`.
+To see all tags referenced across suites, run `./buildscripts/resmoke.py list-tags`.
 
 ### `selector.exclude_with_any_tags`
 
 A list of strings. Any jstest which defines a list of tags which includes any of these tags will be excluded from the suite, unless otherwise included by filename.
 
-To see all available tags, run `./buildscripts/resmoke.py list-tags`.
+To see all tags referenced across suites, run `./buildscripts/resmoke.py list-tags`.
 
 ## `executor`
 

buildscripts/resmokelib/testing/testcases/README.md

@@ -16,6 +16,7 @@ Specify any of the following as the `test_kind` in your [Suite](../../../../buil
 - `db_test`: [`DBTestCase`](./dbtest.py) - A dbtest to execute.
 - `fsm_workload_test`: [`FSMWorkloadTestCase`](./fsm_workload_test.py) - A wrapper for several copies of a `_SingleFSMWorkloadTestCase` to execute.
 - `js_test`: [`JSTestCase`](./jstest.py) - A wrapper for several copies of a `_SingleJSTestCase` to execute
+  - Around **75% of all suites use the `js_test` kind**. See [jstests/README.md](../../../../jstests/README.md) for specific guidance.
 - `json_schema_test`: [`JSONSchemaTestCase`](./json_schema_test.py) - A JSON Schema test to execute.
 - `magic_restore_js_test`: [`MagicRestoreTestCase`](./magic_restore_js_test.py) - A test to execute for running tests in a try/catch block.
 - `mongos_test`: [`MongosTestCase`](./mongos_test.py) - A TestCase which runs a mongos binary with the given parameters.
@@ -39,14 +40,22 @@ Top level interfaces:
 
 Subclasses:
 
-- [`FixtureTestCase`](./fixture.py) - Base class for the fixture test cases.
-- [`FixtureSetupTestCase`](./fixture.py) - TestCase for setting up a fixture.
-- [`FixtureTeardownTestCase`](./fixture.py) - TestCase for tearing down a fixture.
-- [`FixtureAbortTestCase`](./fixture.py) - TestCase for killing a fixture. Intended for use before archiving a failed test.
 - [`JSRunnerFileTestCase`](./jsrunnerfile.py) - A test case with a static JavaScript runner file to execute.
 - [`MultiClientsTestCase`](./jstest.py) - A wrapper for several copies of a SingleTestCase to execute.
 - [`TestCaseFactory`](./interface.py) - Convenience interface to initialize and build test cases
 
+## Fixture TestCases
+
+These are testcases that are used to coordinate fixture lifecycles via resmoke's internal `FixtureTestCaseManager`.
+
+> NOTE This design does lead to seeing "extra" tests in a run, where a fixture sets up, your `N` tests are run, and the fixture tears down, so you see `N+2` "tests" passing via resmoke.
+
+- [`FixtureTestCase`](./fixture.py) - Base class for the fixture test cases.
+- [`FixtureSetupTestCase`](./fixture.py) - TestCase for setting up a fixture.
+- [`FixtureTeardownTestCase`](./fixture.py) - TestCase for tearing down a fixture.
+- [`FixtureAbortTestCase`](./fixture.py) - TestCase for killing/aborting a fixture. Intended for use before archiving a failed test.
+  - When resmoke detects that a test has failed (and [archiving](../../../../buildscripts/resmokeconfig/suites/README.md#executorarchive) is configured​), it dynamically generates a new `FixtureAbortTestCase` for immediate execution. This test case sends a `SIGABRT` to each running mongod process.
+
 ## Testing TestCases
 
 Self-tests for the testcases themselves can be found in [buildscripts/tests/resmokelib/testing/testcases/](../../../../buildscripts/tests/resmokelib/testing/testcases/)

docs/testing/README.md

@@ -11,6 +11,7 @@ Learn more about related topics using their own targeted documentation:
 - [suites](../../buildscripts/resmokeconfig/suites/README.md), how tests are grouped and configured
 - [fixtures](../../buildscripts/resmokelib/testing/fixtures/README.md), specify the server topology that tests run against
 - [hooks](../../buildscripts/resmokelib/testing/hooks/README.md), logic to run before, after and/or between individual tests
+- [testcases](../../buildscripts/resmokelib/testing/testcases/README.md), Python-based unittest interfaces that resmoke can run as different "kinds" of tests.
 
 ## Basic Example
 

jstests/OWNERS.yml

@@ -3,6 +3,9 @@ filters:
   - "README.md":
     approvers:
       - 10gen/devprod-correctness
+  - "tags.md":
+    approvers:
+      - 10gen/devprod-correctness
   - "**/*analyze_shard_key*":
     approvers:
       - 10gen/server-cluster-scalability

jstests/README.md

@@ -22,7 +22,7 @@ At MongoDB we write integration tests in JavaScript. These are tests written to
 
 ### Do not hardcode collection or database names, especially if they are used multiple times throughout a test.
 
-It is best to use variable names that attempt to describe what a value is used for. For example, naming a variable that stores a collection name collectionToDrop is much better than just naming the variable collName.
+It is best to use variable names that attempt to describe what a value is used for. For example, naming a variable that stores a collection named `collectionToDrop` is much better than just naming the variable `collName`.
 
 ### Make every effort to make your test as deterministic as possible.
 
@@ -51,6 +51,16 @@ It is best to use variable names that attempt to describe what a value is used f
 
 All assertions in a test should attempt to verify the most specific property possible. For example, if you are trying to test that a certain collection exists, it is better to assert that the collection’s exact name exists in the list of collections, as opposed to verifying that the collection count is equal to 1. The desired collection’s existence is sufficient for the collection count to be 1, but not necessary (a different collection could exist in its place). Be wary of adding these kind of indirect assertions in a test.
 
+### Test Isolation
+
+Your JS test will likely be running with many other files before and after it. It's important to start from a known state, and to restore that state (to a reasonable extent) at the end of your test content.
+
+- **Before**: If there are critical assumptions about the environment that your test needs, assert for it explicitly before proceeding to the real test content (instead of debugging side effects of that not being the case)
+  - If you have a precondition on the _environment_, use [`@tags`](./tags.md) instead of just an early-return. This will avoid the test being scheduled in the first place if the environment is not supported.
+- **After**: If you are modifying the fixture, do everything possible to safely restore those changes at the end of your test content, even after a test failure. Resmokes' `--continueOnFailure` flag is used in CI, so the fixture is shared across many test files, and is only torn down at the end.
+  - Note, a fixture _can_ immediately "abort" after a test failure, only if [archiving](../../../../buildscripts/resmokeconfig/suites/README.md#executorarchive) is configured, but that shouldn't be assumed because that is a per-suite configuration (and your test can run in many passthrough suite combinations).
+  - One easy approach to restoring your state is to use the [Mocha-style](#use-mocha-style-constructs) `after` hooks in your test content.
+
 ## Modern JS: Modules in Practice
 
 We have fully migrated to the modularized JavaScript world so any new test should use modules and adapt the new style.
@@ -85,10 +95,10 @@ Due to legacy, we have a lot of code that is using the old style to do export, l
 
 ```
 const MyModule = (function() {
-function myFeature() {}
-function myOtherFeature() {}
+  function myFeature() {}
+  function myOtherFeature() {}
 
-return {myFeature, myOtherFeature};
+  return {myFeature, myOtherFeature};
 })();
 ```
 
@@ -169,3 +179,18 @@ or use the filter from resmoke to avoid any file edits:
 ```sh
 buildscripts/resmoke.py run --suites=no_passthrough --mochagrep "do something" jstests/noPassthrough/mytest.js
 ```
+
+## Test Tags
+
+JS Test files can leverage "tags" that suites can key off of to include and/or exclude as necessary. Not scheduling a test to run is much faster than the test doing an early-return when preconditions are not met.
+
+The simplest use case is having something like the following at the top of your js test file:
+
+```js
+/**
+ * Tests for the XYZ feature
+ * @tags: [requires_fcv_81]
+ */
+```
+
+See [tags.md](./tags.md) for more details.

jstests/tags.md

@@ -0,0 +1,65 @@
+# JS Test Tags
+
+JS Test files can leverage "tags" that suites can key off of to include and/or exclude as necessary. Not scheduling a test to run is much faster than the test doing an early-return when preconditions are not met.
+
+The simplest use case is having something like the following at the top of your js test file:
+
+```js
+/**
+ * Tests for the XYZ feature
+ * @tags: [requires_fcv_81]
+ */
+```
+
+These can be an array of tags:
+
+```js
+/**
+ * @tags: [
+ *   requires_fcv_81,
+ *   requires_pipeline_optimization,
+ *   not_allowed_with_signed_security_token,
+ *   cannot_run_during_upgrade_downgrade,
+ * ]
+ */
+```
+
+and can also include (meta) comments:
+
+```js
+/**
+ * @tags: [
+ *   requires_fcv_81,
+ *   requires_pipeline_optimization,
+ *   not_allowed_with_signed_security_token,
+ *   # During fcv upgrade/downgrade the engine might not be what we expect.
+ *   cannot_run_during_upgrade_downgrade,
+ * ]
+ */
+```
+
+The tags are meant to be used in suite configurations, to [`include_with_any_tags`](../buildscripts/resmokeconfig/suites/README.md#selectorinclude_with_any_tags) and/or [`exclude_with_any_tags`](../buildscripts/resmokeconfig/suites/README.md#selectorexclude_with_any_tags):
+
+```bash
+test_kind: js_test
+selector:
+  include_with_any_tags:
+    - multiversion_sanity_check
+  exclude_with_any_tags:
+    - replica_sets_multiversion_backport_required_multiversion
+    - disabled_for_fcv_6_1_upgrade
+```
+
+Build variants can also use tags via the `test_flags` expansion, which facilitates tag-exclusions _across suites_ that run with the variant:
+
+```
+    expansions:
+      test_flags: >-
+        --excludeWithAnyTags=requires_external_data_source,requires_ldap_pool
+```
+
+## Available Tags
+
+There is no current exhaustive list, since tags are arbitrary labels and do not need to be "registered". However, tags are always "global", and many are reused. Names should have communicate clear intent; and be reused/consolidated when appropriate.
+
+> Use `buildscripts/resmoke.py list-tags` to find which tags are actively referenced by suite configs, although there may be more in JS files and Build Variant expansions.