# How to Use Antithesis ## Context Antithesis is a third party vendor with an environment that can perform network fuzzing. We can upload images containing `docker-compose.yml` files, which represent various MongoDB topologies, to the Antithesis Docker registry. Antithesis runs `docker-compose up` from these images to spin up the corresponding multi-container application in their environment and run a test suite. Network fuzzing is performed on the topology while the test suite runs & a report is generated by Antithesis identifying bugs. Check out https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis to see an example of how we use Antithesis today. ## Base Images The `base_images` directory consists of the building blocks for creating a MongoDB test topology. These images are uploaded to the Antithesis Docker registry [nightly](https://github.com/mongodb/mongo/blob/6cf8b162a61173eb372b54213def6dd61e1fd684/etc/evergreen_yml_components/variants/ubuntu/test_dev_master_and_lts_branches_only.yml#L28) during the [`antithesis image build and push`](https://github.com/mongodb/mongo/blob/020632e3ae328f276b2c251417b5a39389af6141/etc/evergreen_yml_components/definitions.yml#L2823) function. ### mongo_binaries This image contains the latest `mongo`, `mongos` and `mongod` binaries. It can be used to start a `mongod` instance, `mongos` instance or execute `mongo` commands. This is the main building block for creating the System Under Test topology. ### workload This image contains the latest `mongo` binary as well as the `resmoke` test runner. The `workload` container is not part of the actual toplogy. The purpose of a `workload` container is to execute `mongo` commands to complete the topology setup, and to run a test suite on an existing topology like so: ```shell buildscript/resmoke.py run --suite antithesis_concurrency_sharded_with_stepdowns_and_balancer ``` **Every topology must have 1 workload container.** Note: During `workload` image build, `evergreen/antithesis_image_build_and_push.sh` runs, which generates "antithesis compatible" test suites and prepends them with `antithesis_`. These are the test suites that can run in antithesis and are available from within the `workload` container. ### Dockerfile This assembles an image with the necessary files for spinning up the corresponding topology. It consists of a `docker-compose.yml`, a `logs` directory, a `scripts` directory and a `data` directory. If this is structured properly, you should be able to copy the files & directories from this image and run `docker-compose up` to set up the desired topology. Example from what `buildscripts/resmokelib/testing/docker_cluster_image_builder.py` generates: ```Dockerfile FROM scratch COPY docker-compose.yml / ADD scripts /scripts ADD logs /logs ADD data /data ADD debug /debug ``` All topology images are built and uploaded to the Antithesis Docker registry during the `antithesis image build and push` task. Some of these directories are created during the `evergreen/antithesis_image_build_and_push.sh` script such as `/data` and `/logs`. Note: These images serve solely as a filesystem containing all necessary files for a topology, therefore use `FROM scratch`. ### docker-compose.yml This describes how to construct the corresponding topology using the `mongo-binaries` and `workload` images. Example from `buildscripts/antithesis/topologies/sharded_cluster/docker-compose.yml`: ```yml version: '3.0' services: configsvr1: container_name: configsvr1 hostname: configsvr1 image: mongo-binaries:evergreen-latest-master volumes: - ./logs/configsvr1:/var/log/mongodb/ - ./scripts:/scripts/ - ./data/configsvr1:/data/configdb/ command: /bin/bash /scripts/configsvr_init.sh networks: antithesis-net: ipv4_address: 10.20.20.6 # Set the an IPv4 with an address of 10.20.20.130 or higher # to be ignored by the fault injector # configsvr2: ... configsvr3: ... database1: ... container_name: database1 hostname: database1 image: mongo-binaries:evergreen-latest-master volumes: - ./logs/database1:/var/log/mongodb/ - ./scripts:/scripts/ - ./data/database1:/data/db/ command: /bin/bash /scripts/database_init.sh Shard1 networks: antithesis-net: ipv4_address: 10.20.20.3 # Set the an IPv4 with an address of 10.20.20.130 or higher # to be ignored by the fault injector # database2: ... database3: ... database4: ... database5: ... database6: ... mongos: container_name: mongos hostname: mongos image: mongo-binaries:evergreen-latest-master volumes: - ./logs/mongos:/var/log/mongodb/ - ./scripts:/scripts/ command: python3 /scripts/mongos_init.py depends_on: - "database1" - "database2" - "database3" - "database4" - "database5" - "database6" - "configsvr1" - "configsvr2" - "configsvr3" networks: antithesis-net: ipv4_address: 10.20.20.9 # The subnet provided here is an example # An alternative subnet can be used workload: container_name: workload hostname: workload image: workload:evergreen-latest-master volumes: - ./logs/workload:/var/log/resmoke/ - ./scripts:/scripts/ command: python3 /scripts/workload_init.py depends_on: - "mongos" networks: antithesis-net: ipv4_address: 10.20.20.130 # The subnet provided here is an example # An alternative subnet can be used networks: antithesis-net: driver: bridge ipam: config: - subnet: 10.20.20.0/24 ``` Each container must have a `command` in `docker-compose.yml` that runs an init script. The init script belongs in the `scripts` directory, which is included as a volume. The `command` should be set like so: `/bin/bash /scripts/[script_name].sh` or `python3 /scripts/[script_name].py`. This is a requirement for the topology to start up properly in Antithesis. When creating `mongod` or `mongos` instances, route the logs like so: `--logpath /var/log/mongodb/mongodb.log` and utilize `volumes` -- as in `database1`. This enables us to easily retrieve logs if a bug is detected by Antithesis. The `ipv4_address` should be set to `10.20.20.130` or higher if you do not want that container to be affected by network fuzzing. For instance, you would likely not want the `workload` container to be affected by network fuzzing -- as shown in the example above. Use the `evergreen-latest-master` tag for all images. This is updated automatically in `evergreen/antithesis_image_build_and_push.sh` -- if needed. ### scripts Take a look at `buildscripts/antithesis/topologies/sharded_cluster/scripts/mongos_init.py` to see how to use util methods from `buildscripts/antithesis/topologies/sharded_cluster/scripts/utils.py` to set up the desired topology. You can also use simple shell scripts as in the case of `buildscripts/antithesis/topologies/sharded_cluster/scripts/database_init.py`. These init scripts must not end in order to keep the underlying container alive. You can use an infinite while loop for `python` scripts or you can use `tail -f /dev/null` for shell scripts. ## How do I create a new topology for Antithesis testing? This should be done with care to ensure we are using our limited resources efficiently. Create a new task extending the `antithesis_task_template`, tagged with `antithesis`, passing the specified `suite` to the `antithesis image build and push` task. See other examples to get started. ## How do I test my suite in antithesis? If you provide the evergreen parameter `schedule_antithesis_tests` to your evergreen patch, once we build the antithesis images in your evergreen patch we send antithesis an api request to run your newly created images for an hour. You will get emailed the report when it finishes running in antithesis. Important Note: This will happen for every antithesis task you schedule in your patch. Please do not schedule more than 1 or 2 tasks with this parameter at a time or it will use up a lot of our testing time allocated with antithesis. `evergreen patch --param schedule_antithesis_tests=true` ## Additional Resources If you are interested in leveraging Antithesis feel free to reach out to #server-testing on Slack.