mongo/buildscripts/antithesis

How to Use Antithesis

Context

Antithesis is a third party vendor with an environment that can perform network fuzzing. We can upload images containing docker-compose.yml files, which represent various MongoDB topologies, to the Antithesis Docker registry. Antithesis runs docker-compose up from these images to spin up the corresponding multi-container application in their environment and run a test suite. Network fuzzing is performed on the topology while the test suite runs & a report is generated by Antithesis identifying bugs. Check out https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis to see an example of how we use Antithesis today.

Base Images

The base_images directory consists of the building blocks for creating a MongoDB test topology. These images are uploaded to the Antithesis Docker registry weekly during the antithesis_image_build task. For more visibility into how these images are built and uploaded to the Antithesis Docker registry, please see evergreen/antithesis_image_build.sh.

mongo_binaries

This image contains the latest mongo, mongos and mongod binaries. It can be used to start a mongod instance, mongos instance or execute mongo commands. This is the main building block for creating the System Under Test topology.

workload

This image contains the latest mongo binary as well as the resmoke test runner. The workload container is not part of the actual toplogy. The purpose of a workload container is to execute mongo commands to complete the topology setup, and to run a test suite on an existing topology like so:

buildscript/resmoke.py run --suite antithesis_concurrency_sharded_with_stepdowns_and_balancer --shellConnString "mongodb://mongos:27017"

Every topology must have 1 workload container.

Note: During workload image build, buildscripts/antithesis_suite.py runs, which generates "antithesis compatible" test suites and prepends them with antithesis_. These are the test suites that can run in antithesis and are available from witihin the workload container.

Topologies

The topologies directory consists of subdirectories representing various mongo test topologies. Each topology has a Dockerfile, a docker-compose.yml file and a scripts directory.

Dockerfile

This assembles an image with the necessary files for spinning up the corresponding topology. It consists of a docker-compose.yml, a logs directory, a scripts directory and a data directory. If this is structured properly, you should be able to copy the files & directories from this image and run docker-compose up to set up the desired topology.

Example from buildscripts/antithesis/topologies/replica_set/Dockerfile:

FROM scratch
COPY docker-compose.yml /
ADD scripts /scripts
ADD logs /logs
ADD data /data

All topology images are built and uploaded to the Antithesis Docker registry during the antithesis_image_build task in the evergreen/antithesis_image_build.sh script. Some of these directories are created in evergreen/antithesis_image_build.sh such as /data and /logs.

Note: These images serve solely as a filesystem containing all necessary files for a topology, therefore use FROM scratch.

docker-compose.yml

This describes how to construct the corresponding topology using the mongo-binaries and workload images.

Example from buildscripts/antithesis/topologies/replica_set/docker-compose.yml:

version: '3.0'

services:
        database1:
                container_name: database1
                hostname: database1
                image: mongo-binaries:evergreen-latest-master
                command: /bin/bash /scripts/database_init.sh
                volumes:
                  - ./logs/database1:/var/log/mongodb/
                  - ./scripts:/scripts/
                  - ./data/database1:/data/db/
                networks:
                        antithesis-net:
                                ipv4_address: 10.20.20.3
                                # Set the an IPv4 with an address of 10.20.20.130 or higher
                                # to be ignored by the fault injector
                                #
        database2: ...
        database3: ...
        workload:
                container_name: workload
                hostname: workload
                image: workload:evergreen-latest-master
                command: /bin/bash /scripts/workload_init.sh
                volumes:
                  - ./logs/workload:/var/log/resmoke/
                  - ./scripts:/scripts/
                depends_on:
                        - "database1"
                        - "database2"
                        - "database3"
                networks:
                        antithesis-net:
                                ipv4_address: 10.20.20.130
                                # The subnet provided here is an example
                                # An alternative subnet can be used

networks:
        antithesis-net:
                driver: bridge
                ipam:
                        config:
                        - subnet: 10.20.20.0/24

Each container must have a commandin docker-compose.yml that runs an init script. The init script belongs in the scripts directory, which is included as a volume. The command should be set like so: /bin/bash /scripts/[script_name].sh. This is a requirement for the topology to start up properly in Antithesis.

When creating mongod or mongos instances, route the logs like so: --logpath /var/log/mongodb/mongodb.log and utilize volumes -- as in database1. This enables us to easily retrieve logs if a bug is detected by Antithesis.

The ipv4_address should be set to 10.20.20.130 or higher if you do not want that container to be affected by network fuzzing. For instance, you would likely not want the workload container
to be affected by network fuzzing -- as shown in the example above.

Use the evergreen-latest-master tag for all images. This is updated automatically in evergreen/antithesis_image_build.sh during the antithesis_image_build task -- if needed.

scripts

Example from buildscripts/antithesis/topologies/replica_set/scripts/workload_init.sh:

sleep 5s
mongo --host database1 --port 27017 --eval "config={\"_id\" : \"RollbackFuzzerTest\",\"protocolVersion\" : 1,\"members\" : [{\"_id\" : 0,\"host\" : \"database1:27017\"}, {\"_id\" : 1,\"host\" : \"database2:27017\"}, {\"_id\" : 2,\"host\" : \"database3:27017\"} ],\"settings\" : {\"chainingAllowed\" : false,\"electionTimeoutMillis\" : 500, \"heartbeatTimeoutSecs\" : 1, \"catchUpTimeoutMillis\": 700}}; rs.initiate(config)"

# this cryptic statement keeps the workload container running.
tail -f /dev/null

The sleep command can be useful to ensure that other containers have had a chance to start. In this example, the workload container waits 5 seconds while the database containers start. After that, it initiates the replica set. The tail -f /dev/null is required for workload containers otherwise the container shuts down.

How do I create a new topology for Antithesis testing?

To create a new topology for Antithesis testing is easy & requires a few simple steps.

1. Add a new directory in buildscripts/antithesis/topologies to represent your desired topology. You can use existing topologies as an example.
2. Update the evergreen/antithesis_image_build.sh file so that your new topology image is uploaded to the Antithesis Docker registry.
3. Reach out to #server-testing on Slack & provide the new topology image name as well as the desired test suite to run.
4. Include a member of the STM team on the code review.

These are the required updates to evergreen/antithesis_image_build.sh:

• Add the following command for each of your mongos and mongod containers in your topology to create your log directories.

mkdir -p antithesis/topologies/[topology_name]/{logs,data}/[container_name]

• Build an image for your new topology ending in -config

cd [your_topology_dir]
sed -i s/evergreen-latest-master/$tag/ docker-compose.yml
sudo docker build . -t [your-topology-name]-config:$tag

• Push your new image to the Antithesis Docker registry

sudo docker tag "[your-topology-name]-config:$tag" "us-central1-docker.pkg.dev/molten-verve-216720/mongodb-repository/[your-topology-name]-config:$tag"
sudo docker push "us-central1-docker.pkg.dev/molten-verve-216720/mongodb-repository/[your-topology-name]-config:$tag"

Additional Resources

If you are interested in leveraging Antithesis feel free to reach out to #server-testing on Slack.