## 1. Why Do We Need Testing? Software rarely fails because of an obvious syntax error. The most expensive defects usually come from incorrect assumptions, boundary inputs, unstable external dependencies, or a seemingly harmless change that alters existing behavior. Testing provides early feedback about these risks and turns requirements into evidence that can be executed repeatedly. A useful test suite should answer more than “Was this line executed?” It should also tell us: - whether the behavior matches the requirements; - whether important inputs were selected systematically; - whether assertions are strong enough to detect small defects; - whether the same model input always produces the same prediction; - how much traffic the API can handle before it violates its SLA. This article uses `pytest` as the main test runner and combines it with coverage, Hypothesis, mutmut, and Locust to answer these questions. ### Validation and Verification Validation and verification are related, but they answer different questions: - **Verification — Are we building the product right?** Does the system conform to its specification, API contract, and defined business rules? - **Validation — Are we building the right product?** Does the implemented behavior solve the actual needs of users and stakeholders? Suppose the specification says that the `limit` query parameter must be in the inclusive range `[1, 100]`. Testing that `limit=0` returns HTTP `422` is verification. Confirming that a maximum of 100 records is appropriate for the real user workflow is validation. Automated testing contributes heavily to verification. Validation also requires clear acceptance criteria, product reviews, user feedback, and observation of the running system. Effective testing starts with the right requirement; a test suite cannot rescue an incorrect requirement. ### Unit, Integration, E2E, and Load Testing Each testing level provides a different kind of confidence: | Test type | Scope | Main question | Characteristics | | --- | --- | --- | --- | | Unit test | A small function, class, or service | Is the business rule correct? | Fast and isolated | | Integration test | Multiple real components | Do the components communicate correctly? | More setup and slower feedback | | End-to-end test | A complete user flow | Does the user journey work? | High confidence but expensive | | Load test | A system under concurrent traffic | Do throughput and latency meet the SLA? | Requires a defined workload and environment | End-to-end tests should not replace every lower-level test. A practical strategy usually contains many fast unit tests, fewer integration tests, and a small number of critical E2E flows. Load testing adds another dimension: a system can be functionally correct while still failing its performance requirements. ### Scope of This Article The example project is a small FastAPI service that fetches campaign data from an upstream API, calculates click-through rates, and exposes a deterministic prediction endpoint. The article focuses primarily on unit tests and component-level API tests with mocked dependencies. It then uses: - Equivalence Partitioning and Boundary Value Analysis to design inputs; - coverage to locate production code that was not executed; - mutation testing to evaluate assertion strength; - property-based testing to search for counterexamples; - Locust to load test the API through a local HTTP upstream service. Create the local environment with `uv`: ```bash git clone https://github.com/itsmekhoathekid/pytest-blog.git cd pytest-blog uv sync --extra dev ``` `uv sync` reads `.python-version`, creates `.venv`, and installs the dependency versions recorded in `uv.lock`. Activating the environment is optional because all commands below use `uv run`. ## 2. Overview: An Ideal Testing Flow The techniques in this article are not an unrelated checklist. They form a feedback loop: fixtures and mocks establish an isolated environment, EP and BVA select meaningful examples, Hypothesis expands the input space, coverage finds unexecuted paths, mutation testing evaluates assertion quality, and Locust checks the performance contract. ```mermaid flowchart TD A[Requirements and acceptance criteria] --> B[Define contracts and invariants] B --> C[Fixtures: prepare test state] C --> D[Mocks: isolate external dependencies] D --> E[EP and BVA: select test cases] E --> F[Parametrized pytest unit and API tests] F --> G{Branch coverage >= X percent?} G -- No --> E G -- Yes --> H[Hypothesis property-based testing] H --> I[Idempotency, determinism, and invariants] I --> J[Mutation testing] J --> K{Mutation score above Y percent?} K -- No: surviving mutants --> E K -- Yes --> L[Integration and Locust load tests] L --> M{Throughput, latency, and error rate meet the SLA?} M -- No --> B M -- Yes --> N[Release confidence] ``` This flow does not mean that mutation and load tests must run after every unit test on a developer laptop. Unit tests should run continuously, while more expensive checks can run in CI or before a release. The diagram shows how confidence accumulates and how failures return feedback to test design. ## 3. Unit Testing with pytest ### What Is a Unit Test? A unit test verifies a small unit of behavior—usually a function, method, or service—in isolation from networks, databases, and external systems. A typical unit test follows the Arrange–Act–Assert structure: ```python def test_click_through_rate(): # Arrange clicks = 20 impressions = 100 # Act result = click_through_rate(clicks, impressions) # Assert assert result == 0.2 ``` - **Arrange:** prepare the input and dependencies. - **Act:** invoke the behavior under test. - **Assert:** verify the output or another observable interaction. Tests should focus on behavior instead of implementation details. If an internal refactor preserves the contract, the ideal unit test should continue to pass. ### Fixtures A fixture is pytest's mechanism for preparing reusable test data or resources, injecting them through test parameters, and cleaning them up afterward. Fixtures remove duplicated setup without hiding a test's dependencies. ```python @pytest.fixture def campaign_rows() -> list[Campaign]: return [ Campaign( id="cmp-low", name="Awareness", region="apac", clicks=10, impressions=100, ), Campaign( id="cmp-high", name="Conversion", region="apac", clicks=80, impressions=100, ), ] @pytest.fixture def client(source_mock: Mock) -> Iterator[TestClient]: app.dependency_overrides[get_campaign_source] = lambda: source_mock with TestClient(app) as test_client: yield test_client app.dependency_overrides.clear() ``` The cleanup after `yield` restores the application to a clean state so that the dependency override cannot leak into another test. Pytest also supports fixture composition, scopes, and teardown patterns; see the [pytest fixture documentation](https://docs.pytest.org/en/stable/how-to/fixtures.html). ### Mocks A mock replaces a real dependency with a test double whose output can be configured and whose interactions can be inspected. The upstream API is not the target of this unit test, so calling the real network would make the test slower and less reliable. `Mock(spec=CampaignSource)` also restricts the mock to the real interface, helping detect invalid method calls. ```python @pytest.fixture def source_mock(campaign_rows: list[Campaign]) -> Mock: source = Mock(spec=CampaignSource) source.fetch_campaigns.return_value = campaign_rows return source ``` This mock serves two purposes: 1. it stubs the output of `fetch_campaigns` with deterministic data; 2. it records interactions so the test can verify how the dependency was called. Not every object should be mocked. Value objects and pure functions are usually better used directly. Mocks are most useful at boundaries such as HTTP clients, database gateways, clocks, and external services. Patch the reference used by the code under test; see [pytest monkeypatching](https://docs.pytest.org/en/stable/how-to/monkeypatch.html). ### Testing a Web API FastAPI's `TestClient` calls an endpoint through its HTTP interface without opening a real network port. A dependency override injects the mocked source: ```python def test_campaign_endpoint_uses_fixture_and_mock( client: TestClient, source_mock: Mock, ) -> None: response = client.get( "/campaigns", params={"region": " APAC ", "min_score": 0.5, "limit": 1}, ) assert response.status_code == 200 assert response.json()[0]["id"] == "cmp-high" assert response.json()[0]["score"] == 0.8 source_mock.fetch_campaigns.assert_called_once_with("apac") ``` The test checks the HTTP contract, response body, and interaction with the upstream dependency. It is a component/API test executed in a unit-test style: the HTTP boundary is real, but the network dependency is mocked. ### Coverage Coverage shows which production code was executed, but it does not prove that the assertions are correct or strong enough. The project measures statement and branch coverage and defines a quality gate in `pyproject.toml`: ```toml [tool.pytest.ini_options] addopts = [ "--cov=pytest_blog", "--cov-branch", "--cov-report=term-missing", "--cov-report=html:reports/coverage", "--cov-fail-under=90", ] ``` Run the suite and generate an HTML coverage report: ```bash uv run pytest --hypothesis-profile=ci open reports/coverage/index.html ``` ![Unit and API tests with 100% coverage](/media/assets/65662d81-6022-4eac-9303-13b963302b42/01-coverage.png) The screenshot above shows the complete suite and its coverage gate. The observed output is: ```text 31 passed TOTAL 87 statements 0 missed 8 branches 0 partial 100% Required test coverage of 90% reached. Total coverage: 100.00% ``` ## 4. Designing Test Cases with EP and BVA Assume the endpoint accepts a `limit` query parameter whose valid range is `[1, 100]`. Testing every integer is unnecessary, while choosing a few random values can easily miss boundary defects. Equivalence Partitioning and Boundary Value Analysis provide a systematic selection method. ### Equivalence Partitioning Equivalence Partitioning (EP) divides an input domain into groups for which the system is expected to behave in the same way: | Partition | Domain | Representative | Expected result | | --- | --- | ---: | ---: | | Invalid below the range | `limit < 1` | `-10` | HTTP 422 | | Valid | `1 <= limit <= 100` | `50` | HTTP 200 | | Invalid above the range | `limit > 100` | `150` | HTTP 422 | ```python EP_CASES = [ pytest.param(-10, 422, id="EP-invalid-below-range"), pytest.param(50, 200, id="EP-valid-range"), pytest.param(150, 422, id="EP-invalid-above-range"), ] ``` EP reduces the number of tests by choosing a meaningful representative from each behavioral partition. ### Boundary Value Analysis Boundary Value Analysis (BVA) focuses on values immediately below, at, and immediately above each boundary. These positions frequently expose `<` versus `<=` mistakes and other off-by-one errors. ```python BVA_CASES = [ pytest.param(0, 422, id="BVA-min-minus-1"), pytest.param(1, 200, id="BVA-min"), pytest.param(2, 200, id="BVA-min-plus-1"), pytest.param(99, 200, id="BVA-max-minus-1"), pytest.param(100, 200, id="BVA-max"), pytest.param(101, 422, id="BVA-max-plus-1"), ] ``` EP checks every behavioral region, while BVA examines the transitions between regions. The two techniques complement rather than replace each other. ### Combining EP and BVA with pytest.mark.parametrize In the real test, every parameter has the appropriate marker. One test function receives the complete matrix, while its test IDs preserve the origin of each case: ```python @pytest.mark.parametrize( ("limit", "expected_status"), [ pytest.param(-10, 422, id="EP-invalid-below-range", marks=pytest.mark.ep), pytest.param(50, 200, id="EP-valid-range", marks=pytest.mark.ep), pytest.param(150, 422, id="EP-invalid-above-range", marks=pytest.mark.ep), pytest.param(0, 422, id="BVA-min-minus-1", marks=pytest.mark.bva), pytest.param(1, 200, id="BVA-min", marks=pytest.mark.bva), pytest.param(2, 200, id="BVA-min-plus-1", marks=pytest.mark.bva), pytest.param(99, 200, id="BVA-max-minus-1", marks=pytest.mark.bva), pytest.param(100, 200, id="BVA-max", marks=pytest.mark.bva), pytest.param(101, 422, id="BVA-max-plus-1", marks=pytest.mark.bva), ], ) def test_limit_validation(client, limit, expected_status): response = client.get( "/campaigns", params={"region": "apac", "limit": limit}, ) assert response.status_code == expected_status ``` Run both groups with verbose test IDs: ```bash uv run pytest tests/test_api.py -m "ep or bva" -vv --no-cov ``` See the [pytest parametrization documentation](https://docs.pytest.org/en/stable/how-to/parametrize.html) for more patterns. ![Parametrized EP and BVA tests](/media/assets/a3398894-edf5-4e1e-85a9-5e55cab79210/02-ep-bva.png) The screenshot confirms that pytest selected all nine designed cases: three equivalence partitions and six boundary values. Every `EP-*` and `BVA-*` case passed, while the remaining nine API tests were intentionally deselected by the marker expression. ## 5. Mutation Testing ### What Is Mutation Testing? Mutation testing deliberately creates versions of the production code with small defects. These versions are called mutants. Each mutant is evaluated by the test suite: - if a test fails, the mutant is **killed** and the assertion detected the change; - if every test passes, the mutant **survived**, indicating a potentially missing case or weak assertion; - timeouts and suspicious results require investigation before applying a gate. Coverage asks, “Was this line executed?” Mutation testing asks, “Would the test notice if the behavior of this line changed?” ### Mutant Example The production code keeps campaigns whose score is equal to or greater than the threshold: ```python eligible = [metric for metric in metrics if metric.score >= min_score] ``` Mutmut can replace `>=` with `>`: ```python # Mutant: a campaign exactly at the threshold is incorrectly excluded. eligible = [metric for metric in metrics if metric.score > min_score] ``` A test containing `score == min_score == 0.5` kills the mutant: ```python result = list_campaign_metrics( source, region="US", min_score=0.5, limit=2, ) assert [(item.id, item.score) for item in result] == [ ("c", 0.9), ("a", 0.5), ] ``` This is a direct connection between BVA and mutation testing: a case at the boundary makes the assertion sensitive to the `>=` to `>` change. ### Mutating Only Changed Code Mutation testing is CPU-intensive because the relevant tests run for each mutant. Running it against the entire codebase for every commit is unnecessary. The project keeps the scope explicit with mutmut wildcards: ```bash uv run mutmut run "pytest_blog.domain*" uv run mutmut run "pytest_blog.service*" uv run mutmut results ``` In a larger project, changed Python files from a Git diff can be mapped to mutmut wildcards. For a small pull request, explicitly listing the changed production modules in the CI job is often simpler and easier to audit. Mutmut also caches results at the function level; see the [mutmut documentation](https://mutmut.readthedocs.io/en/latest/). ### Mutation Score A practical formula is: ```text mutation score = killed / (killed + survived + timeout + suspicious) ``` Run mutation testing for only the two production modules in scope: ```bash uv run mutmut run "pytest_blog.domain*" uv run mutmut run "pytest_blog.service*" uv run mutmut results uv run mutmut export-cicd-stats ``` ![Mutation testing results](/media/assets/8c7f8673-4f96-4fcc-b2f0-e33fe961ca4a/03-mutation.png) The screenshot shows 68 checked mutants: 64 killed and four survived. The observed result is: ```text killed = 64 survived = 4 timeout = 0 suspicious = 0 mutation score = 64 / (64 + 4) = 94.12% ``` Mutants reported as `not checked` belong to modules outside this run's scope and are therefore excluded from its denominator. The four surviving mutants remain useful test backlog or candidates for classification as equivalent mutants. ## 6. Idempotency and Property-Based Testing ### What Are Idempotency and Determinism? Idempotency means that applying an operation repeatedly produces the same final state as applying it once. For a mathematical function, it is commonly written: ```text f(f(x)) = f(x) ``` For example, an HTTP `PUT` operation updates a resource to a representation, so repeating the same request should leave the same final state. For pure model inference, the desired property is usually **determinism** or **repeatability**: ```text predict(model_version, input) == predict(model_version, input) ``` The project rubric uses “idempotency testing” in the operational-consistency sense: repeated executions with the same model version and input must generate the same prediction. This article keeps that label while clarifying that the specific assertion verifies deterministic inference. ### Hypothesis An example-based test only exercises inputs that its author considered. With Hypothesis, the author describes an input domain and invariants. The framework generates many examples, finds a failure, and shrinks it to a small counterexample that is easier to debug. ```python @pytest.mark.property @given( impressions=st.integers(min_value=0, max_value=1_000_000), data=st.data(), ) def test_prediction_is_idempotent_and_bounded(impressions, data): clicks = data.draw( st.integers(min_value=0, max_value=impressions), label="clicks", ) first = predict_conversion_rate(clicks, impressions) second = predict_conversion_rate(clicks, impressions) assert first == second assert 0.0 <= first <= 1.0 ``` `first == second` checks repeatability. The `[0, 1]` invariant prevents a deterministic function that returns a meaningless value from passing. Another property in the suite verifies the CTR calculation against its mathematical definition. Run the idempotency/determinism property directly and ask Hypothesis to print its generation statistics: ```bash uv run pytest \ tests/test_domain.py::test_prediction_is_idempotent_and_bounded \ -vv \ --no-cov \ --hypothesis-show-statistics ``` `--no-cov` does not bypass the quality gate: the complete suite already checks coverage in the previous step. It prevents a single selected test from failing the project-wide coverage threshold. `--hypothesis-show-statistics` provides visible proof of how many generated examples were evaluated. ![Idempotency property tested with Hypothesis](/media/assets/0eaffd3e-1685-4dde-8349-02cbbb234199/04-property.png) The observed output is: ```text tests/test_domain.py::test_prediction_is_idempotent_and_bounded PASSED [100%] Hypothesis Statistics - 100 passing, 0 failing, and 0 invalid test cases - Stopped because settings.max_examples=100 1 passed in 0.07s ``` This is stronger evidence than a single example-based assertion: one pytest test invocation evaluates 100 generated `(clicks, impressions)` combinations against both repeatability and bounded-output properties. See the [Hypothesis documentation](https://hypothesis.readthedocs.io/en/latest/quickstart.html) for generation, shrinking, and pytest integration. ## 7. Stress and Load Testing with Locust ### What Is Locust? Locust is a load-testing framework in which workloads are written in Python. Each virtual user runs weighted tasks, pauses according to `wait_time`, sends HTTP requests, and contributes request-rate, response-time, percentile, and failure-rate statistics. The example runs locally but retains a real HTTP hop: ```mermaid flowchart LR A[20 Locust users] -->|HTTP| B[FastAPI under test on port 8000] B -->|HTTP data pull| C[Mock upstream on port 8001] B --> D[Prediction function] ``` The mock upstream is not a mock object from a unit test. It is a real local HTTP server, so the load test measures request parsing, routing, serialization, and the upstream call made by the data endpoint. ### Writing locustfile.py The workload sends approximately 80% of its traffic to the campaign data endpoint and 20% to the prediction endpoint: ```python from locust import HttpUser, between, task class ApiUser(HttpUser): wait_time = between(0.05, 0.2) @task(4) def list_campaigns(self) -> None: with self.client.get( "/campaigns", params={"region": "apac", "limit": 20}, name="GET /campaigns", catch_response=True, ) as response: if response.status_code != 200: response.failure(f"unexpected HTTP {response.status_code}") @task(1) def predict(self) -> None: self.client.post( "/predictions", json={"clicks": 20, "impressions": 100}, name="POST /predictions", ) ``` - `HttpUser` provides the HTTP client and statistics. - `@task(4)` and `@task(1)` produce a 4:1 workload ratio. - `between(0.05, 0.2)` models a pause between actions. - `name=` groups requests with different query values under one metric. - `catch_response=True` allows a business condition to mark a response as failed. ### Throughput, Latency, and SLA - **Throughput** is the number of completed requests per second. - **Latency** is the request duration; a percentile such as p95 is generally more informative than an average alone. - **Failure rate** is the percentage of requests marked as failed. The demo SLA is version-controlled in `scripts/check_sla.py`: | Service-level indicator | Quality gate | | --- | ---: | | Throughput | `>= 100 req/s` | | p95 latency | `<= 100 ms` | | Failure rate | `<= 1%` | ```python MIN_REQUESTS_PER_SECOND = 100.0 MAX_P95_MILLISECONDS = 100.0 MAX_FAILURE_RATE = 0.01 ``` Start the local upstream service in the first terminal: ```bash uv run uvicorn pytest_blog.upstream:app \ --host 127.0.0.1 \ --port 8001 ``` Start the API under test in a second terminal and point it at the upstream: ```bash CAMPAIGN_SOURCE_URL=http://127.0.0.1:8001 \ uv run uvicorn pytest_blog.main:app \ --host 127.0.0.1 \ --port 8000 ``` In a third terminal, Locust creates 20 users over four seconds, runs for 30 seconds, and writes both HTML and CSV reports: ```bash uv run locust \ --headless \ --host http://127.0.0.1:8000 \ --users 20 \ --spawn-rate 5 \ --run-time 30s \ --html reports/locust.html \ --csv reports/locust ``` Evaluate the aggregate CSV row against the project-specific SLA thresholds: ```bash uv run python scripts/check_sla.py ``` ### HTML Report After the native Locust command finishes, open the generated report: ```bash open reports/locust.html ``` The Locust HTML report includes Statistics, Response Time Percentiles, Charts, and failure details. The SLA checker reads the CSV output, turning performance from a screenshot-only observation into an automated pass/fail condition. See [Locust headless mode and HTML reports](https://docs.locust.io/en/stable/running-without-web-ui.html). ![Locust request and response-time statistics](/media/assets/6cde5152-4c66-4919-807e-e9939489f5aa/5-locust-test.png) **Request and percentile summary.** The HTML report recorded 4,279 requests with zero failures and an aggregate rate of 143.63 requests per second. Aggregate response time was 6 ms at p50, 17 ms at p95, and 51 ms at the maximum. ![Locust throughput and response-time charts](/media/assets/bfcb3e92-0e4f-40a8-b8ee-1dd8198cecd2/5-locust-test-2.png) **Traffic and latency over time.** Throughput ramped up with the users and then stabilized near 150 requests per second. Failures remained at zero, while p50 stayed near 6 ms and p95 stayed between approximately 16 and 18 ms. ![Locust users and task ratio](/media/assets/9f0a1a86-bad5-489f-82c4-7a188269dc64/5-locust-test-3.png) **Users and workload distribution.** The test maintained 20 concurrent users during the measurement window. The final task ratio matches the Locust weights: 80% `listCampaigns` and 20% `predict`. The SLA checker reads the CSV generated by the same load-test run. Its observed output is: ```text Observed: 143.50 req/s, p95=17 ms, failure rate=0.00% PASS: throughput >= 100 req/s PASS: p95 <= 100 ms PASS: failure rate <= 1% ``` The small difference between the HTML report rate (`143.63 req/s`) and the final CSV aggregate (`143.50 req/s`) comes from when Locust finalizes each report. Both outputs satisfy the defined SLA. This remains a baseline from one laptop, not a production capacity claim; the workload, hardware, network topology, and dataset must remain stable before results from different commits can be compared. ## 8. Conclusion No single metric proves that a system has been tested well: - fixtures and mocks create fast, isolated, and deterministic unit tests; - EP and BVA make example selection systematic; - parametrization keeps the test matrix concise and readable; - coverage identifies production paths that were not executed; - Hypothesis explores inputs beyond hand-written examples; - mutation testing verifies that assertions detect behavioral changes; - Locust turns throughput, latency, and failure rate into enforceable SLAs. The example project produces a chain of complementary evidence: ```text 31 tests passed Branch coverage: 100% Mutation score: 94.12% Hypothesis idempotency property: 100 generated examples passed Load-test SLA: 143.50 req/s, p95 17 ms, 0% failures ``` The goal is not to reach 100% coverage at any cost. The goal is to build a feedback loop that detects the right risks at an appropriate cost. Source code: [https://github.com/itsmekhoathekid/pytest-blog](https://github.com/itsmekhoathekid/pytest-blog)

itsmekhoathekid@github