# Testing

Testing vcpkg is important whenever one makes changes to the tool itself, and
writing new tests and keeping them up to date is also very important. If one's
code is subtly broken, we'd rather find it out right away than a few weeks down
the line when someone complains!

## Running end to end (e2e) tests

Before running the tests you must have installed the `Pester` module.
This can be done by running `Install-Module -Name Pester -MinimumVersion 5.6.1 -MaximumVersion 5.99 -Force` in a PowerShell session.  
On Linux, some e2e suites also require `mono-complete` for NuGet export scenarios.  
To run the e2e test run the following command:  
`pwsh azure-pipelines/end-to-end-tests.ps1`  
**Parameters**:  
`WorkingRoot`: The dir in which the tests are executed and temporary files are created. Default `work`  
`VcpkgRoot`: The path to the vcpkg root. Default `$VCPKG_ROOT`  
`VCPKGExe`: The path to the vcpkg executable. Default `./vcpkg`  
`Filter`: The name of the tests. The names are the filenames in the folder `azure-pipelines/end-to-end-tests-dir`.


## Environment Variables

You will need to set the environment variable `VCPKG_ROOT` to the location
of a copy of the [main vcpkg repo](https://github.com/microsoft/vcpkg) on your
machine, in order to run the tests. This repository is tool-only, so local e2e
runs need a separate `microsoft/vcpkg` checkout to provide built-in triplets,
scripts, and registry history. In this dev container environment, `/vcpkg` is
likely to already contain a full `microsoft/vcpkg` clone suitable for this
purpose.

If you are using a temporary Docker container or other ephemeral environment,
do not rely on local session notes or container-local memory persisting across
restarts; keep any setup notes you need in the repository or recreate the test
setup when the container is replaced.

Some registry tests require full git history in that `microsoft/vcpkg`
checkout. A shallow clone may fail baseline lookups; fetch additional history
if needed.

Failure to do so will result in this error, when some of the tests are run:

```
Error: Could not detect vcpkg-root.
```

## Running Tests

Before anything else, we should know whether you can actually run the tests!
All you should need is a way to build vcpkg -- anything will do! All you have to
do is follow the guide 😄

With `$VCPKG_DIRECTORY` being the directory where you have cloned vcpkg-tool,
create a build directory in (commonly named `out`), and
`cd` into it. Make sure to clean it out if it already exists!

```sh
$ cmake out -DCMAKE_BUILD_TYPE=Debug -G Ninja
$ cmake --build out
$ ./out/vcpkg-test # ./out/vcpkg-test [$SPECIFIC_TEST] for a specific set of tests
$ # i.e., ./out/vcpkg-test [arguments]
```

If you make any modifications to `vcpkg`, you'll have to do the
`cmake --build .` step again.

## Writing Tests

In your journey to write new tests, and to modify existing tests, reading the
[Catch2 documentation] will be very helpful! Come back after reading those 😀

You'll want to place your tests in one of the existing files, or, if it doesn't
belong in any of those, in a [new file](#adding-new-test-files).

The layout of these tests is as follows:

```cpp
// ... includes

TEST_CASE("Name of test", "[filename without the .cpp]") {
    // setup and the like
    REQUIRE(some boolean expression);
}

// etc.
```

You want to give these test cases good, descriptive, unique names, like
`SourceParagraph construct minimum` -- it doesn't need to be extremely clear
english, and shorthand is good, but make sure it's clear what the test is from
the name. For the latter parameter, known as "tags", you should at least put the
name of the file which the test case is in -- e.g., in `arguments.cpp`, you'd
tag all of the test cases with `[arguments]`.

If you wish to add helper functions, make sure to place them in an anonymous
namespace -- this will ensure that they don't trample over anybody else's
space. Additionally, there are a few helper functions that live in
`<vcpkg-test/util.h>` and `src/vcpkg-test/util.cpp` -- make sure to look into
them so that you're not rewriting functionality.

That should be all you need to know to start writing your own tests!
Remember to check out the [Catch2 documentation]
if you'd like to get more advanced with your tests,
and good luck on your testing journey!

## Adding New Test Files

Adding new test files should be easy and straightforward. All it requires is
creating a new source file in `src/vcpkg-test`.

### Example

Let's try writing a new test file called `example` (very creative, I know).

First, we should create a file, `example.cpp`, in `src/vcpkg-test`:

```cpp
// vcpkg-test/example.cpp
#include <catch2/catch.hpp>
```

This is the minimum file needed for tests; let's rebuild!

```sh
$ cmake --build .
[80/80] Linking CXX executable vcpkg.exe
```

Okay, now let's make sure this worked; add a test case to `example.cpp`:

```cpp
TEST_CASE("Example 1 - fail", "[example]") {
    REQUIRE(false);
}
```

Now build the tests again, and run them:

```sh
$ cmake --build .
[2/2] Linking CXX executable vcpkg-test.exe
$ ./vcpkg-test

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vcpkg-test.exe is a Catch v2.9.1 host application.
Run with -? for options

-------------------------------------------------------------------------------
Example 1 - fail
-------------------------------------------------------------------------------
$VCPKG_DIRECTORY/src/vcpkg-test/example.cpp(3)
...............................................................................

$VCPKG_DIRECTORY/src/vcpkg-test/example.cpp(14): FAILED:
    REQUIRE( false )

===============================================================================
test cases:  102 |  101 passed | 1 failed
assertions: 3611 | 3610 passed | 1 failed
```

Hopefully, that worked! It should compile correctly, and have one failing test.
Now let's try a more complex test, after deleting the old one;

```cpp
// add #include <vcpkg/base/strings.h> to the top of the file
namespace Strings = vcpkg::Strings;

TEST_CASE("Example 2 - success", "[example]") {
    std::string hello = "Hello";
    REQUIRE(Strings::case_insensitive_ascii_equals(hello, "hELLo"));
    REQUIRE_FALSE(Strings::case_insensitive_ascii_starts_with(hello, "E"));
}
```

Now compile and build the tests, and this time let's only run our example tests:

```sh
$ cmake --build .
[2/2] Linking CXX executable vcpkg-test.exe
$ ./vcpkg-test [example]
Filters: [example]
===============================================================================
All tests passed (2 assertions in 1 test case)
```

Hopefully you have one test running and succeeding! If you have that, you have
succeeded at adding a new file to vcpkg's tests. Congratulations! Have fun on
the rest of your journey 🐱‍👤😁

[Catch2 documentation]: https://github.com/catchorg/Catch2/blob/master/docs/tutorial.md#top
