<Chapter Label="Chap-Testing">
<Heading>Testing &GAP;</Heading>

<E>TODO: This chapter is partially outdated and has to be revised</E>
<P/>

In this chapter we describe several tests which should be applied regularly
to the development and release candidate versions of &GAP;
in the hope to reduce the number of bugs and potential other problems
(such as conflicts between the main &GAP; system and packages)
in the released version.
<P/>

These tests cover &GAP; computations,
in the sense that actual output is compared with expected output,
as well as formal aspects of the documentation.


<Section Label="Sect-TstFiles">
<Heading><C>.tst</C>-files</Heading>

The directory <F>tst</F> in the &GAP; root directory contains directories,
each of which contains files whose names have the suffix <C>.tst</C>.
The format of each of these files must be suitable for being read with
the function <C>Test</C>,
see section <Ref BookName="Ref" Sect="Test Files"/>.
If functionality from &GAP; packages is needed in a <C>tst</C> file
then it is recommended to execute these tests only if the packages are
really available, and to load the packages explicitly,
for example as follows.

<Listing>
gap> if LoadPackage( "ctbllib" ) &lt;&gt; fail then
>      if Irr( CharacterTable( "WeylD", 4 ) )[1] &lt;&gt;
>           [ 3, -1, 3, -1, 1, -1, 3, -1, -1, 0, 0, -1, 1 ] then
>        Print( "problem with Irr( CharacterTable( \"WeylD\", 4 ) )[1]\n" );
>      fi;
>    fi;
</Listing>

Moreover, it is recommended to group tests that require packages to be
performed after all other tests are completed.

The directory <F>tst/testinstall</F> contains a subset of tests which are
recommended after installing &GAP;.
The idea is that the examples in these <C>.tst</C> files are typical
&GAP; computations,
and that running these tests requires only a few minutes.
The &GAP; script <F>tst/testinstall.g</F> will run the tests in
<F>tst/testinstall</F>.
<P/>

Further tests are contained in <F>tst/teststandard</F>. The script
<F>tst/teststandard.g</F> will run all tests in both <F>tst/teststandard</F>
and <F>tst/testinstall</F>.
<P/>

</Section>


<Section Label="Sect-CheckManualExamples">
<Heading>Checking examples in the main manuals</Heading>

The reproducibility of the output of examples in the main &GAP; manuals
(the two manuals in the directories <F>ref</F> and <F>tut</F>)
is promised in section <Ref BookName="tut" Sect="Starting and Leaving GAP"/>,
under the condition that all examples of a manual chapter are run
immediately after starting &GAP;.
Section <Ref BookName="GAPDoc" Sect="Testing Manual Examples"/> contains another
paragraph about the use of manual examples for testing purposes.
<P/>

There is no guarantee that package manuals behave this way.
It is up to the package authors whether they want to promise something
similar for their package.
<P/>

</Section>


<Section Label="Sect-CheckPackages">
<Heading>Tests for packages</Heading>

Besides tests of the functionality of packages, we are also interested
in the effects of packages on the functionality of the main system.
The output that is shown in examples in the &GAP; manuals as well as the
output that is prescribed in the test files in <C>tst</C> should be the same
when &GAP; is run without packages or with all packages available.
Therefore, the tests in the standard test suite
(see <Ref Sect="Sect-TestSuite"/>) are run several times,
in a &GAP; that loads no packages and in a &GAP; that loads all available
packages and/or all available autoloaded packages.
<P/>

Concerning tests of package functionality,
in principle the package authors are responsible for providing and running
tests for their packages.
However, it makes sense to provide test files analogous to those described
in section <Ref Sect="Sect-TstFiles"/>.
<P/>

If the record in the <F>PackageInfo.g</F> file of the package
(see <Ref BookName="Ref" Sect="The PackageInfo.g File"/>)
contains the component <C>TestFile</C> then the file given by the value
is read as a part of the regular &GAP; test suite,
see section <Ref Sect="Sect-TestSuite"/>.
So this file should contain <C>Test</C> statements for suitable
files that are part of the package distribution.
Since these tests are intended to be run regularly,
they should require not more than a few hours of CPU time.
(Of course also longer examples can make sense but they should not be
listed in the file given by the <C>TestFile</C> component.)
<P/>

Note that the inclusion of the above package tests refers to the latest
released version of each package, not to its development version.
<P/>

Since the format of the documentation for a package is not prescribed,
there is no generic tool for checking manual examples of packages.
One possibility to include the examples from package manuals in the
standard tests is to create a <C>.tst</C> file with these examples from
the manual.
(For one implementation of this approach, see the script <F>etc/makexpl</F>
in the <C>ctbllib</C> package.
It produces a file <F>tst/docxpl.tst</F> in the package directory,
containing all examples from the package manual.)
</Section>


<Section Label="Sect-CheckManualRefs">
<Heading>Checking references and cross-references of manuals</Heading>

This is still to be provided (and then documented here).
Perhaps the easiest way is a check of cross-references in and between
the HTML versions of the manuals;
and perhaps this should be viewed as a part of the more general task
to check the validity of (in particular cross-references in)
the &GAP; website.
<P/>

(Possible ingedients are <F>doc/test/CheckBooks.g</F> and
<F>dev/LinksOfAllHelpSections.g</F>.)
<P/>

For the manuals in the formats given by the &GAPDoc; package,
multiply defined labels and non-resolved references will be
reported at the manual build stage.
<P/>

</Section>


<Section Label="Sect-NameConflicts">
<Heading>Detecting potential naming conflicts</Heading>

Also this is still to be provided (and then documented here).
</Section>


<Section Label="Sect-TestSuite">
<Heading>The standard test suite</Heading>

The standard test suite is given by several targets in the <F>Makefile</F>
in the &GAP; root directory.
(So the file <F>Makefile.in</F> is relevant for the maintenance.)
It consists of
<List>
  <Item>
    The tests contained in <F>tst/testinstall</F>,
    see <Ref Sect="Sect-TstFiles"/>,
  </Item>
  <Item>
    All tests in all subdirectories of <C>tst</C>,
    see <Ref Sect="Sect-TstFiles"/>,
  </Item>
  <Item>
    the tests of the main manuals described in section
    <Ref Sect="Sect-CheckManualExamples"/>,
  </Item>
  <Item>
    the package tests described by the <C>TestFile</C> components of
    <F>PackageInfo.g</F> files of all accepted or deposited packages, and
  </Item>
  <Item>
    the tests of packages loading with
    <Ref BookName="ref" Func="LoadPackage"/>
    and
    <Ref BookName="ref" Func="LoadAllPackages"/>
  </Item>
</List>

As is stated in Section <Ref Sect="Sect-CheckPackages"/>,
each test is run at least twice,
once in a &GAP; that loads no packages and then
in a &GAP; that loads all available
packages and/or all available autoloaded packages.
<P/>

Before running these tests,
you should make sure that the system is up to date, in particular,
<List>
  <Item>
    compile package executables where applicable,
  </Item>
  <Item>
    call <C>make doc</C> in the root directory,
    in order to process the main documentation,
  </Item>
</List>

Then proceed as follows in the &GAP; root directory.
<List>
  <Item>
    Call <C>dev/ci.sh testinstall</C>.
    This reads the file <C>tst/testinstall.g</C>,
    which should require just a few minutes.
  </Item>
  <Item>
    Call <C>dev/ci.sh testmanuals</C>.
    This runs the examples in the main &GAP; manuals,
    and should also require only a few minutes.
    (Note that a new &GAP; process is started for each manual chapter.
    In order to accelerate this, a &GAP; workspace is created
    in the beginning, is reused for each chapter, and removed in the end.)
  </Item>
  <Item>
    Call <C>dev/ci.sh teststandard</C>.
    This runs the tests in <C>tst</C>,
    and may require a couple of hours.
  </Item>
  <Item>
    Call <C>dev/ci.sh testpackages</C>.
    This starts a new &GAP; session for each accepted or deposited package,
    and reads the file given by the component <C>TestFile</C> in the
    <F>PackageInfo.g</F> file (if this is available).
    It may require several hours.
  </Item>
  <Item>
    Call <C>dev/ci.sh testpackagesloading</C>.
    This loads each accepted or deposited package in a new &GAP; session,
    then starts a new &GAP; session and calls <Ref BookName="ref" Func="LoadAllPackages"/>
    to check that all packages may be loaded simultaneosuly.
    It require just a few minutes.
  </Item>
</List>

<P/>

The output is written to files in the directory <F>dev/log</F>,
which does not belong to the distribution of &GAP;.
The file names start with the name of the target (<C>testinstall</C>,
<C>teststandard</C>, etc.),
followed by <C>1</C> for the case that no packages are loaded before the tests,
by <C>2</C> if all packages are loaded before the tests,
and by <C>A</C> if only autoloaded packages are loaded before the tests,
followed by the date and time when the test was started,
in the format <C>_YY_MM_DD_HH_MM</C>.
For the package tests, the output for each package in question is written
to a file of its own, and the package name is appended to the file name.
<P/>

Note that in the case that no package is loaded initially,
some packages may be loaded during the tests,
due to <Ref BookName="ref" Func="LoadPackage"/> statements in the test files.
So a list of all packages that are loaded at the end of the tests
can be found at the end of the output files.
</Section>


<Section Label="Sect-TestEvaluation">
<Heading>Test evaluation</Heading>

After the tests,
please inspect the output files and deal with the differences that were
found.
The following situations may be expected.
<List>
  <Item>
    There are differences between the prescribed output and the actual
    output.

    This may be the result of code improvements and thus intentional;
    then the prescribed output should be replaced by the actual one.
    It may be a side-effect of code changes,
    for example new or changed methods may choose different generators
    for a group or a different ordering of conjugacy classes;
    then one should make sure that the changed output is not caused by
    worse methods being used than before.

    If the prescribed and the actual value are essentially different
    (e.g., <K>true</K> vs. <K>false</K> or different numerical values)
    then probably we have a serious problem n the sense that (at least)
    one of the two values is wrong.
  </Item>
  <Item>
    There are differences between the results of a test without packages
    and the same test with packages loaded.

    It may be that a package provides a better solution than the one that
    was expected when the input example had been prepared;
    in this case, it is perhaps advisable to ask the maintainers of the
    relevant code to modify the example in question,
    since it is apparently no longer typical.

    It may also be that a new package version breaks functionality;
    in this case, the package authors should be informed.
  </Item>
  <Item>
    A more subtle difference is a change in the runtime, which is listed
    more or less detailed in the output files.

    If the runtime increases w.r.t. an earlier &GAP; version then this
    should be reported to the &GAP; group.
    If the runtime with packages is substantially bigger than without packages
    then the package authors should be informed.
  </Item>
</List>
<P/>

When test input is adjusted to the current behaviour,
it should be adjusted for the situation that no packages are loaded.
(In the case of test input for a package, this means that only the needed
packages are loaded,
see the component <C>Dependencies.NeededOtherPackages</C> in the file
<F>PackageInfo.g</F>.)
Note that testing the changed behaviour introduced by another package
would be an issue for tests that belong to this package.
</Section>


<Section Label="Sect-TestAndRelease">
<Heading>Test Issues for Releases</Heading>

The following additional issues are related to testing
when a new version of &GAP; is to be released.
<List>
  <Item>
    Run all tests.
  </Item>
</List>
</Section>


<Section Label="Sect-TestToDo">
<Heading>Open items concerning tests</Heading>

<List>
  <Item>
    Add a paragraph about general tests for features:
    When new documentation is added,
    add meaningful examples to this documentation,
    from which users get an idea what typical uses are.
    In a corresponding file in the <C>tst</C> directory,
    add examples which cover also pathological input such as empty lists.
    Add also interesting computations to the tests which are perhaps to be
    excluded from the regular testing;
    for features in a package, perhaps distinguish between the tests that are
    covered by the <C>TestFile</C> component of the <F>PackageInfo.g</F> file
    and hard tests.
  </Item>
</List>
</Section>

</Chapter>
