This document contains information about the:

The rules, specifications, and deadlines given here are absolute. Only the competition panel has the right to make exceptions. It is assumed that all entrants have read the web pages related to the competition, and have complied with the competition rules. Non-compliance with the rules could lead to disqualification. A "catch-all" rule is used to deal with any unforeseen circumstances:- CASC-J6 will run in two parts:
*CASC@Turing*will be held during the Alan Turing Centenary Conference, and the "regular" CASC will be held during IJCAR. CASC@Turing has three divisions: FOF@Turing, FNT@Turing, MZR@Turing. The regular CASC has six divisions: THF, TFA, FOF, FNT, EPR, LTB. See Divisions for details. - The FOF division includes the previous CNF division.
The FOF division has four problem categories:
FEQ, FNE, CEQ, CNE.
See Divisions for details.
- The FNT division includes the previous SAT division (which
has been suspended since CASC-J5).
The FNT division has four problem categories:
FNQ, FNN, SEQ, SNE.
See Divisions for details.
- The UEQ division has been suspended.
- Prover9 was entered as a replacement for Otter, as a fixed point against which progress can be judged over the years.

- The
**FOF@Turing**division: First-Order Form syntactically non-propositional theorems (axioms with a provable conjecture). The FOF@Turing division has three problem categories:- The
**FNE**category: FOF with No Equality, not (known to be) effectively propositional - The
**FEQ**category: FOF with EQuality, not (known to be) effectively propositional - The
**FEP**category: FOF Effectively Propositional

- The
- The
**FNT@Turing**division: First-order form syntactically non-propositional Non-Theorems (axioms with a countersatisfiable conjecture, and satisfiable axiom sets). The FNT@Turing division has two problem categories:- The
**FNN**category: FNT with No equality - The
**FNQ**category: FNT with eQuality

- The
- The
**MZR@Turing**division: First-order form non-propositional theorems (axioms with a provable conjecture) taken from an export of Mizar problems, presented in batches. The problems in each batch can be attempted in any order, including multiple attempts on a problem. This facilitates learning from proofs found. There is no per-problem time limit, only an overall wall clock time limit. A set of*training problems and solutions*is available for tuning and training systems entered in this division. - The
**THF**division: Typed Higher-order Form theorems (axioms with a provable conjecture), using the THF0 syntax. The THF division has two problem categories:- The
**TNE**category: THF with No Equality - The
**TEQ**category: THF with EQuality

- The
- The
**TFA**division: Typed First-order with Arithmetic theorems (axioms with a provable conjecture), using the TFF0 syntax. The TFA division has two problem categories:- The
**TFI**category: TFA with only Integer arithmetic - The
**TFR**category: TFA with only Rational or only Real arithmetic (no mixed rational and real arithmetic)

- The
- The
**FOF**division: First-Order Form syntactically non-propositional theorems (axioms with a provable conjecture), and clause normal form really non-propositional theorems (unsatisfiable clause sets), but not unit equality problems. The FOF division has four problem categories:- The
**FNE**category: FOF with No Equality - The
**FEQ**category: FOF with EQuality - The
**CNE**category: CNF with No Equality - The
**CEQ**category: CNF with EQuality

- The
- The
**FNT**division: First-order form syntactically non-propositional Non-Theorems (axioms with a countersatisfiable conjecture, and satisfiable axiom sets), and clause normal form really non-propositional non-theorems (satisfiable clause sets). The FNT division has four problem categories:- The
**FNN**category: FNT with No equality - The
**FNQ**category: FNT with eQuality - The
**SNE**category: SAT with No Equality - The
**SEQ**category: SAT with EQuality

- The
- The
**EPR**division: Effectively PRopositional clause normal form theorems and non-theorems (clause sets). The EPR division has two problem categories:- The
**EPT**category: Effectively Propositional Theorems (unsatisfiable clause sets) - The
**EPS**category: Effectively Propositional non-theorems (Satisfiable clause sets)

- The
- The
**LTB**division: First-order form non-propositional theorems (axioms with a provable conjecture) from Large Theories, presented in Batches. The LTB division has three problem categories:- The
**ISA**category: Problems exported from Isabelle. - The
**MZR**category: Problems exported from the Mizar Mathematical Library (MML). - The
**SMO**category: Problems exported from the Suggested Upper Merged Ontology (SUMO). In this problem category the conjectures are specified as questions (specified by the formula role), to indicate that answers are desired in the output.

- The

- Two Intel Xeon E5620, quad-core, 2.40GHz CPUs, hyperthreading enabled
- 48GB memory
- Debian 6 (squeeze) 3.2.14.1.amd64-smp operating system

Problems for CASC are taken from the TPTP Problem Library. Additionally, problems for the MZR@Turing division will be taken from the MPTP2078 problem set, and problems for the SMO category of the LTB division will be taken from a problem set developed for CASC-J6. The TPTP version used for CASC is released after the competition has started, so that new problems have not been seen by the entrants. Access to and use of the non-TPTP problem sets is controlled to ensure that the system complied with the CASC tuning restrictions.

The problems have to meet certain criteria to be eligible for selection:

- The TPTP uses system performance data to compute problem difficulty
ratings, and from the ratings classifies problems as one of:
- Easy: Solvable by all state-of-the-art ATP systems
- Difficult: Solvable by some state-of-the-art ATP systems
- Unsolved: Not yet solved by any ATP system
- Open: Theorem-hood unknown

- The TPTP distinguishes versions of problems as one of standard, incomplete, augmented, especial, or biased. All except biased problems are eligible.
- In the batch divisions, the problems are selected so that there is consistent symbol usage between problems in each batch, but there may not be consistent axiom naming between problems.

- The selection is constrained so that no division or category contains an excessive number of very similar problems.
- The selection mechanism is biased to select problems that are new in the TPTP version used, until 50% of the problems in each category have been selected, after which random selection (from old and new problems) continues. The actual percentage of new problems used depends on how many new problems are eligible and the limitation on very similar problems.

*Number of Problems*

The minimal numbers of problems that must be used in each division and
category, to ensure sufficient confidence in the competition results,
are determined from the numbers of eligible problems in each division
and category
(the competition organizers have to ensure that there are sufficient computers
available to run the ATP systems on this minimal number of problems).
The minimal numbers of problems are used in determining the
time limits imposed on each solution attempt.

A lower bound on the total number of problems to be used is determined from the number of computers available, the time allocated to the competition, the number of ATP systems to be run on the competition computers over all the divisions, and the time limit per problem, according to the following relationship:

NumberOfComputers * TimeAllocated NumberOfProblems = --------------------------------- NumberOfATPSystems * TimeLimitIt is a lower bound on the total number of problems because it assumes that every system uses all of the time limit for each problem. Since some solution attempts succeed before the time limit is reached, more problems can be used.

The numbers of problems used in each division and problem category are (roughly) proportional to the numbers of eligible problems, after taking into account the limitation on very similar problems. The numbers of problems used in each division and category are determined according to the judgement of the competition organizers. For the MZR@Turing division, which has features that encourage machine learning from proofs found, at least 300 problems will be used.

- strip out all comment lines, including the problem header
- randomly reorder the formulae/clauses
(
`include`directives are left before formulae, type declarations and definitions are kept before the symbols' uses) - randomly swap the arguments of associative connectives, and randomly reverse implications
- randomly reverse equalities

In the SMO problem category of the LTB division, the `conjecture`
role is replaced by the `question` role, to indicate that
answers are desired.

In the demonstration division the same problems are used as for the competition divisions, with the same preprocessing applied. However, the original file names can be retained for systems running on computers provided by the entrant.

- A header line
`% SZS start BatchConfiguration` - A problem category line of the form

`division.category`*division_mnemonic*.*category_mnemonic*

For the MZR@Turing division it will be

`division.category MZR.MZR`

For the LTB division it will be

`division.category LTB.`*category_mnemonic*

where the category mnemonics are`ISA`,`MZR`,`SMO`. - A specification of whether or not the problems in the batch must be
attempted in order is given, in a line of the form

`execution.order`*ordered/unordered*

For the MZR@Turing division it will be

`execution.order unordered`

For the LTB division it will be

`execution.order ordered`

i.e., in the LTB division systems may not start any attempt on a problem, including reading the problem file, before ending the attempt on the preceding problem. - A specification of what output is required from the ATP systems
for each problem, in a line of the form

`output.required`*space_separated_list*

where the available list values are the SZS values`Assurance`,`Proof`,`Model`, and`Answer`. For CASC-J6 it will be

`output.required Assurance` - A specification of what output is desired from the ATP systems
for each problem, in a line of the form

`output.desired`*space_separated_list*

where the list values are as for the required output. For the MZR@Turing division and the MZR problem category of the LTB division it will be

`output.desired Proof`

For the ISA problem category it will be

`output.desired Proof ListOfFOF`

where the`ListOfFOF`is a list of axioms sufficient for a proof (a well-formed proof can be used to provide such a list). For the SMO problem category it will be

`output.desired Proof Answer`

where the answer is a definite binding for the outermost existentially quantified variables of the conjecture. - The wall clock time limit per problem, in a line of the form

`limit.time.problem.wc`*limit_in_seconds*

A value of zero indicates no per-problem limit - this will be the case for the MZR@Turing division. - A terminator line
`% SZS end BatchConfiguration` - A header line
`% SZS start BatchIncludes` -
`include`directives that are used in every problem. Problems in the batch have all these`include`directives, and can also have other`include`directives that are not listed here. - A terminator line
`% SZS end BatchIncludes` - A header line
`% SZS start BatchProblems` - Pairs of absolute problem file names, and absolute output file names where the output for the problem must be written.
- A terminator line
`% SZS end BatchProblems`

CPU and wall clock time limits are imposed. The minimal CPU time limit per problem is 240s. The maximal CPU time limit per problem is determined using the relationship used for determining the number of problems, with the minimal number of problems as the

In the demonstration division, each entrant can choose to use either a CPU or a wall clock time limit, whose value is the CPU time limit of the competition divisions.

**MZR@Turing division**

For each batch there is a there is an overall wall clock time limit, which
is available as a command line parameter.
The overall limit is the at least 30s multiplied by the number of problems
in the division.
There are no CPU time limits.

**LTB division**

For each batch there is a wall clock time limit per problem, which is
provided in the configuration section at the start of each batch.
The minimal wall clock time limit per problem is 30s.
For each problem category there is an overall wall clock time limit, which
is available as a command line parameter.
The overall limit is the sum over the batches of the batch's per-problem
limit multiplied by the number of problems in the batch.
Time spent before starting the first problem of a batch (e.g., preloading
and analysing the batch axioms), and times spent between
ending a problem and starting the next (e.g.,
learning from a proof just found), are not part of the times taken on the
individual problems, but are part of the overall time taken.
There are no CPU time limits.

The systems are ranked in the competition divisions, from the performance data.
The THF, TFA, EPR,
and LTB divisions have an *assurance* ranking
class, ranked according to the number of problems solved, but not necessarily
accompanied by a proof or model (thus giving only an assurance of the
existence of a proof/model).
The CASC@Turing, FOF, and FNT divisions have a *proof*/*model*
ranking class, ranked according to the number of problems solved with an
acceptable proof/model output.
Ties are broken according to the average time over problems solved
(CPU time for the non-batch divisions, wall clock time for the batch
divisions).
In the competition divisions, class winners are announced and prizes are
awarded.

- Google has provided prize money for CASC@Turing. Prizes will be awarded for the proof and model ranking classes. In each case the winner will receive £1500, the second place £1000, and the third place £500.
- The Isabelle group at the Technische Universität München has provided a travel prize for the ISA problem category of the LTB division. The prize will be awarded according to the number of problems solved with a list of axioms sufficient for a proof output. The winner will be invited to visit the group at the university for up to one week. The travel and hotel expenses will be covered.
- Articulate Software has provided prize money for the SMO category of the LTB division. Prizes will be awarded for the assurance ranking class, and also according to the question answering measure described below. In each case the winner will receive $750, the second place $500, and the third place $250.

The competition panel decides whether or not the systems' proofs and models are acceptable for the proof/model ranking classes. The criteria include:

- Derivations must be complete, starting at formulae from the problem, and ending at the conjecture (for axiomatic proofs) or a false formula (for proofs by contradiction, including CNF refutations).
- For proofs of FOF problems by CNF refutation, the conversion from FOF to CNF must be adequately documented.
- Derivations must show only relevant inference steps.
- Inference steps must document the parent formulae, the inference rule used, and the inferred formula.
- Inference steps must be reasonably fine-grained.
- An unsatisfiable set of ground instances of clauses is acceptable for establishing the unsatisfiability of a set of clauses.
- Models must be complete, documenting the domain, function maps, and predicate maps. The domain, function maps, and predicate maps may be specified by explicit ground lists (of mappings), or by any clear, terminating algorithm.

In addition to the ranking criteria, other measures are made and presented in the results:

- The
*state-of-the-art contribution*(SOTAC) quantifies the unique abilities of each system. For each problem solved by a system, its SOTAC for the problem is the inverse of the number of systems that solved the problem. A system's overall SOTAC is its average SOTAC over the problems it solves. - The
*efficiency measure*balances the number of problems solved with the time taken. It is the average of the inverses of the times for problems solved (CPU times for the non-batch divisions, wall clock times for the LTB division, with times less than the timing granularity rounded up to the granularity, to avoid skewing caused by very low times) multiplied by the fraction of problems solved. This can be interpreted intuitively as the average of the solution rates for problems solved, multiplied by the fraction of problems solved. - In the LTB division, which uses a wall clock time limit,
the
*core usage*is the average of the ratios of CPU time to wall clock time used, over the problems solved. This measures the extent to which the systems take advantage the multiple cores. - In the ISA problem category of the LTB division, the
*number of axiom lists*(output of a list of axioms sufficient for a proof) is counted. A well-formed proof can be used to provide such a list. - In the SMO problem category of the LTB division, the
*number of questions answered*(output of the bindings for the outermost existentially quantified variables) is counted.

At some time after the competition, all high ranking systems in the competition divisions are tested over the entire TPTP. This provides a final check for soundness (see the section on system properties regarding soundness checking before the competition). If a system is found to be unsound during or after the competition, but before the competition report is published, and it cannot be shown that the unsoundness did not manifest itself in the competition, then the system is retrospectively disqualified. At some time after the competition, the proofs and models from the winners of the proof/model ranking classes are checked by the panel. If any of the proofs or models are unacceptable, i.e., they are significantly worse than the samples provided, then that system is retrospectively disqualified. All disqualifications are explained in the competition report.

Systems can be entered at only the division level, and can be entered into more than one division (a system that is not entered into a competition division is assumed to perform worse than the entered systems, for that type of problem - wimping out is not an option). Entering many similar versions of the same system is deprecated, and entrants may be required to limit the number of system versions that they enter. Systems that rely essentially on running other ATP systems without adding value are deprecated; the competition panel may disallow or move such systems to the demonstration division. The division winners of the previous CASC are automatically entered into their divisions, to provide benchmarks against which progress can be judged.

- Architecture. This section introduces the ATP system, and describes the calculus and inference rules used.
- Strategies. This section describes the search strategies used, why they are effective, and how they are selected for given problems. Any strategy tuning that is based on specific problems' characteristics must be clearly described (and justified in light of the tuning restrictions).
- Implementation. This section describes the implementation of the ATP system, including the programming language used, important internal data structures, and any special code libraries used. The availability of the system is also given here.
- Expected competition performance. This section makes some predictions about the performance of the ATP system in each of the divisions and categories in which it is competing.
- References.

The system description has to be emailed to the competition organizers by the system description deadline. The system descriptions, along with information regarding the competition design and procedures, form the proceedings for the competition.

Proof samples for the FOF proof class must include a proof for
`SEU140+2`.
Model samples for the FNT model class must include models for
`NLP042+1` and
`SWV017+1`.
The sample solutions must illustrate the use of all inference rules.
An explanation must be provided for any non-obvious features.

For systems competing for the ISA problem category prize in the LTB division,
representative sample proofs or lists of axioms must be emailed to the
competition organizers by the sample solutions
deadline.
Use of the
SZS standards is required.
Samples must include a proof or list for
`SEU140+2`.
For systems competing for the SMO problem category prize in the LTB division,
representative sample answers must be emailed to the competition organizers
by the sample solutions deadline.
Samples must include an answer for
`CSR082+1`.

**Soundness and Completeness**

- Systems must be sound. At some time before the competition all the systems in the competition divisions are tested for soundness. Non-theorems are submitted to the systems in the FOF@Turing, MZR@Turing, THF, TFA, FOF, EPR, and LTB divisions, and theorems are submitted to the systems in the FNT@Turing, FNT, and EPR divisions. Finding a proof of a non-theorem or a disproof of a theorem indicates unsoundness. If a system fails the soundness testing it must be repaired by the unsoundness repair deadline or be withdrawn. The soundness testing eliminates the possibility of a system simply delaying for some amount of time and then claiming to have found a solution. For systems running on computers supplied by the entrant in the demonstration division, the entrant must perform the soundness testing and report the results to the competition organizers.
- Systems do not have to be complete in any sense, including calculus, search control, implementation, or resource requirements.
- All techniques used must be general purpose, and expected to extend
usefully to new unseen problems.
The precomputation and storage of information about individual TPTP
problems or their solutions is not allowed.
Strategies and strategy selection based on individual TPTP problems
or their solutions are not allowed.
If machine learning procedures are used, the learning must ensure that
sufficient generalization is obtained so that no there is no
specialization to individual problems or their solutions.
- For the MZR@Turing division, the problems will be taken from the publicly available MPTP2078 problem set. As such, precomputation and storage of information about problems in that set, or their solutions, is not directly allowed. However ...
- These training problems and solutions that have been taken from that set may be used for tuning and training systems for the MZR@Turing division. The training problems will not appear in the MZR@Turing division. There are at least twice as many training problems as there will be MZR@Turing division problems.
- The training problems and solutions may be used for producing generally useful strategies that extend to "unseen" problems in the MPTP2078 set. Such strategies can rely on the consistent naming of symbols and formulas in the MPTP2078 set, and may use techniques for memorization and generalization of problems and solutions in the training set. The system description must fully explain any such tuning or training that has been done.
- The competition panel may disqualify any system whose tuning or training is deemed to be problem specific rather than general purpose. If you are in doubt, contact the competition organizer.

- The system's performance must be reproducible by running the system again.

- Systems must run on a single locally provided standard UNIX computer
(the
*competition computers*). ATP systems that cannot run on the competition computers can be entered into the demonstration division. - Systems must be executable by a single command line, using an absolute path name for the executable, which might not be in the current directory. In the non-batch divisions the command line arguments are the absolute path name of a symbolic link as the problem file name, the individual problem time limit (if required by the entrant), and entrant specified system switches. In the batch divisions the command line arguments are the absolute path name of the batch specification file, the overall category time limit (if required by the entrant), and entrant specified system switches. No shell features, such as input or output redirection, may be used in the command line. No assumptions may be made about the format of file names.
- Systems must be fully automatic, i.e., all command line switches have to be the same for all problems in each division.

- In the non-batch divisions all solution output must be to
`stdout`. In the batch divisions all solution output must be to the named output file for each problem. - In the LTB division the systems must print SZS notification lines to
`stdout`when starting and ending work on a problem (including any cleanup work, such as deleting temporary files). For example% SZS status Started for /home/graph/tptp/TPTP/Problems/CSR/CSR075+2.p ... (system churns away, result and solution output to file) % SZS status Theorem for /home/graph/tptp/TPTP/Problems/CSR/CSR075+2.p % SZS status Ended for /home/graph/tptp/TPTP/Problems/CSR/CSR075+2.p

- For each problem, the systems must output a distinguished string
indicating what solution has been found or that no conclusion has been
reached.
The distinguished strings the problem status should use
the
SZS ontology and standards.
For example
% SZS status Theorem for SYN075+1

or% SZS status GaveUp for SYN075+1

Regardless of whether the SZS status values are used, the distinguished strings must be different for:- Proved theorems of FOF problems
(SZS status
`Theorem`) - Disproved conjectures of FNT problems
(SZS status
`CounterSatisfiable`) - Unsatisfiable sets of formulae (FOF problems without conjectures) and
unsatisfiable set of clauses (CNF problems)
(SZS status
`Unsatisfiable`) - Satisfiable sets of formulae (FNT problems without conjectures) and
satisfiable set of clauses (SAT problems)
(SZS status
`Satisfiable`)

In batch divisions this line must use the SZS standards, including the problem file name, and must be output to both

`stdout`and the solution file. In the LTB division this line must be output as the last thing before the ending notification line. - Proved theorems of FOF problems
(SZS status
- When outputting proofs/models, the start and end of the proof/model must
be delimited by distinguished strings.
The distinguished strings should use the
SZS ontology and standards.
For example
% SZS output start CNFRefutation for SYN075+1 ... % SZS output end CNFRefutation for SYN075+1

Regardless of whether the SZS output forms are used, the distinguished strings must be different for:- Proofs
(SZS output forms
`Proof`,`Refutation`,`CNFRefutation`) - Models
(SZS output forms
`Model`,`FiniteModel`,`InfiniteModel`,`Saturation`)

- Proofs
(SZS output forms
- When outputting a list of axioms sufficient
for a proof in the ISA problem category of the LTB division, the
start and end of the list must be delimited by distinguished strings.
The distinguished strings should use the
SZS ontology and standards.
For example
% SZS output start ListOfFOF for SYN075+1 ... % SZS output end ListOfFOF for SYN075+1

- When outputting answers in the SMO problem category of the LTB division, the answers must be output using the Tuple or Instantiated answer form of the proposed TPTP standard for answer reporting.

- The systems that run on the competition computers must be
interruptible by a
`SIGXCPU`signal, so that the CPU time limit can be imposed, and interruptable by a`SIGALRM`signal, so that the wall clock time limit can be imposed. For systems that create multiple processes, the signal is sent first to the process at the top of the hierarchy, then one second later to all processes in the hierarchy. The default action on receiving these signals is to exit (thus complying with the time limit, as required), but systems may catch the signals and exit of their own accord. If a system runs past a time limit this is noticed in the timing data, and the system is considered to have not solved that problem. - If an ATP system terminates of its own accord, it may not leave any
temporary or intermediate output files.
If an ATP system is terminated by a
`SIGXCPU`or`SIGALRM`, it may not leave any temporary or intermediate output files anywhere other than in`/tmp`. Multiple copies of the ATP systems must be executable concurrently, in the same (NFS cross mounted) directory. It is therefore necessary that temporary files have unique names. - For practical reasons excessive output from an ATP system is not allowed. A limit, dependent on the disk space available, is imposed on the amount of output that can be produced. The limit is at least 10MB per system.

- Instructions for installation
- Instructions for executing the system, using
`%s`and`%d`to indicate where the problem file name and time limit must appear in the command line. - The distinguished strings indicating what solution has been found, and delimiting proofs/models.

The installation procedure may require changing path variables, invoking
`make` or something similar, etc., but nothing unreasonably
complicated.
All system binaries must be created in the installation process; they
cannot be delivered as part of the installation package.
If the ATP system requires any special software, libraries, etc, which is
not part of a standard installation, the competition organizers must
be told in the system registration.
The system is installed onto the competition computers by the competition
organizers, following the instructions in the `ReadMe` file.
Installation failures before the system delivery deadline are passed
back to the entrant
(i.e., delivery of the installation package before the system delivery
deadline provides an opportunity to fix things if the installation fails!).
After the system delivery deadline no further changes or late systems are
accepted.
If you are in doubt about your installation package or procedure, please
email the competition organizers.

For systems running on entrant supplied computers in the demonstration
division, entrants must deliver a source code package to the competition
organizers by the start of the competition.
The source code package must be a `.tgz` file containing
the system source code.

After the competition all competition division systems' source code is made publicly available on the CASC web site. In the demonstration division, the entrant specifies whether or not the source code is placed on the CASC web site. An open source license is encouraged.

During the competition a `perl` script parses the systems' outputs.
If any of an ATP system's distinguished strings are found then the
time used to that point is noted.
A system has solved a problem iff it outputs its termination string within
the time limit, and a system has produced a proof/model iff it outputs
its end-of-proof/model string within the time limit.
The result and timing data is used to generate an HTML file, and a web
browser is used to display the results.

The execution of the demonstration division systems is supervised by their entrants.

- Check: The ATP system can run on a computer that has the same
configuration as the competition computers.
- Check: The ATP system can be run by an absolute path name for the
executable.
prompt> pwd /home/tptp prompt> which MyATPSystem /home/tptp/bin/MyATPSystem prompt> /home/tptp/bin/MyATPSystem /home/tptp/TPTP/Problems/SYN/SYN075-1.p SZS status Unsatisfiable for SYN075-1

- Check: The ATP system accepts an absolute path name of a symbolic
link as the problem file name.
prompt> cd /home/tptp/tmp prompt> ln -s /home/tptp/TPTP/Problems/SYN/SYN075-1.p CCC001.p prompt> cd /home/tptp prompt> /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001.p SZS status Unsatisfiable for CCC001

- Check: The ATP system makes no assumptions about the format of the
problem file name.
prompt> ln -s /home/tptp/TPTP/Problems/SYN/SYN075-1.p _foo-Blah prompt> /home/tptp/bin/MyATPSystem _foo-Blah SZS status Unsatisfiable for _foo-Blah

- Check: The ATP system can run under the
`TreeLimitedRun`program.prompt> which TreeLimitedRun /home/tptp/bin/TreeLimitedRun prompt> /home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 4867 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001 FINAL WATCH: 147.8 CPU 150.0 WC

- Check: The ATP system's CPU time can be limited using the
`TreeLimitedRun`program.prompt> /home/tptp/bin/TreeLimitedRun -q0 10 20 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 10s TreeLimitedRun: WC time limit is 20s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- CPU time limit exceeded FINAL WATCH: 10.7 CPU 13.1 WC

- Check: The ATP system's wall clock time can be limited using the
`TreeLimitedRun`program.prompt> /home/tptp/bin/TreeLimitedRun -q0 20 10 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 20s TreeLimitedRun: WC time limit is 10s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- Alarm clock FINAL WATCH: 9.7 CPU 10.1 WC

- Check: The system outputs a distinguished string when terminating of
its own accord.
prompt> /home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001 FINAL WATCH: 147.8 CPU 150.0 WC

Similar checks should be made for the cases where the system gives up. - Check: The system outputs distinguished strings at the start and end
of its solution.
prompt> /home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem -output_proof /home/tptp/tmp/CCC001.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001 SZS output start CNFRefutation for CCC001 ... acceptable proof/model here ... SZS output end CNFRefutation for CCC001 FINAL WATCH: 147.8 CPU 150.0 WC

- Check: No temporary or intermediate files are left if the system
terminates of its own accord, and no temporary or intermediate files are
left anywhere other than in
`/tmp`if the system is terminated by a`SIGXCPU`or`SIGALRM`. Check in the current directory, the ATP system's directory, the directory where the problem's symbolic link is located, and the directory where the actual problem file is located.prompt> pwd /home/tptp prompt> /home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 13526 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001 FINAL WATCH: 147.8 CPU 150.0 WC prompt> ls /home/tptp ... no temporary or intermediate files left here ... prompt> ls /home/tptp/bin ... no temporary or intermediate files left here ... prompt> ls /home/tptp/tmp ... no temporary or intermediate files left here ... prompt> ls /home/tptp/TPTP/Problems/GRP ... no temporary or intermediate files left here ... prompt> ls /tmp ... no temporary or intermediate files left here by decent systems ...

- Check: Multiple concurrent executions do not clash.
prompt> (/home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001.p) & (/home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001.p) TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 5829 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001 FINAL WATCH: 147.8 CPU 150.0 WC SZS status Unsatisfiable for CCC001 FINAL WATCH: 147.8 CPU 150.0 WC