This document contains information about the:

The rules, specifications, and deadlines given here are absolute. Only the competition panel has the right to make exceptions.

Every effort has been made to organize the competition in a fair and constructive manner. No responsibility is taken if, for one reason or the other, your system does not win.

- The FNT division - First order form Non-Theorems - has been added. This division uses counter-satisfiable and satisfiable FOF problems.
- The FOF and FNT divisions include problems that have no functors of arity greater than one. The problems must have variables. For the systems that convert to CNF (almost all), this means that the resulting CNF may be effectively propositional, depending on the quantifier patterns in the formaule and on the FOF to CNF conversion.
- The problems will use the full set of TPTP roles defined in
the TPTP language:
`axiom`,`lemma`,`conjecture`, etc. - The distinguished strings used to report the problem status should
(developers are encouraged to do this, not absolutely required :-) use
the SZS ontology, in a line starting
`SZS status`. For exampleSZS status Unsatisfiable for SYN075+1

orSZS status GaveUp for SYN075+1

- The distinguished strings used to delimit proofs/models should
(developers are encouraged to do this, not absolutely required :-) specify
the precise output form named in the SZS ontology, using lines
starting
`SZS output start`and`SZS output end`. For exampleSZS output start CNFRefutation for SYN075-1 ... SZS output end CNFRefutation for SYN075-1

CASC is run in divisions according to problem and system characteristics.
There are *competition* divisions in which systems are explicitly
ranked, and a *demonstration* division in which systems demonstrate
their abilities without being formally ranked.
Some divisions are further divided into problem categories, which
make it possible to analyze, at a more fine grained level, which systems
work well for what types of problems.
The problem categories have no effect on the competition rankings, which
are made at only the division level.

- The
**FOF**division: First-order form non-propositional theorems (axioms with a provable conjecture).

The FOF division has two problem categories:- The
**FNE**category: FOF with no Equality - The
**FEQ**category: FOF with Equality

- The
- The
**CNF**division: Mixed clause normal form really non-propositional theorems (unsatisfiable clause sets).

*Mixed*means Horn and non-Horn problems, with or without equality, but not unit equality problems (see the UEQ division below).*Really non-propositional*means with an infinite Herbrand universe. The CNF division has five problem categories:- The
**HNE**category: Horn with No Equality - The
**HEQ**category: Horn with some (but not pure) Equality - The
**NNE**category: Non-Horn with No Equality - The
**NEQ**category: Non-Horn with some (but not pure) Equality - The
**PEQ**category: Pure Equality

- The
- The
**FNT**division: First-order form non-propositional non-theorems (axioms with an unprovable conjecture, and satisfiable sets of axioms).

The FNT division has two problem categories:- The
**FNN**category: FNT with no Equality - The
**FNQ**category: FNT with Equality

- The
- The
**SAT**division: Clause normal form really non-propositional non-theorems (satisfiable clause sets).

The SAT division has two problem categories:- The
**SNE**category: SAT with No Equality - The
**SEQ**category: SAT with Equality

- The
- The
**EPR**division: Effectively propositional clause normal form theorems and non-theorems (clause sets).

*Effectively propositional*means non-propositional with a finite Herbrand Universe. The EPR division has two problem categories:- The
**EPT**category: Effectively Propositional Theorems (unsatisfiable clause sets) - The
**EPS**category: Effectively Propositional non-theorems (Satisfiable clause sets)

- The
- The
**UEQ**division: Unit equality clause normal form really non-propositional theorems (unsatisfiable clause sets).

ATP systems that cannot run on the competition computers, or cannot be entered into the competition divisions for any other reason, can be entered into the demonstration division. Demonstration division systems can run on the competition computers, or the computers can be supplied by the entrant. Computers supplied by the entrant may be brought to CASC, or may be accessed via the internet.

The entry specifies which competition divisions' problems are to be used. The results are presented along with the competition divisions' results, but may not be comparable with those results.

- AMD Athlon XP 3000+, 2091MHz CPU
- 503MB memory
- Linux 2.6.20-1.2307.fc5 operating system

*Problem Selection*

The problems are from the
TPTP Problem Library.
The TPTP version used for the competition is not released until after the
system installation deadline, so that new problems have not been seen by
the entrants.

The problems have to meet certain criteria to be eligible for selection:

- The TPTP uses system performance data to compute problem difficulty
ratings, and from the ratings classifies problems as one of:
- Easy: Solvable by all state-of-the-art ATP systems
- Difficult: Solvable by some state-of-the-art ATP systems
- Unsolved: Not yet solved by any ATP system
- Open: Theorem-hood unknown

- The TPTP distinguishes versions of problems as one of standard, incomplete, augmented, especial, or biased. All except biased problems are eligible.

The problems used are randomly selected from the eligible problems at the start of the competition, based on a seed supplied by the competition panel.

- The selection is constrained so that no division or category contains an excessive number of very similar problems.
- The selection mechanism is biased to select problems that are new in the TPTP version used, until 50% of the problems in each category have been selected, after which random selection (from old and new problems) continues. The actual percentage of new problems used depends on how many new problems are eligible and the limitation on very similar problems.

*Number of Problems*

The minimal numbers of problems that have to be used in each division and
category, to ensure sufficient confidence in the competition results,
are determined from the numbers of eligible problems in each division
and category
(the competition organizers have to ensure that there is sufficient CPU time
available to run the ATP systems on this minimal number of problems).
The minimal numbers of problems are used in determining the
CPU time limit imposed on each solution
attempt.

A lower bound on the total number of problems to be used is determined from the number of computers available, the time allocated to the competition, the number of ATP systems to be run on the competition computers over all the divisions, and the CPU time limit, according to the following relationship:

NumberOfComputers * TimeAllocated NumberOfProblems = --------------------------------- NumberOfATPSystems * CPUTimeLimitIt is a lower bound on the total number of problems because it assumes that every system uses all of the CPU time limit for each problem. Since some solution attempts succeed before the CPU time limit is reached, more problems can be used.

The numbers of problems used in each division and problem category is (roughly) proportional to the numbers of eligible problems, after taking into account the limitation on very similar problems.

The numbers of problems used in each division and category are determined according to the judgement of the competition organizers.

- rename all predicate and function symbols to meaningless symbols
- randomly reorder the clauses and literals in CNF problems
- randomly reorder the formulae in FOF problems
- randomly reverse the equalities in the UEQ problems
- remove equality axioms that are not needed by the ATP systems
- add equality axioms that are needed by the ATP systems
- output the problems in the formats required by the ATP systems
(The clause type information, one of
`axiom`

,`hypothesis`

, or`conjecture`

, may be included in the final output of each formula.)

`CCCNNN-1.p`

for the symbolic links, with `NNN`

running from `001`

to the number of problems in the respective
division or category.
The problems are specified to the ATP systems using the symbolic link
names.
In the demonstration division the same problems are used as for the competition divisions, with the same tptp2X transformations applied. However, the original file names can be retained.

In the demonstration division, each entrant can choose to use either a CPU or a wall clock time limit, whose value is the CPU time limit of the competition divisions.

For each ATP system, for each problem, three items of data are recorded:
whether or not a solution was found,
the CPU time taken,
and whether or not a solution (proof or model) was output on `stdout`.
The systems are ranked from this performance data.
Division and class winners are announced and prizes are awarded.

The competition panel decides whether or not the systems' proofs and models are acceptable. The criteria include:

- Derivations must be complete, starting at formulae from the
problem, and ending at the conjecture (for axiomatic proofs) or a false
formula (for proofs by contradiction, including CNF refutations).
- For proofs of FOF problems by CNF refutation, the conversion from FOF to CNF must be adequately documented.

- Derivations must show only relevant inference steps.
- Inference steps must document the parent formulae, the inference rule used, and the inferred formula.
- Inference steps must be reasonably fine-grained.
- An unsatisfiable set of ground instances of clauses is acceptable for establishing the unsatisfiability of a set of clauses.
- Models must be complete, documenting the domain, function maps, and predicate maps. The domain, function maps, and predicate maps may be specified by explicit ground lists (of mappings), or by any clear, terminating algorithm.

In the assurance classes, and the EPR and UEQ divisions, the ATP systems
are not required to output solutions (proofs or models).
However, systems that do output solutions on `stdout` are
highlighted in the presentation of results.

If a system is found to be unsound during or after the competition, but before the competition report is published, and it cannot be shown that the unsoundness did not manifest itself in the competition, then the system is retrospectively disqualified. At some time after the competition, all high ranking systems in the competition divisions are tested over the entire TPTP. This provides a final check for soundness (see the section on System Properties regarding soundness checking before the competition). At some time after the competition, the proofs from the winners of the FOF and CNF division proof classes, and the models from the winner of the FNT and SAT division model class, are checked by the panel. If any of the proofs or models are unacceptable, i.e., they are significantly worse than the samples provided, then that system is retrospectively disqualified. All disqualifications are explained in the competition report.

Systems can be entered at only the division level, and can be entered into more than one division (a system that is not entered into a competition division is assumed to perform worse than the entered systems, for that type of problem - wimping out is not an option). Entering many similar versions of the same system is deprecated, and entrants may be required to limit the number of system versions that they enter. The division winners from the previous CASC are automatically entered into their divisions, to provide benchmarks against which progress can be judged.

It is assumed that each entrant has read the WWW pages related
to the competition, and has complied with the competition rules.
Non-compliance with the rules could lead to disqualification.
A "catch-all" rule is used to deal with any unforseen circumstances:
*No cheating is allowed*.
The panel is allowed to disqualify entrants due to unfairness, and to adjust
the competition rules in case of misuse.

A system description has to be provided for each ATP system entered, using this HTML schema. The system description must fit onto two pages, using 12pt times font. The schema has the following sections:

- Architecture. This section introduces the ATP system, and describes the calculus and inference rules used.
- Implementation. This section describes the implementation of the ATP system, including the programming language used, important internal data structures, and any special code libraries used.
- Strategies. This section describes the search strategies used, why they are effective, and how they are selected for given problems. Any strategy tuning that is based on specific problems' characteristics must be clearly described (and justified in light of the tuning restrictions).
- Expected competition performance. This section makes some predictions about the performance of the ATP system in each of the divisions and categories in which the system is competing.
- References.

The system description has to be emailed to the competition organizers by the system description deadline. The system descriptions, along with information regarding the competition design and procedures, form the proceedings for the competition.

For systems in the proof and model classes
representative sample solutions must be emailed to the competition organizers
before the sample solutions deadline.
Proof samples for the FOF proof class must include a proof for
`SYN075+1`.
Proof samples for the CNF proof class must include a proof for
`SYN075-1`.
Model samples for the the FNT model class must include models for
`MGT019+2` and
`SWV010+1`.
Model samples for the the SAT model class must include a model for
`MGT031-1`.
The sample solutions must illustrate the use of all inference rules.
A key must be provided if any non-obvious abbreviations for inference rules
or other information are used.

- The ATP systems have to run on a single locally provided standard UNIX
computer (the
*competition computers*). ATP systems that cannot run on the competition computers can be entered into the demonstration division. - Systems have to be fully automatic, i.e., any command line switches have to be the same for all problems.
- Systems have to be sound. At some time before the competition all the systems in the competition divisions are tested for soundness. Non-theorems are submitted to the systems in the FOF, CNF, EPR, and UEQ divisions, and theorems are submitted to the systems in the FNT, SAT and EPR divisions. Finding a proof of a non-theorem or a disproof of a theorem indicates unsoundness. If an ATP system fails the soundness testing it must be repaired by the unsoundness repair deadline or be withdrawn. The soundness testing eliminates the possibility of an ATP system simply delaying for some amount of time and then claiming to have found a solution. For systems running on computers supplied by the entrant in the demonstration division, the entrant must perform the soundness testing and report the results to the competition organizers.
- Systems do not have to be complete in any sense, including calculus, search control, implementation, or resource requirements.
- The ATP systems have to be executable by a single command line, using an absolute path name for the executable, which might not be in the current directory. The command line arguments are the absolute path name of a symbolic link as the problem file name, the time limit (if required by the entrant), and entrant specified system switches (the same for all problems). No shell features, such as input or output redirection, may be used in the command line. No assumptions may be made about the format of the problem file name.
- The ATP systems that run on the competition computers have to be interruptable
by a
`SIGXCPU`signal, so that the CPU time limit can be imposed on each solution attempt, and interruptable by a`SIGALRM`signal, so that the wall clock time limit can be imposed on each solution attempt. For systems that create multiple processes, the signal is sent first to the process at the top of the hierarchy, then one second later to all processes in the hierarchy. Any orphan processes are killed after that, using`SIGKILL`. The default action on receiving these signals is to exit (thus complying with the time limit, as required), but systems may catch the signals and exit of their own accord. If a system runs past a time limit this is noticed in the timing data, and the system is considered to have not solved that problem. - When terminating of their own accord, the ATP systems have to output a
distinguished string (specified by the entrant) on
`stdout`

indicating either that a solution has been found, or that no conclusion has been reached. The distinguished strings the problem status should use the SZS ontology, in a line starting`SZS status`. For exampleSZS status Unsatisfiable for SYN075+1

orSZS status GaveUp for SYN075+1

Only the first such string is recognized, and accepted as the system's claimed result. - When outputing proofs/models for the proof/model ranking classes, the
start and end of the proof/model must be identified by distinguished
strings (specified by the entrant).
These pairs of strings must be different for proofs and models.
The distinguished strings should specify the precise output form named
in the SZS ontology, using lines starting
`SZS output start`and`SZS output end`. For exampleSZS output start CNFRefutation for SYN075-1 ... SZS output end CNFRefutation for SYN075-1

The string specifying the problem status must be output before the start of a proof/model. - If an ATP system terminates of its own accord, it may not leave any
temporary or other output files.
If an ATP system is terminated by a
`SIGXCPU`or`SIGALRM`, it may not leave any temporary or other output files anywhere other than in`/tmp`

. Multiple copies of the ATP systems have to be executable concurrently on different machines but in the same (NFS cross mounted) directory. It is therefore necessary to avoid producing temporary files that do not have unique names, with respect to the machines and other processes. An adequate solution is a file name including the host machine name and the process id. - For practical reasons excessive output from the ATP systems is not
allowed.
A limit, dependent on the disk space available, is imposed on the amount
of
`stdout`

and`stderr`

output that can be produced. The limit is at least 10KB per problem (averaged over all problems so that it is possible to produce*some*long proofs). - The precomputation and storage of any information specifically about TPTP problems is not allowed. Strategies and strategy selection based on the characteristics of a few specific TPTP problems are not allowed, i.e., strategies and strategy selection must be general purpose and expected to extend usefully to new unseen problems. If automatic strategy learning procedures are used, the learning must ensure that sufficient generalization is obtained, and that no learning at the individual problem level is performed.
- For every problem solved, the system's solution process must be reproducible by running the system again.

`.tar.gz`

file containing
the system source code, any other files required for installation, and
a `ReadMe`

file.
The `ReadMe`

file must contain:
- Instructions for installation
- Instructions for executing the system
- Format of problem files, in the form of tptp2X format and transformation parameters.
- Command line, using
`%s`and`%d`to indicate where the problem file name and CPU time limit must appear.

- The distinguished strings output.

The installation procedure may require changing path variables, invoking
`make`

or something similar, etc, but nothing unreasonably
complicated.
All system binaries must be created in the installation process; they
cannot be delivered as part of the installation package.
The system is reinstalled onto the competition computers by the competition
organizers, following the instructions in the `ReadMe`

file.
Installation failures before the installation deadline are passed
back to the entrant
(i.e., deliver your installation package before the installation deadline
so if the installation fails you have a chance to fix it!).
If you are in doubt about your installation package or procedure, please
email the competition organizers.

For systems running on entrant supplied computers in the demonstration
division, entrants must deliver a source code package to the competition
organizers by the start of the competition.
The source code package must be a `.tar.gz`

file containing
the system source code.

After the competition all competition division systems' source code, is made publically available on the CASC WWW site. In the demonstration division, the entrant specifies whether or not the source code is placed on the CASC WWW site.

`perl`

script, provided by the competition organizers.
The jobs are queued onto the computers so that each
computer is running one job at a time.
All attempts at the Nth problems in all the divisions and
categories are started before any attempts at the (N+1)th problems.
During the competition a `perl`

script parses the systems'
outputs.
If any of an ATP system's distinguished strings are found then the CPU
time used to that point is noted.
A system has solved a problem iff it outputs its termination string
within the
CPU time limit, and a system has produced a proof/model iff it outputs its
end-of-proof/model string within the CPU time limit.
The result and timing data is used to generate an HTML file, and a WWW
browser is used to display the results.

The execution of the demonstration division systems is supervised by their entrants.

- Check: The ATP system can run on a computer that has the same
configuration as the competition computers.
The competition computers' configuration, obtained from
`uname`, is:> uname -mp -sr Linux 2.6.20-1.2307.fc5 i686 athlon

If the ATP system requires any special software, libraries, etc, which not part of a standard installation, please let the competition organizers know in the system registration. - Check: The ATP system can be run by an absolute path name for the
executable.
For example
prompt> pwd /home/tptp prompt> which MyATPSystem /home/tptp/bin/MyATPSystem prompt> /home/tptp/bin/MyATPSystem /home/tptp/TPTP/Problems/SYN/SYN075-1.p SZS status Unsatisfiable for SYN075-1

- Check: The ATP system accepts an absolute path name of a symbolic
link as the problem file name. For example
prompt> cd /home/tptp/tmp prompt> ln -s /home/tptp/TPTP/Problems/SYN/SYN075-1.p CCC001-1.p prompt> cd /home/tptp prompt> /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p SZS status Unsatisfiable for CCC001-1

- Check: The ATP system makes no assumptions about the format of the
problem file name. For example
prompt> ln -s /home/tptp/TPTP/Problems/SYN/SYN075-1.p _foo-Blah prompt> /home/tptp/bin/MyATPSystem _foo-Blah SZS status Unsatisfiable for _foo-Blah

- Check: The ATP system can run under the
`TreeLimitedRun`program. For exampleprompt> which TreeLimitedRun /home/tptp/bin/TreeLimitedRun prompt> /home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 4867 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001-1 FINAL WATCH: 147.8 CPU 150.0 WC

- Check: The ATP system's CPU time can be limited using the
`TreeLimitedRun`program. For exampleprompt> /home/tptp/bin/TreeLimitedRun -q0 10 20 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 10s TreeLimitedRun: WC time limit is 20s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- CPU time limit exceeded FINAL WATCH: 10.7 CPU 13.1 WC

- Check: The ATP system's wall clock time can be limited using the
`TreeLimitedRun`program. For exampleprompt> /home/tptp/bin/TreeLimitedRun -q0 20 10 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 20s TreeLimitedRun: WC time limit is 10s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- Alarm clock FINAL WATCH: 9.7 CPU 10.1 WC

- Check: The system outputs a distinguished string when terminating of
its own accord.
The SZS format, described in System
Properties, has been used here.
Similar checks should be made for the cases where the system gives up.
prompt> /home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001-1 FINAL WATCH: 147.8 CPU 150.0 WC

- Check: The system outputs distinguished strings at the start and end
of its solution.
The SZS format, described in System
Properties, has been used here.
prompt> /home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem -output_proof /home/tptp/tmp/CCC001-1.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001-1 SZS output start CNFRefutation for CCC001-1 ... acceptable proof/model here ... SZS output end CNFRefutation for CCC001-1 FINAL WATCH: 147.8 CPU 150.0 WC

- Check: No temporary or other files are left if the system terminates
of its own accord, and no temporary or other files are left anywhere
other than in
`/tmp`

if the system is terminated by a`SIGXCPU`or`SIGALRM`. Check in the current directory, the ATP system's directory, the directory where the problem's symbolic link is located, and the directory where the actual problem file is located.prompt> pwd /home/tptp prompt> /home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 13526 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001-1 FINAL WATCH: 147.8 CPU 150.0 WC prompt> ls /home/tptp ... no temporary or other files left here ... prompt> ls /home/tptp/bin ... no temporary or other files left here ... prompt> ls /home/tptp/tmp ... no temporary or other files left here ... prompt> ls /home/tptp/TPTP/Problems/GRP ... no temporary or other files left here ... prompt> ls /tmp ... no temporary or other files left here by decent systems ...

- Check: Multiple concurrent executions do not clash. For example
prompt> (/home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p) & (/home/tptp/bin/TreeLimitedRun -q0 200 400 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p) TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 5827 TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: ---------------------------------------------------------- TreeLimitedRun: /home/tptp/bin/MyATPSystem TreeLimitedRun: CPU time limit is 200s TreeLimitedRun: WC time limit is 400s TreeLimitedRun: PID is 5829 TreeLimitedRun: ---------------------------------------------------------- SZS status Unsatisfiable for CCC001-1 FINAL WATCH: 147.8 CPU 150.0 WC SZS status Unsatisfiable for CCC001-1 FINAL WATCH: 147.8 CPU 150.0 WC