Design and Organization

This document contains information about the:

• Changes
• Divisions
  • Competition Divisions
  • Demonstration Division
• Infrastructure
  • Computers
  • Problems
  • Time Limits
• System Evaluation
• System Entry
  • System Description
  • Sample Solutions
• System Requirements
  • System Properties
  • System Delivery
  • System Execution
  • System Checks

Disclaimer

A Tense Note

• The FNT division has returned from hiatus.

• Proofs may not have parents listed that are not used in the inference.

• The maximal number of parents used in an inference step will be recorded and reported.

• Proofs should (but it won't be enforced) graft in subproofs from external systems.

The THF division: Typed (monomorphic) Higher-order Form theorems (axioms with a provable conjecture). The THF division has two problem categories:

• The TNE category: THF with No Equality. Example: NUM738^1
• The TEQ category: THF with EQuality. Example: SET171^3

The FOF division: First-Order Form theorems (axioms with a provable conjecture). The FOF division has two problem categories:

• The FNE category: FOF with No Equality. Example: COM003+1
• The FEQ category: FOF with EQuality. Example: SEU147+3

The FNT division: First-order form Non-Theorems (axioms with a countersatisfiable conjecture, and satisfiable axiom sets). The FNT division has two problem categories:

• The FNN category: FNT with No equality. Example: KRS173+1
• The FNQ category: FNT with eQuality. Example: MGT033+2

The UEQ division: Unit EQuality clause normal form unsatisfiable clause sets. Example: RNG026-7

ATP systems that cannot run in the competition divisions for any reason (e.g., the system requires special hardware, or the entrant is an organizer) can be entered into the demonstration division. The demonstration division uses the same problems as the competition divisions, and the entry specifies which competition divisions' problems are to be used. Demonstration division systems can run on the competition computers, or on computers supplied by the entrant. The demonstration division results are presented along with the competition divisions' results, but might not be comparable with those results. The systems are not ranked.

The entrants in the various divisions are listd in the entrants table. The division winners of the previous CASC and the Prover9 1109a system are automatically entered into the demonstration division, to provide benchmarks against which progress can be judged.

Computers

• Two octa-core Intel(R) Xeon(R) E5-2620 v4 @ 2.10GHz CPUs, without hyperthreading.
• 256GiB memory
• The Ubuntu 24.04.3 LTS operating system, with Linux kernel 6.8.0-71-generic

One ATP system runs on one CPU at a time. StarExec uses Linux's sched_setaffinity to restrict each system run to a single CPU, and setrlimit to limit memory use to 128 GiB. This separation avoids contention that might affect performance measurements. Systems can use all the cores on the CPU, which can be advantageous in divisions where a wall clock time limit is used. StarExec copies the systems and problems to the compute nodes before starting execution so that there are no network delays.

The StarExec computers used for CASC are the same as are publicly available to the TPTP community, which allows system developers to test and tune their systems in exactly the same environment as is used for the competition.

Problems for the TPTP-based Divisions

• The TPTP tags problems that are designed specifically to be suited or ill-suited to some ATP system, calculus, or control strategy as biased. They are excluded from the competition.
• The problems must be syntactically non-propositional.
• The TPTP uses system performance data in the Thousands of Solutions from Theorem Provers (TSTP) solution library to compute problem difficulty ratings in the range 0.00 (easy) to 1.00 (unsolved). The upper bound of 0.99 excludes problems that cannot be solved by any system and thus don't differentiate between systems, the lower bound of 0.21 was chosen (many years ago, and it has worked successfully) to exclude problems that would be solved by most of the systems and thus don't differentiate between systems. Problems of lesser and greater ratings might also be made eligible in some divisions or problem categories if there are not enough problems with ratings in that range.

  ATP systems can be submitted before the competition so that their performance data is used in computing the problem ratings - problems that are newly solved get a rating less than 1.00 and thus become eligible (until the rating drops below 0.21). The rating calculation also uses performance data from ATP systems that are not entered into the competition, which can produce ratings that make some problems eligible for selection but easy or unsolvable for the systems in the competition. Using problems that are solved by all or none of the competition systems does not affect the competition rankings, has the benefit of placing the systems' performances in the context of the state-of-the-art in ATP, but does reduce the differentiation between the systems in the competition.

• stripping out all comment lines, including the problem header
• randomly reordering the formulae/clauses (include directives are left before formulae, type declarations and definitions are left before the symbols' uses)
• randomly swapping the arguments of associative connectives, and randomly reversing implications
• randomly reversing equalities

• The selection is constrained so that no problem category contains an excessive number of very similar problems, according to the "very similar problems" (VSP) lists distributed with the TPTP. For each problem category in each division, if the category is going to use N problems and there are L VSP lists that have an intersection of at least N/(L + 1) with the eligible problems for the category, then maximally N/(L + 1) problems are taken from each VSP list.
• In order to combat excessive tuning towards problems that are in the preceding TPTP version, the selection is biased to select problems that are new in the TPTP version used, until 50% of the problems in each problem category have been selected or there are no more new problems to select, after which random selection from old and new problems continues. The number of new problems used depends on how many new problems are eligible and the limitation on very similar problems.
• Problems with rating 0.21 to 0.99 are selected before problems with other ratings.

Time Limits

• whether or not the problem was solved
• whether or not a solution (proof or model) was output
• the CPU and wall clock times taken (as measured by StarExec's runsolver utility, and prepended to each line of the system's stdout)
• the maximal number of parents in any inference

The systems are ranked in the competition divisions according to the number of problems solved with an acceptable solution output. Ties are broken according to the average time taken over problems solved. Trophies are awarded to the competition divisions' winners. A system that is not entered into a division is assumed to perform worse than the entered systems, for that type of problem - wimping out is not an option. In the demonstration division the systems are not ranked, and no trophies are awarded.

The competition panel decides whether or not the systems' solutions are "acceptable". The criteria include:

• Proofs should not use a TPTP language that is more expressive than that of the problem.
• Proofs must be in TPTP format. This is checked using TPTP4X.
  Entrants can do this in SystemB4TSTP:
  • Provide your proof as a file or formulae.
  • The default action in SystemB4TSTP is a syntax check, so just click "ProcessProblem". If the proof is dubious or faulty, you'll get an WARNING/ERROR messages.
• Proofs must be structurally correct:
  • Annotated formulae in a proof must be uniquely named.
  • Proofs must be acyclic.
  • Proofs must have formulae from the problem as leaves, and end at the conjecture (for axiomatic proofs) or $false formulae (for proofs by contradiction, e.g., CNF refutations).
  • Proofs that negate the conjecture must correctly annotate the step as status(cth) and have a a single parent with the role conjecture.
  • Proofs must show only relevant inference steps.
  • Inference steps may not list parents that are not used in the inference.
  • Proofs that make assumptions must propagate and eventually discharge the assumptions.
  Proof structure will be checked using GDV.
  Entrants can do this in SystemOnTSTP:
  • Provide your proof as a file or formulae.
  • Select "GDV" as the system.
  • To check only the structure of the derivation as requred for CASC-30, change GDV's "Command" to run_GDV %s 30 -d -u (i.e., add -d -u on the end).
  • Click "ProcessSolution". If the proof is dubious or faulty, you'll get an WARNING/ERROR messages.
• For solutions that use translations from one form to another, e.g., translation of FOF problems to CNF, the translations must be adequately documented.
• Inference steps must be reasonably fine-grained. The maximal number of parents in any inference will be considered in this regard.
• An unsatisfiable set of ground instances of clauses is acceptable for establishing the unsatisfiability of a set of clauses.
• Proofs should (but it won't be enforced) graft in subproofs from external systems.
• Models must be complete, documenting the domain, function maps, and predicate maps. The domain, function maps, and predicate maps may be specified by explicit ground lists (of mappings), or by any clear, terminating algorithm. Saturations are acceptable if their models are a subset of the models of the problem formulae.

• The state-of-the-art contribution (SotAC) quantifies the unique abilities of each system (excluding the previous year's winners that are earlier versions of competing systems). For each problem solved by a system, its SotAC for the problem is the fraction of systems that do not solve the problem, and a system's overall SotAC is the average over the problems it solves but that are not solved by all the systems.
• The core usage measures the extent to which the systems take advantage of multiple cores. It is the average of the ratios of CPU time used to wall clock time used, over the problems solved.
• The efficiency measure balances the number of problems solved with the time taken. It is the average solution rate over the problems solved (the solution rate for one problem is the reciprocal of the time taken to solve it), multiplied by the fraction of problems solved. Efficiency is computed for both CPU time and wall clock time, to measure how efficiently the systems use one core and multiple cores respectively.
• The proof granularity reports the maximal number of parents in any inference, as a measure of proof granularity and readability.

At some time after the competition all high ranking systems in the competition divisions are tested over the entire TPTP. This provides a final check for soundness (see the section on system properties regarding soundness checking before the competition). If a system is found to be unsound during or after the competition, but before the competition report is published, and it cannot be shown that the unsoundness did not manifest itself in the competition, then the system is retrospectively disqualified. At some time after the competition, the solutions from the winners are checked. Proofs are checked for structure by GDV, and models are checked by the panel. If any of the solutions are unacceptable then the victory is rescinded. All disqualifications and ranking changes are explained in the competition report.

The ATP systems are delivered to the competition organizer as StarExec .tgz installation packages, by the system delivery deadline. The installation package may contain only the components necessary for running the system (i.e., not including source code, etc.). The competition organizer installs and tests the packages on StarExec. Source code is delivered separately, under the trusting assumption that the installation package corresponds to the source code. The entrants must email a .tgz file containing the source code and any files required for building the StarExec installation package to the competition organizer by the system delivery deadline. After the competition all competition division systems' StarExec and source code packages are made publicly available on the CASC web site. This allows anyone to use the systems on StarExec, and to examine the source code. An open source license is encouraged, to allow the systems to be freely used, modified, and shared. Entrants are encouraged to make a public release of their systems ASAP after the competition, so that users can enjoy the latest capabilities. Many of the StarExec packages include statically linked binaries that provide further portability and longevity of the systems. In the demonstration division the entrant specifies whether or not the source code is placed on the site.

Execution of the ATP systems is controlled by StarExec. The ATP systems must be fully automatic - they are executed as black boxes, on one problem at a time. Any command line parameters have to be the same for all problems in each division. The ATP systems must be sound, and are tested for soundness before the competition by submitting non-theorems to the systems in the THF, FOF, and UEQ division, and theorems to the systems in the FNT division. Claiming to have found a proof of a non-theorem or a disproof of a theorem indicates unsoundness. If a system fails the soundness testing it must be repaired by the unsoundness repair deadline or be withdrawn. The execution of demonstration division systems is supervised by their entrants.

System Description

• Architecture. This section introduces the ATP system, and describes the calculus and inference rules used.
• Strategies. This section describes the search strategies used, why they are effective, and how they are selected for given problems. Any strategy tuning that is based on specific problems' characteristics must be clearly described (and justified in light of the tuning restrictions).
• Implementation. This section describes the implementation of the ATP system, including the programming language used, important internal data structures, and any special code libraries used. The availability of the system is also given here.
• Expected competition performance. This section makes some predictions about the performance of the ATP system for each of the divisions and categories in which it is competing.
• References.

The system description has to be emailed to the competition organizer by the system description deadline. The system descriptions form part of the competition proceedings.

Sample Solutions

Proof/model samples are required as follows:

• THF: SET014^4
• FOF: SEU140+2
• FNT: NLP042+1 and SWV017+1
• UEQ: BOO001-1

System Properties

Execution, Soundness, and Completeness

1. Systems must be fully automatic.
2. Systems' performances must be reproducible by running the system again.
3. Systems must be sound.
4. Systems do not have to be complete in any sense, including calculus, search control, implementation, or resource requirements.
5. All techniques used must be general purpose, and expected to extend usefully to new unseen problems. The precomputation and storage of information about individual problems that might appear in the competition, or their solutions, is not allowed. Strategies and strategy selection based on individual problems or their solutions are not allowed. If machine learning procedures are used to tune a system, the learning must ensure that sufficient generalization is obtained so that there is no specialization to individual problems. The system description must explain any such tuning or training that has been done. The competition panel may disqualify any system that is deemed to be problem specific rather than general purpose.

1. All output must be to stdout.
2. For each problem, the system must output a distinguished string indicating what solution has been found or that no conclusion has been reached. Systems must use the SZS ontology and standards for this. For example

   % SZS status Theorem for SYN075+1.p

   or

   % SZS status GaveUp for SYN075+1.p

3. When outputting a solution, the start and end of the solution must be delimited by distinguished strings. Systems must use the SZS ontology and standards for this. For example

   % SZS output start CNFRefutation for SYN075+1.p
     ...
   % SZS output end CNFRefutation for SYN075+1.p

   The string specifying the problem status must be output before the start of a solution.
4. A system has solved a problem iff it outputs its termination string within the time limit, and a system has produced a solution iff it outputs its end-of-solution string within the time limit.
5. Solutions may not have irrelevant output (e.g., from other threads running in parallel) interleaved in the solution.
6. Use of the TPTP format for proofs is required, and use of the TPTP format for interpretations is encouraged.
7. Proofs are checked for syntax and structure as explained in System Evaluation.

1. Systems that run on the competition computers must be interruptible by a SIGXCPU signal so that CPU time limits can be imposed, and interruptable by a SIGALRM signal so that wall clock time limits can be imposed. For systems that create multiple processes the signal is sent first to the process at the top of the process hierarchy, then one second later to all processes (even if they have disconnected from the process hierarchy). The default action on receiving these signals is to exit (thus complying with the time limit, as required), but systems may catch the signals and exit of their own accord. If a system runs past a time limit this is noticed in the timing data, and the system is considered to have not solved the problem.
2. If a system terminates of its own accord it may not leave any temporary or intermediate output files. If a system is terminated by a SIGXCPU or SIGALRM it may not leave any temporary or intermediate output files anywhere other than in /tmp.
3. For practical reasons excessive output from an ATP system is not allowed. A limit, dependent on the disk space available, is imposed on the amount of output that can be produced.

System Checks

• Check: You can login to StarExec Miami. If not, apply for an account in the TPTP community.

• Check: You can access the TPTP space. If not, email the competition organizer.

• Check: You can create and upload a StarExec installation package. The competition organizer has exemplar StarExec installation packages that you can use as a starting point - email the competition organizer to get one that is appropriate for your ATP system.

• Check: You can create a job and run it, and your ATP system gets the correct result. Use the SZS post processor.

• Check: Your ATP system can solve a problem that has include directives. Because of the way StarExec runs jobs, your ATP system must implement the TPTP requirement that "Include files with relative path names are expected to be found either under the directory of the current file, or if not found there then under the directory specified in the TPTP environment variable."

• Check: You can email your StarExec installation package to the competition organizer for testing.