Towards a Protocol for Benchmark Selection in IPC

The planning competition has traditionally played an
important role in motivating research and advances in
Planning & Scheduling techniques. Despite its pivotal
role in the planning community, some aspects of the
competition have not been engineered yet. This is the
case for the protocol for selecting benchmark instances.
Benchmarks are of critical importance, since they can
significantly affect competition results.
In this paper we describe desirable properties of a selection
protocol, discuss methods exploited in past SAT
and planning competitions, and identify challenges that
organisers of future competitions have to address in order
to improve reliability and usefulness of the insights
gained by looking at competitions’ results.