Management of large clusters of workstations is a challenging task. This is not only caused by the noticeably increased processing power and network bandwidth[AP97] that is needed for supervision of machine pools in the range of several hundred units; it is also not a trivial task to coordinate diverse software components[HM96]. In the case of high energy physics experiments the situation becomes even more critical:
First, the systems are prototyped long before the detector is built. It is likely that the infrastructure with which we are confronted now differs from the one that we will see in the production setup. Prototype environments make heavy use of existing office clusters or spare time on supercomputers. The production system, however, will be implemented as a cluster of computers connected by a high speed short area network and an additional management network. So the framework into which programs are embedded should be adaptable to different environments.
Second, the lifetime of the experiment is considerably long. Over fifteen years software and hardware will be upgraded gradually. Again, flexibility is the main requirement for tools concerning management. They must be able to cope with changing configurations.
Third, software has to be coordinated. It has to be assured that at any time all programs in the system can be controlled. The operators must have the possibility to install and choose different versions of a program in parts of the cluster. It must be possible to divide the cluster into different processing responsibilities. Whole groups of tasks have to be steered.
Fourth, the system must exhibit a certain degree of autonomy. User interaction cannot always be expected when a problem occurs. Software that fails must be restarted automatically. For a high energy physics experiment the situation gets more difficult: the computers are not physically accessible for a period of roughly half a year once the cavern is sealed. Simple errors, such as harddisk failure or an operating system crash have to be handled remotely by the operator or autonomously by the system. The latter solution is preferred as it may be problematic to install and check over one thousand systems individually. Finally, as we have to deal with software diversity, the tools should not require special instrumentation of the existing code. The tool should offer a simple, yet flexible means to allow any software to interface with the management framework. The reason for this is not that we do not want to impose a specific structure onto the programs, but rather, that we do not always have access to the source code: Programs may be little documented. Sometimes they are written in different programming languages or do already have a specific command interface that cannot be changed easily. We state the following requirement: