ASCI RED

The ASCI project has been founded in 1996 in order to establish nationwide supercomputer installations for simulating nuclear weapon testing. In order to achieve this goal, so far not available processing and data communication equipment had to be built. As a result the first machine, capable of performing 1.8 TeraFlops per second has been installed by Intel at Sandia Labs, US[WSB$^$97,Int]. The original set up comprises 4576 computing nodes equipped with 200 MHz dual Pentium Pro CPUs, i.e. of-the-shelf workstations, 36 service nodes that interface to the outside world, 39 disk I/O nodes that alleviate the processing nodes from costly I/O operations, and two boot nodes. Communication is split into two parts: the machine sends and receives information to and from the outside world using Ethernet and ATM over 12 special purpose nodes. Internal node-to-node bandwidth is 800 MBytes/sec per mesh router chip[MH97]. Each chip has 6 ports. The mesh is organised in multiple planes, so that each router cheap has two ports in x and y directions and two in z directions two connect to the neighbouring planes. The router chips and network interfaces are custom built by Intel. The mesh size is $38 \times 32 \times 2$ (see also figure 18. Although the throughput number is quite impressive, we have to consider the fact that there exists no $n$ to $n$ communication between all processors. Only neighbouring processors can communicate with each other. This ideal for massive parallel applications, such as the simulation of matter developing in space over time. Neighbouring space volumes have to exchange data frequently in order to update their states. the ASCI Red is build as a distributed memory multicomputer, thus making it necessary to use message passing for achieving this task. The communication software used is MPI with a relatively low latency of 68$\mu$sec. For distributed data acquisition the degree is connectivity is too low. The Option Red supercomputer has recently been upgraded and is currently the fastest computer in the world, achieving roughly 2 TeraOpsA.3.

The operating system called Cougar (a commercial version of Puma, made by Intel, the software originates from the Intel Paragon project) is a light weight kernel that supports only a subset of the functions that can be found in a full UNIX like operating system. This can be done, because the computing nodes are not intended for interactive use. Also file access is done by through special purpose libraries and through the I/O nodes. The interactive nodes run an operating system called T/OS (TeraFlop Operating System).

The installation of such a system almost 3 years ago teaches us, that interconnection technology as we need it is available, but that the commodity workstations are not capable of exploiting it. Software is way too slow to make good use of the hardware and researchers have to invent complicated topologies, including special purpose I/O nodes to generate a balance between communication and computation. It is obvious that there is a bottleneck in between that has to be removed.