Multilevel Event Building

We saw that at small message size, bandwidth is limited and we cannot transfer all event fragments from the readout units to the filter units any more. Filter units may request some fragment and apply a selection algorithm to it. The result of the algorithm is an accept or reject statement. If the event fragment does not meet the necessary criteria, it can be cleared in all readout units and does not have to be transferred to data store. If the fragment is accepted another part of the event may be requested and further algorithms may be applied. This procedure may be performed several times. It is called high level triggering. In order to raise the upper limit of processing element utilisation, computation of one level may be interleaved with data transfer of other fragments[JBD$^$91]. One way to achieve this is to use a multithreaded worker pool[Kru98].

A model of calculating the capacity of each switch port can be given. The input rate for each filter unit is the total output rate of the level 1 hardware trigger divided by the number of filter units: $l_1 = L_{GT}/n_{FU}$. Suppose the number of readout units is in the order of five hundred. Then each unit holds $1 {\rm MB} / n_{RU} \approx 2 KBytes$ of data for each event. The necessary capacity per switch port is

$$C_{port} = l_1 \cdot \sum_{i=1}^n ( \prod_{i=1}^n p_i m_i S)$$ (8)

where $n$ is the total number of filter algorithms that are applied. $p_i$ gives the event input rate for each processing level in percent of the total rate, $m_i$ is the number of messages needed to get the event fragments for this stage and $S$ is the average size of the fragment in one readout unit. To illustrate the switch capacity needs, let us make the following assumptions: The first processing stage requires information about the level 1 decision and the calorimeters (10 plus 15 RUs). We require the algorithms to accept only 25% of the rate. The next processing stage shall receive the remaining parts of the event: Preshower, Tracker and Pixel detector information. This last processing level may then apply algorithms that reduce the rate down to the cluster output rate. This does not affect the communication link capacity anymore.

$$\begin{eqnarray*} C_{port} = (200 {\rm ~Hz} \cdot 1 \cdot 35 \cdot 2 {\rm ~KB}) ... ...\rm ~KB}) = \\ = 57.6 {\rm ~MBytes/sec} ( = 0.48 {\rm ~Gb/sec}) \end{eqnarray*}$$

From figure 11 we can derive that at a message size of 2 KBytes the average throughput is 30 MBytes/sec. The theoretical maximum is 64 MBytes/sec. It seems quite reasonable that with higher bus speed and wider buses this performance can be reached. This scheme is then feasible for event building. If the switch port is not fully saturated, a multibus-multiprocessor machine could be used to act as filter unit. Then at least two independent filter units can be combined in one physical crate. The switch port can be fully loaded and the number of necessary ports is divided by two. The calculation outlined above has been performed under conservative assumptions. We could also introduce an additional trigger level. The first software level shall now reduce the rate only to 50%. The second one cuts to a quarter of the data, but uses only the preshower (a calorimeter with position measurement capability[ea96]). Then we get the following result:

$$\begin{eqnarray*} \begin{array}{lcl} C_{port} & = & (200 {\rm ~Hz} \cdot 1 \cdot... ...approx & 43 {\rm ~MBytes/sec} ( = 0.36 {\rm ~Gb/sec}) \end{array}\end{eqnarray*}$$

Just by doubling the bus capacity of the evaluated processing hardware, the required 43 MBytes/sec could be transferred with the proposed network technology at a message size of 2 KBytes. Control messages for requesting the fragments are only several tens of bytes large. They can either be routed through the switching fabric (modern networks like Myrinet provide the full bandwidth nonblocking for two directions[GCO$^$99b]) or over a separate control network. In any case it must be assured that no two filter units request the same event from the readout unit cluster. This is done by sending the requests through an event manager (see pzd174 in figure 14). This component associates data for each event coming from the very front end detectors with an identifier (pzd172 in figure 14). This information is provided to the readout units and the filter units. The filter units (pzd173 in figure 14) may now communicate directly with the readout units using this handle. If an event is cleared, either upon reject or because it has been stored, the event identifier is returned to the event manager. The communication could as well be done without an event manager. However management overhead is increased: (i) upon event clearance the filter units have to broadcast clear messages to all readout units. With an event manager this is not needed, cause such a device knows at any point in time which events are currently being processed and which ones will not be touched anymore. It can therefore tell the readout units to just overwrite these events the next time it tells them to read out data from the front end devices. Achieving good performance for such communication mode in a distributed system that relies on autonomous switching networks is not an easy task, though [Kle85, pp. 1210][XGN94] (ii) The filter units have to generate an event identifier themselves. A possible algorithm is $ID = id_{unit} + N_{units} \cdot i$, where $ID$ is the event identifier, $id_{unit}$ is the number of the filter processing unit, $N_{units}$ equals the total number of CPUs in the cluster and $i$ is a running counter that is increased with each request for a new event. A wraparound of the event identifier does not introduce difficulties as it may be re-used. Finally the stored event gets a new identifier that is valid for off-line processing only. (iii) The event manager is able to resynchronise readout units in case one of them overflows. Once such a situation occurs, the remaining working readout units must flush these events from their buffers. The causes for this kind of behaviour can be manifold: A failing filter unit which does not collect the fragments or an error in the switching network are just two examples. Hence some sort of control mechanism is needed. Although the switch size can be reduced by the proposed approach, the costs remain significantly high. This drives our quest for alternatives.