iitac : The IITAC cluster

The IITAC project has funded one large, 64bit, compute cluster (called iitac) for large scale parallel jobs and one medium, 32bit, compute cluster (called moloch) for serial jobs.

Brief Overview

Overview

The components of the IITAC cluster werepurchased from IBM over the Summer of 2005 and assembled by technical staff at the Trinity Centre for High Performance Computing. It is a large scale facility consisting of 712 AMD Opteron 2.4GHz processors with a theoretical peak performance of 3.4 TFlops.

Operating system The nodes all run Debian and was installed using the FAI installer Login There are two e326 login nodes in the cluster, login01.tchpc.tcd.ie and login02.tchpc.tcd.ie. These nodes are also used for compiling and job submission. Compute There are 346 e326 compute nodes each containing two 2.4GHz AMD 64bit processors, 4GB RAM and a 80GB SATA scratch disk. Each node has dual onboard gigabit ethernet and a PCI-X infiniband card.

iitac

Interconnect The nodes are connected to two Voltaire 288 port InfiniBand (IB) switches - providing a dedicate low latency high bandwidth interconnect for parallel jobs. The Voltaire technology allows for inter node communication speeds of up to 10Gb. The IB network is also used by our parallel file system. Each node is also connected to a Force10 Gigabit switch - providing a high speed dedicated management network.

One of our Voltaire IB switches

Disk storage
IBM's General Parallel File System (GPFS) runs across all nodes on the cluster. The cluster is connected to our S2A9500 storage system from Data Direct Networks. Twelve e326 nodes are connected via two fibre channel switches to the DDN S2A9550. These twelve nodes operate as GPFS I/O servers. All other nodes connect to these twelve nodes as GPFS clients via the two Voltaire IB switches. 70TB of high performance storage, in a RAID 6 configuration, for home directories.

Node Specifications

Last updated: Nov 05 2008