Skip to main content »

Trinity College Dublin

Bluegene

Table of Contents

1. Introduction to TCHPC’s Bluegene/P Capability System
2. Overview

    2.1. Getting Help
    2.2. Hardware
    2.3. Software

3. Using the queuing system

    3.1. Batch jobs
    3.2. Interactive usage
    3.3. Commands to query SLURM for useful information

4. Disclaimer


---------------------------------------------------------------------

1. Introduction to TCHPC’s Bluegene/P Capability System

---------------------------------------------------------------------

This service is provided via 1 rack of Bluegene/P, the service is
primarly provided to TCD and Irish researchers on a best effort
basis.


---------------------------------------------------------------------

2. Overview

---------------------------------------------------------------------

The system mostly contains a stock installation provided by IBM with
the exception of the scheduling system, any online documentation and
redbooks from IBM should in general hold true for the TCHPC Bluegene/
P machine.


2.1. Getting Help

------------

If you require assistance on the Bluegene/P system please email
ops@tchpc.tcd.ie with your query.


2.2. Hardware

------------

2.2.1. Front end node (bg-fe.tchpc.tcd.ie)

The front end node is a 16core Power5+ machine running at 1.8GHz and
64GB RAM. It’s purpose is for development, pre/post processing of
data and job submission.

Users must login to this machine via ssh.

$ ssh bg-fe.tchpc.tcd.ie

2.2.2. Storage

The storage system is an IBM SAN with 32tb allocated to /gpfs/
projects/ and 1tb allocated to /gpfs/homes/. Each user has a quota of
50gb in the home partition, users are recommended to run their jobs
in the projects filesystem.

The two GPFS filesystems are accessible from the front-end node and
the Bluegene/P compute nodes.

2.2.3. Bluegene/P

The Bluegene/P machine itself contains 1024 compute nodes each with
2gb of memory, the nodes are divided up into an upper and lower
midplane. Each midplane contains 512 compute nodes, this is further
divided up into 16 compute cards and each compute card contains 32
nodes.

The geometry of our 1 rack system is 1x1x2 and for a half a rack it
is 1x1x1. This geometry information can be given to the scheduler for
block creation and allocation.

Although there are 1024 compute nodes it is possible to run the
compute nodes in three modes of operation: SMP, Dual and VirtualNode.

  * SMP: single mpi task with four threads
  * Dual: two mpi tasks with two threads
  * VirtualNode: four mpi tasks


2.3. Software

------------

The Bluegene/P software stack is V1R4M2_200_2010-100508P, this
software stack provides the IBM xlc/xlf compilers, ESSL and MPI
libraries.

2.3.1. MPI wrappers

  * mpixlc
  * mpixlcxx
  * mpixlf2003
  * mpixlf77
  * mpixlf90
  * mpixlf95

2.3.2. Bluegene Compilers

  * bgc89
  * bgc99
  * bgcc
  * bgf2003
  * bgf77
  * bgf90
  * bgf95
  * bgfort77
  * bgxlc
  * bgxlc++
  * bgxlf95
  * bgxlC
  * bgxlf2003
  * bgxlf
  * bgxlf90

2.3.3. IBM Math Libraries

The Math libraries are located at

/opt/ibmmath

ESSL can be found at

/opt/ibmmath/essl


---------------------------------------------------------------------

3. Using the queuing system

---------------------------------------------------------------------

Note

SLURM is only resposible for managing allocations, once SLURM has
allocated you a block it is up to the user to launch their program(s)
with mpirun in the correct way. Users should refer to the mpirun help
by simply doing the following

$ mpirun -h


3.1. Batch jobs

------------

Running batch jobs is done by writing simple shell scripts which are
submitted to the scheduling system. We are currently using SLURM
(which is also used on all other TCHPC managed clusters). Users must
specify at minimum how many nodes (the small unit size is 32nodes)
and for how long the job should run for.

A sample script for a 32 node debug job running in SMP mode. 

#!/bin/bash
#SBATCH -t 01:00:00
#SBATCH -N 32

mpirun ./MyCoolCode.x

When asking for an allocation that is not a multiple of 512 compute
nodes the system will not run optimally. Users are encouraged to
develop and run codes that scale out to larger job sizes. The
following example will request 512 nodes from the queuing system.

A sample script for a one mid-plane debug job running in SMP mode. 

#!/bin/bash
#SBATCH -t 01:00:00
#SBATCH -g 1x1x1

mpirun ./MyCoolCode.x

To request the full system one can do the following

A sample script for one rack of BG/P debug job running in SMP mode. 

#!/bin/bash
#SBATCH -t 01:00:00
#SBATCH -g 1x1x2

mpirun ./MyCoolCode.x


3.2. Interactive usage

------------

Users can alternatively do an salloc to interactively run jobs for
development purposes

To request the full system

$ salloc -g 1x1x2

or to request half the system

$ salloc -g 1x1x1


3.3. Commands to query SLURM for useful information

------------

Have sinfo show block information

$ sinfo -b

Have smap show block information

$ smap -Db

Show what’s in the queue

$ squeue

Show what resources are available (without block information)

$ sinfo


---------------------------------------------------------------------

4. Disclaimer

---------------------------------------------------------------------

The Bluegene/P capability service is provide on a best effort basis
due to limited time and funding.

Warning

User data is not backed up on the file systems, it is up to the users
to backup their own data.

Last updated 07 Sep 2011Contact TCHPC: info | support.