TCIN Neuro HPC Cluster

Trinity College Institute of Neuroscience (TCIN) have access to a small dedicated HPC cluster with the following characteristics:

  • 1 head node with ~12TB of shared storage
  • 2 compute nodes, each with 32 CPU cores, (2 by 16 core Intel Xeon Silver 4314 CPU's @ 2.40GHz), ~250GB of RAM and ~2TB of local scratch disk for each compute node, each compute node mounts the shared storage from the head node
  • 10GbE networking to be commissioned in the future.
  • Ubuntu 20.04 LTS Operating System

To access it you must have a Research IT account, please apply for one if you don't have one.

To request access to the cluster please email rit-support@tcd.ie.

Login

To login please connect to neuro01.tchpc.tcd.ie using the usual SSH instructions. It is accessible from the College network, including the VPN. To connect to it from the internet please first login to the College VPN or relay through rsync.tchpc.tcd.ie as per our instructions.

File systems

  • /home is a stand alone file system ~12TB in size that is shared from the head node to all compute nodes. It is separate from the file systems on any of our other clusters. You will have to copy data onto the neuro cluster.
  • No data on the cluster in the cluster is backed up. Including in the /home file system which is is not backed up.
  • Each node has a local /tmp scratch disk that is only accessible on that node, it is not shared with others, and while you have been allocated to that node, you will not be able to access data in those disks once your allocated jobs have finished.
  • The siemens-dicom, siemens-raw, tcin-bruker and tcin-philips MRI archive shares are available on the head node in the /mnt directory, e.g. /mnt/siemens-dicom. They are not accessible from the compute nodes. If you want to process data from one of those shares you will have to first copy it from the share to /home on the head node, process the data in the compute queue, then copy any results back.

Software

Software is installed with our usual modules system. You can view the available software with module av and an example to load a module is: module load apps afni. You may need to load the apps modules list first to see all available packages with the module load apps command.

Running jobs

Running jobs must be done via the Slurm scheduler.

The scheduler is configured to share jobs on the same node, i.e. multiple jobs can run on the same node simultaneously. This may lead to contention issues where jobs interfere with each other, please let us know if you have issues with that.

Batch job example

#!/bin/bash
#SBATCH -n 12
#SBATCH --mem=96GB
module load apps afni ANTs eigen freesurfer fsl
echo "Starting"
./exe.x
matlab -nojvm -nodesktop -nosplash simple.m

Interactive allocation

salloc -n 12 --mem=96GB

There is an issue with the modules system not working on interactive allocations. Modules work in batch jobs. But if you try to use the modules in an interactive session it errors as follows:

user@neuro01:~$ salloc -N 1
salloc: Granted job allocation 1936
salloc: Waiting for resource configuration
salloc: Nodes neuro-n02 are ready for job
user@neuro-n02:~$ module av
module: command not found
user@neuro-n02:~$ module load matlab
module: command not found

The workaround is to issue the following command and the modules will be available.

source /etc/profile.d/modules.sh

See the HPC clusters usage documenation for furhter instructions.

Using VNC on the neuro HPC cluster

Notes:

  • VNC is only available on the head node, it is not possible to get VNC to any of the compute nodes.
  • VNC is only available over SSH forwarding, it is not possible to connect a VNC viewer client on your computer to neuro01 for security reasons.
  • These notes assume you are using linux of MacOS.

Pre-requisites, necessary on neuro01 before being able to get VNC to work:

  1. Determine your user id, you will need to reference this later: id -u
  2. Set a VNC password: tigervncpasswd

Option 1 - Steps to start a VNC session, applies to Linux and MacOS.

Log into neuro with X and SSH forwarding on:

> ssh -X -L 5901:localhost:ID username@neuro01.tchpc.tcd.ie
  • Replace ID with the output of the id -u command above.
  • Replace username with your user name.

Start a desktop session you can connect to:

> tigervncserver -xstartup /usr/bin/mate-session

Note the value of the Use xtigervncviewer ... output from the tigervncserver ... command above, it will give you the command to use to connect to the VNC session.

Example command to connect to the session:

> xtigervncviewer -SecurityTypes VncAuth -passwd ~/.vnc/passwd :1

Note: the :1 value may not be the correct one, adjust that for the number provided from the output of the tigervncserver command or use tigervncserver -list to list the sessions running.

Option 2 - Steps to start a VNC session, apply to MacOS only.

Log into neuro with SSH.

ssh userame@neuro01.tchpc.tcd.ie

Replace username with your user name.

Start a new vncserver session on neuro01 but with a desktop environment launcher:

vncserver -xstartup /usr/bin/mate-session

It should output some details, e.g.

New Xtigervnc server 'neuro01:2 (username)' on port 5901 for display :2.
Use xtigervncviewer -SecurityTypes VncAuth -passwd /home/users/username/.vnc/passwd :2 to connect to the VNC server.

Take note of the number after port in the first line, 5901 above but may be different.

You can also check it with vncserver --list command, it should be the value below the RFB PORT # entry in that column.

On a new terminal session on your mac run the following.

ssh -N -L XXXX:localhost:XXXX emoore5@neuro01.tchpc.tcd.ie

Replacing XXXX with the port number noted earlier.

On your Mac open Finder. Press ⌘K and then enter the following in the dialog box.

vnc://localhost:5901

Once you are finished stop your vnc sessions:

  1. Disconnect from your running session.
  2. Use tigervncserver -list to list the id's of your running session(s)
  3. Use tigervncserver -kill :1 to kill the session, replacing :1 with the correct value from 2 above.