Skip to main content »

Trinity College Dublin

I/O Tuning

This page list the information about how to optimise I/O operation on code and file system level (mostly taken from http://www2.fz-juelich.de/jsc/juropa/usage/i_o-tuning/).

I/O Tuning Guidelines

In order to optimize the I/O performance of programs on TCHPC clusters and avoid having an impact on the overall system performance, please obey the following rules when writing/porting your software on/to TCHPC clusters:

  • Write your data in large blocks instead of small portions to disk.
  • Use buffered I/O whenever possible and avoid to flush your output frequently.
  • Avoid keeping a large number of files open simultaneously. For example, rather than writing one output file per iteration, write them all to a single file prefixing each line with the iteration number, and then parse the single output file afterwards.

See below for specific rules for different programming languages.

FORTRAN

When using the ifortcompiler, make sure that the environment variable

FORT_BUFFERED = true

is set (this should be set in your relevant Makefile, arch.make or other build configuration files).

Alternatively, you can use the compiler option '-assume buffered_io' to switch on I/O buffering for FORTRAN programs.

The ifort compiler offers the non-standard parameters BUFFERED, BLOCKSIZE and BUFFERCOUNT for the 'open' statement in order to customize the buffering of the output. Here are typical values recommended for TCHPC. For details, please see the Intel manual

BUFFERED=YES enables buffering of WRITE commands
BLOCKSIZE=1048576 sets the size of the buffer to 1 MB
BUFFERCOUNT=n uses 'n' buffers; in general n=1 is appropriate

For example:

open(unit=20,file=test.out,status='new',form='formatted',
     blocksize=1048576, buffered='yes', buffercount=1)

C

Do not use 'fflush' (except at the end of your output/program).

C++

Do not use 'flush' (except at the end of your output/program).

Do not use 'endl', because this performs an additional flush each time called. If you need a line break, please use '\n' instead.


Last updated 01 Jun 2011Contact TCHPC: info | support.