From June 2011, Research IT (TCHPC) have implemented a new set of data management polices. This allows us to cleanly remove old data from the system.
All existing and new users must read and comply with the policies below, and in particular should take note of all data deletion policies for user and group directories.
TCHPC data storage facilities are not available for long-term data storage - they are made available for the lifetime of user and group projects, and data will be deleted upon user and project expiration.
Current Filesystem Structure
- There are three storage pools available to the main clusters (lonsdale, parsons, kelvin):
- To view your quota, use the
myquotautility. The reports both user quotas and project quotas. See this link for more details on Quotas.
Note: As of the end of 2013, the support contract and warranty on the DDN 9900 SAN array (the storage array on which the
/gscratch filesystem resides) has expired. There is currently no funding source available to upgrade or renew the system, so TCHPC will continue to operate the system on a best-effort basis.
User Home Directories
User home directories are stored on the global
In order to enable more efficient data management processes and provide disaster recovery and snapshot services:
/homeis restricted to 5TB, with 50GB allocated per user (enforced by quotas)
- Users should use
/hometo store data input files and submission scripts, results files, and where appropriate source code
- On a monthly basis, all directories and files in
/homeare backed up for disaster recovery purposes
- Upon user account expiry, there will be a grace period of 1 month for users to copy files from their home directory as required before their home directory and all its files will be deleted. The user will be sent automated emails warning about their account expiry in advance.
Group Project Directories
Shared group project files are stored on the global
- All projects are allocated cpu and storage resources
- Please ensure that job outputs are be written to a project directory and not to home directories
- Principal Investigators and Research Staff/Postgraduates can apply for projects. These are organised per PI.
- The directory structure for projects is:
- Group quotas are enforced per PI, with additional storage granted/removed per project. That is, when a project is created, the PI group quota will be increased by the set amount. Similarly, when a project expires, the PI group quota will be reduced by the allocated amount.
- Any files which should be shared between group/project members must be placed in a project directory.
- Upon project expiry, there will be a grace period of 1 month for users to copy files from the project directory as required before the files will be deleted. All users in the project group (including the project PI) will be sent automated emails warning about this in advance.
myquotatool can be used to report their quota usage (on home directory, and all project directories).
Note on the /gscratch partition
/gscratchfilesystem is also available for transient files.
- Files older than 60 days will be deleted from
/gscratchautomatically without exception.
- Please note that this data should not be used a primary storage location for important data, as the SAN controller hardware for /gscratch is now out of warranty.
- Thus, /gscratch should be considered to be at risk.
- In order to make
/gscratchas accessible as possible the file system has global read and write permissions for everyone, i.e. any user can read and or delete any data there. Sensitive data should not be saved to
Users must also comply with the following policies and guidelines
- All users are expected to follow the guidelines provided in the TCD document Good Research Practice, the TCD Data Access and Data Management Policy
- New accounts are created with ownership and access permissions (of directories and files) restricted by default to the account holder. These permissions can be changed by the user. Each user is responsible for ensuring that their data is kept secure. For more information on ownership and permissions, please consult the notes in our Introduction to Linux course or contact our support team
- Data is not backed up
- Data in project directories is available for a discrete project period and will be deleted once the project expires
- Data is home directories will be backed up for disaster recovery purposes (only) on a monthly basis. Accidentally deleted data will not be retrieved.
- Unless otherwise explicitly agreed in writing, services provided by the centre should be used for research purposes only, they may not be used for clinical research or for commercial gain
- Unless otherwise explicitly agreed in writing, services provided by the Centre are not suitable for the storage or processing of clinical, medical, confidential, personal, commercial and/or other types of sensitive information. Users should be mindful of their obligations under the Data Protection acts and (in the case of TCD users) the Good Research Policy guidelines.
Procedure for retrieving files/data for users who have left TCD
If you had files stored on TCHPC servers and have now left Trinity but wish to access those files (e.g. if you have moved to another institute, but are still collaborating with groups within Trinity, then further to requirements from the College's Good Research Practice document, we will need you to arrange:
- Signoff from your former Head of School & Research Institute before we can process this request
- An undertaking from all parties that the data can cleanly be transferred off campus and that the data does not contain medical, confidential, commercially sensitive, sensitive, or personal data or records of any form or description.