HPCC Best Practices
The iCER team is recommending that HPCC users run jobs on scratch rather than on the “Home” or “Research” file systems. Several users have recently reported job failure errors when their programs read or write files on these file systems. “Home” and “Research” were designed for stability, not speed; when many users are reading/writing files to them, the system becomes unstable.
The scratch file system, meanwhile, is designed for speed. It is about 20 times faster, can hold almost 2000 terabytes, and does not count against your disk quota. To run your job on scratch, copy the input files and programs to your scratch folder and do your work from there. After your work is successful, copy all files you need to save back to your “Home” or “Research” space. This last step is critical, as the scratch file system is not backed up, and files older than 45 days are deleted.
Feel free to contact iCER if you need assistance or have questions on how to run on scratch. Your cooperation is greatly appreciated.
|