Jasper back in production.
The /lustre filesystem on Jasper and Hungabee is had run out of Inodes.
This is caused by having to many (185,000,000+) small files on the filesystem.
We are currently have 90% Inodes used on Lustre.
Please clean up any files that you do not need, specifically large numbers
of small files. Please archive and tar your data that should go to longterm
storage and move it out off the Jasper cluster.
Running jobs may have been lost. We also stopped scheduling new jobs for about 24
hours to run until we were sure new jobs would not die from lack of
As of 9:00 AM Thursday October 27 we restarted scheduling jobs to run on Jasper.