Scientific Data & Computing Center

[Tony Wong <tony AT bnl.gov>]

To all HPC users:

On Thursday March 30 at around 11:50AM, a user accidentally issued a 
recursive delete ("rm -rf /") as a non-privileged user starting at the top 
(root) of the BNL Institutional Cluster file system. The recursive delete 
was nitiated by the issuance of “make clean” on an misconfigured makefile 
target. The recursive delete was noticed by SDCC staff at about 12:10PM 
and was terminated. However, during its run time, the recursive delete may 
have traversed parts of the three GPFS file systems that are mounted on 
the IC cluster.

1. /gpfshome01/u – SDCC home directories
2. /hpcgpfs01 – IC GPFS data file system
3. /cfngpfs01 – CFN GPFS file system

Fortunately, the default permissions on directories and files in these 
file systems would have prevented the deletion of any files by this errant 
recursive delete process; however, the delete process was capable of 
deleting any directory or file if the “other” permission bits were set to 
“w” for these object.

If you changed any of your files or directory permissions to be writable 
by “other” and suspect that it might have been deleted by this errant 
process, then you should check the “snapshot” directory for a possible 
“backup” copy of the file or directory. The snapshot directories are as 
follows:

1. /gpfshome01/u/.snapshots
2. /hpcgpfs01/.snapshots
3. /cfngpfs01/.snapshots

If the a deleted file or directory is found in the snapshot directory, you 
can use “cp” to copy the file/directory back to its appropriate location. 
Snapshots are made daily and any given snapshot is kept for seven days. 
Due to the limited lifetime of a snapshot, users should check for 
accidentally deleted files as soon as possible.

Tony