File Storage:

Home Directory

The /home file system quota is 5 GB. Files can be stored on /home permanently, which makes it an ideal place for your source code and executables. The /home file system is meant for interactive use such as editing and active code development. Do not use /home for batch job I/O.
 

Work (Scratch) Directory

The /work volume meant for the input and output of executing batch jobs and not for long term storage. We expect files to be copied to other locations or deleted in a timely manner, usually within 30-120 days. For performance reasons on all volumes, our policy is to limit the number of files per directory to around 10,000 and total files to about 500,000.

There is no quota is enforced on /work but if it becomes over utilized we will enforce a 30 days purging policy, which means that any files that have not been accessed for the last 30 days will be permanently deleted. An email message will be sent out weekly to users targeted for a purge informing them of their /work utilization.

Please do not try to circumvent the removal process by date changing methods. We expect most files over 30 days old to disappear. If you try to circumvent the purge process, this may lead to access restrictions to the /work volume or the cluster.

Please note that the /work volume is not unlimited. Please limit your usage rate to a reasonable amount. When the utilization of /work is over 80%, a 14 day purge may be performed on users using more than 2 TB or having more than 500,000 files. Should disk space become critically low, all files not accessed in 14 days will be purged or even more drastic measures if needed. Users using the largest portions of the /work volume will be contacted when problems arise and they will be expected to take action to help resolve issues.

User Environment.

The LSU HPC systems make use of softenv to allow for adding software to the user's environment. Executing softenv on philip will display the available software:

$ softenv
  +ImageMagick-6.4.6.9-intel-11.1
  +ParMetis-3.1.1-intel-11.1-mpich-1.2.7p1
  +R-2.8.1-gcc-4.3.2
  ...

In order to add software to your environment, you'll need to add the appropriate key to your ~/.soft file. For example, to add the package ImageMagick to your user environment, you would need to add the following:

$ cat ~/.soft
  +ImageMagick-6.4.6.9-intel-11.1
  @default

The order in which you add keys to ~/.soft is important. The first occurrence of a setting takes precedence.

Once the entries are to your liking, you must then execute the command resoft, i.e.:

$ resoft

If your code needs to link to a library of given package, you will find all software installed under /usr/local/packages/, e.g.:

$ ls /usr/local/packages/
  apache_ant  boost     fuse     gold      hdf5         ...
  arpack      boostjam  gamess   graphviz  hypre        
  atlas       condor    git      gromacs   ImageMagick  
  blacs       fftw      gnuplot  gsl       iozone       

Programming/Compiling

Version 11.1 of the Intel compilers are loaded by default, codes can be compiled according to the following chart:

Running Jobs:

There are 3 possible job queues to choose from:

• single - Used for jobs that will only execute on a single node, i.e. nodes=1:ppn<=8. Currently, this queue is only enabled for the 24 GB nodes, and has a limit of 168 hours (7 days) of wallclock time.

• workq - Used for jobs that will use at least one node, i.e. nodes>=1:ppn=8. Currently, this queue is only enabled for the 24 GB nodes, and has a limit of 72 hours (3 days) of wallclock time.

• bigmem - Used for jobs that want to use the 48 and 96 GB nodes. This queue has a limit of 48 hours (2 days) of wallclock time.

Single Queue Job Script Template

$ cat ~/script

#!/bin/bash
#PBS -q single
#PBS -l nodes=1:ppn=1 
#PBS -l walltime=HH:MM:SS
#PBS -o desired_output_file_name
#PBS -N NAME_OF_JOB

/path/to/your/executable

Work Queue Job Script Template

$ cat ~/script

#!/bin/bash
#PBS -q workq
#PBS -l nodes=1:ppn=8
#PBS -l walltime=HH:MM:SS
#PBS -o desired_output_file_name
#PBS -j oe 
#PBS -N NAME_OF_JOB

# mpi jobs would execute:
#   mpirun -np 8 -machinefile $PBS_NODEFILE /path/to/your/executable
# OpenMP jobs would execute:
#   export OMP_NUM_THREADS=8; /path/to/your/executable

Bigmem Queue Job Script Template

$ cat ~/script

#!/bin/bash
#PBS -q bigmem
# Request to use a 48 GB node, similarly could request mem96
#PBS -l nodes=1:ppn=8:mem48
#PBS -l walltime=HH:MM:SS
#PBS -o desired_output_file_name
#PBS -j oe 
#PBS -N NAME_OF_JOB

# mpi jobs would execute:
#    mpirun -np 8 -machinefile $PBS_NODEFILE /path/to/your/executable
# OpenMP jobs would execute:
#    export OMP_NUM_THREADS=8; /path/to/your/executable

Submit the job by executing:

$ qsub script

Monitoring Jobs

The following commands can be used to view/modify the queue

• qdel jobid - deletes a PBS job in the queue.

• qstat - shows you the status of your job and the jobs of others in the queue. It can show you various other bits of information about your job as well, such as the number of nodes it intends to use, the name of the queue it's in, etc

• showq - displays jobs info within the batch system.

• showstart jobid - gives an estimated starting time for your job.

• checkjob jobid - displays detailed job state information