IME Scratch Filesystem Cache

The Owens and Pitzer systems support a DDN Infinite Memory Engine (IME), a 40TB SSD cache of the /fs/scratch filesystem. The IME can provide much faster access to scratch files, however it is not simple to use. There are three ways to access the IME (in order of increasing programming effort and performance):

  1. The /ime/scratch FUSE filesystem
  2. The MPI-IO API
  3. The native IME API


IME Concepts

Whether using the /ime/scratch filesystem or the APIs the IME can access all files in /fs/scratch, but it won't provide any performance improvement unless the file data is resident in the IME.

File data is made resident either by explicitly importing it with the IME management utility ime-ctl or by directly writing data to the IME.

Once file data is resident there is no automatic synchronization with the corresponding file in /fs/scratch!  This lack of synchronization goes both ways.  Changes to the resident file data will not be visible on /fs/scratch unless the file is explicitly synchronized.  Changes to the file in /fs/scratch may sometimes be reflected in the resident data, but that behavior is not guaranteed.

The IME capacity is much less than that of the /fs/scratch filesystem.  Because of this resident data may be purged at any time.


The /ime/scratch FUSE Filesystem

The IME provides a user-space filesystem that is mounted on an as-needed basis by including the :ime attribute in a batch node specification. For example, on owens:

#PBS -l nodes=1:ppn=28:ime

All files in /fs/scratch will appear under /ime/scratch, but that doesn't mean the data is cached. To check if data is cached use

ime-stat /fs/scratch/path-to-file (you can omit the /fs/scratch/ prefix)

You can't count on data remaining cached in the IME so it's best to explicitly import files in your jobs. Use

ime-ctl -b -i /ime/scratch/path-to-file (you must include the /ime/scratch prefix here)

The '-b' option makes the command block until the import is complete, otherwise it imports asynchronously and may not be complete when your job tries to use the file.

If you cd to /ime/scratch/... any files you create will be resident in the IME. They will have a directory entry under /fs/scratch.... but no data. To flush the data to scratch use

ime-ctl -b -r /ime/scratch/path-to-file

If your files are not owner-writable, the synchronization will fail with error message " ERROR: open file: Unknown status ERROR: cmdline-func `file': Unknown status". If you encounter this situation, please add write-permission to the file via "chmod u+w file" then re-flush the data to scratch.


MPI-IO and IME Native API

There is a native IME API modeled on the POSIX low-level file API. Using this requires modifying program source code. Contact oschelp for API documentation.

Some of our MPI implementations provide an MPI-IO layer that use this API. Some program modifications are still required due to the file path syntax of the API.