Apptainer sandbox fails on GPFS due to stale NFS file handles

Category: 
Software
Resolution: 
Resolved
Affected Software: 
Apptainer (formerly Singularity)

Users may encounter errors when attempting to run a sandbox on GPFS-mounted directories such as /fs/scratch or /fs/ess. The error output may look like the following:

source: read /.singularity.d/env/01-base.sh: stale NFS file handle
source: read /.singularity.d/env/10-docker2singularity.sh: stale NFS file handle
source: read /.singularity.d/env/90-environment.sh: stale NFS file handle
source: read /.singularity.d/env/91-environment.sh: stale NFS file handle
source: read /.singularity.d/env/94-appsbase.sh: stale NFS file handle
source: read /.singularity.d/env/95-apps.sh: stale NFS file handle
source: read /.singularity.d/env/99-base.sh: stale NFS file handle
FATAL: failed to open /bin/sh for inspection: failed to open elf binary /bin/sh: read /bin/sh: stale NFS file handle

This issue has been reported upstream: https://github.com/apptainer/apptainer/issues/2657.It affects Apptainer versions prior to 1.3.5.

Workaround

The issue has been resolved in Apptainer 1.3.6. Our current version is 1.3.3, and we have scheduled an upgrade to 1.3.6 on August 19, 2025. In the meantime, users can try one of the following workarounds:

  1. Use a container image file instead of a sandbox.
  2. Create the sandbox under $TMPDIR, rather than in your home, scratch, or project directory.

  3. Use a custom configuration file with the overlay option disabled:

apptainer -c ~support/share/apptainer/apptainer.conf shell path/to/sandbox