HOWTO: Transfer files using Globus

Globus is a reliable, high-performance file transfer platform allowing users to transfer large amounts of data seamlessly between systems. It aims to make transfers a "click-and-forget" process by setting up configuration details in the background and automating fault recovery.  

Globus can be used for both file transfers between OSC and:

  • A computing institution with Globus installed (check with your site provider for availability) or
  • A personal computer (known as a personal endpoint)

Users transferring between OSC and another computing institution with​ Globus installed do not need to install Globus Connect Personal, and can skip to Usage.

If you are looking to transfer smaller sized files you can utilize OnDemand's file transfer capabilities, or use an SFTP client to connect to  sftp.osc.edu . Our general recommendation is that for small files - measured in MB to several hundred MB - to use OnDemand or SFTP. You can continue to use SFTP and get reasonable performance up to file sizes of several GB. For transfers of several GB or larger, you should consider using Globus.

Install Globus Connect Personal

To use Globus to transfer from a personal computer, you will need to install the Globus Connect Personal client on your computer following the steps below. Those transferring between OSC and another computing institution can skip to Usage.

  1. Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD. b.png                                              
  2. Download the Globus Connect Personal Client 
    • In the main Globus web page, click "ENDPOINTS" on the sidebar.
    • Click "Create a personal endpoint" on the top-right of the pageendpoint_new.png
    • Choose a unique name for your endpoint and generate the setup key. Download the Globus Connect client for your operating systemcreate.png
  3. Install Globus Connect Personal Client
    • Windows
      1. Run the Installer
      2. Copy-Paste the setup key to complete the installation
    • Mac​​
      1. Mount your drives
      2. Copy the Globus Client to your Application Folder
      3. Start The Globus Client, and enter the provided setup key
    • Linux
      1. Un-tar the .tgz file with the command tar -zxvf
      2. Run globusconnect , found within the unzipped directory
      3. Copy-Paste the setup key when prompted
  4. (Optional) Changing directories accessible to Globus

By default, Globus will only add certain default folders to the list of files and directories accessible by Globus. To change/add/remove files and directories from this list:

Windows

  1. Start Globus Connect Personal
  2. Go to "Options"
  • Add directories/files using the  "+" button
  • Remove directories/files using the "-" button
  • Revert to the default accessible directories/files using the "Reset to Defaults" button
  • Any changes you make are not made permanent until you press the "Save" button

​​Mac

  1. Start Globus Connect Personal
  2. Go to Preferences -> Access
  • Add directories/files using the  "+" button
  • Remove directories/files using the "-" button
  • Revert to the default accessible directories/files using the "Reset to Defaults" button
  • Any changes you make are not made permanent until you press the "Save" button

​Linux

  1. Configure the   ~/.globusonline/lta/config-paths  file. This file is a plain text file, with each line corresponding to the configuration of a particular directory path you wish to make accessible. Each line consists of 3 comma-separated fields as below
    <path1>,<sharing flag>,<R/W flag>
    <path2>,<sharing flag>,<R/W flag>
    <path3>,<sharing flag>,<R/W flag>
    ...
    

    Path: an absolute directory/path to be permitted. A leading tilde "~" can be used to represent the home directory of the user that runs globusconnectpersonal.

    Sharing Flag: it controls sharing, with a value of "1" allowing sharing for the path and a value of "0" disallowing sharing for the path.

    R/W Flag: it determines whether the path will be accessible read-only or read/write, with a "1" permitting read/write access and a "0" marking the path as read-only. 

  2. After making a change to the ~/.globusonline/lta/config-paths file, you must stop and restart Globus Connect Personal before the changes will take effect as below
$ ./globusconnectpersonal -stop
$ ./globusconnectpersonal -start &

Usage

We provide instructions on how to transfer data using both Globus Connect Server v4 and v5 on this page. It is highly recommended to use Globus Connect Server v4 for general data transfer that does not require premium storage connectors, or sharing data. If you would like to share your data with your collaborators, please see this page

If you are doing a large transfer you should transfer to/from the parallel file system for best performance. 

Usage of Globus Connect Server v4

  1. Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD.b.png
  2. Navigate to the "File Manager" on the sidebar. Select the two-panel view by switching the icon beside "Panels" on the top-right of the page if the view is one panel switch.png
  3. Enter your endpoint in one of the collections
    • If transferring to a computer with Globus Connect Personal installed, make sure to have Globus Connect Personal running. The name will be the unique name chosen during installation. It is also listed under "Administered by You" under the "ENDPOINTS" on the sidebar
  4. Enter osc#gcs (OSC Globus Connect Server) as the other endpoint
    • Enter your OSC username and password for the authentication
    • By default, you should see the files in your home directory. You can also change the directory by entering the appropriate path in the Path boxtransfer.png
  5. You can now transfer files and directories both ways by selecting them and pressing the arrow indicating which way you'd like to transferstart.png
  6. Once you click 'Start', an activity report will appear as shown below, and you can click on it to see the detailsstatus.png
  7. Click "View details', you will get the activity report about the transfer as shown below. You will also receive an email once the transfer is completed (successful or failed)summary.png
Once a transfer has begun, you do not need to keep the Globus webpage up, but you will need to make sure the Globus Connect Personal Client is running on your computer until it has completed.  If the transfer is interrupted for any reason, Globus will attempt to re-initiate the transfer automatically.

Usage of Globus Connect Server v5

  1. Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iDb.png
  2. If you do not log in with your OSC HPC credentials, map your OSC credentials to your Globus account following this page. Select 'Ohio Supercomputer Center (OSC)' as the identity provider. You will be redirected to the page below. Provide your OSC HPC credentials: OSC_login.png
  3. Navigate to the "ENDPOINTS" on the sidebar and search 'OSC GCSv5'. Click 'OSC GCSv5' to go to this endpointv5.png
  4. Click the 'Collections' menu. You will see all of the collections added by you if you have added any before. To add a new collection, click 'Add a Collection'. Enter your OSC username and password for the authenticationcollection_0.png
  5. The first time you create a collection on an endpoint, Globus will ask you to allow the Collections app to manage your collections. Click Allowcreate-collections-posix-3.png
  6. Choose the filesystem you want to transfer file
    • OSC Home Gateway: home directory
    • OSC /fs/project Gateway: project directory
    • OSC /fs/scratch Gateway: scratch directorygateway.png
  7. Create a guest collection. (Note: the default path will be $HOME for home directory, /fs/project for project filesystem and /fs/scratch for scratch filesystem. You can change to a more specific directory by providing the path in ‘Base Directory’). Click 'Create Collection' to finish the creationgateway_2.png
  8. Navigate to the "File Manager" on the sidebar. Select the two-panel view by switching the icon beside "Panels" on the top-right of the page if the view is one panel switch.png
  9. Enter your endpoint in one of the collections
    • If transferring to a computer with Globus Connect Personal installed, make sure to have Globus Connect Personal running. The name will be the unique name chosen during installation. It is also listed under "Administered by You" under the "ENDPOINTS" on the sidebartransfer.png
  10. Enter another collection by clicking 'select a collection': 
    • Click 'Your Collections' to choose the right collection choose_s.png
  11. You can now transfer files and directories both ways by selecting them and pressing the arrow indicating which way you'd like to transferstart.png
  12. Once you click 'Start', an activity report will appear as shown below, and you can click on it to see the detailsstatus.png
  13. Click "View details', you will get the activity report about the transfer as shown below. You will also receive an email once the transfer is completed (successful or failed)summary.png
Once a transfer has begun, you do not need to keep the Globus webpage up, but you will need to make sure the Globus Connect Personal Client is running on your computer until it has completed.  If the transfer is interrupted for any reason, Globus will attempt to re-initiate the transfer automatically.

Further Reading: