HOWTO: Use Globus (Overview)

 

GlobusOnline-265x258.jpg

Globus is a cloud-based service designed to let users move, share, and discover research data via a single interface, regardless of its location or number of files or size.

Globus was developed and is maintained at the University of Chicago and is used extensively at supercomputer centers and major research facilities.

Globus is available as a free service that any user can access. More on how Globus works can be found on the Globus "How It Works" page.

OSC deprecated Globus v4 endpoint on Tuesday, Jan 26, 2021. Please use Globus v5.4 endpoint by searching 'OSC Globus Connect Server' for transferring and sharing data. See this page for more information on how to transfer data and this page for more information on how to share data. 
Please contact OSC Help if you need to transfer protected data, such as PHI and PII data into OSC HPC systems

Further Reading

HOWTO: Install Globus Connect Personal

Globus is a reliable, high-performance file transfer platform allowing users to transfer large amounts of data seamlessly between systems. It aims to make transfers a "click-and-forget" process by setting up configuration details in the background and automating fault recovery.  

Globus can be used for both file transfers between OSC and:

  • A computing institution with Globus installed (check with your site provider for availability) or
  • A personal computer (known as a personal endpoint)

Users transferring between OSC and another computing institution with​ Globus installed do not need to install Globus Connect Personal, and can skip this page.

Install Globus Connect Personal

To use Globus to transfer from a personal computer, you will need to install the Globus Connect Personal client on your computer following the steps below. Those transferring between OSC and another computing institution can skip to Usage.

 

Watch How to Install Globus Personal
Video about 'How to Transfer Files Using Globus'

 

  1. Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD. b.png                                              
  2. Download the Globus Connect Personal Client 
    • In the main Globus web page, click "ENDPOINTS" on the sidebar.
    • Click "Create a personal endpoint" on the top-right of the pageendpoint_new.png
    • Download the Globus Connect client for your operating systemdownload after clicking create endpoint 2.PNG
  3. Install Globus Connect Personal Client
    • Windows
      1. Run the Installer
      2. Copy-Paste the setup key to complete the installation
    • Mac​​
      1. Mount your drives
      2. Copy the Globus Client to your Application Folder
      3. Start The Globus Client, and enter the provided setup key
    • Linux
      1. Un-tar the .tgz file with the command tar -zxvf
      2. Run globusconnect , found within the unzipped directory
      3. Copy-Paste the setup key when prompted
  4. Setup Globus Personal
    • The setup process will ask you to login. Click Loginsetup login.PNG
    • This will bring up a browser window with Globus asking for permissions for Globus Personal to have access to. Click "allow" to continue.Allow access.PNG
    • This will bring up a Globus Personal window up to finish the setup process. Fill out the required information and click "Save".   complete setup.PNG
  5. (Optional) Changing directories accessible to Globus

By default, Globus will only add certain default folders to the list of files and directories accessible by Globus. To change/add/remove files and directories from this list:

Windows

  1. Start Globus Connect Personal
  2. Go to "Options"
  • Add directories/files using the  "+" button
  • Remove directories/files using the "-" button
  • Revert to the default accessible directories/files using the "Reset to Defaults" button
  • Any changes you make are not made permanent until you press the "Save" button

​​Mac

  1. Start Globus Connect Personal
  2. Go to Preferences -> Access
  • Add directories/files using the  "+" button
  • Remove directories/files using the "-" button
  • Revert to the default accessible directories/files using the "Reset to Defaults" button
  • Any changes you make are not made permanent until you press the "Save" button

​Linux

  1. Configure the   ~/.globusonline/lta/config-paths  file. This file is a plain text file, with each line corresponding to the configuration of a particular directory path you wish to make accessible. Each line consists of 3 comma-separated fields as below
    <path1>,<sharing flag>,<R/W flag>
    <path2>,<sharing flag>,<R/W flag>
    <path3>,<sharing flag>,<R/W flag>
    ...
    

    Path: an absolute directory/path to be permitted. A leading tilde "~" can be used to represent the home directory of the user that runs globusconnectpersonal.

    Sharing Flag: it controls sharing, with a value of "1" allowing sharing for the path and a value of "0" disallowing sharing for the path.

    R/W Flag: it determines whether the path will be accessible read-only or read/write, with a "1" permitting read/write access and a "0" marking the path as read-only. 

  2. After making a change to the ~/.globusonline/lta/config-paths file, you must stop and restart Globus Connect Personal before the changes will take effect as below
$ ./globusconnectpersonal -stop
$ ./globusconnectpersonal -start &

HOWTO: Transfer files using Globus

Globus is a reliable, high-performance file transfer platform allowing users to transfer large amounts of data seamlessly between systems. It aims to make transfers a "click-and-forget" process by setting up configuration details in the background and automating fault recovery.  

Globus can be used for both file transfers between OSC and:

  • A computing institution with Globus installed (check with your site provider for availability) or
  • A personal computer (known as a personal endpoint). See this page on how to install Globus Connect Personal

Users transferring between OSC and another computing institution with​ Globus installed do not need to install Globus Connect Personal.

If you are looking to transfer smaller sized files you can utilize OnDemand's file transfer capabilities, or use an SFTP client to connect to  sftp.osc.edu . Our general recommendation is that for small files - measured in MB to several hundred MB - to use OnDemand or SFTP. You can continue to use SFTP and get reasonable performance up to file sizes of several GB. For transfers of several GB or larger, you should consider using Globus.

Transfer files using Globus

We provide instructions on how to transfer data on this page. If you would like to share your data with your collaborators, please see this page

If you are doing a large transfer you should transfer to/from the parallel file system for best performance. 

Watch How to Transfer Files Using Globus
Video about 'How to Transfer Files Using Globus'

 

  1. Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD.b.png
  2. If you do not log in with your OSC HPC credentials, map your OSC credentials to your Globus account following this page. Select 'Ohio Supercomputer Center (OSC)' as the identity provider. You will be redirected to the page below. Provide your OSC HPC credentials: OSC_login.png

  3. Navigate to the "File Manager" on the sidebar. Select the two-panel view by switching the icon beside "Panels" on the top-right of the page if the view is one panel switch.png
  4. Enter your endpoint in one of the collections
    • If transferring to a computer with Globus Connect Personal installed, make sure to have Globus Connect Personal running. The name will be the unique name chosen during installation. It is also listed under "Administered by You" under the "ENDPOINTS" on the sidebar
  5. Enter 'OSC Globus Connect Server' in the other collection box to search all the collections managed by OSC as below. Choose the collection where you want to do the data transfer:
    •   Endpoint
      OSC's home directory OSC $HOME
      OSC's project directory OSC /fs/project
      OSC's scratch directory OSC /fs/scratch
      OSC's ess storage OSC /fs/ess
      AWS S3 storage OSC S3
    • If needed, enter your username and password for the authentication
    • You should see the files in the collection. You can also change the directory by entering the appropriate path in the Path boxtransfer.png
  6. You can now transfer files and directories both ways by selecting them and pressing the arrow indicating which way you'd like to transferstart.png
  7. Once you click 'Start', an activity report will appear as shown below, and you can click on it to see the detailsstatus.png
  8. Click "View details', you will get the activity report about the transfer. You will also receive an email once the transfer is completed (successful or failed)
Once a transfer has begun, you do not need to keep the Globus webpage up, but you will need to make sure the Globus Connect Personal Client is running on your computer until it has completed.  If the transfer is interrupted for any reason, Globus will attempt to re-initiate the transfer automatically.

HOWTO: Share data using Globus

Globus Connect Server allows OSC users to share data with their collaborators who do not have OSC HPC account (the collaborator needs to sign up for a free Globus account though). The advantage of data sharing via Globus is that you do not have to move your data in order to share it. You can select directory paths to be securely shared with your collaborator, and grant them read-only or read-write access.

Note: You can only share directories, not individual files. 

Sharing data with collaborators 

Watch How to Share Files Using Globus
Video about 'How to Share Files Using Globus

 

  • Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iDb.png
  • Map your OSC credentials to your Globus account following this page if you haven't done this before. Select 'Ohio Supercomputer Center (OSC)' as the identity provider from the list. You will be redirected to the page below. Provide your OSC HPC credentials:OSC_login.png
  • Navigate to the "ENDPOINTS" on the sidebar. Enter 'OSC Globus Connect Server' in the search box to search all the endpointss managed by OSC as below:
  Endpoint
OSC's home directory OSC $HOME
OSC's project directory OSC /fs/project
OSC's scratch directory OSC /fs/scratch
OSC's ess storage OSC /fs/ess
AWS S3 storage OSC S3
  • Choose the endpoint where you want to share the data. You will be redirected to the overview page of this endpoint. Click the 'Collections' menu. You will see all of the collections added under this endpoint before. To add a new collection, click 'Add a Collection'. 
  • Create a guest collection. (Note: the default path will be $HOME for home directory, /fs/project for project filesystem, /fs/scratch for scratch filesystem, and /fs/ess for ess storage. You can change to a more specific directory by providing the path in ‘Directory’). Click 'Create Collection' to finish the creation.arcum
  • Navigate to the "File Manager" on the sidebar. Select the view to be one-panel by switching the icon beside "Panels" on the top-right of the pageswitch.png
  • Enter the collection you want to share by clicking 'select a collection': choose_sharing.png
  • Click 'Your Collections' to choose the right collection choose_s.png
  • Click 'Share' button in the right side paneshare button 2.PNG
  • To share with your collaborators, click 'Add Permissions - Share With'. You can specify the directory you want to share, choose to share with users by selecting their email address or Globus username, or share with a group or all Globus users or everyone. You can also select 'Send Email' to send email to your collaborator. If you want your collaborator to also be able to transfer data into the directory, you can also provide write access by checking the 'write' button. Once the options have been selected, click 'Add Permission'. You should be redirected to the information page on this collection, and see the people you have shared it with.
    • You can repeat this process for any number of collaborators.sharewith.pngsharewith_2.png
  • To terminate the sharing, you can click the 'trash can' icon next to the invitee on the screen.trashing sharing connection 2.PNG
  • To see all collections you have shared, go to 'Endpoints' in the left bar, then 'Shareable by You'.allshare.png
If your collaborator has the write access, he/she can transfer data and delete files within that directory and any subdirectories. So be careful about providing write access.
It is highly recommended that you terminate the sharing or delete the collection when your collaborator has completed downloading the data. 

Information for your collaborators 

  1. Your collaborator needs to sign up for a free Globus account for data sharing
  2. Your collaborator needs to install the Globus Connect Personal client on his/her personal computer if he/she needs to download the data. See this page for more information. 
  3. If you select 'Send Email' to send an email to your collaborator, he/she will get an email including the URL to access the share. Your collaborator can click on the link and log in with the credentials
  4. Your collaborator can also access to the share by logging into Globus. Navigate to the "ENDPOINTS" on the sidebar and click on 'Shared with You'. 

Further Reading

 

    HOWTO: Use AWS S3 in Globus

    Globus Connect Server v5 allows OSC clients to connect to Amazon S3. Please follow the steps below:

    Create a New Bucket

    • Log into Amazon AWS
    • Navigate to Services -> S3
    • Click 'Create Bucket'
    • Give the bucket a name then click 'Next'
    • Click 'Next' and then 'Next' again then 'Create bucket' to create a new bucket

    Create a New IAM

    • Log into Amazon AWS
    • Navigate to Services -> IAM

    Add Policy

    • Select Policies and then click 'Create Policy'
    • Choose the JSON tab and paste the JSON policy as below (It will be necessary to replace the value for 'Resource' to match bucket name from previous steps):
    You may also refer to this page provided by Globus for the JSON policy.
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllBuckets",
                "Effect": "Allow",
                "Action": [
                    "s3:ListAllMyBuckets",
                    "s3:GetBucketLocation"
                ],
                "Resource": "*"
            },
            {
                "Sid": "Bucket",
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket"
                ],
                "Resource": "arn:aws:s3:::osc-globus-test"
            },
            {
                "Sid": "Objects",
                "Effect": "Allow",
                "Action": [
                    "s3:DeleteObject",
                    "s3:GetObject",
                    "s3:PutObject"
                ],
                "Resource": "arn:aws:s3:::osc-globus-test/*"
           }
        ]
    }
    • Choose 'Review Policy'
    • Fill in the name such as 'AllowGlobusS3_osc-globus-test', click 'Create Policy' to create the new policy

    Add Group

    • Choose 'Groups' in left menu bar then click 'Create New Group'. Provide the group name such as “s3_osc-globus-test” then click 'Next Step' button
    • In search bar for Filter type 'Globus' and check box next to 'AllowGlobusS3_osc-globus-test' then click 'Next Step'
    • Click 'Create Group' after reviewing new group information. 

    Add User

    • Choose 'Users' in left menu bar and then click 'Add user'. Provide the username, check 'Programmatic access' and then click 'Next: Permissions'
    • Choose 'Add user to group', filter for the group created previous and check box next to 's3_osc-globus-test', click 'Next: Tags' and then 'Next: Review'
    • Click 'Create user'
    • Make a copy of 'Access key ID' and click 'Show' for 'Secret access key' and save both for future reference
    • Click 'Close'

    Create a New Collection

    • Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD
    • Navigate to the 'ENDPOINTS' on the sidebar and search 'OSC, s3'. Click 'OSC S3' to go to this gateway
    • Click the 'Collections' menu. You will see all of the collections added by you before. To add a new collection, click 'Add a Collection'. Enter your OSC username and password for the authentication
    • Provide 'Access key' and "Access secret' from previous steps and associate Globus identity with your OSC identity
    • Provide bucket name for 'Base Directory'
    • Provide the name of the collection in 'Collection Display Name' field
    • Click 'Create Collection' to finish the creation
    • Click the logo of 'Globus' and navigate back to the 'ENDPOINTS' on the sidebar. Click the 'Administered by You' and then you can locate the new collection you just created. 

    Further Reading

     

    HOWTO: Add InCommon Authentication to Globus

    (OPTIONAL) Adding InCommon Authentication 

    Adding InCommon authentication to your Globus account allows you to login to Globus Online using your university credentials.  Using this process you can store your Globus username password for safe keeping, and instead use your university username and password to login.  If your already logged in to your university authentication system, logging in to Globus can be as simple as two clicks away.

    To use this feature, your university needs to be a InCommon participant.  Some Ohio universities active in InCommon include: Ohio State University, Case Western University, Columbus State Community College, Miami University, Ohio Northern University, Ohio University, University of Findlay, University of Dayton, and many more.  

    For a complete list, visit https://incommon.org/participants/ .

    To add InCommon Authentication:
    1. Login to Globus Online
    2. Go to "Manage Identities" under your username
    3. Click "Add External Identity"
      • Choose a name for the Identity Settings. 
      • Choose InCommon / CILogon from the drop down menu
    4. On the next page, choose your University / Identity Provider
    • Click "Remember this selection"
    • Click "Log on"
    • You may be prompted to login to your university authentication system if you are not already

    When you go to login next, click "alternative login" and then "InCommon / CILogon".  Select your university on the next page, and login using your university credentials.  Globus will remember this preference, and automatically prompt you to login using your university authentication next time.

    HOWTO: Deploy your own endpoint on a server

    OSC clients who are affiliated with Ohio State can deploy their own endpoint on a server using OSU subscriptions. Please follow the steps below:

    1. Send a request to OSC Help the following information:
      • Name of organization that will be running the endpoint, ie: OSU Arts and Sciences
        • NOTE: if the name already exists, they will have to coordinate with the existing Admin for that project
      • OSU affiliated email address associated with the Globus account, ie: name.#@osu.edu
    2. OSC will create a new project at https://developers.globus.org, make the user provided in #1 the administrator, and inform the user to set up the endpoint credentials
    3. The user goes to https://developers.globus.org/ and chooses “Register a new Globus Connect Server v5”. Under the project, the user chooses Add dropdown and chooses Add new Globus Connect Server. Provide a display name for the endpoint, ie: datamover02.hpc.osc.edu. Select “Generate New Client Secret” and save that value and Client ID and use those values when configuring the Globus Connect Server install on their local system
    4. The user finishes configuring Globus Connect Server and runs the necessary commands to register the new endpoint with Globus. Once the new endpoint is registered, please email OSC Help the endpoint name so we can mark the endpoint as managed under the OSU subscription

     

    Supercomputer: