HOWTO: Use Globus (Overview)

 

GlobusOnline-265x258.jpg

Globus is a cloud-based service designed to let users move, share, and discover research data via a single interface, regardless of its location or number of files or size.

Globus was developed and is maintained at the University of Chicago and is used extensively at supercomputer centers and major research facilities.

Globus is available as a free service that any user can access. More on how Globus works can be found on the Globus "How It Works" page.

Globus How To pages

For information on:

  • Linking OSC username with globus user account
  • data transfer and sharing
  • globus personal connect installation

See globus how to pages.

OSC endpoints

  • Enter 'OSC Globus Connect Server' in the endpoint search box to find all the endpointss managed by OSC as below:
  Endpoint
OSC's home directory OSC $HOME
OSC's project directory OSC /fs/project
OSC's scratch directory OSC /fs/scratch
OSC's ess storage OSC /fs/ess
AWS S3 storage OSC S3
OSC high assurance

OSC /fs/ess/ High Assurance for project storage

OSC /fs/scratch/ High Assurance for scratch storage

Note: the default path will be $HOME for home directory, /fs/ess for project storage, /fs/scratch for scratch filesystem. You can change to a more specific directory by providing the path in ‘Directory’. The location for project/scratch data would be under /fs/ess/<project-code> or /fs/scratch/<project-code>.

Further Reading

HOWTO: Use AWS S3 in Globus

Globus Connect Server v5 allows OSC clients to connect to Amazon S3. Please follow the steps below:

Create a New Bucket

  • Log into Amazon AWS
  • Navigate to Services -> S3
  • Click 'Create Bucket'
  • Give the bucket a name then click 'Next'
  • Click 'Next' and then 'Next' again then 'Create bucket' to create a new bucket

Create a New IAM

  • Log into Amazon AWS
  • Navigate to Services -> IAM

Add Policy

  • Select Policies and then click 'Create Policy'
  • Choose the JSON tab and paste the JSON policy as below (It will be necessary to replace the value for 'Resource' to match bucket name from previous steps):
You may also refer to this page provided by Globus for the JSON policy.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllBuckets",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Bucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::osc-globus-test"
        },
        {
            "Sid": "Objects",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::osc-globus-test/*"
       }
    ]
}
  • Choose 'Review Policy'
  • Fill in the name such as 'AllowGlobusS3_osc-globus-test', click 'Create Policy' to create the new policy

Add Group

  • Choose 'Groups' in left menu bar then click 'Create New Group'. Provide the group name such as “s3_osc-globus-test” then click 'Next Step' button
  • In search bar for Filter type 'Globus' and check box next to 'AllowGlobusS3_osc-globus-test' then click 'Next Step'
  • Click 'Create Group' after reviewing new group information. 

Add User

  • Choose 'Users' in left menu bar and then click 'Add user'. Provide the username, check 'Programmatic access' and then click 'Next: Permissions'
  • Choose 'Add user to group', filter for the group created previous and check box next to 's3_osc-globus-test', click 'Next: Tags' and then 'Next: Review'
  • Click 'Create user'
  • Make a copy of 'Access key ID' and click 'Show' for 'Secret access key' and save both for future reference
  • Click 'Close'

Create a New Collection

  • Login to Globus. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD
  • Navigate to the 'ENDPOINTS' on the sidebar and search 'OSC, s3'. Click 'OSC S3' to go to this gateway
  • Click the 'Collections' menu. You will see all of the collections added by you before. To add a new collection, click 'Add a Collection'. Enter your OSC username and password for the authentication
  • Provide 'Access key' and "Access secret' from previous steps and associate Globus identity with your OSC identity
  • Provide bucket name for 'Base Directory'
  • Provide the name of the collection in 'Collection Display Name' field
  • Click 'Create Collection' to finish the creation
  • Click the logo of 'Globus' and navigate back to the 'ENDPOINTS' on the sidebar. Click the 'Administered by You' and then you can locate the new collection you just created. 

Further Reading

 

HOWTO: Deploy your own endpoint on a server

OSC clients who are affiliated with Ohio State can deploy their own endpoint on a server using OSU subscriptions. Please follow the steps below:

  1. Send a request to OSC Help the following information:
    • Name of organization that will be running the endpoint, ie: OSU Arts and Sciences
      • NOTE: if the name already exists, they will have to coordinate with the existing Admin for that project
    • OSU affiliated email address associated with the Globus account, ie: name.#@osu.edu
  2. OSC will create a new project at https://developers.globus.org, make the user provided in #1 the administrator, and inform the user to set up the endpoint credentials
  3. The user goes to https://developers.globus.org/ and chooses “Register a new Globus Connect Server v5”. Under the project, the user chooses Add dropdown and chooses Add new Globus Connect Server. Provide a display name for the endpoint, ie: datamover02.hpc.osc.edu. Select “Generate New Client Secret” and save that value and Client ID and use those values when configuring the Globus Connect Server install on their local system
  4. The user finishes configuring Globus Connect Server and runs the necessary commands to register the new endpoint with Globus. Once the new endpoint is registered, please email OSC Help the endpoint name so we can mark the endpoint as managed under the OSU subscription

 

Supercomputer: