Skip to main content

Globus general use

Prerequisites​

Globus Online, often referred to as Globus App, is used to manage your file transfers. The general idea is to transfer data between two systems where each system has an endpoint registered with Globus. The terrabyte DSS system is already set up as a Globus Server endpoint. Setting up a Globus Connect Personal endpoint on your side will enable data transfers between these two endpoints.

In order to transfer data between the terrabyte DSS Globus endpoint and your Globus Connect Personal endpoint, you will need:

Access Requirement

Globus online does not work for transfers to/from your HOME directory, but only for transfers to/from a storage container on the DSS. Even transferring data from or to the login node via Globus requires access rights to at least one DSS container.See Apply for storage on how to gain access to a terrabyte storage container.

Access Management

Data on terrabyte DSS is organized in "Data Projects" with "Data Curators" responsible for managing access rights. The data curators group typically includes the "Principal Investigator" (PI) of your project. You can only connect to data containers you were granted access to and will have to authenticate yourself using your LRZ account credentials. Visit DSS Management for details.

Setting up a transfer​

Detailed descriptions of the process are provided at How To Log In and Transfer Files with Globus (by Globus) and How to use Globus Online (by LRZ).

General Setup​

  • Whenever asked to log in to Globus, use
    Leibniz-Rechenzentrum der Bayerischen Akademie der Wissenschaften
    as your organization and
    Your terrabyte LRZ credentials
    as your username and password.

  • Establish a Globus Connect Personal endpoint on your local machine

    • Install Globus Connect Personal (available for Linux, MacOS, Windows)
    • If setting this up, additional firewall rules may be required
    • Follow the procedure to log in to Globus
    • Grant the required consents
    • Enter the details for your endpoint ("Collection Name" will later be used to identify the endpoint)
    • Make sure your endpoint is running (OS specific; follow the instructions given in the process)

Web Interface​

  • Visit https://app.globus.org

    • Log in if not yet done
    • Switch to File Manager
    • Search for the terrabyte DSS collection:
      Leibniz Supercomputing Centre's LRZ DSS - CILogon
      and your container, e.g.
      /dss/dsstbyfs02/pn49ci/pn49ci-dss-0004/

  • Using "Transfer or Sync to", set up the File Manager to transfer between the DSS container and your endpoint

    • Select the source and destination endpoints
    • (Hint: Bookmarks/Your Collections should list your active endpoint)

  • Start the transfer

    • The transfer status can be viewed from the link provided
    • An email will be sent when the transfer was completed or terminated

Command Line Interface​

Installation:

pipx install globus-cli

Search for the LRZ Globus endpoint Leibniz Supercomputing Centre’s LRZ DSS - CILogon:

globus endpoint search “LRZ DSS - CILogon”
ID                                   | Owner                                                        | Display Name
------------------------------------ | ------------------------------------------------------------ | -------------------------------------------------
c3f32bba-797e-11e6-8435-22000b97daec | 4f4c9920-dae3-4f15-b95a-c12984f8013c@clients.auth.globus.org | Leibniz Supercomputing Centre's LRZ DSS - CILogon

Save the endpoint ID and owner in environment variables:

lrz_ep=c3f32bba-797e-11e6-8435-22000b97daec
lrz_ow=4f4c9920-dae3-4f15-b95a-c12984f8013c

Log in with the endpoint's owner ID:

globus login --gcs $lrz_ow

Get some general info:

globus collection show $lrz_ep

Directory listing:

globus ls $lrz_ep:/dss

You will be prompted to grant consent to the Globus CLI to access data from this collection. This can be done by executing the following command (will also be displayed in the prompt):

globus session consent 'urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/c3f32bba-797e-11e6-8435-22000b97daec/data_access]'

Next, store the ID of the local endpoint to an environment variable:

loc_ep=$(globus endpoint local-id)

Transfer a file from LRZ to the local endpoint:

globus transfer $lrz_ep:/dss/dssfs02/pn56su/pn56su-dss-0007/file1.txt $loc_ep:file1.txt

Recursively transfer a folder from the local endpoint to LRZ and give the transfer a human-readable label:

globus transfer $loc_ep:test_folder $lrz_ep:/dss/dssfs02/pn56su/pn56su-dss-0007/test_folder --recursive --label "CLI folder transfer"

Globus APIs​

Alternatively, you can use a number of Globus APIs to implement Globus data transfer routines into your scripts and applications.