Skip to main content

gsutil

Introduction to gsutil​

gsutil is a command-line tool for managing Google Cloud Storage resources. It allows users to perform a variety of tasks, such as uploading, downloading, and managing data in Google Cloud Storage buckets. gsutil is part of the Google Cloud SDK and is widely used for automating workflows and managing large-scale data transfers.

gsutil supports advanced features like resumable uploads, parallel transfers, and bucket lifecycle management. It is a powerful tool for developers, system administrators, and researchers working with cloud-based storage solutions. For more details, visit the official gsutil documentation.

Using gsutil with Modules​

To use gsutil on the terrabyte HPC system, load the gsutil module with the following command:

# consider adding the module use line to your ~/.bashrc to always make terrabyte modules available 
module use /dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/share/modules/files/
module load gsutil

Usage Examples​

Once loaded, you can execute gsutil commands to interact with Google Cloud Storage. Below are some examples of common gsutil operations:

Example 1: List Buckets​

To list all buckets in your Google Cloud project:

gsutil ls

Example 2: Upload a File to a Bucket​

To upload a file to a specific bucket:

gsutil cp localfile.txt gs://your-bucket-name/

Example 3: Download a File from a Bucket​

To download a file from a bucket:

gsutil cp gs://your-bucket-name/remotefile.txt .

Example 4: Synchronize a Local Directory with a Bucket​

To synchronize a local directory with a bucket:

gsutil rsync -r ./local-directory gs://your-bucket-name/

For additional usage instructions and configuration details, refer to the gsutil documentation.