Roadmap

We have just started with terrabyte and will further extend it. The following table offers a high-level overview of completed milestones and activities planned for the future. Please note that the roadmap is subject to change. We will update it regularly.

If you have topics you think should be considered in terrabyte, please raise the topic in our User Forum or reach out to servicedesk@terrabyte.lrz.de.

Latest update: 2024-07-15

V1 Summer 2023

COMPLETED	Self Registration Service	DLR users can create a terrabyte / LRZ account themselves.
COMPLETED	HPC SLURM environment	Users can use the HPC SLURM environment for data processing and analysis.
COMPLETED	Open On Demand with JupyterLab, QGIS, and VS Code	Users can launch JupyterLab, QGIS, and Visual Studio Code instances in the browser on the terrabyte HPDA cluster.
COMPLETED	Basic Python tutorials for using EO data	Basic tutorials are provided on how to use the STAC API for the curated data catalog. In addition basic data cube operations are shown based on xarray and Dask.
COMPLETED	STAC API for for curated data catalog	Most of the EO data provided in the curated data catalog are described with STAC metadata. The API can be used to search and filter the available collections.
COMPLETED	STAC Browser for curated data catalog
COMPLETED	Initial data ingestion	Initial Sentinel, Landsat, MODIS, and VIIRS data are provided as part of the curated data catalog.
COMPLETED	Internal documentation	A documentation is available on how to use terrabyte for DLR-internal users (DLR Wiki)
COMPLETED	External info page	An external info page (https://www.dlr.de/eoc/terrabyte) is available with initial information about terrabyte.
COMPLETED	Initial terrabyte data cube	A first version of a terrabyte data cube is available based on STAC, xarray, and Dask.
COMPLETED	User- and Support-Forum	To enhance the exchange between users and to have a general procedure for questions & answers we would like to establish a web-based user forum.

V1.2 Summer 2024

Infrastructure
COMPLETED	Compute extension	Additional compute resources have been made available.
COMPLETED	Storage extension	Additional storage capacity have been made available for curated datasets.
Services
COMPLETED	2FA	Login to all services with Two-Factor-Authentication
COMPLETED	Access for external project partners	External project partners have the possibility to get a terrabyte / LRZ account for project collaborations.
COMPLETED	Remote Desktop on compute nodes	The remote desktop application with QGIS is now available on terrabyte compute nodes.
DEVELOPING	STAC API for user and project data	To allow users to create and manage their own STAC collections and items, we will extend the STAC API with user authentication and enable per user private as well as shared data catalogs.
COMPLETED	STAC metadata exports	Curated data catalogues are available in GeoParquet/DuckDB files in addition to the STAC API.
Software
DEVELOPING	terrabyte library terrapi	We will compile general functions to use our terrabyte services into a Python library.
Data
COMPLETED	Continuous data ingestion #1	New EO data are ingested hourly for Sentinel-1, Sentinel-2, Sentinel-3, Sentinel-5p, and Landsat Collection 2 Level-2 data.
COMPLETED	Roadmap for data provisioning	An initial roadmap for data availabilities will be provided.
Support
COMPLETED	Documentation webpage	To allow external users to access the documentation, we will migrate from the Wiki to a dedicated documentation webpage, which is accessible to all users with a terrabyte account.
COMPLETED	Onboarding workshops	We will start with workshops about how to use terrabyte.
SCHEDULED	terrabyte Café	We plan to have a regular exchange with terrabyte users for questions & answers and exchange between users within a terrabyte Café.

V1.4 Winter 2024

DEVELOPING	Continuous data ingestion #2	New EO data are ingested continously for MODIS and VIIRS data.
DEVELOPING	HPC Slurm REST API	HPC Slurm commands can be send via a RESTful web API.
DEVELOPING	OGC EO Application Package	Allow users to develop and run OGC EO Application Packages on terrabyte HPC.
DEVELOPING	Data ordering from D-SDA	Data from the D-SDA archive can be automatically loaded on demand via a dedicated web service.
DEVELOPING	OpenEO API	Provide an OpenEO API for data processing on terrabyte HPC.
DEVELOPING	S3 interface for data storage	Offer an S3 compatible interface to read and write within Data Science Storage containers.
SCHEDULED	Workshops	We will provide workshops for specific topics (e.g., STAC API, Data Transfers, Data Cubes, Containers, Environments)
SCHEDULED	Extended Python tutorials	We will further extend the basic Python tutorials. Please also share your experiences with us!
SCHEDULED	STAC API optimizations	Based on your feedback and our experiences with the initial version of the STAC API we will further work on performance optimizations.

Backlog

	Kubernetes as a Service	We will offer Kubernetes as a Service to allow terrabyte users to launch their Kubernetes workloads.
	CI/CD with DLR Gitlab	Users shall be able to build CI/CD pipelines (e.g., for containers) from DLR Gitlab, which can be used in terrabyte.
	Web-based data visualization	All data that is available in the STAC API can be visualized with OGC compliant web services.
	OGC Geo Data Cube API	We follow current international discussions about an OGC Geo Data Cube API and will evaluate and implement it once decided.
	Processing as a Service	Users shall be able to launch pre-defined processes on demand provided either by the terrabyte team or terrabyte users.
	Data catalog subscription service	As soon as new data is available in the STAC API, a notification will be send or a trigger will be executed, which the user can subscribe to.
	Project request portal	terrabyte projects need to be submitted for special requests. We will offer a web-based solution for this.