Roadmap
We have just started with terrabyte and will further extend it. The following table offers a high-level overview of completed milestones and activities planned for the future. Please note that the roadmap is subject to change. We will update it regularly.
If you have topics you think should be considered in terrabyte, please raise the topic in our User Forum or reach out to servicedesk@terrabyte.lrz.de.
Latest update: 2024-10-21
- V1 Summer 2023
- V1.2 Autumn 2024
- V1.4 Summer 2025
- Backlog
| COMPLETED | Self Registration Service | DLR users can create a terrabyte / LRZ account themselves. |
| COMPLETED | HPC SLURM environment | Users can use the HPC SLURM environment for data processing and analysis. |
| COMPLETED | Open On Demand with JupyterLab, QGIS, and VS Code | Users can launch JupyterLab, QGIS, and Visual Studio Code instances in the browser on the terrabyte HPDA cluster. |
| COMPLETED | Basic Python tutorials for using EO data | Basic tutorials are provided on how to use the STAC API for the curated data catalog. In addition basic data cube operations are shown based on xarray and Dask. |
| COMPLETED | STAC API for for curated data catalog | Most of the EO data provided in the curated data catalog are described with STAC metadata. The API can be used to search and filter the available collections. |
| COMPLETED | STAC Browser for curated data catalog | |
| COMPLETED | Initial data ingestion | Initial Sentinel, Landsat, MODIS, and VIIRS data are provided as part of the curated data catalog. |
| COMPLETED | Internal documentation | A documentation is available on how to use terrabyte for DLR-internal users (DLR Wiki) |
| COMPLETED | External info page | An external info page (https://www.dlr.de/eoc/terrabyte) is available with initial information about terrabyte. |
| COMPLETED | Initial terrabyte data cube | A first version of a terrabyte data cube is available based on STAC, xarray, and Dask. |
| COMPLETED | User- and Support-Forum | To enhance the exchange between users and to have a general procedure for questions & answers we would like to establish a web-based user forum. |
| Infrastructure | ||
| COMPLETED | Compute extension | Additional compute resources have been made available. |
| COMPLETED | Storage extension | Additional storage capacity have been made available for curated datasets. |
| Services | ||
| COMPLETED | 2FA | Login to all services with Two-Factor-Authentication |
| COMPLETED | Access for external project partners | External project partners have the possibility to get a terrabyte / LRZ account for project collaborations. |
| COMPLETED | Remote Desktop on compute nodes | The remote desktop application with QGIS is now available on terrabyte compute nodes. |
| COMPLETED | STAC API for user and project data | To allow users to create and manage their own STAC collections and items, we will extend the STAC API with user authentication and enable per user private as well as shared data catalogs. |
| COMPLETED | STAC metadata exports | Curated data catalogues are available in GeoParquet/DuckDB files in addition to the STAC API. |
| Software | ||
| COMPLETED | terrabyte library terrapi | We will compile general functions to use our terrabyte services into a Python library. |
| Data | ||
| COMPLETED | Continuous data ingestion #1 | New EO data are ingested hourly for Sentinel-1, Sentinel-2, Sentinel-3, Sentinel-5p, and Landsat Collection 2 Level-2 data. |
| COMPLETED | Roadmap for data provisioning | An initial roadmap for data availabilities will be provided. |
| Support | ||
| COMPLETED | Documentation webpage | To allow external users to access the documentation, we will migrate from the Wiki to a dedicated documentation webpage, which is accessible to all users with a terrabyte account. |
| COMPLETED | Onboarding workshops | We will start with workshops about how to use terrabyte. |
| SCHEDULED | terrabyte Café | We plan to have a regular exchange with terrabyte users for questions & answers and exchange between users within a terrabyte Café. |
| DEVELOPING | Continuous data ingestion #2 | New EO data are ingested continously for MODIS, VIIRS, and ERA5-Land data. |
| DEVELOPING | HPC Slurm REST API | HPC Slurm commands can be send via a RESTful web API. |
| DEVELOPING | OGC EO Application Package | Allow users to develop and run OGC EO Application Packages on terrabyte HPC. |
| DEVELOPING | Data ordering from D-SDA | Data from the D-SDA archive can be automatically loaded on demand via a dedicated web service. |
| DEVELOPING | S3 interface for data storage | Offer an S3 compatible interface to read and write within Data Science Storage containers. |
| DEVELOPING | Resource catalogue | Within this catalogue, resources (e.g., Jupyter Notebooks, Workflows, Scripts, Algorithms) can be registered and made available for discoverability. |
| PLANNED | STAC Browser for user and project data | Provide a STAC browser for the user data STAC API. |
| PLANNED | OpenEO API | Provide an OpenEO API for data processing on terrabyte HPC. |
| PLANNED | Workshops | We will provide workshops for specific topics (e.g., STAC API, Data Transfers, Data Cubes, Containers, Environments) |
| PLANNED | Extended Python tutorials | We will further extend the basic Python tutorials. Please also share your experiences with us! |
| PLANNED | STAC API optimizations | Based on your feedback and our experiences with the initial version of the STAC API we will further work on performance optimizations. |
| Kubernetes as a Service | We will offer Kubernetes as a Service to allow terrabyte users to launch their Kubernetes workloads. | |
| CI/CD with DLR Gitlab | Users shall be able to build CI/CD pipelines (e.g., for containers) from DLR Gitlab, which can be used in terrabyte. | |
| Web-based data visualization | All data that is available in the STAC API can be visualized with OGC compliant web services. | |
| OGC Geo Data Cube API | We follow current international discussions about an OGC Geo Data Cube API and will evaluate and implement it once decided. | |
| Processing as a Service | Users shall be able to launch pre-defined processes on demand provided either by the terrabyte team or terrabyte users. | |
| Data catalog subscription service | As soon as new data is available in the STAC API, a notification will be send or a trigger will be executed, which the user can subscribe to. | |
| Project request portal | terrabyte projects need to be submitted for special requests. We will offer a web-based solution for this. |