QGreenland-Net OGDC Project Overview

Robyn Marowitz

Rushiraj Nenuji

Alyona Kosobokova

Trey Stafford

2025-04-18

Meet the team

Alyona (ADC)	Rushiraj (ADC)
Robyn (NSIDC)	Trey (NSIDC)

Open Geospatioal Data Cloud

We envision the OGDC being at the center of multiple overlapping interests in cryospheric data management and analysis.

OGDC strives to provide a robust and reproducible interface for transforming, visualizing, and publishing geospatial data, serving QGreenland, the ADC, and various other research interests, including the public.

The CI coordinating group is one of a few curated expert groups that the QGreenland-Net team is working with to ensure that the OGDC aligns with community needs.

High Level OGDC Overview

Our vision for the OGDC, as presented in the QGreenland-Net proposal.

The OGDC will provide an open and reproducible system for performing geospatial data transformations on data within the DataONE network. We are leverage existing networks like DataONE, NSIDC, Arctic Data Center, and PANGAEA.

Users of the OGDC are diverse and include QGreenland, ADC data portals, and public users/scientists.

OGDC Operations

OGDC operations focus on geospatial data transformations to support data analysis and visualization.

Examples of operations are featured here. We’ll highlight a concrete example in the remainder of the presentation.

OGDC Example

The Water temperature, salinity, and position-geographic data taken from seal tags, in coastal Greenland from 2010-08-31 to 2012-05-14 dataset provides a simple use-case.

The primary data file is a .csv with no spatial metadata:

"Cruise","Station","Type","Date","Longitude","Latitude","depth_from_surface","Pressure","Temperature","Salinity","Fluorescence","Oxygen"
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,0,2,5.639,29.87,999,999
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,NA,3,5.6071,29.87,999,999
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,NA,10,4.8221,30.2596,999,999
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,NA,12,4.5924,30.4161,999,999
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,NA,13,4.3179,30.586,999,999
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,NA,17,3.6478,30.899,999,999
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,NA,18,3.0415,31.252,999,999
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,NA,20,2.2182,31.6582,999,999
"ct71-01-10",1,"B","2010-08-31 12:00",-43.3587,60.0247,NA,23,0.8525,32.2843,999,999

The seal tags dataset is a CSV file with one record per seal.

This dataset is a good example of a simple one-step operation to take a non-geospatial ready data file and make it ready for analysis in QGreenland.

The data is in CSV, and uses geographic coordinates (EPSPG:4326). The QGreenland project makes all included data consistently in the Polar Stereographic North (EPSG:3413) projection and strives to include all vector datasets as a geopackage (gpkg).

This can be accomplished through a single ogr2ogr command.

Making seal-tags “analysis-ready”

ogr2ogr \
  -lco ENCODING=UTF-8 \
  -makevalid \
  -s_srs "EPSG:4326" \
  -t_srs "EPSG:3413" \
  -oo X_POSSIBLE_NAMES=Longitude -oo Y_POSSIBLE_NAMES=Latitude \
  /output_dir/ct71_ODV_epsg3413.gpkg /input_dir/ct71_ODV.csv

To make the seal-tags dataset analysis-ready for QGreenland, we need to convert to .gpkg and reproject to EPSG:3413. A single ogr2ogr command is sufficient to achieve this, but other examples may be much more complex.

How?

• Cloud-based distributed computing environment
  • Kubernetes
• Workflow management technology
  • Argo
• Integration with DataONE network of repositories
  • ADC-hosted with access to DataONE data
• A reusable mechanism for expressing what transformations are applied to a dataset
  • Recipes
• Open-source focus.
  • GitHub hosted code, documentation, and recipes

The seal-tags example is a simple, one-step case. But we anticipate more complex use-cases where there are many transformation operations distributed across compute resources in a parallel fashion.

OGDC Architecture

Trey takes over here.

The QGreenland-Net development team has been working on the technical details of the OGDC, which is broadly captured by this architecture diagram.

The OGDC architecture we are working on will automate geospatial data processing using robust open source tools deployed to a cloud-based environment, stores datasets in scalable storage, and publishes metadata via Arctic Data Center, enabling seamless access through APIs and CLI tools.

We have been focused on developing the components responsible for performing user transformation requests (e.g., reprojection, subsetting, etc.) within the ADC’s cloud. Notable milestones include choosing core technologies, deploying initial prototypes, and successfully transforming data.

Next steps will include integrating with DataONE to support publishing outputs.

Example Recipe

meta.yaml file identifies key metadata, including the source to pull data from.

name: "Water measurements from seal tag data"
id: "seal-tags"
inputs:
  urls:
    - "https://arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A31162eb9-7e3b-4b88-948f-f4c99f13a83f"

recipe.sh is a bash file that lists a sequence of operations to transform the dataset.

ogr2ogr \
  -lco ENCODING=UTF-8 \
  -makevalid \
  -s_srs "EPSG:4326" \
  -t_srs "EPSG:3413" \
  -oo X_POSSIBLE_NAMES=Longitude -oo Y_POSSIBLE_NAMES=Latitude \
  /output_dir/ct71_ODV_epsg3413.gpkg /input_dir/ct71_ODV.csv

OGDC Architecture

Highlight recipe -> OGDC runner phase. How do we transform a recipe like we saw in the previous example into something that’s executable in kubernetes?

OGDC-runner

• Responsible for transforming a recipe into an Argo workflow that
• Python-based. Utilizes the Hera Python SDK for programatically constructing Argo workflows.

A Python Software Development Kit designed to interact with Argo Workflows, simplifying the creation and management of workflows programmatically.

Argo Workflow

Rushiraj takes over on this slide.

A tool for orchestrating parallel tasks on Kubernetes, facilitating the management of complex workflows.

Handling workflows with Argo and K8s

Integration of Argo with Kubernetes to streamline workflow automation and scalability.

Seal-tag Argo Workflow

Revisiting our seal-tags example.

• Argo Dashboard shows workflow with intermediate outputs and final data publication step.
• The result is a .gpkg that GIS programs can understand. ogrinfo shows valid spatial metadata is present.

Scaling Viz workflow

Tiling Workflow

References

• QGreenland-Net Dev Site
• QGreenland-Net GitHub
  • ogdc-runner
  • ogdc-helm

Thank you!