Data flow
From COS-2 Notifications to Data Collections
The downstream protocol of the ESA Charter Mapper is triggered by the COS-21 Core notifications. Via the COS-2 Core notifications interface, multiple types of Documents (see here for “Document”, “Indexes”, “Dataset”, and “Data Item” definitions) are pushed to the ESA Charter Mapper where they are indexed in the Catalogue, which are:
- Activations,
- Disaster Zones,
- Acquisitions,
- Value-Added Products.
The Ingestion component of the ESA Charter Mapper is the entry point for recording new Documents in the database. Once in the ESA Charter Mapper Database, those Documents become Data Items for the platform. In this protocol, the Catalogue is the component that allows users to search for Data Items filtering their Metadata properties like the sensing time and area of interest, the orbit number and direction and many other sensor-specific fields of source EO data.
In the ESA Charter Mapper the Data Collections menu is the main entrance for search in the different collections of Data Items. This menu is located in the top bar of the ESA Charter Mapper GUI and is organized in multiple data collections (also called data contexts), which are dynamically managed with respect to the Activation information (zones, type and the related Acquisitions and Value Added Products available. In particular, it contains the following types of Data Collections:
-
Acquisitions: Earth Observation Products notified by COS-2 that may be grouped by mission, sensor type or any other metadata.
- Notifications: This collection contains all the acquisitions in their early stage in the system. As soon as a notification is received properly from COS-2, a corresponding entry is available in this collection as Notification New [NN]. In this collection, items are tagged as Notification Imported [NI] when a proper data file has been copied using the reference in the notification. In case it is not possible to download the source EO data from COS-2 or only the metadata is available in the downloaded EO product the ESA Charter Mapper generates respectively a Notification Missing Data [NMD] or a Notification Metadata Only [NMO].
-
Datasets: This collection contains all the imported and calibrated datasets that have been properly harvested for its metadata. This collection is primarily meant to contain all the dataset for which a successful calibration process has been performed. It also includes not calibrated datasets: SAR SLC images, and EO data for which the calibration has not been possible (e.g. due to missing metadata at source). The ESA Charter Mapper ensures proper dataset access authorization according to the EULA management set in COS-2. Thus, if a user (PM and/or VA) has not accepted the EULA for a dataset that is not public, the ESA Charter Mapper:
- does not provide access to dataset assets and full-resolution overviews
- does not allow the usage of the dataset as input of processing services
- shows on the map the "default" overview at a pre-defined lower resolution.
- warns the user that EULA acceptance is required for full access to the dataset indicating the link to the COS-2 EULA acceptance page. An information banner is displayed in the upper part of the left panel under “Details” of the Dataset.
-
Value Added Products: End User map products that may be displayed on the map.
-
Open EO Data: This collection allows to gather, upload and the ingest external satellite imagery from open EO data collections (Sentinel-1 GRD, Sentinel-2 MSI L1C and L2A).
-
My results: This collection include all results obtained from the User in the Workspace.
-
Activation: Point of interest with Activation details (Activation / Information), Zones identified by the call and the Project Manager with all the related metadata extracted from the KML provided with the Activation (Activation / Areas).
The menus are dynamically managed with respect to the Activation information (zones, type and the related Acquisitions and Value Added Products available.
At first opening the Activation Workspace has the Data Collection “Acquisition/Dataset” enabled by default and, as a result, all found and notified acquisitions that have passed the 2 essential preliminary steps (import + harvesting + calibration) to become a Dataset are presented to the user. Datasets are listed in the Results tab (see following section about Search and Results tab) with full metadata available. By selecting instead the “Activation” Data Collection and then the “Areas” subclass, the Results panel is then refreshed and only the list of all the vector layers representing the Disaster zones of the activation is given. Instead, the “Information” subclass will provide a Pinpoint that geolocates the disaster with activation information included as metadata.
Activation, Acquisition and Value Added Products Data flow
To manage multiple types of Documents and Data Items, different types of Data Flow must exist in the ESA Charter Mapper within a Charter activation. Data flows depend on the nature of the COS-2 notification and are classified in three types:
- Activation Data Flow,
- Acquisition Data Flow
- and Value Added Products Data Flow.
In the ESA Charter Mapper, the Activation Data Flow begins via a notification from COS-2 Core that a new activation is activated for the Charter. The COS-2 notification provides all essential information about the call such as the Activation ID, Call-ID, PM username, activation name, disaster type.
Once the coming COS-2 activation notification is received by the ESA Charter Mapper, a new application is then created and published into the Portal. The resulting Activation workspace will then be dedicated to the specific activation (Act-ID 688), tailored to the specific disaster type (Tropical storm) and will include all the products and information from the call.
In the existing Activation Workspace, the Acquisition Data Flow begins via an acquisition notification from COS-2 Core. A new entry is then created and imported into the Catalogue and tagged as notification. As a result, a New Acquisition Notification is also available in the Portal (footprint visible on the map and basic metadata).
Once source EO data is found, and successfully imported in the ESA Charter Mapper, data is copied into the storage. After this step a new Imported Acquisition Notification is then published into the ESA Charter Mapper Portal and the associated acquisition quicklook is visible in the map.
Later, the Metadata harvesting enables the creation of the Dataset in the ESA Charter Mapper and the acquisition of source EO data is completed. After these steps, an initial processing is performed to calibrate the dataset and make it ready for further generic processing. The ESA Charter Mapper ensures proper dataset access authorization according to the EULA management set in COS-2. More information about the EULA management can be found here
As a result, a new Calibrated Dataset (either Optical or SAR) is then available for the user and ready for scenario systematic processing (see here) and on-demand processing in the ESA Charter Mapper for the creation of different types of output Products.
-
The International Charter Space and Major Disasters, COS-2 dashboard. Available at: disasterscharter.org. ↩