For administrators > Data Center user guide

Data Store configuration user guide

TIP

Data Stores (eNODEs) provide the data and metadata that are visible to the EPISODES Platform users

How does the EPISODE Platform Data Center connect to eNODEs?

The Data Center collects all the metadata about data that can be served through the EPISODES Platform (note, that it doesn't collect the data itself!). The metadata is read from all the available eNODEs (called Data Stores within this guide) and stored in the Data Center local database in a unified format. This allows the EPISODES Platform to search through the data properties (metadata) and return the files that meet certain criteria. If the user chooses to use or display a specific file, it is downloaded from the eNODE that stores the data and shown to the user.

Fig. 1. Connections between eNODEs, Data Center and EPISODES Platform

Data Stores (eNODE) configuration

To configure a connection to an eNODE (it is called a Data Store in the Data Center interface), use the tab Data Stores (marked with (2) in Fig.1 in Data Center user guide). The view provides a list of all currently configured Data Stores, with their URL, type (CIBIS or CDGP) and unique ID.

Fig. 2. List of Data Stores (eNODEs)

To add a connection to a new Data Store, choose Add connection to CIBIS or Add connection to CDGP (marked with (1) and (2), respectively, in Fig.2) depending on the type of eNODE that should be added. To view an existing connection details either use the details button to the right of the chosen Data Store (marked with (3) in Fig.2), or click the Data Store id.

CIBIS Data Store configuration

To add a connection to the CIBIS-type Data Store, check the configuration of the CIBIS server and fill in the displayed form (see Fig. 3). The form includes configuration of the connection to the LDAP server, which contains all the metadata, configuration of the endpoint for downloading data and contact information.

The LDAP URL should follow the format <protocol>://<host>:<port>, where the protocol is usually ldap, and the port has number 389, so, if the CIBIS is installed on a server cibis.com, the URL will be: ldap://cibis.com:389, but this has to be confirmed with your server administrator. The Base form requires providing the name from which all the LDAP entry will start. For most of the CIBIS installations, this will have a value of dc=cibis,dc=com (as suggested in the form). Next, you have to provide the credentials for a user that has a read access to the LDAP installation. The User DN is in an LDAP enries form (similar to the Base), e.g. uid=<username>,ou=Users,dc=cibis,dc=com. We strongly discourage providing credentials for a user having administrator permissions in the User DN and Password fields. If the LDAP is enabled to serve the data through a secure connection (TLS), check also the Secure connection checkbox.

After configuring the LDAP connection, specify the URL for downloading data, which should be a base URL of the WebDav server, from which the actual files can be downloaded. The URL is again in the form: <protocol>://<host>(:<port>), where protocol will be usually https (http is not recommended). Port is not required, but might be necessary in some installations (check with your server administrator). The host is usually the same as in LDAP URL, so, if the CIBIS is installed on a server cibis.com, the URL will be: https://cibis.com. See also download URL section for the information how the download URLs for each of the files are constructed.

The last part of the configuration is the Contact e-mails form, which collects the e-mail addresses of the individuals responsible for the data and metadata within the CIBIS being configured. The e-mails will be used for automatic notifications in case episode data stored in the configured CIBIS are not available or in case of any other reported problems with data or metadata.

After the configuration form is filled in, you can test the connection by using the Test button at the bottom of the page (see Fig. 3). This will not save any data to Data Center yet, but will check if we are able to read any of the metadata from LDAP. In case the test is successful, you will be able to save the CIBIS (using the Save button) and proceed to load its metadata (see metadata operations section).

Fig. 3. Configuration of a new CIBIS Data Store

How the metadata is read from CIBIS

The process of reading CIBIS metadata into Data Center is strictly connected to the LDAP structure (hierarchical, tree-like structure), which stores the metadata in CIBIS. Therefore, the process is follows:

1. We connect to the LDAP in CIBIS and list the LDAP branches for the base level (indicated by the Base field - see Fig. 3).
2. For each LDAP branch we go recursively to until we reach the individual files metadata (leaves in the LDAP tree)
3. Each file metadata is saved in the Data Center internal database with metadata being a combination of the individual file metadata and the parent metadata (where individual metadata are not present, they are inherited from the parent directory).

CDGP Data Store configuration

To add a connection to the CDGP-type Data Store, check the configuration of the CDGP server and fill in the displayed form (see Fig. 4). The form includes configuration of the connection of the endpoint for downloading all the metadata, configuration of the endpoint for downloading data and contact information.

The Service URL and URL for downloading metadata files (note, that these two fields should be filled with the same value, they are separated for historical reasons, in the next versions they might be merged) are the base URL of the service from which the metadata files are downloaded - the URL is in the form: <protocol>://<host>, where protocol will be usually https (http is not recommended) - e.g. https://cdgp.u-strasbg.fr/TCS-AH. The Metadata listing path is the name of the file containing list of either metadata files of directories containing further metadata lists.

After configuring the connection for downloading metadata, specify the URL for downloading data, which should be a base URL of the server, from which the actual files can be downloaded (the URL is again in the form: <protocol>://<host>) and, if required, the name of the query parameter where the file name should be specified. See also download URL section for the information how the download URLs for each of the files are constructed.

The last part of the configuration is the Contact e-mails form, which collects the e-mail addresses of the individuals responsible for the data and metadata within the CIBIS being configured. The e-mails will be used for automatic notifications in case episode data stored in the configured CIBIS are not available or in case of any other reported problems with data or metadata.

After the configuration form is filled in, you can test the connection by using the Test button at the bottom of the page (see Fig. 4). This will not save any data to Data Center yet, but will check if we are able to read any of the metadata from CDGP. In case the test is successful, you will be able to save the connection (using the Save button) and proceed to load its metadata (see section Data Store operations).

Fig. 4. Configuration of a new CDGP Data Store

How the metadata is read from CDGP

The process of reading CDGP metadata in Data Center is as follows:

1. We collect the list of available metadata files or directories containing metadata from URL for downloading metadata files + Metadata listing path (e.g. https://cdgp.u-strasbg.fr/TCS-AH/CDGP-metadata-list, if the URL is https://cdgp.u-strasbg.fr/TCS-AH, and the listing path is CDGP-metadata-list). Each line of the listing file contains a name of the file/directory and its date of last modification (the modification date is used to optimize the reading process in case we need to load only the most recent data).
2. For each name of the file/directory
   • In case of a file (XML files): we download it and load its content to the Data Center database - the metadata file download url is the base service URL + directory path (if present) + file name - e.g. https://cdgp.u-strasbg.fr/TCS-AH/SSFS2004/SSFS2004-Downhole_network-metadata.xml
   • In case a directory (it is suffixed with '/'), we repeat the same operation as in the first step with a URL constructed from the service URL + directory path + listing path - e.g. https://cdgp.u-strasbg.fr/TCS-AH/SSFS2004/CDGP-metadata-list
3. All the metadata read from the files are stored in the Data Center's internal database.

Data Store operations

To perform any operation on the Data Store, go to the Data Stores tab (marked with (2) in Fig.2) and choose the details button to the right of the chosen Data Store (marked with (3) in Fig.2), or click the Data Store id. This will open the Data Store information, list the stored metadata (if already loaded) and allow to perform further actions on the Data Store. The available actions are (see Fig. 5):

• Remove this Data Store – removes the Data Store configuration and all the metadata associated to it.
• Remove metadata from this Data Store – removes all the metadata associated with this Data Store (eNODE) from the Data Center internal database (no data is removed from the Data Store itself). The Data Store configuration is still present, but no data can be read from it. The Update metadata operation can load the metadata again (see below)
• Update metadata
  • Load new/changed metadata – if any metadata are already loaded, this operation checks if there are any modifications in the Data Store - if yes, applies the modifications to synchronize the Data Store content with the Data Center internal database. Use this option to load small changes (changes in single files metadata, or adding new episodes). In case no metadata was already loaded, behaves the same as Reload all metadata
  • Reload all metadata – removes all metadata and loads everything anew from the Data Store. Use this option when adding new Data Store or when synchronizing after performing large, structural changes.
• Verify metadata consistency – does basic checks if the metadata is correct. This action is read-only - does not modify the Data Center internal database
• Add Episode - select data to be published as an episode - see further in this section for more information

Fig. 5. View of a saved Data Store configuration (CIBIS in this case) with action buttons to the right.

After the connection to a Data Store (CIBIS or CDGP) is created, the metadata from it are not yet loaded and there are no episodes defined. To load the metadata, use the Update metadata button (see Fig. 5, at this point, both variants can be used, but Reload all metadata is preferred). After this operation, the Metadata from <Data Store type>: <Data Store ID> section should be filled (if it is still empty and you see errors occurring, please report them to the administrators, however, most of the errors at this point are caused by invalid metadata structure). The eNODE metadata is now stored in the Data Center internal database with the correct structure, however, the episodes contained in the metadata will not yet be visible in the EPISODES Platform. This is because the eNODE (Data Store), especially the CIBIS type, may contain more than one copy of the episode metadata (called episode 'publication'). Therefore, to ensure only the correct version of the episode publication is provided to the EPISODES Platform, an episode has to be additionally configured by selecting its path in the Data Store, this may be done by choosing the Add episode button (see Fig. 5) and filling the form as displayed in Fig. 6. The Episode Directory may be chosen from a list of available paths which correspond to a structure that might be checked in the  Metadata from <Data Store type>: <Data Store ID> section (see Fig. 7). The Episode id has to be unique across the whole Data Center (within all integrated Data Stores).

Fig. 6. Adding new episode

Fig. 7. Episode directory path displayed next to the metadata structure

After creating the episode, its metadata are displayed within a tree structure, where they can be checked and managed (see Fig. 8). The episode itself can be edited or removed (see buttons in Fig. 7).

The episode metadata are displayed in a form of a tree, where the configured episodes are the main elements (see Fig. 8. left part). On choosing any element of the tree, its metadata are shown to the right (see Fig. 9. right part, grey area) - displaying all the properties that were read from the Data Store (eNODE). These properties are used for the data search within EPISODES Platform and/or for displaying additional information about the files there. For each tree element, there are also actions that can be invoked (see Fig. 9. right upper part - buttons) - these are:

• Download - downloads the file from the Data Store (eNODE). This action may be useful to check if the download configuration works properly after adding new Data Store, but it also can be used for debugging purposes in cases when a file download or viewing through the EPISODES Platform stop working. In such case, it is worth checking whether downloading the same file through the Data Center interface works - if yes, it means the problem is in the EPISODES Platform itself, otherwise, the problem should be sought in the Data Store (eNODE) that provides the data.
• Remove - removes the file metadata from the internal Data Center database. Neither the file itself or its metadata is removed from the Data Store where it resides, therefore, this operation can be reverted by reloading the metadata (see below) or by updating the whole Data Store metadata set (see the beginning of this section). This operation may be used when there are files in the Data Store that should not yet be published to the EPISODES Platform (if the file is not present in the Data Center database, it will not be visible on the EPISODES Platform).
• Refresh cache - invalidates the cached value of the file content in the Data Center internal cache. Use this option if there was a change in the file content and it fails to be visible on the EPISODES Platform. Note: this option works only for the production instance (does nothing for the test instance) of Data Center.
• Reload metadata for subtree – updates the metadata for the chosen item and, in case of a directory, the whole subtree – this is the recommended option to update the metadata for a file or for a whole episode as it will not affect other files/episodes.

Fig. 8. Metadata of individual episode files

Download URL construction

The URL for downloading a file from a Data Store (eNODE) is constructed based on the Data Store configuration and the file download ID displayed in the file metadata. The URL is constructed in similar way for both kinds of Data Stores (CIBIS and CDGP), but there is a slight difference in parameters - therefore, both are described further in this section. Knowing the download URL might help debugging the file download in case of any errors.

Note, that the endpoints for file download in Data Stores (eNODEs) are usually secured and accessible only from a chosen list of IP addresses, therefore, to test the actual download URL, you might be required to set up an SSH tunnel or use a VPN.

CIBIS download URL

The file download URL can be constructed by adding the downloadFileBaseUrl Data Store property (see CIBIS configuration section), a slash sign (/), and downloadId from the chosen file metadata (see metadata operations section) - both parameters are marked in Fig. 9. For the example shown in Fig. 9, the URL would be: https://enode.gig.eu/webdav/3/3/BOBREK MINE: local seismicity linked to longwall mining/BOBREK_20230619/data relevant for the considered hazards/seismic/BOBREK_catalog.mat (note, that for some programs, spaces in the URL might have to be replaced by %20 escape sequence).

Fig. 9. Properties required for constructing a file download URL for CIBIS Data Store

CDGP download URL

The file download URL can be constructed in two ways, depending on whether the fileDownloadQueryParam property is configured in the CDGP Data Store configuration. All the required parameters are marked in Fig. 10.

• If the fileDownloadQueryParam is configured (not empty), the URL is constructed as follows: downloadFileBaseUrl?fileDownloadQueryParam=downloadId, where downloadFileBaseUrl and fileDownloadQueryParam are read from the Data Store configuration (see CDGP configuration section) and the downloadId is read from the chosen file metadata (see metadata operations section). For the example shown in Fig. 10, the URL would be: https://cdgp.eost.unistra.fr/CDGP-AAAI/comm-tcsah.php?resource=SSFS2000-Catalogue_Drif
• If the fileDownloadQueryParam is empty, the URL is constructed as follows: downloadFileBaseUrl/downloadId, where downloadFileBaseUrl is read from the Data Store configuration (see CDGP configuration section) and the downloadId is read from the chosen file metadata (see metadata operations section). For the example shown in Fig. 10. (assuming the fileDownloadQueryParam would not be set - would be displayed as null), the URL would be: https://cdgp.eost.unistra.fr/CDGP-AAAI/comm-tcsah.php/SSFS2000-Catalogue_Drif

Fig. 10. Properties required for constructing a file download URL for CDGP Data Store

Related Documents

• Data Store configuration user guide
• Data Center user guide
• Seiscomp configuration user guide