CGMES metadata models extension #2898

zamarrenolm · 2024-02-13T11:44:46Z

Please check if the PR fulfills these requirements

The commit message follows our guidelines
Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)

Does this PR already have an issue describing the problem?

What kind of change does this PR introduce?

Feature + bug because of dependencies management during export.

What is the current behavior?

Multiple extensions to keep CGMES metadata:

cgmesSshMetadata
cgmesSvMetedata

What is the new behavior (if this is a feature change)?
CGMES metadata is kept in a single IIDM Network extension.

Does this PR introduce a breaking change or deprecate an API?

Yes
No

If yes, please check if the following requirements are fulfilled

The Breaking Change or Deprecated label has been added
The migration steps are described in the following section

What changes might users need to make in their application due to this PR? (migration steps)

Other information:

Bring to current development branch the initial work made in #2034

Signed-off-by: Luma <zamarrenolm@aia.es>

flo-dup · 2024-03-07T12:39:24Z

After discussing with @annetill, the backward compatibility for extensions CgmesSshMetadata and CgmesSvMetedata does not need to be addressed. It seems that there's no usecase which needed to keep iidm files with those extensions (see slack question in the #powsybl channel). You can thus delete these extensions and their SerDe. If a (strong) need is expressed later, we'll deal with it with a corrective release.

cgmes/cgmes-extensions/src/main/java/com/powsybl/cgmes/extensions/CgmesMetadataModelsSerDe.java

cgmes/cgmes-extensions/src/main/resources/xsd/cgmesMetadataModels.xsd

…sion for storing metadata Signed-off-by: Luma <zamarrenolm@aia.es>

Signed-off-by: Luma <zamarrenolm@aia.es>

zamarrenolm · 2024-03-11T08:14:36Z

To support the use-case of CGM export we need to be able to build an SV model with "dependent on" references to its IGM updated SSH models and also to the original IGM TP models.

In a CGM IIDM Network we have each IGM as a Subnetwork.

When a user wants to make a CGM export, first he has to export the SSH of each Subnetwork, then he has to export the SV at the Network level.

To keep the required information in the extension (the data about the exported SSH), we plan to update the extension every time an export is made. To avoid the extension growing indefinitely, we will only keep the last model for each part and modelling authority.

Signed-off-by: Luma <zamarrenolm@aia.es>

zamarrenolm · 2024-03-11T13:23:46Z

After an internal discussion we prefer not to use the metadata extension to store results of export.

Instead, to support the use case of CGM SSH, SV export, we will use an export context.

Signed-off-by: Luma <zamarrenolm@aia.es>

zamarrenolm · 2024-03-13T15:08:25Z

cgmes/cgmes-model/src/main/java/com/powsybl/cgmes/model/triplestore/CgmesModelTripleStore.java

@@ -257,6 +257,11 @@ public PropertyBags fullModel(String cgmesProfile) {
        return namedQuery("fullModel", cgmesProfile);


yes, it has been removed in the last commit

Signed-off-by: Luma <zamarrenolm@aia.es>

colinepiloquet · 2024-03-14T16:26:39Z

cgmes/cgmes-model/src/main/java/com/powsybl/cgmes/model/InMemoryCgmesModel.java

-    public PropertyBags fullModel(String cgmesProfile) {
-        return new PropertyBags();
-    }
-


Why do you not implement the fullModels method here? I am not sure how these classes are used.

This was used to obtain the CGMES metadata model for a specific part. Now we want to make a single query to the CGMES database to obtain all the metadata models. We do not need anymore the method fullModel(String cgmesProfile). We have replaced it by fullModels().

colinepiloquet · 2024-03-15T09:25:51Z

cgmes/cgmes-model/src/main/resources/CIM16.sparql

@@ -28,6 +28,37 @@ SELECT *
    OPTIONAL { ?FullModel md:Model.description ?description }


Here the fullModel query should be removed too no?

Yes, it is not used anymore. I have removed it.

colinepiloquet · 2024-03-15T10:43:03Z

cgmes/cgmes-extensions/src/main/java/com/powsybl/cgmes/extensions/CgmesMetadataModelsSerDe.java

@@ -0,0 +1,128 @@
+/**
+ * Copyright (c) 2022, RTE (http://www.rte-france.com)


Should it not be 2024? The identifier is missing too.

yes, fixed.

Signed-off-by: Luma <zamarrenolm@aia.es>

flo-dup · 2024-03-18T09:34:41Z

cgmes/cgmes-extensions/src/main/resources/xsd/cgmesMetadataModels.xsd

+    <xs:element name="cgmesMetadataModels">
+        <xs:complexType>
+            <xs:sequence>
+                <xs:element name="model" type="cmm:Model" maxOccurs="unbounded"/>


Instead of a unbounded list of elements with a part enum, it would be clearer to restrict to 4 optional named elements (SV / SSH / TP / EQ).

We want the extension to be generic enough to store metadata for other parts. The complete list is defined in CgmesSubset. What do you think about defining the part as a simple enum type with the proper values? (like the Side in the branch observability IIDM extension)

EQUIPMENT TOPOLOGY STATE_VARIABLES STEADY_STATE_HYPOTHESIS DYNAMIC DIAGRAM_LAYOUT GEOGRAPHICAL_LOCATION EQUIPMENT_BOUNDARY TOPOLOGY_BOUNDARY UNKNOWN

We are storing in the extension only the information about the models that have been read during the import of the network. We could limit it even more, and store only some of the models we have read.
I think that having a limited list of named elements instead of a property would make future modifications more difficult. What do you think?

Ok, I wasn't aware there are more! Ok to keep the list with a part enum then.

And about defining the part as a simple enum type in the xsd, it's a good idea yes

the part is now an enum in the xsd

Signed-off-by: Luma <zamarrenolm@aia.es>

rolnico

In addition to the other comment, could you also add some javadoc? At least to the public methods that have been modified

rolnico · 2024-03-21T14:40:58Z

...s/cgmes-conversion/src/main/java/com/powsybl/cgmes/conversion/export/CgmesExportContext.java

+    private final CgmesMetadataModel exportedEQModel = new CgmesMetadataModel(CgmesSubset.EQUIPMENT, DEFAULT_MODELING_AUTHORITY_SET_VALUE);
+    private final CgmesMetadataModel exportedTPModel = new CgmesMetadataModel(CgmesSubset.TOPOLOGY, DEFAULT_MODELING_AUTHORITY_SET_VALUE);
+    private final CgmesMetadataModel exportedSVModel = new CgmesMetadataModel(CgmesSubset.STATE_VARIABLES, DEFAULT_MODELING_AUTHORITY_SET_VALUE);
+    private final CgmesMetadataModel exportedSSHModel = new CgmesMetadataModel(CgmesSubset.STEADY_STATE_HYPOTHESIS, DEFAULT_MODELING_AUTHORITY_SET_VALUE);


Do we want to keep those models hardcoded here? If another model is added, we'll have to add it here, add the corresponding methods (like getExportedEQModel()), etc.

Would it not be better to store them for example in a map, using the CgmesSubset enum as a key? It would allow to have only a getExporterModel(CgmesSubset cmgesSubset) method instead of one for each model.

Actually the "models" attributes will be removed in #2927. I don't think that is necessary to refactor that part now.

Signed-off-by: Luma <zamarrenolm@aia.es>

Signed-off-by: Romain Courtier <romain.courtier@rte-france.com>

annetill

Really nice, thanks all!

Signed-off-by: Romain Courtier <romain.courtier@rte-france.com>

sonarqubecloud · 2024-03-26T13:34:29Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
95.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

rolnico · 2024-03-26T15:06:20Z

My comments will be covered by a future PR

CGMES metadata models extension

39b04b0

Signed-off-by: Luma <zamarrenolm@aia.es>

zamarrenolm added the CGMES label Feb 13, 2024

zamarrenolm self-assigned this Feb 13, 2024

checkstyle: whitespace

5348e0f

Signed-off-by: Luma <zamarrenolm@aia.es>

annetill added the March release label Feb 19, 2024

annetill mentioned this pull request Feb 19, 2024

[WIP] CGMES metadata is now in one extension #2034

Closed

4 tasks

Merge branch 'main' into cgmes_metadata_models

01dd060

rcourtier self-requested a review February 23, 2024 15:20

Merge branch 'main' into cgmes_metadata_models

53f5051

Merge branch 'main' into cgmes_metadata_models

10c32b9

flo-dup reviewed Mar 7, 2024

View reviewed changes

cgmes/cgmes-extensions/src/main/java/com/powsybl/cgmes/extensions/CgmesMetadataModelsSerDe.java Outdated Show resolved Hide resolved

flo-dup reviewed Mar 7, 2024

View reviewed changes

cgmes/cgmes-extensions/src/main/resources/xsd/cgmesMetadataModels.xsd Show resolved Hide resolved

zamarrenolm added 6 commits March 7, 2024 18:40

remove previous sv, ssh extensions and replace with new generic exten…

c1dfc40

…sion for storing metadata Signed-off-by: Luma <zamarrenolm@aia.es>

fixes from failing unit tests

04d3863

Signed-off-by: Luma <zamarrenolm@aia.es>

names in extension

db2bb00

Signed-off-by: Luma <zamarrenolm@aia.es>

always export models sorted; adjust names in test expected data

d43c1ae

Signed-off-by: Luma <zamarrenolm@aia.es>

remove debug code

fc6defd

Signed-off-by: Luma <zamarrenolm@aia.es>

Merge branch 'main' into cgmes_metadata_models

88d7f50

zamarrenolm marked this pull request as ready for review March 8, 2024 02:29

zamarrenolm marked this pull request as draft March 11, 2024 08:15

zamarrenolm added 3 commits March 11, 2024 12:44

add source of model (import or export); use enum for part (cgmes subset)

8b85452

Signed-off-by: Luma <zamarrenolm@aia.es>

Merge branch 'main' into cgmes_metadata_models

3a8ba5e

Signed-off-by: Luma <zamarrenolm@aia.es>

ignore unknown subset in test

7510cb1

Signed-off-by: Luma <zamarrenolm@aia.es>

zamarrenolm added 2 commits March 12, 2024 14:19

Merge branch 'main' into cgmes_metadata_models

0ca3f24

remove source of metadata model

f8d998a

Signed-off-by: Luma <zamarrenolm@aia.es>

zamarrenolm marked this pull request as ready for review March 12, 2024 14:03

zamarrenolm mentioned this pull request Mar 12, 2024

CGMES export. Handle CGM export dependencies #2927

Merged

7 tasks

zamarrenolm marked this pull request as draft March 13, 2024 14:18

use a single class for storing CGMES metadata model information

5bd9507

Signed-off-by: Luma <zamarrenolm@aia.es>

zamarrenolm commented Mar 13, 2024

View reviewed changes

Merge branch 'main' into cgmes_metadata_models

54fe16e

annetill marked this pull request as ready for review March 14, 2024 15:29

zamarrenolm added 2 commits March 14, 2024 16:58

Merge branch 'main' into cgmes_metadata_models

9d39929

we do not need to store multiple identifiers in a metadata model

705e84c

Signed-off-by: Luma <zamarrenolm@aia.es>

colinepiloquet reviewed Mar 15, 2024

View reviewed changes

zamarrenolm added 3 commits March 15, 2024 17:31

query fullModel for profile replace by fullModels

17a2425

Signed-off-by: Luma <zamarrenolm@aia.es>

fix license data

9759a96

Signed-off-by: Luma <zamarrenolm@aia.es>

Merge branch 'main' into cgmes_metadata_models

e1707e4

flo-dup reviewed Mar 18, 2024

View reviewed changes

zamarrenolm added 3 commits March 18, 2024 16:13

Merge branch 'main' into cgmes_metadata_models

3c80f77

Model part is an enum

feeef19

Signed-off-by: Luma <zamarrenolm@aia.es>

Merge branch 'main' into cgmes_metadata_models

7ebd5c0

annetill removed the March release label Mar 21, 2024

rolnico requested changes Mar 21, 2024

View reviewed changes

zamarrenolm added 2 commits March 22, 2024 11:25

naming: uniform use of cgmes "subset" instead of "part"

6e535d4

Signed-off-by: Luma <zamarrenolm@aia.es>

Merge branch 'main' into cgmes_metadata_models

78e0c49

annetill added the March release label Mar 22, 2024

Add javadoc

514e137

Signed-off-by: Romain Courtier <romain.courtier@rte-france.com>

annetill approved these changes Mar 26, 2024

View reviewed changes

Merge branch 'main' into cgmes_metadata_models

d27b2bd

Signed-off-by: Romain Courtier <romain.courtier@rte-france.com>

rolnico approved these changes Mar 26, 2024

View reviewed changes

annetill merged commit 8ef9e67 into main Mar 26, 2024
6 checks passed

annetill deleted the cgmes_metadata_models branch March 26, 2024 15:23

zamarrenolm mentioned this pull request Apr 26, 2024

CGMES export: fix regression on external explicit SV dependencies #2991

Merged

5 tasks

zamarrenolm mentioned this pull request Jul 1, 2024

Refactoring of CGMES Metadata extensions and metadata properties #2384

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CGMES metadata models extension #2898

CGMES metadata models extension #2898

zamarrenolm commented Feb 13, 2024 •

edited by annetill

Loading

flo-dup commented Mar 7, 2024

zamarrenolm commented Mar 11, 2024

zamarrenolm commented Mar 11, 2024

zamarrenolm Mar 13, 2024

colinepiloquet Mar 14, 2024

zamarrenolm Mar 15, 2024

colinepiloquet Mar 15, 2024

zamarrenolm Mar 15, 2024

colinepiloquet Mar 15, 2024

zamarrenolm Mar 15, 2024

flo-dup Mar 18, 2024

zamarrenolm Mar 18, 2024

zamarrenolm Mar 18, 2024

flo-dup Mar 18, 2024

flo-dup Mar 18, 2024

zamarrenolm Mar 18, 2024

rolnico left a comment

rolnico Mar 21, 2024

rcourtier Mar 26, 2024

annetill left a comment

sonarqubecloud bot commented Mar 26, 2024

rolnico commented Mar 26, 2024

		@@ -257,6 +257,11 @@ public PropertyBags fullModel(String cgmesProfile) {
		return namedQuery("fullModel", cgmesProfile);

		@@ -28,6 +28,37 @@ SELECT *
		OPTIONAL { ?FullModel md:Model.description ?description }

		@@ -0,0 +1,128 @@
		/**
		* Copyright (c) 2022, RTE (http://www.rte-france.com)

CGMES metadata models extension #2898

CGMES metadata models extension #2898

Conversation

zamarrenolm commented Feb 13, 2024 • edited by annetill Loading

flo-dup commented Mar 7, 2024

zamarrenolm commented Mar 11, 2024

zamarrenolm commented Mar 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rolnico left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

annetill left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 26, 2024

Quality Gate passed

rolnico commented Mar 26, 2024

zamarrenolm commented Feb 13, 2024 •

edited by annetill

Loading