WIP: [RHOAIENG-4485] Fix feature tracker errors while uninstalling operator #1192

Sara4994 · 2024-08-20T15:52:10Z

Description

How Has This Been Tested?

Screenshot or short clip

Merge criteria

You have read the contributors guide.
Commit messages are meaningful - have a clear and concise summary and detailed explanation of what was changed and why.
Pull Request contains a description of the solution, a link to the JIRA issue, and to any dependent or related Pull Request.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

openshift-ci · 2024-08-20T15:52:15Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2024-08-20T15:52:17Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from sara4994. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

VaishnaviHire · 2024-08-28T02:09:46Z

bundle/manifests/opendatahub-operator.clusterserviceversion.yaml

@@ -105,6 +105,7 @@ metadata:
    createdAt: "2024-08-09T10:12:27Z"
    olm.skipRange: '>=1.0.0 <2.16.0'
    operators.operatorframework.io/builder: operator-sdk-v1.31.0
+    operators.operatorframework.io/internal-objects: '[featuretrackers.features.opendatahub.io]'


Annotation expects double quotes for the crd to be hidden in console

operators.operatorframework.io/internal-objects: '["featuretrackers.features.opendatahub.io"]'

VaishnaviHire · 2024-08-28T02:20:01Z

Even when CRD is hidden, it does appear in list of operands during uninstall

I see 2 errors -

* DSCI default-dsci DSCInitialization Error: r: admission webhook "operator.opendatahub.io" denied the request: Cannot delete DSCI object when DSC object still exists
* FeatureTracker FT  FeatureTracker Error: r: featuretrackers.features.opendatahub.io "opendatahub-serverless-serving-gateways" not found

I believe to remove featureTracker error we need to update the delete function to check if the instance exists. @bartoszmajsak Can you confirm if this is what we need to do ?
Regarding webhook error, I wonder if we need to remove the webhook check, keep CRs independent of each other? (cc: @zdtsw @ykaliuta )

Note

There seems to be no way current to do ordered deletion of operands in OpenShift Web Console.

bartoszmajsak · 2024-08-28T04:35:31Z

 * FeatureTracker FT  FeatureTracker Error: r: featuretrackers.features.opendatahub.io "opendatahub-serverless-serving-gateways" not found
I believe to remove featureTracker error we need to update the delete function to >
check if the instance exists. @bartoszmajsak Can you confirm if this is what we need to do ?

@VaishnaviHire Before I dive deeper, quick question: who is deleting instances of FeatureTracker? How the delete operation is triggered?

bartoszmajsak · 2024-08-28T04:49:38Z

Morning coffee later. Is it what is happening?

Deletion is triggered from the Openshift Console as "uninstall" (so outside of our control)
It does try to delete listed FT but it's already gone, because during uninstall, the DSC was first marked for deletion and so the associated FT was removed during .Cleanup() phase of all the features associated with DSC, therefore it's not there anymore when Openshift Console gets to it from its list of operands.

What we can do here is to make DSC and DSCI (depending on where features were created) owners of associated FTs and simplify the cleanup logic. Not sure this will mitigate the problem of trying to delete a resource that was already garbage collected.

We have this already done on our feature branch waiting as stacked PR, so can open it against incubation.

ykaliuta · 2024-08-28T06:14:30Z

Regarding webhook error, I wonder if we need to remove the webhook check, keep CRs independent of each other? (cc: @zdtsw @ykaliuta )

Well, it came there for reasons. What can be a solution then? There was discussion about ownership (@bartoszmajsak ) but IIUC it will cause DSC deletion if a user deletes DSCI "by accident" which is not desirable.

Note

There seems to be no way current to do ordered deletion of operands in OpenShift Web Console.

bartoszmajsak · 2024-08-28T07:11:47Z

There was discussion about ownership (@bartoszmajsak ) but IIUC it will cause DSC deletion if a user deletes DSCI "by accident" which is not desirable.

@ykaliuta I don't remember a discussion about DSCI accidental deletion - can you point me to it so that I can refresh my fish memory? :)

ykaliuta · 2024-08-28T07:30:43Z

There was discussion about ownership (@bartoszmajsak ) but IIUC it will cause DSC deletion if a user deletes DSCI "by accident" which is not desirable.

@ykaliuta I don't remember a discussion about DSCI accidental deletion - can you point me to it so that I can refresh my fish memory? :)

#1094 (comment)

zdtsw · 2024-08-28T07:35:37Z

here are two different ways of uninstallations:

done by CLI/configmap/manually one resource after another, this is mostly handled by Operator
the console way, with a click and assume all will be gone without error.

the problem here lies in the way how 1) and 2) are not doing the same thing.
in (1) operator are trying to be smart and deleting with ownership + FTer are created/removed by DSC/DSCI logic + DSC and DSCI have a hidden depedency (webhook)
in (2) console only delete all operands in order/non-order/which order?

but as you all have said abvoe ^, when console first snapshot all operands and try to delete all of them, some of these are already gone due to operator's intelligence . so the error catch all these "missing" ones.

back to the change itself:
by hidden FTer from console (this proposed PR) it does not show FTer from UI but FTer is still part of the Operator's API, and as API it is resource will be seen as operands. so the console will still try to delete them.

as for the webhook part:
this is from the logic of "blocking deletion of DSCI if DSC exist". but all in all, this is still the same as the FTer error, catch all the "missing" ones which console tries to delete.

and i have no good idea how to tackle this "UI uninstallation" without a big refactor of code, tbh.

lburgazzoli · 2024-08-28T07:38:09Z

My 2 cents:

since the two resources are independent, the operator should be able to handle the case where only one exists (i.e. I should be able to create the DSC first and after the DSCI), as consequence it should be able to handle the case where one of the resource is deleted (i.e. if the DSC is being deleted, it should probably ignore the fact that the DSCI is not there)
if the point above is not possible, then the two resource should probably be merged in a single one as properly pointed out by @adelton here
if we still want to a strong relation, we can use finalizer to either prevent the DSCI to be deleted till the DSC is still there or marked for deletion (mimic background deletion), or even trigger the re-creation of the DSCI but I would push toward option 1 and 2 above

lburgazzoli · 2024-08-28T07:40:53Z

here are two different ways of uninstallations:
1. done by CLI/configmap/manually one resource after another, this is mostly handled by Operator

2. the console way, with a click and assume all will be gone without error.
the problem here lies in the way how 1) and 2) are not doing the same thing. in (1) operator are trying to be smart and deleting with ownership + FTer are created/removed by DSC/DSCI logic + DSC and DSCI have a hidden depedency (webhook) in (2) console only delete all operands in order/non-order/which order?

but as you all have said abvoe ^, when console first snapshot all operands and try to delete all of them, some of these are already gone due to operator's intelligence . so the error catch all these "missing" ones.

FWIW, you can actually do the same with kubectl, like kubectl delete --all dsci,dsc,featuretrackers would still probably report some errors.

zdtsw · 2024-08-28T08:04:08Z

here are two different ways of uninstallations:
1. done by CLI/configmap/manually one resource after another, this is mostly handled by Operator

2. the console way, with a click and assume all will be gone without error.
the problem here lies in the way how 1) and 2) are not doing the same thing. in (1) operator are trying to be smart and deleting with ownership + FTer are created/removed by DSC/DSCI logic + DSC and DSCI have a hidden depedency (webhook) in (2) console only delete all operands in order/non-order/which order?
but as you all have said abvoe ^, when console first snapshot all operands and try to delete all of them, some of these are already gone due to operator's intelligence . so the error catch all these "missing" ones.
FWIW, you can actually do the same with kubectl, like kubectl delete --all dsci,dsc,featuretrackers would still probably report some errors.

mostly likely, yes.
but also this is a race condition problem.
e.g i have DSCI created and its FTer, but no DSC at all.
I wont get any error by "kubectl delete" or from "uninstall operator" UI.
and this can also happen if i have only one/few component enabled in DSC, so the deletion of DSC CR is much shorter. before DSCI deletion starts.

ykaliuta · 2024-08-28T09:15:04Z

My 2 cents:

since the two resources are independent, the operator should be able to handle the case where only one exists (i.e. I should be able to create the DSC first and after the DSCI), as consequence it should be able to handle the case where one of the resource is deleted (i.e. if the DSC is being deleted, it should probably ignore the fact that the DSCI is not there)

I doubt it's an option: if DSCI is deleted with its Features, components probably will fail. Some of them I remember make some decisions on initialization, so the cluster will be in a pretty undefined state. Correct me if I'm wrong.

if the point above is not possible, then the two resource should probably be merged in a single one as properly pointed out by @adelton here

Would it be possible to merge DSCI configuration into DSC without changing API version? Probably, yes, it just adds fields and live should be possible.

if we still want to a strong relation, we can use finalizer to either prevent the DSCI to be deleted till the DSC is still there or marked for deletion (mimic background deletion), or even trigger the re-creation of the DSCI but I would push toward option 1 and 2 above

lburgazzoli · 2024-08-28T09:41:43Z

My 2 cents:

since the two resources are independent, the operator should be able to handle the case where only one exists (i.e. I should be able to create the DSC first and after the DSCI), as consequence it should be able to handle the case where one of the resource is deleted (i.e. if the DSC is being deleted, it should probably ignore the fact that the DSCI is not there)

I doubt it's an option: if DSCI is deleted with its Features, components probably will fail. Some of them I remember make some decisions on initialization, so the cluster will be in a pretty undefined state. Correct me if I'm wrong.

No you are not wrong and we may need to work to improve components failure handling.
But I guess something similar would happen in case the DSCI is amended in a way that requires the cluster and the components to be reconfigured after the first initialization.

My main point, is that we should try to make the operator resilient to failure and be able to gracefully handle a degraded service.

if the point above is not possible, then the two resource should probably be merged in a single one as properly pointed out by @adelton here

Would it be possible to merge DSCI configuration into DSC without changing API version? Probably, yes, it just adds fields and live should be possible.

As long as the change is backward compatible, then yes is is possible.

Sara4994 · 2024-08-28T14:35:55Z

My 2 cents

@bartoszmajsak Regarding the Feature Tracker error, Will merging two resources (DSC & DSCI) solve/fix the Feature tracker related errors while uninstalling via UI? while reproduction the issue as I can see and you have also mentioned the same here, the operator removes or cleans up FTs quickly whereas openshift console is trying remove the operands which are already removed. Could there be a way to notify/signal web console from the operator immediately after the FTs operands are removed? so that web console doesnt look for them again. @zdtsw @ykaliuta @VaishnaviHire

lburgazzoli · 2024-08-29T09:11:28Z

My 2 cents

@bartoszmajsak Regarding the Feature Tracker error, Will merging two resources (DSC & DSCI) solve/fix the Feature tracker related errors while uninstalling via UI? while reproduction the issue as I can see and you have also mentioned the same here, the operator removes or cleans up FTs quickly whereas openshift console is trying remove the operands which are already removed. Could there be a way to notify/signal web console from the operator immediately after the FTs operands are removed? so that web console doesnt look for them again. @zdtsw @ykaliuta @VaishnaviHire

I think if the FTs are listed in the operators.operatorframework.io/internal-objects annotation, then they are ignored when uninstalling an operator and should be managed by the operator itself.

zdtsw · 2024-08-29T09:14:59Z

My 2 cents

@bartoszmajsak Regarding the Feature Tracker error, Will merging two resources (DSC & DSCI) solve/fix the Feature tracker related errors while uninstalling via UI? while reproduction the issue as I can see and you have also mentioned the same here, the operator removes or cleans up FTs quickly whereas openshift console is trying remove the operands which are already removed. Could there be a way to notify/signal web console from the operator immediately after the FTs operands are removed? so that web console doesnt look for them again. @zdtsw @ykaliuta @VaishnaviHire

I think if the FTs are listed in the operators.operatorframework.io/internal-objects annotation, then they are ignored when uninstalling an operator and should be managed by the operator itself.

i did a test with this proposal (fixed with "") and from UI, it does not show FTer anymore after installed our Operator.
But when click "Uninstall Operator" still it shows all FTer as Operands

Sara4994 · 2024-08-29T09:16:56Z

Even

@lburgazzoli Well i tried with that annotation, though FT didnt show up in the UI(list of operands) , but while uninstalling i still see error releated to FT. some how it is still considering FT while uninstalling

Sara4994 · 2024-08-29T09:17:45Z

My 2 cents

@bartoszmajsak Regarding the Feature Tracker error, Will merging two resources (DSC & DSCI) solve/fix the Feature tracker related errors while uninstalling via UI? while reproduction the issue as I can see and you have also mentioned the same here, the operator removes or cleans up FTs quickly whereas openshift console is trying remove the operands which are already removed. Could there be a way to notify/signal web console from the operator immediately after the FTs operands are removed? so that web console doesnt look for them again. @zdtsw @ykaliuta @VaishnaviHire

I think if the FTs are listed in the operators.operatorframework.io/internal-objects annotation, then they are ignored when uninstalling an operator and should be managed by the operator itself.

i did a test with this proposal (fixed with "") and from UI, it does not show FTer anymore after installed our Operator. But when click "Uninstall Operator" still it shows all FTer as Operands

+1

lburgazzoli · 2024-08-29T09:20:13Z

Even

@bartoszmajsak Well i tried with that annotation, though FT didnt show up in the UI(list of operands) , but while uninstalling i still see error releated to FT. some how it is still considering FT while uninstalling

ok, so we need to dig more into this

Introduces ability to define owner reference for FeatureTracker (internal custom resource). This enables garbage collection of resources belonging to the given Feature when its owner such as DSC or DSCI has been removed. When Features are grouped through means of `FeatureHandler`s, the ownership is defined under the hood, linking it to DSCI or DSC, depending on the type used. In addition, this PR simplifies how Feature relies on `client.Client` instance. Instead of creating its own instance it expects to have one passed as part of the builder call chain. It creates a default one otherwise. With that we can rely on a client configured for controllers with the right schemas, caching rules etc. As a consequence the change of retry logic for status needed to adjusted. Besides a conflict it needs to be triggered on NotFound error, as shared client's cache might not have yet an instance of FeatureTracker created just before the condition is reported. The reason why it was working before is the fact that Feature has its own simple client instance w/o caching. > [!IMPORTANT] > > With DSCI and DSC becoming owners of particular FeatureTrackers > `func (c *Component) Cleanup` called in finalizer block becomes > somewhat speculative. The only use case where custom function is > invoked is unpatching SMCP authorization provider. This was based > on early assumption that we might want to plug into customer's > existing SMCP instance. It's unclear to me if this is still long-term > thinking so we might want to revisit need for this function. > > From the completness point of view, if we allow to create/manipulate > resources programatically as part of the Feature DSL, we should be able > to register cleanup counter-part functions, especially if we cannot simply > remove them (as this is handled through ownership of associated > FeatureTracker). > > There is, however, a need to perform cleanup when particular > component is "Removed" (not managed anymore). Right now this is > handled inside its Reconcile function. Fixes [RHOAIENG-8629](https://issues.redhat.com/browse/RHOAIENG-8629) Relates to opendatahub-io#1192 (comment) Co-authored-by: Bartosz Majsak <bartosz.majsak@gmail.com>

…1228) * feat(tracker): feature tracker can have ownerReferences Introduces ability to define owner reference for FeatureTracker (internal custom resource). This enables garbage collection of resources belonging to the given Feature when its owner such as DSC or DSCI has been removed. When Features are grouped through means of `FeatureHandler`s, the ownership is defined under the hood, linking it to DSCI or DSC, depending on the type used. In addition, this PR simplifies how Feature relies on `client.Client` instance. Instead of creating its own instance it expects to have one passed as part of the builder call chain. It creates a default one otherwise. With that we can rely on a client configured for controllers with the right schemas, caching rules etc. As a consequence the change of retry logic for status needed to adjusted. Besides a conflict it needs to be triggered on NotFound error, as shared client's cache might not have yet an instance of FeatureTracker created just before the condition is reported. The reason why it was working before is the fact that Feature has its own simple client instance w/o caching. > [!IMPORTANT] > > With DSCI and DSC becoming owners of particular FeatureTrackers > `func (c *Component) Cleanup` called in finalizer block becomes > somewhat speculative. The only use case where custom function is > invoked is unpatching SMCP authorization provider. This was based > on early assumption that we might want to plug into customer's > existing SMCP instance. It's unclear to me if this is still long-term > thinking so we might want to revisit need for this function. > > From the completness point of view, if we allow to create/manipulate > resources programatically as part of the Feature DSL, we should be able > to register cleanup counter-part functions, especially if we cannot simply > remove them (as this is handled through ownership of associated > FeatureTracker). > > There is, however, a need to perform cleanup when particular > component is "Removed" (not managed anymore). Right now this is > handled inside its Reconcile function. Fixes [RHOAIENG-8629](https://issues.redhat.com/browse/RHOAIENG-8629) Relates to #1192 (comment) Co-authored-by: Bartosz Majsak <bartosz.majsak@gmail.com> * fix: no need to return dsci --------- Co-authored-by: Cameron Garrison <cgarriso@redhat.com>

…pendatahub-io#1228) * feat(tracker): feature tracker can have ownerReferences Introduces ability to define owner reference for FeatureTracker (internal custom resource). This enables garbage collection of resources belonging to the given Feature when its owner such as DSC or DSCI has been removed. When Features are grouped through means of `FeatureHandler`s, the ownership is defined under the hood, linking it to DSCI or DSC, depending on the type used. In addition, this PR simplifies how Feature relies on `client.Client` instance. Instead of creating its own instance it expects to have one passed as part of the builder call chain. It creates a default one otherwise. With that we can rely on a client configured for controllers with the right schemas, caching rules etc. As a consequence the change of retry logic for status needed to adjusted. Besides a conflict it needs to be triggered on NotFound error, as shared client's cache might not have yet an instance of FeatureTracker created just before the condition is reported. The reason why it was working before is the fact that Feature has its own simple client instance w/o caching. > [!IMPORTANT] > > With DSCI and DSC becoming owners of particular FeatureTrackers > `func (c *Component) Cleanup` called in finalizer block becomes > somewhat speculative. The only use case where custom function is > invoked is unpatching SMCP authorization provider. This was based > on early assumption that we might want to plug into customer's > existing SMCP instance. It's unclear to me if this is still long-term > thinking so we might want to revisit need for this function. > > From the completness point of view, if we allow to create/manipulate > resources programatically as part of the Feature DSL, we should be able > to register cleanup counter-part functions, especially if we cannot simply > remove them (as this is handled through ownership of associated > FeatureTracker). > > There is, however, a need to perform cleanup when particular > component is "Removed" (not managed anymore). Right now this is > handled inside its Reconcile function. Fixes [RHOAIENG-8629](https://issues.redhat.com/browse/RHOAIENG-8629) Relates to opendatahub-io#1192 (comment) Co-authored-by: Bartosz Majsak <bartosz.majsak@gmail.com> * fix: no need to return dsci --------- Co-authored-by: Cameron Garrison <cgarriso@redhat.com> (cherry picked from commit 10dd554)

…1228) * feat(tracker): feature tracker can have ownerReferences Introduces ability to define owner reference for FeatureTracker (internal custom resource). This enables garbage collection of resources belonging to the given Feature when its owner such as DSC or DSCI has been removed. When Features are grouped through means of `FeatureHandler`s, the ownership is defined under the hood, linking it to DSCI or DSC, depending on the type used. In addition, this PR simplifies how Feature relies on `client.Client` instance. Instead of creating its own instance it expects to have one passed as part of the builder call chain. It creates a default one otherwise. With that we can rely on a client configured for controllers with the right schemas, caching rules etc. As a consequence the change of retry logic for status needed to adjusted. Besides a conflict it needs to be triggered on NotFound error, as shared client's cache might not have yet an instance of FeatureTracker created just before the condition is reported. The reason why it was working before is the fact that Feature has its own simple client instance w/o caching. > [!IMPORTANT] > > With DSCI and DSC becoming owners of particular FeatureTrackers > `func (c *Component) Cleanup` called in finalizer block becomes > somewhat speculative. The only use case where custom function is > invoked is unpatching SMCP authorization provider. This was based > on early assumption that we might want to plug into customer's > existing SMCP instance. It's unclear to me if this is still long-term > thinking so we might want to revisit need for this function. > > From the completness point of view, if we allow to create/manipulate > resources programatically as part of the Feature DSL, we should be able > to register cleanup counter-part functions, especially if we cannot simply > remove them (as this is handled through ownership of associated > FeatureTracker). > > There is, however, a need to perform cleanup when particular > component is "Removed" (not managed anymore). Right now this is > handled inside its Reconcile function. Fixes [RHOAIENG-8629](https://issues.redhat.com/browse/RHOAIENG-8629) Relates to opendatahub-io#1192 (comment) Co-authored-by: Bartosz Majsak <bartosz.majsak@gmail.com> * fix: no need to return dsci --------- Co-authored-by: Cameron Garrison <cgarriso@redhat.com> (cherry picked from commit 10dd554)

…flux/component-updates/odh-trustyai-service-operator-v2-17 chore(deps): update odh-trustyai-service-operator-v2-17 to 6b0de77

[RHOAIENG-4485] Fix feature tracker errors while uninstalling operator

40b2be5

openshift-ci bot added the do-not-merge/work-in-progress label Aug 20, 2024

VaishnaviHire reviewed Aug 28, 2024

View reviewed changes

bartoszmajsak mentioned this pull request Sep 10, 2024

feat(tracker): allows to define ownerReferences for FeatureTrackers #1228

Merged

5 tasks

Sara4994 closed this Jan 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: [RHOAIENG-4485] Fix feature tracker errors while uninstalling operator #1192

WIP: [RHOAIENG-4485] Fix feature tracker errors while uninstalling operator #1192

Sara4994 commented Aug 20, 2024

openshift-ci bot commented Aug 20, 2024

openshift-ci bot commented Aug 20, 2024

VaishnaviHire Aug 28, 2024

VaishnaviHire commented Aug 28, 2024 •

edited by bartoszmajsak

Loading

bartoszmajsak commented Aug 28, 2024 •

edited

Loading

bartoszmajsak commented Aug 28, 2024 •

edited

Loading

ykaliuta commented Aug 28, 2024

bartoszmajsak commented Aug 28, 2024

ykaliuta commented Aug 28, 2024

zdtsw commented Aug 28, 2024

lburgazzoli commented Aug 28, 2024 •

edited

Loading

lburgazzoli commented Aug 28, 2024

zdtsw commented Aug 28, 2024

ykaliuta commented Aug 28, 2024

lburgazzoli commented Aug 28, 2024

Sara4994 commented Aug 28, 2024 •

edited

Loading

lburgazzoli commented Aug 29, 2024

zdtsw commented Aug 29, 2024

Sara4994 commented Aug 29, 2024 •

edited

Loading

Sara4994 commented Aug 29, 2024

lburgazzoli commented Aug 29, 2024

WIP: [RHOAIENG-4485] Fix feature tracker errors while uninstalling operator #1192

WIP: [RHOAIENG-4485] Fix feature tracker errors while uninstalling operator #1192

Conversation

Sara4994 commented Aug 20, 2024

Description

How Has This Been Tested?

Screenshot or short clip

Merge criteria

openshift-ci bot commented Aug 20, 2024

openshift-ci bot commented Aug 20, 2024

VaishnaviHire Aug 28, 2024

Choose a reason for hiding this comment

VaishnaviHire commented Aug 28, 2024 • edited by bartoszmajsak Loading

bartoszmajsak commented Aug 28, 2024 • edited Loading

bartoszmajsak commented Aug 28, 2024 • edited Loading

ykaliuta commented Aug 28, 2024

bartoszmajsak commented Aug 28, 2024

ykaliuta commented Aug 28, 2024

zdtsw commented Aug 28, 2024

lburgazzoli commented Aug 28, 2024 • edited Loading

lburgazzoli commented Aug 28, 2024

zdtsw commented Aug 28, 2024

ykaliuta commented Aug 28, 2024

lburgazzoli commented Aug 28, 2024

Sara4994 commented Aug 28, 2024 • edited Loading

lburgazzoli commented Aug 29, 2024

zdtsw commented Aug 29, 2024

Sara4994 commented Aug 29, 2024 • edited Loading

Sara4994 commented Aug 29, 2024

lburgazzoli commented Aug 29, 2024

VaishnaviHire commented Aug 28, 2024 •

edited by bartoszmajsak

Loading

bartoszmajsak commented Aug 28, 2024 •

edited

Loading

bartoszmajsak commented Aug 28, 2024 •

edited

Loading

lburgazzoli commented Aug 28, 2024 •

edited

Loading

Sara4994 commented Aug 28, 2024 •

edited

Loading

Sara4994 commented Aug 29, 2024 •

edited

Loading