Bug EDR API use creating space issue for Postgres Pod in Kubernetes

[sachinargade123]

Bug Report

We have started using EDR API for Contract negotiation, Data transfer process initiate and download data from provider. After successfully used EDR API in 11-12 hours we started getting space issue with Postgres pod then EDC control plane not worked as expected.

Thanks in advance.

Expected Behavior

• It should not occupy 'MB' space for record counts of 200-500 or 1k.

Observed Behavior

EDC control plane issues because of database space reached to allocated size

Steps to Reproduce

Steps to reproduce the behavior:

1. Have an EDC with PostgreSQL persistence.
2. Have to use EDR api to contract negotiate and data download
3. Control plane should keep refreshing EDR token then it will reached space issue

Context Information

• EDC: tractusx/edc-controlplane-postgresql-hashicorp-vault:0.5.1
• OS: Kubernetes pod
• Postgres: bitnami/postgresql:15.1.0-debian-11-r12 - attached 1GB pvc volume for it.

Detailed Description

We were able to use EDR api for 5 data offers, was able to see successfully EDR authorised token for download data from data provider data-plane directly but after 10-12 hours we notice some space issue with EDC Postgres pod.
After debug the space issue we encounter the 'edc_transfer_process' table which occupying more space with respect to number of records please find screenshots. Also, notice there is background running process in EDC control plane to refresh EDR token.

Also In EDC control plane logs we notice data transfer not marked as COMPLETED its remain STARTED status.

#EDC logs

DEBUG 2023-10-30T12:52:06.086873511 TransferProcess 220ece09-3c92-4aed-92dd-cbeadf34e655 is now in state STARTED WARNING 2023-10-30T12:52:06.653144905 No checker found for process 1252b0f7-4c7a-4bf3-b2e6-e99e2e3c33b0. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:06.744451444 No checker found for process 1d6dc495-11d2-45c2-b929-828295a4b539. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:06.744471852 No checker found for process 206b817a-7130-439b-9545-0cef1313f59a. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:07.055185392 No checker found for process 2a1a323b-c180-4d4a-a3c3-56c707e1f261. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:07.667745137 No checker found for process 4977b847-b579-47c4-9c78-bd0c1d7a268f. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:11.30765068 No checker found for process 635b1e69-1c7a-419f-aeb2-d722ef9ca007. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:14.240320055 No checker found for process 1ec3e4f2-d188-4294-bbd3-e00ac7b0577d. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:14.812176233 No checker found for process 2cc5a859-76b7-4be7-902d-a411398e5a4f. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:14.928739067 No checker found for process 323c39b4-f7bc-427a-a257-e4316e8563d2. The process will not advance to the COMPLETED state. WARNING 2023-10-30T12:52:14.928758864 No checker found for process 33fe2b79-c151-4a5b-80d9-b864dd338593. The process will not advance to the COMPLETED state.

Could you please guide us,

• Why this particular table have occupying more space with respect to record count?
• How to avoid the increasing table size?
• How to avoid EDC control warning logs for ''No checker found for process

Possible Implementation

1. Instead keep refreshing EDR token, provide way to get refresh token on demand.
2. Need to clean up process to delete expired records from 'edc_transfer_process'.

[github-actions]

Thanks for your contribution 🔥 We will take a look asap 🚀

[wolf4ood]

Hi @sachinargade123

This has already been fixed in EDC 0.3.1 here #3479

and it will be in tractusx-edc 0.6.x (currently 0.6.0-rc1 available).

Also 'No checker found for process has been resolved

Thanks

[sachinargade123]

Thanks for update!!

[MehranRoshandel]

We need the bug fixing in the tractusx-edc 0.5.x