Add lastAccessed field to map table #296

sorenjohanson · 2024-06-20T07:37:35Z

closes #286

teammapper-backend/src/map/entities/mmpMap.entity.ts

teammapper-backend/src/map/services/maps.service.ts

JannikStreek · 2024-06-20T09:06:50Z

Please don't forget to generate the migration file for this, see https://github.com/b310-digital/teammapper/blob/main/teammapper-backend/README.md.

JannikStreek · 2024-06-20T09:38:36Z

please also add the use of this field to the delete handler in the service (deleteOutdatedMaps), so that we use the lastAccessed field on map as a default and as a backup the lastModified field (existing logic). We can also do this in a new PR?

sorenjohanson · 2024-06-20T09:40:11Z

please also add the use of this field to the delete handler in the service (deleteOutdatedMaps), so that we use the lastAccessed field on map as a default and as a backup the lastModified field (existing logic). We can also do this in a new PR?

I'm working on that right now, it's just that I need to restructure the queries a bit to work with the new field, so it'll take some time.

teammapper-backend/src/map/services/maps.service.spec.ts

teammapper-backend/src/map/controllers/maps.controller.ts

teammapper-backend/src/jobs/seedMapData.job.ts

JannikStreek · 2024-09-11T13:59:34Z

For documentation purposes:

When a map is created, createdAt will be filled, accessedAt will be nil.
When accessed for the first time (after creation, first reload or load by user user of map), accessedAt will be set.
During the migration, old maps will have empty accessedAt and createdAt fields. lastModified will be still available and this field is essentially a createdAt field as the date is not updated after creation.

teammapper-backend/src/map/services/maps.service.ts

JannikStreek · 2024-09-12T12:08:09Z

ok there are actually >300 maps on our server, that do not have any nodes. Interestingly, all these maps are not old but were created in September - so there seems to be a bug not creating the root node or deleting the root node.

But this is a bug for a different issue. However, we should still include this case, that a map does not contain any nodes. => nodeLastUpdatedAt = null AND GREATER(map.lastModified, map.lastAccessed) ...

The following test cases are the most relevant:

Only lastModified is set on a map (either too old or recent enough) with one node that was either modified before and after the lastModified date.
lastModified AND lastAccess are set (one being older than the other) + with one node existing, being modified at a later point and at a newer point in time
A map that has no nodes at all for some reason => only look at lastModified as a reference. I also thought about deleting those directly, but for debugging purposes it makes sense to keep them for some time.

teammapper-backend/src/map/services/maps.service.ts

teammapper-backend/src/map/services/maps.service.spec.ts

JannikStreek · 2024-09-13T13:57:36Z

teammapper-backend/src/map/services/maps.service.spec.ts

@@ -152,6 +208,33 @@ describe('MapsController', () => {
      expect(await nodesRepo.findOne({ where: { id: node.id } })).not.toBeNull()
    })

+    it('deletes a map which has outdated nodes and outdated lastAccess', async () => {


also add a test (if not given, can't find it right now): Outdated modifedAt, recent nodes => do not delete. This is how its working right now.

This is covered by it('does not delete a map that contains a recent node'), where lastAccessed is set to 2019 but the node is set to today using new Date().

JannikStreek · 2024-09-13T13:58:11Z

teammapper-backend/src/map/services/maps.service.ts

@@ -233,19 +239,9 @@ export class MapsService {
        'lastmodifiednode.nodeMapid = map.id'
      )
      .where(
-        // delete all maps that have nodes that were last updated after afterDays
-        "(lastmodifiednode.lastUpdatedAt + (INTERVAL '1 day' * :afterDays)) < :today",
+        "(GREATEST(map.lastAccessed, map.lastModified, lastmodifiednode.lastUpdatedAt) + (INTERVAL '1 day' * :afterDays)) < :today",


thats a cool discovery that this takes more than 2 arguments, makes it way easier 👍

sorenjohanson · 2024-09-16T10:04:19Z

Dates are now set explicitly in every test, in addition to the following format changes:

An old date is always set to 2021-01-01.
A recent date is always set to 2024-09-01.

If we're testing recent dates, system time will always be set exactly to the same date.
On the other hand, if we're testing old dates, system time will always be set to old date + 30 days, which will always result in system time being set to 2021-01-31.

JannikStreek · 2024-09-16T11:39:26Z

looks good, I also tested everything locally with cross running every minute and 1 day as delete threshold. Could not find any problems.

added: lastAccessed field, update when accessing map

a16852e

JannikStreek reviewed Jun 20, 2024

View reviewed changes

teammapper-backend/src/map/entities/mmpMap.entity.ts Show resolved Hide resolved

JannikStreek reviewed Jun 20, 2024

View reviewed changes

teammapper-backend/src/map/services/maps.service.ts Outdated Show resolved Hide resolved

sorenjohanson added 2 commits June 21, 2024 08:50

add: migration for lastAccessed

003f450

change deleteOutdatedMaps() to use lastAccessed

541f365

sorenjohanson marked this pull request as ready for review June 21, 2024 09:28

sorenjohanson added 4 commits June 21, 2024 09:31

fix: lastAccessed for maps.controller.spec.ts

1d4dbb2

Merge branch 'main' into 286-add-lastaccessed-field-to-map-table

b6e75e6

fix: merge latest main, add lastAccessed to missing maps

7e71538

remove host check debug

a45b890

JannikStreek reviewed Jun 24, 2024

View reviewed changes

teammapper-backend/src/map/services/maps.service.spec.ts Show resolved Hide resolved

JannikStreek reviewed Jun 24, 2024

View reviewed changes

teammapper-backend/src/map/controllers/maps.controller.ts Outdated Show resolved Hide resolved

sorenjohanson added 6 commits June 25, 2024 18:51

Merge branch 'main' into 286-add-lastaccessed-field-to-map-table

8c821a0

changed: update lastAccessed first, added: lastAccessed to client maps

334d3f1

fix: add lastAccessed to test maps

73c2c78

fix: add lastAccessed fn and variable for e2e tests

fef5694

refactor: old map deletion, simplify query, pass all test cases

c396f54

fix: disableHostCheck for dev only

bf1759e

JannikStreek reviewed Jul 15, 2024

View reviewed changes

teammapper-backend/src/jobs/seedMapData.job.ts Outdated Show resolved Hide resolved

sorenjohanson added 4 commits July 16, 2024 05:52

Merge branch 'main' into 286-add-lastaccessed-field-to-map-table

96904bc

fix: merge main

35ae1ed

Merge branch 'main' into 286-add-lastaccessed-field-to-map-table

2477602

fix: lastAccessed tests

99b9e3b

JannikStreek reviewed Sep 11, 2024

View reviewed changes

teammapper-backend/src/map/services/maps.service.ts Outdated Show resolved Hide resolved

JannikStreek mentioned this pull request Sep 12, 2024

chore: add locking mechanism for backend to prevent sync conflicts on map import #455

Closed

sorenjohanson added 2 commits September 12, 2024 14:05

add: tests, simplify query

35a0b3a

fix: lint

276faab

JannikStreek reviewed Sep 12, 2024

View reviewed changes

teammapper-backend/src/map/services/maps.service.ts Outdated Show resolved Hide resolved

sorenjohanson added 2 commits September 13, 2024 12:59

Merge branch 'main' into 286-add-lastaccessed-field-to-map-table

5074421

remove comment

40bc6a0

JannikStreek reviewed Sep 13, 2024

View reviewed changes

teammapper-backend/src/map/services/maps.service.spec.ts Show resolved Hide resolved

JannikStreek reviewed Sep 13, 2024

View reviewed changes

fix: set date explicitly in deletion tests

d4d36c9

JannikStreek merged commit 8fbb0fa into b310-digital:main Sep 16, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lastAccessed field to map table #296

Add lastAccessed field to map table #296

sorenjohanson commented Jun 20, 2024

JannikStreek commented Jun 20, 2024

JannikStreek commented Jun 20, 2024

sorenjohanson commented Jun 20, 2024

JannikStreek commented Sep 11, 2024

JannikStreek commented Sep 12, 2024 •

edited

Loading

JannikStreek Sep 13, 2024

sorenjohanson Sep 16, 2024

JannikStreek Sep 13, 2024

sorenjohanson commented Sep 16, 2024 •

edited

Loading

JannikStreek commented Sep 16, 2024

Add lastAccessed field to map table #296

Add lastAccessed field to map table #296

Conversation

sorenjohanson commented Jun 20, 2024

JannikStreek commented Jun 20, 2024

JannikStreek commented Jun 20, 2024

sorenjohanson commented Jun 20, 2024

JannikStreek commented Sep 11, 2024

JannikStreek commented Sep 12, 2024 • edited Loading

JannikStreek Sep 13, 2024

Choose a reason for hiding this comment

sorenjohanson Sep 16, 2024

Choose a reason for hiding this comment

JannikStreek Sep 13, 2024

Choose a reason for hiding this comment

sorenjohanson commented Sep 16, 2024 • edited Loading

JannikStreek commented Sep 16, 2024

JannikStreek commented Sep 12, 2024 •

edited

Loading

sorenjohanson commented Sep 16, 2024 •

edited

Loading