Chain Event Types Removal Finale #3309

timolegros · 2023-04-01T04:55:53Z

Link to Issue

Closes: #2949

Description of Changes

This PR is nearly identical to #3044 with the only changes being to redo some of the changes that we're reverted in the last attempt at deploying chain-event-types removal.

Hugely reduces Notifications table size (5,426,564 -> 150,834 rows). DB total size reduced from 10.1GB to 2.4GB. This PR also massively reduces the number of notifications and subscriptions created giving us much more time to properly redesign notifications to facilitate scaling. We should also see a considerable performance gain for pg:copy operations or full DB dumps.

Test Plan

Deployed to Frick

Deployment Plan

Turn off CE subscribers - wait a couple of seconds for existing queued messages to be processed.
Turn off CW consumer
Put the commonwealthapp in maintenance mode
Put chain-events in maintenance mode
Deploy CW and CE (any order)
Turn back on the CW consumer then the CE subscribers once deployments are over
Turn off maintenance mode on CE first then CW

Other Considerations

# Conflicts: # packages/chain-events/services/ChainEventsConsumer/ChainEventHandlers/migration.ts # packages/commonwealth/client/scripts/controllers/server/chain_entities.ts

# Conflicts: # packages/commonwealth/server/models/subscription.ts

…hains

…craper imports

…node

# Conflicts: # packages/chain-events/services/ChainEventsConsumer/ChainEventHandlers/entityArchival.ts # packages/chain-events/services/ChainEventsConsumer/ChainEventHandlers/migration.ts # packages/chain-events/services/ChainEventsConsumer/ChainEventHandlers/notification.ts # packages/chain-events/services/ChainEventsConsumer/ChainEventHandlers/storage.ts # packages/chain-events/services/ChainEventsConsumer/republishMessages.ts # packages/chain-events/services/ChainSubscriber/util.ts # packages/chain-events/services/database/database.ts # packages/chain-events/services/database/models/chain_event.ts # packages/chain-events/services/database/models/chain_event_type.ts # packages/chain-events/src/interfaces.ts # packages/chain-events/test/integration/service-tests/chainEventsConsumer.test.ts # packages/common-common/src/rabbitmq/RabbitMQConfig.ts # packages/common-common/src/rabbitmq/rabbitMQController.ts # packages/common-common/src/rabbitmq/types/chainEventNotificationsCUD.ts # packages/common-common/src/rabbitmq/types/chainEventTypeCUD.ts # packages/common-common/src/rabbitmq/types/index.ts # packages/commonwealth/client/scripts/controllers/server/chain_entities.ts # packages/commonwealth/client/scripts/controllers/server/notifications.ts # packages/commonwealth/client/scripts/controllers/server/socket/chainEventsNs.ts # packages/commonwealth/client/scripts/models/ChainEvent.ts # packages/commonwealth/client/scripts/models/DashboardActivityNotification.ts # packages/commonwealth/client/scripts/models/Notification.ts # packages/commonwealth/client/scripts/state.ts # packages/commonwealth/server/CommonwealthConsumer/CommonwealthConsumer.ts # packages/commonwealth/server/CommonwealthConsumer/messageProcessors/chainEventTypeCUDQueue.ts # packages/commonwealth/server/models.ts # packages/commonwealth/server/models/chain_event_type.ts # packages/commonwealth/server/models/notification.ts # packages/commonwealth/server/models/subscription.ts # packages/commonwealth/server/routes/deleteChain.ts # packages/commonwealth/server/routes/getSubscribedChains.ts # packages/commonwealth/server/routes/subscription/createSubscription.ts # packages/commonwealth/server/scripts/enforceDataConsistency.ts # packages/commonwealth/server/socket/chainEventsNs.ts # packages/commonwealth/server/util/emitNotifications.ts # packages/commonwealth/test/integration/enforceDataConsistency.spec.ts # packages/commonwealth/test/systemTests/consumer.test.ts

# Conflicts: # packages/chain-events/services/ChainEventsConsumer/republishMessages.ts

# Conflicts: # packages/commonwealth/server/migrations/20221226212006-remove-unused-entity-id.js

* from event-types client side * move chain + network to event -> working chain activity dashboard * switch ce commands to ts-node-dev * chain + network added to event in ChainEventConsumer * save * filter out non entity creation/completion events for notifications * remove EventTypeCUD RabbitMQ types + config + pub/sub calls * remove ChainEventType model * remove chain-event-type migration * CET model removal fixes * CW remove CET migration * CW remove CET model * CET removal migrations + model update * migration fixes * migration fixes * chain subscriptions + chain-event notification settings + subscribedChains * add full chain to createSubscription route result + fix UI * turn off sourceMap, use proper tsconfig when building ce, and fixed scraper imports * add type check to CI + fix type errors * add type check to CI + fix type errors + remove format in CE repo * set allowJs: false in tsconfig so sourceMap files don't interrupt ts-node * RabbitMQ merge fixes * Make CI lint only the changes in the branch rather than everything * fix tsconfig + run prettier * merge fixes * merge fixes * discoverReconnectRange fix * consumer erc20 notification handler bug fix * load-env-var fix * Procfile fix * master merge fixes * fix chain-event notifications * fix chain-event email notifications * format * format * CE css fix * remove subscriptions count for chain-events * fix getSubscribedChains call * notification settings page fixes + remove unnecessary logs * format * delete script * migration script fix * prettier format * publishCustomRabbitMQMessage.ts improvements * prettier format * more script improvements for testing/QA * add dump file to gitignore * 8 min local * 15 seconds local * add fkey from NR to Sub * format * eslint + format + .prettierignore update * Ensure CET Removal Backwards Compatibility. (#2765) * Use CE proxy for events too. * Ensure backwards compatibility for deployment + fix link to chain. * Prettier fixes. * Remove log. * fixed type errors * fix lint * fix lint * format * small logging fix * migration log improvement * ce procfile fix * add missing index to notif table * Subscriptions pkey + not null * fix type * test improved CE queries * Procfile fix * heroku test * heroku test 2 * cleaup * redo migration * redo migration CE + network migration fix * fix order of CE migrations * logging for debugging 503 * error handling CE - 503 debugging * event route - 503 debugging * optimize event route db query * prettier --------- Co-authored-by: Jake Naviasky <jake@commonwealth.im> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Muon Shot <120686579+CowMuon@users.noreply.github.com> (cherry picked from commit 6632d14)

* Chain Event Types Removal Finale (#3309) * from event-types client side * move chain + network to event -> working chain activity dashboard * switch ce commands to ts-node-dev * chain + network added to event in ChainEventConsumer * save * filter out non entity creation/completion events for notifications * remove EventTypeCUD RabbitMQ types + config + pub/sub calls * remove ChainEventType model * remove chain-event-type migration * CET model removal fixes * CW remove CET migration * CW remove CET model * CET removal migrations + model update * migration fixes * migration fixes * chain subscriptions + chain-event notification settings + subscribedChains * add full chain to createSubscription route result + fix UI * turn off sourceMap, use proper tsconfig when building ce, and fixed scraper imports * add type check to CI + fix type errors * add type check to CI + fix type errors + remove format in CE repo * set allowJs: false in tsconfig so sourceMap files don't interrupt ts-node * RabbitMQ merge fixes * Make CI lint only the changes in the branch rather than everything * fix tsconfig + run prettier * merge fixes * merge fixes * discoverReconnectRange fix * consumer erc20 notification handler bug fix * load-env-var fix * Procfile fix * master merge fixes * fix chain-event notifications * fix chain-event email notifications * format * format * CE css fix * remove subscriptions count for chain-events * fix getSubscribedChains call * notification settings page fixes + remove unnecessary logs * format * delete script * migration script fix * prettier format * publishCustomRabbitMQMessage.ts improvements * prettier format * more script improvements for testing/QA * add dump file to gitignore * 8 min local * 15 seconds local * add fkey from NR to Sub * format * eslint + format + .prettierignore update * Ensure CET Removal Backwards Compatibility. (#2765) * Use CE proxy for events too. * Ensure backwards compatibility for deployment + fix link to chain. * Prettier fixes. * Remove log. * fixed type errors * fix lint * fix lint * format * small logging fix * migration log improvement * ce procfile fix * add missing index to notif table * Subscriptions pkey + not null * fix type * test improved CE queries * Procfile fix * heroku test * heroku test 2 * cleaup * redo migration * redo migration CE + network migration fix * fix order of CE migrations * logging for debugging 503 * error handling CE - 503 debugging * event route - 503 debugging * optimize event route db query * prettier --------- Co-authored-by: Jake Naviasky <jake@commonwealth.im> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Muon Shot <120686579+CowMuon@users.noreply.github.com> (cherry picked from commit 6632d14) * merge CET removal fixes * add key to divs in notification settings * redraw -> forceRerender * standardized * Fix proposals page for contracts. --------- Co-authored-by: Jake Naviasky <jake@commonwealth.im>

timolegros added 30 commits December 6, 2022 03:36

from event-types client side

76d9437

move chain + network to event -> working chain activity dashboard

40ef656

switch ce commands to ts-node-dev

eb00def

chain + network added to event in ChainEventConsumer

8aa9c25

save

cbaae3b

filter out non entity creation/completion events for notifications

df44f65

Merge branch 'master' into tim.remove-event-types

a4053f4

# Conflicts: # packages/chain-events/services/ChainEventsConsumer/ChainEventHandlers/migration.ts # packages/commonwealth/client/scripts/controllers/server/chain_entities.ts

remove EventTypeCUD RabbitMQ types + config + pub/sub calls

2587dd9

remove ChainEventType model

8907f7b

remove chain-event-type migration

ee05e1f

CET model removal fixes

0aac144

CW remove CET migration

d2230f6

CW remove CET model

66feb57

CET removal migrations + model update

892ccba

Merge branch 'master' into tim.remove-event-types

374e1e0

# Conflicts: # packages/commonwealth/server/models/subscription.ts

migration fixes

05bd075

migration fixes

b503daa

chain subscriptions + chain-event notification settings + subscribedC…

f4d925e

…hains

add full chain to createSubscription route result + fix UI

3fcd13a

turn off sourceMap, use proper tsconfig when building ce, and fixed s…

c4beca2

…craper imports

add type check to CI + fix type errors

11a836d

add type check to CI + fix type errors + remove format in CE repo

a5eb796

Merge branch 'master' into tim.fix_ce

6b036f4

set allowJs: false in tsconfig so sourceMap files don't interrupt ts-…

0499dab

…node

RabbitMQ merge fixes

8466b5b

Make CI lint only the changes in the branch rather than everything

bd5a5b9

fix tsconfig + run prettier

f6e44b4

merge fixes

d7ca704

Merge branch 'tim.fix_ce' into tim.remove-event-types

144699b

# Conflicts: # packages/chain-events/services/ChainEventsConsumer/republishMessages.ts

timolegros added 7 commits March 16, 2023 01:57

heroku test 2

1e521a5

cleaup

5a22fb3

Merge branch 'master' into tim.remove-event-types-2

d39c947

# Conflicts: # packages/commonwealth/server/migrations/20221226212006-remove-unused-entity-id.js

Merge branch 'master' into tim.remove-event-types-2

74bf196

redo migration

39d9195

redo migration CE + network migration fix

fb90609

fix order of CE migrations

6298171

timolegros temporarily deployed to chain-events-staging April 1, 2023 05:19 Inactive

timolegros temporarily deployed to commonwealth-staging April 2, 2023 04:10 Inactive

timolegros temporarily deployed to chain-events-staging April 2, 2023 04:11 Inactive

logging for debugging 503

4b071c8

timolegros temporarily deployed to chain-events-staging April 2, 2023 05:30 Inactive

error handling CE - 503 debugging

4a1ca97

timolegros temporarily deployed to chain-events-staging April 2, 2023 05:44 Inactive

event route - 503 debugging

d1784c9

timolegros temporarily deployed to chain-events-staging April 2, 2023 05:54 Inactive

optimize event route db query

87a09ad

timolegros temporarily deployed to chain-events-staging April 2, 2023 06:31 Inactive

timolegros marked this pull request as ready for review April 2, 2023 07:26

timolegros requested review from jnaviask, zakhap, dillchen and CowMuon as code owners April 2, 2023 07:26

prettier

637a28c

jnaviask approved these changes Apr 2, 2023

View reviewed changes

jnaviask merged commit 6632d14 into master Apr 2, 2023

jnaviask deleted the tim.remove-event-types-2 branch April 2, 2023 15:11

timolegros mentioned this pull request Apr 3, 2023

Fix Chain-Events Node Memory Settings #3230

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chain Event Types Removal Finale #3309

Chain Event Types Removal Finale #3309

timolegros commented Apr 1, 2023 •

edited

Loading

Chain Event Types Removal Finale #3309

Chain Event Types Removal Finale #3309

Conversation

timolegros commented Apr 1, 2023 • edited Loading

Link to Issue

Description of Changes

Test Plan

Deployment Plan

Other Considerations

timolegros commented Apr 1, 2023 •

edited

Loading