Proposal: Reorganize the codebase

[macobo]

Overview

This PR moves code along the lines proposed in #261. I find it makes it much easier to follow the code flow - what code is being executed in workers, what in main threads and what everywhere.

No real behavioral changes here, just moving files. Some minor changes:

• Split up server.ts into src/main/pluginsServer and src/shared/server
• Renamed celery/worker to src/main/ingestion/celery-queue-worker. This avoids confusion with worker threads.

The eventual goal would be to make src/shared as tiny as possible and move everything else under specific subfolders.

Resolves #47.

Commits

• Move schedule.ts under src/main
• Move queue.ts under src/main
• Move kafka-queue logic under src/main
• Move ingest-event file
• Move web/server under main.ts
• Move plugins code under src/worker
• Move vm code under src/worker
• Move celery worker under src/main
• move process-event under src/worker
• Move src/celery under src/shared
• Move remaining src/ingestion under src/shared
• move utils under src/shared
• Move some top-level code under shared
• Move main pluginsserver code under src/main
• Fix typeerror in tests
• Move createServer under src/shared

Checklist

• Updated Settings section in README.md, if settings are affected
• Jest tests

[Twixes]

I broadly agree. My only two notes are:

1. I think it's a bit awkward that src/celery/worker.ts got pulled out of the celery directory into src/main/ingestion/celery-queue-worker.ts, especially considering that the celery directory used to be a whole separate package. The analogy to the Kafka queue is good though.
2. Most of src/server.ts ended up in src/main/pluginsServer.ts, but not all of it. I think we should keep that file intact instead, as basically it's all PluginsServer.

However in any case I'd wait for @mariusandra's input on this too before merging.

[macobo]

Thanks for the feedback. Both of these were motivated by trying to keep shared/ as small as possible.

1. For server.ts I feel strongly. Basically half of the code there only executed/was available in the main thread and is the defacto start point for the whole control flow. The other half is under shared because that god object is used everywhere. I think the issue here is mostly around naming - is Server the right name for the shared state?

2. For worker.ts I can see it going both ways, I originally thought it was a pure alternative for kafka-queue.ts, but these two behave differently. It just feels a bit awkward having it under shared/ given that it's not used by worker. If it does become a library it will still be easy to extract the code as needed.

[mariusandra]

Regarding these discussed points:

1. I guess it's fine to split up the celery code info folder. I moved the lib into our codebase precisely so we could be flexible with this. (plus for other optimisations...)

2. Totally fine with splitting the "server" and "god" objects from "server.ts". The god object could even be called something like "services.ts" or something, as it basically just hosts connections to all the things we connect to.

[mariusandra]

Approved. Some merge conflicts though.