Skip to content

Commit

Permalink
feat: replicator stack (#57)
Browse files Browse the repository at this point in the history
Adds replicator stack by moving its content out of CARPARK on the
direction to #50

Implementation details:
- replicator-stack has 2 queues, one to replicate CARs and other to
replicate .idx files. At first I was attempting to do both in one single
queue. Ended up moving to 2 queues because I would need to have handlers
to identify origin bucket and have a hardcoded mapping where I would see
if it was carpark/satnav and with that select the appropriate CF
destination bucket.
- while my decision for 2 queues was based on the above, it might
actually be the best approach to not have queues throttling flow on
copies from independent buckets.
- also important noticing that despite 2 queues, there is only one
handler function code. The lambda code is the same, but it is injected
with destination BUCKET_NAME according to the queue it is handling.

Closes #54

Co-authored-by: Oli Evans <oli@protocol.ai>
  • Loading branch information
vasco-santos and olizilla authored Dec 1, 2022
1 parent 6cca987 commit b91742d
Show file tree
Hide file tree
Showing 18 changed files with 484 additions and 348 deletions.
9 changes: 2 additions & 7 deletions carpark/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,12 @@
},
"dependencies": {
"@aws-sdk/client-eventbridge": "^3.216.0",
"@aws-sdk/client-s3": "^3.211.0",
"@aws-sdk/client-sqs": "^3.213.0",
"@ipld/car": "^5.0.1",
"cardex": "^1.0.0",
"uint8arrays": "^4.0.2"
"@aws-sdk/client-sqs": "^3.213.0"
},
"devDependencies": {
"@ipld/dag-pb": "^3.0.0",
"@aws-sdk/client-s3": "^3.211.0",
"@serverless-stack/resources": "*",
"ava": "^4.3.3",
"multiformats": "^10.0.2",
"nanoid": "^4.0.0",
"testcontainers": "^8.13.0"
}
Expand Down
163 changes: 0 additions & 163 deletions carpark/replicator.js

This file was deleted.

98 changes: 0 additions & 98 deletions carpark/test/replicator.test.js

This file was deleted.

42 changes: 33 additions & 9 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"console": "sst console",
"lint": "tsc && eslint '**/*.js'",
"clean": "rm -rf docs node_modules pnpm-lock.yaml packages/*/{pnpm-lock.yaml,coverage,.nyc_output,dist,node_modules,.next}",
"test": "npm test -w api -w carpark -w satnav"
"test": "npm test -w api -w carpark -w replicator -w satnav"
},
"devDependencies": {
"@serverless-stack/cli": "^1.18.0",
Expand Down Expand Up @@ -61,6 +61,7 @@
"workspaces": [
"api",
"carpark",
"replicator",
"satnav"
]
}
Loading

0 comments on commit b91742d

Please sign in to comment.