Skip to content
This repository has been archived by the owner on Jan 29, 2024. It is now read-only.

Implement bbs_database upload-elasticsearch #620

Closed
3 of 4 tasks
FrancescoCasalegno opened this issue Aug 30, 2022 · 0 comments · Fixed by #633
Closed
3 of 4 tasks

Implement bbs_database upload-elasticsearch #620

FrancescoCasalegno opened this issue Aug 30, 2022 · 0 comments · Fixed by #633
Labels
🗄️ database Creation and maintenance of a database of scientific literature

Comments

@FrancescoCasalegno
Copy link
Contributor

FrancescoCasalegno commented Aug 30, 2022

Context

  • In our bbs_database commands implementing the literature ETL pipeline, we currently have a subcommand add for uploading to the MySQL database.
  • But since we now want to use Elasticsearch instead of a SQL db, we need a command that allows to upload to Elasticsearch.

Actions

  • Is upload-elasticsearch a good name, or can we find anything better? upload-es? add-elasticsearch? ...
  • Implement command upload-elasticsearch to upload the JSON parse paragraphs + article metadata in the respective Elasticsearch tables.
  • Notice that we decided to upload paragraphs, not sentences (different from what done in the past!).
  • The article metadata uploaded on Elasticsearch should include topics info.

Dependencies

@FrancescoCasalegno FrancescoCasalegno added the 🗄️ database Creation and maintenance of a database of scientific literature label Aug 30, 2022
@drsantos89 drsantos89 mentioned this issue Sep 20, 2022
6 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
🗄️ database Creation and maintenance of a database of scientific literature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant