[ENH] Add a script to generate csv files #46
Conversation
First, convert bson to json using mongodb's tool. Second, use this script to generate csv tables.
|
|
||
| data = defaultdict(emptylist, {}) | ||
| for entry in dictlist: | ||
| md5sum = entry['provenance']['md5sum'] |
There was a problem hiding this comment.
dumps from the rating collection don't have a provenance field, md5sum should exist at the top level of the object.
There was a problem hiding this comment.
Yep, you need to use the flag --no-dedup for those. Probably the flag could use a renaming.
| import json | ||
|
|
||
| import numpy as np | ||
| import pandas as pd |
There was a problem hiding this comment.
pandas and numpy aren't requirements for eve, maybe add them to the requirements.txt so the main docker container can run the tool? Otherwise I'm not opposed to a requirements file for the tools directory since it seems it might be used outside the containers.
There was a problem hiding this comment.
Yeah, I see this more as a place to have the script, not as a dependency.
|
You know the dependencies and how to call it, I'm merging. |
First, convert bson to json using mongodb's tool. Second, use this script to generate csv tables.