Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make csv BCO-DMO provides a consistant csv #27

Open
adyork opened this issue May 27, 2021 · 2 comments
Open

Make csv BCO-DMO provides a consistant csv #27

adyork opened this issue May 27, 2021 · 2 comments
Assignees
Labels
wontfix This will not be worked on

Comments

@adyork
Copy link

adyork commented May 27, 2021

This isn't necessarily a datapackage viewer issue, but putting it here since this is the first time the datapackage csv is being provided directly.

And a reminder, I mentioned before I think we should consider how confusing it will be if we put out two separate csv products from BCO-DMO. I can see this making headaches for us down the road.

One from laminar and one from the csv download badge on the main page (ERDDAP). If they were the same csv I wouldn't think that was a problem but the csvs will have different contents.

I don't know if everyone is aware of this so I'm explaining the difference.
Here are the two types:

  1. The csv out of the datapackage. This is what the link in the data viewer is currently providing access to. One header line with column names. This is the exact version of the data the data managers dumped after Laminar. The format here matches the format we are entering in OSPREY (and what is shown in the datapackage viewer).
    image

  2. The csv that is provided if you press "csv" download button on the current dataset landing pages for datasets that are in ERDDAP. There are two header lines, one for column names, the second row is units. Values in this dataset are changed in various ways when coming out of ERDDAP. For example all datetime columns will have values changed to add 00 padding until it gets to second resolution regardless of whether the format we are saying has them. This format no longer matches the formats we are entering into OSPREY.
    image

Also note that the missing data ids will vary between these two .csv files. The csv filenames differ too which is good i guess so they can't be mistaken for the same.

Example:
Dataset landing page with csv download badge https://www.bco-dmo.org/dataset/813166
Data view with csv download link to csv in datapackage https://www.bco-dmo.org/dataset/813166/data/view

CSVs differ.
datapackage csv porewater.csv
image

ERDDAP csv (out of landing page) bcodmo_dataset_813166_712b_5843_9069.csv
image

@adyork
Copy link
Author

adyork commented May 27, 2021

Also, injecting some personal feelings that may be irrelevant. But I really hate that ERDDAP changes values. I also really hate fake time resolution because it messes up plotting in synthesis projects.

@ashepherd
Copy link
Collaborator

discussed this with Danie, and we are OK with ERDDAP doing what it does, and we will communicate this from the new website that using ERDDAP file formats aren't producing isomorphic versions of DOI'd data file.

@ashepherd ashepherd added wontfix This will not be worked on and removed help wanted Extra attention is needed labels Jun 5, 2021
@ashepherd ashepherd self-assigned this Jun 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

2 participants