update ascending flag variable clean up #154

nlenssen2013 · 2023-03-16T17:47:30Z

Github Issue: #153

Description

xarray decode times chokes on blank ascending time variable

Overview of work done

remove the variable if the flag indicates it is blank

Overview of verification done

Open up three different files, ascending, descending and both

Overview of integration done

Explain how this change was integration tested. Provide screenshots or logs if appropriate. An example of this would be a local Harmony deployment.

PR checklist:

Linted
Updated unit tests
Updated changelog
Integration testing

See Pull Request Review Checklist for pointers on reviewing this pull request

jamesfwood

Looks good Nick. Could you do me a favor though? Could you roll those dependabot updates into this PR? Bump werkzeug to 2.2.3, and cryptography to 39.0.1.
It may need a poetry update, but not sure. Thanks!

nlenssen2013 · 2023-03-20T17:49:07Z

Looks good Nick. Could you do me a favor though? Could you roll those dependabot updates into this PR? Bump werkzeug to 2.2.3, and cryptography to 39.0.1. It may need a poetry update, but not sure. Thanks!

Got it, so merge the other two pull requests into this one?

jamesfwood · 2023-03-20T18:00:44Z

Looks good Nick. Could you do me a favor though? Could you roll those dependabot updates into this PR? Bump werkzeug to 2.2.3, and cryptography to 39.0.1. It may need a poetry update, but not sure. Thanks!

Looks good Nick. Could you do me a favor though? Could you roll those dependabot updates into this PR? Bump werkzeug to 2.2.3, and cryptography to 39.0.1. It may need a poetry update, but not sure. Thanks!

Got it, so merge the other two pull requests into this one?

Yeah, or just update those versions and after it passes I can just close those tickets. Either way, up to you.

nlenssen2013 · 2023-03-21T16:49:59Z

Looks good Nick. Could you do me a favor though? Could you roll those dependabot updates into this PR? Bump werkzeug to 2.2.3, and cryptography to 39.0.1. It may need a poetry update, but not sure. Thanks!

Looks good Nick. Could you do me a favor though? Could you roll those dependabot updates into this PR? Bump werkzeug to 2.2.3, and cryptography to 39.0.1. It may need a poetry update, but not sure. Thanks!

Got it, so merge the other two pull requests into this one?

Yeah, or just update those versions and after it passes I can just close those tickets. Either way, up to you.

Okay, so I did poetry add werkzeug=='2.2.3' and poetry add cryptography=='39.0.1'. Then did a poetry update, pushed the changes in the lock and toml files. Seems like the poetry version is still wrong....

frankinspace · 2023-03-21T17:09:21Z

@nlenssen2013 you just need to run poetry update and then check in the new poetry.lock file that gets generated. I ran it and the updated version are there now.

nlenssen2013 · 2023-03-21T17:26:52Z

@nlenssen2013 you just need to run poetry update and then check in the new poetry.lock file that gets generated. I ran it and the updated version are there now.

Looks like it's still failing at the same spot

frankinspace · 2023-03-21T18:11:47Z

Didn't realize the action was also failing, that was a separate problem. Needed to bump the version of poetry the github action itself was using.

nlenssen2013 · 2023-03-22T16:42:09Z

Didn't realize the action was also failing, that was a separate problem. Needed to bump the version of poetry the github action itself was using.

You alright if I merge this?

frankinspace · 2023-03-22T16:54:27Z

Need to figure out why tests are failing

nlenssen2013 · 2023-03-27T17:43:20Z

Need to figure out why tests are failing

Will this get figured out this week?

jamesfwood · 2023-03-28T20:05:25Z

@nlenssen2013 Would you have time to fix the tests or if not I can do it. Let me know, thanks!

nlenssen2013 · 2023-03-29T14:21:17Z

@nlenssen2013 Would you have time to fix the tests or if not I can do it. Let me know, thanks!

I can work on it today, but tests are passing when I run it locally. I have the updated cryptography and werkzeug versions as well - so not sure where to trouble shoot

jamesfwood · 2023-03-29T15:57:46Z

@nlenssen2013 Would you have time to fix the tests or if not I can do it. Let me know, thanks!

I can work on it today, but tests are passing when I run it locally. I have the updated cryptography and werkzeug versions as well - so not sure where to trouble shoot

Ok, that would be great. Did you see the error in Github Actions? It's something with this:
AttributeError: module 'numpy' has no attribute 'warnings'

Maybe your local version has a different version of numpy?

Thanks for checking it out!
Also, sorry about the trouble with updating those two libraries. Didn't think it would cause so many problems.

nlenssen2013 · 2023-03-29T16:46:20Z

@nlenssen2013 Would you have time to fix the tests or if not I can do it. Let me know, thanks!

I can work on it today, but tests are passing when I run it locally. I have the updated cryptography and werkzeug versions as well - so not sure where to trouble shoot

Ok, that would be great. Did you see the error in Github Actions? It's something with this: AttributeError: module 'numpy' has no attribute 'warnings'

Maybe your local version has a different version of numpy?

Thanks for checking it out! Also, sorry about the trouble with updating those two libraries. Didn't think it would cause so many problems.

Does numpy need to be 1.24? because everything goes through and passes when it remains numpy==1.23.5

jamesfwood · 2023-03-29T21:54:25Z

@nlenssen2013 Would you have time to fix the tests or if not I can do it. Let me know, thanks!

I can work on it today, but tests are passing when I run it locally. I have the updated cryptography and werkzeug versions as well - so not sure where to trouble shoot

Ok, that would be great. Did you see the error in Github Actions? It's something with this: AttributeError: module 'numpy' has no attribute 'warnings'
Maybe your local version has a different version of numpy?
Thanks for checking it out! Also, sorry about the trouble with updating those two libraries. Didn't think it would cause so many problems.

Does numpy need to be 1.24? because everything goes through and passes when it remains numpy==1.23.5

yeah go ahead and downgrade numpy to 1.23.5 and push it and see if that works in github actions.

nlenssen2013 · 2023-03-30T15:26:24Z

@nlenssen2013 Would you have time to fix the tests or if not I can do it. Let me know, thanks!

I can work on it today, but tests are passing when I run it locally. I have the updated cryptography and werkzeug versions as well - so not sure where to trouble shoot

Ok, that would be great. Did you see the error in Github Actions? It's something with this: AttributeError: module 'numpy' has no attribute 'warnings'
Maybe your local version has a different version of numpy?
Thanks for checking it out! Also, sorry about the trouble with updating those two libraries. Didn't think it would cause so many problems.

Does numpy need to be 1.24? because everything goes through and passes when it remains numpy==1.23.5

yeah go ahead and downgrade numpy to 1.23.5 and push it and see if that works in github actions.

nlenssen2013 · 2023-03-30T15:27:26Z

@nlenssen2013 Would you have time to fix the tests or if not I can do it. Let me know, thanks!

I can work on it today, but tests are passing when I run it locally. I have the updated cryptography and werkzeug versions as well - so not sure where to trouble shoot

Ok, that would be great. Did you see the error in Github Actions? It's something with this: AttributeError: module 'numpy' has no attribute 'warnings'
Maybe your local version has a different version of numpy?
Thanks for checking it out! Also, sorry about the trouble with updating those two libraries. Didn't think it would cause so many problems.

Does numpy need to be 1.24? because everything goes through and passes when it remains numpy==1.23.5

yeah go ahead and downgrade numpy to 1.23.5 and push it and see if that works in github actions.

Was just trying to delete my comment - looks like sonar cloud isn't too happy

jamesfwood · 2023-03-30T17:10:45Z

@nlenssen2013 Would you have time to fix the tests or if not I can do it. Let me know, thanks!

I can work on it today, but tests are passing when I run it locally. I have the updated cryptography and werkzeug versions as well - so not sure where to trouble shoot

Ok, that would be great. Did you see the error in Github Actions? It's something with this: AttributeError: module 'numpy' has no attribute 'warnings'
Maybe your local version has a different version of numpy?
Thanks for checking it out! Also, sorry about the trouble with updating those two libraries. Didn't think it would cause so many problems.

Does numpy need to be 1.24? because everything goes through and passes when it remains numpy==1.23.5

yeah go ahead and downgrade numpy to 1.23.5 and push it and see if that works in github actions.

Was just trying to delete my comment - looks like sonar cloud isn't too happy

If you want to get this PR in, then go ahead and squash and merge it and we can deal with the sonar cloud issue another time. It doesn't seem like anything critical right now.

* /version 2.5.0-alpha.0 * /version 2.5.0-alpha.1 * Adding collections OPS: C2599212091-POCLOUD * /version 2.5.0-alpha.2 * Adding collections OPS: C2208421887-POCLOUD * /version 2.5.0-alpha.3 * Adding collections OPS: C2205121449-POCLOUD * /version 2.5.0-alpha.4 * update ascending flag variable clean up (#154) * update ascending flag variable clean up * Update changelog * Add sndr ascending/descending type * Update testing file * added sndr file to list files to test * change tests to call subset method * change assertion test if box returned is true * change assertion test if box returned is true * expand time test zone * Ascending flag does not coorespond to the asc_node_time * Ascending flag does not coorespond to the asc_node_time * Updated werkzeug=2.2.3 and cryptography=='39.0.1' * Update poetry * Ran poetry update * Update poetry version used in pipeline * Fix pylint errors * Bump python version used in build * rollback numpy=1.24 to numpy=1.23.5 --------- Co-authored-by: nlensse1 <nicholas.f.lenssen@nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 2.5.0a5 * Adding collections OPS: C2274919541-POCLOUD * Adding collections UAT: C1238658392-POCLOUD C1243175554-POCLOUD * /version 2.5.0a6 * Adding collections UAT: C1240817851-POCLOUD C1238621185-POCLOUD C1241042620-POCLOUD C1240739719-POCLOUD C1256420924-POCLOUD C1245295751-POCLOUD C1238621118-POCLOUD C1238621219-POCLOUD C1240739606-POCLOUD C1242735870-POCLOUD C1240739611-POCLOUD C1240739704-POCLOUD C1238657961-POCLOUD C1238658049-POCLOUD C1241042621-POCLOUD C1238621092-POCLOUD C1238658050-POCLOUD C1240739764-POCLOUD C1238621115-POCLOUD C1238657960-POCLOUD C1256524295-POCLOUD C1245295750-POCLOUD C1240739768-POCLOUD C1238621176-POCLOUD * /version 2.5.0a7 * Adding collections UAT: C1238621178-POCLOUD C1238658088-POCLOUD C1238687534-POCLOUD C1238687284-POCLOUD C1256445396-POCLOUD C1238621172-POCLOUD C1240739654-POCLOUD C1238658086-POCLOUD C1238621186-POCLOUD C1238687546-POCLOUD C1238658080-POCLOUD C1240739688-POCLOUD C1238621182-POCLOUD * /version 2.5.0a8 * Feature/podaac 5538 (#156) * update l2ss to load each dataset variable individually to write to file * remove unused variable * remove test variable that is not used * /version 2.5.0a9 * Adding collections OPS: C2296989383-POCLOUD C2296989388-POCLOUD C2296989390-POCLOUD * /version 2.5.0a10 * Adding collections UAT: C1234208436-POCLOUD C1238570311-POCLOUD * /version 2.5.0a11 * Adding collections UAT: C1256783391-POCLOUD C1256783386-POCLOUD * /version 2.5.0a12 * Feature/podaac 5537 (#157) * update l2ss py to open hdf5 files if we fail to open as netcdf, and change way we copy attributes in case it has a / * fix pylint and removed comment * update changelog * /version 2.5.0a13 * Feature/podaac 5537 (#161) * update l2ss py to open hdf5 files if we fail to open as netcdf, and change way we copy attributes in case it has a / * fix pylint and removed comment * update changelog * change so that were able to process S1 and S2 dtypes * fix pylint and flake8 * revert spaces * remove spaces * remove spaces * /version 2.5.0a14 * Bump tornado from 6.2 to 6.3.2 (#160) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.2 to 6.3.2. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](tornadoweb/tornado@v6.2.0...v6.3.2) --- updated-dependencies: - dependency-name: tornado dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.28.2 to 2.31.0 (#159) Bumps [requests](https://github.com/psf/requests) from 2.28.2 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.28.2...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * /version 2.5.0a15 * Feature/issue 155 (#158) * add group variable fix test * Updated MLS fix with TEMPO * update linting * fix pylinting * Fix pylint * fix pylint * add extra blank line * Update group_vars extending function * fix linting * attempted fix with variable subsetting * pylinting updated * Update subset.py and lint * Update subset.py and lint * fix unique groups * fix unique groups * fix unique groups * fix unique groups * add unit test * remove print statements * remove whitespace * CHANGELOG updated * add space in file * remove whitespace * Don't add a variable twice * fix linting * include test for latitude in root group * include test for latitude in root group * fix group handling * fix group handling * streamline group list control flow * add missing import * remove indent, remove print statement --------- Co-authored-by: nlensse1 <nicholas.f.lenssen@nasa.gov> Co-authored-by: jamesfwood <jamesfwood@hotmail.com> Co-authored-by: danielfromearth <daniel.kaufman@nasa.gov> * /version 2.5.0a16 * Updated changelog for release 2.5.0 * /version 2.6.0-alpha.0 * /version 2.5.0-rc.1 * trigger rebuild * /version 2.5.0rc2 * /version 2.5.0rc3 * fix version format * /version 2.5.0rc4 * Bugfix/fix docker versioning type (#164) * update docker versioning to pep440 * update changelog * /version 2.5.0rc5 * undo replacing slashes with _ in values for attributes (#165) * /version 2.5.0rc6 * Bump cryptography from 39.0.2 to 41.0.0 (#167) Bumps [cryptography](https://github.com/pyca/cryptography) from 39.0.2 to 41.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@39.0.2...41.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * /version 2.6.0a1 * Feature/issue 162 (#166) * add group variable fix test * Updated MLS fix with TEMPO * update linting * fix pylinting * Fix pylint * fix pylint * add extra blank line * Update group_vars extending function * fix linting * attempted fix with variable subsetting * pylinting updated * Update subset.py and lint * Update subset.py and lint * fix unique groups * fix unique groups * fix unique groups * fix unique groups * add unit test * remove print statements * remove whitespace * CHANGELOG updated * add space in file * remove whitespace * Don't add a variable twice * fix linting * add root variable subsetting * add root variable subsetting * include file ext in tests for argument * update tests * update tests * allow subsetting from one lat var name * allow subsetting from one lat var name * allow subsetting from one lat var name * allow temporal subsetting for MLS and OCO3 * updated changelog * Update test change * remove itertools extra import * Update checking the var_name --------- Co-authored-by: nlensse1 <nicholas.f.lenssen@nasa.gov> * /version 2.6.0a2 * Feature/issue 168 (#169) * make separate copy of test data file to get expected results before subsetting * update CHANGELOG.md * lint the test_subset.py file via pylint * /version 2.6.0a3 * /version 2.5.0rc7 * deploy * /version 2.5.0rc8 * re-add get timevar process for MLS (#171) * re-add get timevar process for MLS * Update changelog * Add comments for get time var name * Add comments for get time var name --------- Co-authored-by: nlensse1 <nicholas.f.lenssen@nasa.gov> * /version 2.6.0a4 * /version 2.5.0rc9 * Adding collections UAT: C1256507988-POCLOUD * Adding collections UAT: C1256507990-POCLOUD C1256122852-POCLOUD C1256507989-POCLOUD * /version 2.6.0a5 * Feature/issue 173 (#174) * Update xarray dataset merge * update subsetting for variables without sptaital dimensions * Update variable names and Changelog * Update variable name * Update variable name --------- Co-authored-by: nlensse1 <nicholas.f.lenssen@nasa.gov> * /version 2.6.0a6 * /version 2.5.0rc10 * Adding collections UAT: C1238621112-POCLOUD * /version 2.6.0a7 * Adding collections OPS: C2601585875-POCLOUD C2628593693-POCLOUD C2601584109-POCLOUD C2601583089-POCLOUD C2601581863-POCLOUD C2628598397-POCLOUD * /version 2.6.0a8 * /version 2.5.0rc11 * trigger redeploy * /version 2.5.0rc13 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: l2ss-py bot <l2ss-py@noreply.github.com> Co-authored-by: James Wood <James.F.Wood@jpl.nasa.gov> Co-authored-by: podaac-cloud-dsa <podaac-cloud-dsa@jpl.nasa.gov> Co-authored-by: Nick Lenssen <nicklenssen4@gmail.com> Co-authored-by: nlensse1 <nicholas.f.lenssen@nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sliu008 <69875423+sliu008@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danielfromearth <daniel.kaufman@nasa.gov> Co-authored-by: Daniel Kaufman <114174502+danielfromearth@users.noreply.github.com>

nlensse1 added 9 commits March 16, 2023 13:42

update ascending flag variable clean up

bed1e44

Update changelog

b4d1037

Add sndr ascending/descending type

ff2b70e

Update testing file

377c97f

added sndr file to list files to test

2d9d709

change tests to call subset method

6db7723

change assertion test if box returned is true

1baf6ac

change assertion test if box returned is true

2be8fff

expand time test zone

dcfd3b1

nlenssen2013 marked this pull request as ready for review March 17, 2023 14:32

nlenssen2013 requested review from jamesfwood and frankinspace March 17, 2023 14:32

nlensse1 added 2 commits March 17, 2023 14:11

Ascending flag does not coorespond to the asc_node_time

fa1accd

Ascending flag does not coorespond to the asc_node_time

82b130a

jamesfwood reviewed Mar 20, 2023

View reviewed changes

nlensse1 added 2 commits March 21, 2023 12:27

Updated werkzeug=2.2.3 and cryptography=='39.0.1'

f094029

Update poetry

c4e5c37

Ran poetry update

3c8ceb9

frankinspace approved these changes Mar 21, 2023

View reviewed changes

frankinspace added 2 commits March 21, 2023 10:56

Update poetry version used in pipeline

85f6a3f

Fix pylint errors

b9d496f

Bump python version used in build

d8bf283

rollback numpy=1.24 to numpy=1.23.5

8e1da6e

nlenssen2013 closed this Mar 30, 2023

nlenssen2013 reopened this Mar 30, 2023

jamesfwood approved these changes Mar 30, 2023

View reviewed changes

nlenssen2013 merged commit 9cde8fa into develop Mar 30, 2023

nlenssen2013 deleted the feature/issue-153 branch March 30, 2023 17:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update ascending flag variable clean up #154

update ascending flag variable clean up #154

nlenssen2013 commented Mar 16, 2023 •

edited

Loading

jamesfwood left a comment

nlenssen2013 commented Mar 20, 2023

jamesfwood commented Mar 20, 2023

nlenssen2013 commented Mar 21, 2023 •

edited

Loading

frankinspace commented Mar 21, 2023

nlenssen2013 commented Mar 21, 2023

frankinspace commented Mar 21, 2023

nlenssen2013 commented Mar 22, 2023

frankinspace commented Mar 22, 2023

nlenssen2013 commented Mar 27, 2023

jamesfwood commented Mar 28, 2023

nlenssen2013 commented Mar 29, 2023

jamesfwood commented Mar 29, 2023

nlenssen2013 commented Mar 29, 2023

jamesfwood commented Mar 29, 2023

nlenssen2013 commented Mar 30, 2023

nlenssen2013 commented Mar 30, 2023

jamesfwood commented Mar 30, 2023

update ascending flag variable clean up #154

update ascending flag variable clean up #154

Conversation

nlenssen2013 commented Mar 16, 2023 • edited Loading

Description

Overview of work done

Overview of verification done

Overview of integration done

PR checklist:

jamesfwood left a comment

Choose a reason for hiding this comment

nlenssen2013 commented Mar 20, 2023

jamesfwood commented Mar 20, 2023

nlenssen2013 commented Mar 21, 2023 • edited Loading

frankinspace commented Mar 21, 2023

nlenssen2013 commented Mar 21, 2023

frankinspace commented Mar 21, 2023

nlenssen2013 commented Mar 22, 2023

frankinspace commented Mar 22, 2023

nlenssen2013 commented Mar 27, 2023

jamesfwood commented Mar 28, 2023

nlenssen2013 commented Mar 29, 2023

jamesfwood commented Mar 29, 2023

nlenssen2013 commented Mar 29, 2023

jamesfwood commented Mar 29, 2023

nlenssen2013 commented Mar 30, 2023

nlenssen2013 commented Mar 30, 2023

jamesfwood commented Mar 30, 2023

nlenssen2013 commented Mar 16, 2023 •

edited

Loading

nlenssen2013 commented Mar 21, 2023 •

edited

Loading