Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(airflow): fixed issue with _is_memory_dataset #1011

Merged
merged 25 commits into from
Feb 21, 2025
Merged

fix(airflow): fixed issue with _is_memory_dataset #1011

merged 25 commits into from
Feb 21, 2025

Conversation

CF-FHB-X
Copy link
Contributor

Description

Fixed the issue described in #998

Development notes

Tweaked the code in grouping/_is_memory_dataset() to also check for datasets that are set as MemoryDataset in the Data Catalog.

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Updated jsonschema/kedro-catalog-X.XX.json if necessary
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes
  • Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

@CF-FHB-X CF-FHB-X force-pushed the fix/998-is-memory-dataset branch from 96b5075 to 9bb91ed Compare February 17, 2025 13:44
@CF-FHB-X CF-FHB-X force-pushed the fix/998-is-memory-dataset branch from 98874b0 to e68f09c Compare February 18, 2025 12:54
ravi-kumar-pilla and others added 23 commits February 18, 2025 10:15
@ravi-kumar-pilla @CF-FHB-X
build(datasets): Release 6.0.0 (kedro-org#968)
32fdf69 
release draft

Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@ravi-kumar-pilla @CF-FHB-X
chore(datasets): Remove tracking datasets which are used in Kedro Viz…
ef05d23 
… Experiment Tracking (kedro-org#969)

* remove et related kedro datasets

* update release note and static json schema

* temporary doc fix

Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@ankatiyar @CF-FHB-X
docs(datasets): Move to linkcode extension (kedro-org#985)
778811c 
Move to linkcode extension

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@ravi-kumar-pilla @CF-FHB-X
fix(datasets): Fix polars.CSVDataset save on Windows (kedro-org#979)
5a6b100 
* test csv win

Signed-off-by: ravi_kumar_pilla <ravi_kumar_pilla@mckinsey.com>

* change ci yaml for testing

Signed-off-by: ravi_kumar_pilla <ravi_kumar_pilla@mckinsey.com>

* change ci yaml for testing

Signed-off-by: ravi_kumar_pilla <ravi_kumar_pilla@mckinsey.com>

* add default encoding when opening file

* revert workflow tests

Signed-off-by: ravi_kumar_pilla <ravi_kumar_pilla@mckinsey.com>

* fix lint

Signed-off-by: ravi_kumar_pilla <ravi_kumar_pilla@mckinsey.com>

* update release note

* update release note

---------

Signed-off-by: ravi_kumar_pilla <ravi_kumar_pilla@mckinsey.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@ElenaKhaustova @CF-FHB-X
feat(all): Replace trufflehog with detect-secrets (kedro-org#983)
377cf97 
* Removed trufflehog

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated github actions per plugin

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated release notes

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated validate-pr check scopes

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated lint command

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Added key to trigger check

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated GH action to track per plugin

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Removed secret

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated GH for kedro-datasets

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated secrets baseline

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

---------

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@deepyaman @astrojuanlu @CF-FHB-X
build(datasets): use intersphinx over type_targets (kedro-org#801)
7ed49f9 
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@ElenaKhaustova @CF-FHB-X
fix(datasets): Add parameter to enable/disable lazy saving for `Parti…
fb9a099 
…tionedDataset` (kedro-org#978)

* Replaced callable check

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updateds lazy_save test

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Added test_callable_save

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Fixed lint

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Fixed docs links

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Fixed all docs links

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated release notes

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Fixed all docs links

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Fixed typo

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Added argument to disable lazy saving

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Removed save function argument

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated unit test

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Fixed lint

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated related docs

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Revert test changes

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated baseline

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated release notes

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated release notes

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

* Updated docstrings

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>

---------

Signed-off-by: Elena Khaustova <ymax70rus@gmail.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@deepyaman @CF-FHB-X
fix(datasets): use kwarg for Ibis read_* methods (kedro-org#1005)
4a15b80 
* fix(datasets): use kwarg for Ibis `read_*` methods

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Update RELEASE.md

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@deepyaman @merelcht @CF-FHB-X
build(datasets): pin PyArrow until 19.0.1 is out (kedro-org#1006)
e6b22c3 
* build(datasets): pin PyArrow until `19.0.1` is out

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): exclude `19.0.0` instead of bound

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@deepyaman @CF-FHB-X
build(datasets): update list of extras for Ibis 10 (kedro-org#1003)
0fed73c 
* build(datasets): update list of extras for Ibis 10

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Update RELEASE.md

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@deepyaman @merelcht @CF-FHB-X
chore: remove internal devtools from release notes (kedro-org#1004)
a652129 
* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Fixed case where MemoryDatasets in catalog wouldn't trigger `_is…
c384f0c 
…_memory_dataset`

Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Tests to ensure that MemoryDatasets are passed in mocked data ca…
b9005e6 
…talog

Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Changelog
9fe874a 
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Linting fixes
4dd6619 
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@deepyaman @CF-FHB-X
build(datasets): update list of extras for Ibis 10 (kedro-org#1003)
15a0e3c 
* build(datasets): update list of extras for Ibis 10

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Update RELEASE.md

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Richard <CF-FHB-X@users.noreply.github.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@deepyaman @merelcht @CF-FHB-X
chore: remove internal devtools from release notes (kedro-org#1004)
6060e67 
* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Richard <CF-FHB-X@users.noreply.github.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Tests to ensure that MemoryDatasets are passed in mocked data ca…
e49f25d 
…talog

Signed-off-by: Richard <CF-FHB-X@users.noreply.github.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Linting fixes
cffbaa2 
Signed-off-by: Richard <CF-FHB-X@users.noreply.github.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Changed function according to PR comments
b07b8cb 
Signed-off-by: Richard <CF-FHB-X@users.noreply.github.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Tests to ensure that MemoryDatasets are passed in mocked data ca…
2ec88b0 
…talog

Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Linting fixes
69aad57 
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
Signed-off-by: Richard <CF-FHB-X@users.noreply.github.com>
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X
998: Tweaked release
fbb8081 
Signed-off-by: Richard Asselin <richard.asselin@gmail.com>
@CF-FHB-X CF-FHB-X force-pushed the fix/998-is-memory-dataset branch from 9136a92 to fbb8081 Compare February 18, 2025 15:22
ElenaKhaustova
ElenaKhaustova approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@ElenaKhaustova ElenaKhaustova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a nit comment, the rest looks good to me, thank you!

@CF-FHB-X @ElenaKhaustova
Update RELEASE.md
8eb9beb 
Co-authored-by: ElenaKhaustova <157851531+ElenaKhaustova@users.noreply.github.com>
Signed-off-by: Richard <CF-FHB-X@users.noreply.github.com>
ankatiyar
ankatiyar approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@ankatiyar ankatiyar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @CF-FHB-X

@ElenaKhaustova ElenaKhaustova merged commit 4458d2c into kedro-org:main Feb 21, 2025
18 checks passed
@CF-FHB-X CF-FHB-X deleted the fix/998-is-memory-dataset branch February 21, 2025 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ankatiyar ankatiyar ankatiyar approved these changes

@ElenaKhaustova ElenaKhaustova ElenaKhaustova approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@CF-FHB-X @ankatiyar @ElenaKhaustova @ravi-kumar-pilla @deepyaman