Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SCHEMATIC-183] Use paths from file view for manifest generation #1529

Merged
merged 101 commits into from
Dec 10, 2024
Merged
Changes from 1 commit
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
8a80ce3
add test for clause method
SageGJ Oct 29, 2024
a2de0c4
add method to process dataset id into query clause
SageGJ Oct 29, 2024
45193a1
use new method for validation
SageGJ Oct 29, 2024
80b6bda
update clause method
SageGJ Oct 29, 2024
d810576
update file based manifest gen test
SageGJ Oct 29, 2024
b1e60af
consolidate filebased manifest gen tests
SageGJ Oct 29, 2024
33796b4
update test layout
SageGJ Oct 29, 2024
62d8b28
use fileview for file paths
SageGJ Oct 30, 2024
8b012d3
add functionality to just return filename
SageGJ Oct 31, 2024
8e670ad
make syn id regex a util function
SageGJ Oct 31, 2024
c90ebf0
add non-api integration test for detFilesInStorageDataset
SageGJ Oct 31, 2024
f9b9bcc
fix typo and mismatched ids
SageGJ Oct 31, 2024
be4f6b6
add case for nested data structure
SageGJ Oct 31, 2024
b39afe2
get nested files as well
SageGJ Nov 1, 2024
566e741
add return type annotation
SageGJ Nov 1, 2024
86fb15d
add docstring
SageGJ Nov 1, 2024
2fa1ad6
add str prefix
SageGJ Nov 1, 2024
cd6713b
revert param name
SageGJ Nov 4, 2024
9438811
change datasetid clause method
SageGJ Nov 4, 2024
e076770
get files in doubly+ nested files
SageGJ Nov 5, 2024
19188ea
add comments
SageGJ Nov 5, 2024
c46a100
add test cases for filtering results
SageGJ Nov 5, 2024
ccead76
add test case for filtered results
SageGJ Nov 7, 2024
26e5539
Update README.md
jaymedina Oct 21, 2024
2f835bd
Update README.md
jaymedina Oct 21, 2024
f228245
Update README.md
jaymedina Oct 22, 2024
ae11b85
updated data model type rules to include error param
andrewelamb Oct 25, 2024
2daacb9
fix validate type attribute to use msg level param
andrewelamb Oct 25, 2024
2982d8e
added error handling
andrewelamb Oct 25, 2024
a1e0783
run black
andrewelamb Oct 25, 2024
450fbdf
Update CODEOWNERS
thomasyu888 Nov 1, 2024
b16bf55
Update scan_repo.yml
thomasyu888 Nov 1, 2024
8d50e1a
Update .github/CODEOWNERS
thomasyu888 Nov 1, 2024
1336fc6
Update .github/workflows/scan_repo.yml
thomasyu888 Nov 1, 2024
c61f39c
Attach additional telemetry data to OTEL traces (#1519)
BryanFauble Nov 1, 2024
ce4d642
feat: added tracing for cross manifest validation and file name valid…
linglp Nov 1, 2024
31f3f1d
Updating contribution doc to expect squash and merge (#1534)
BryanFauble Nov 5, 2024
256403c
[FDS-2491] Integration tests for Schematic API Test plan (#1512)
BryanFauble Nov 5, 2024
856fef6
[FDS-2500] Add Integration Tests for: Manifest Validation (#1516)
jaymedina Nov 6, 2024
d6fc9ad
[FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530)
jaymedina Nov 7, 2024
08008ae
filter based on filenames if given
SageGJ Nov 8, 2024
d0aa01d
change manifest exclusion method
SageGJ Nov 8, 2024
38cedd5
Update file annotation store process to require filename be present i…
BryanFauble Nov 7, 2024
22f0bba
Revert "Update file annotation store process to require filename be p…
BryanFauble Nov 7, 2024
4580e06
Don't attempt to annotate the table
BryanFauble Nov 7, 2024
ce5c349
Updates for integration test failures (#1537)
BryanFauble Nov 12, 2024
d661b9a
add test for bug case
SageGJ Nov 11, 2024
951a061
update test for table tidyness
SageGJ Nov 11, 2024
89fb9a8
remove unused import
SageGJ Nov 11, 2024
0c9e773
remove etag column if already present when building temp file view
SageGJ Nov 11, 2024
ab4ece7
catch all exceptions to switch to sequential mode
SageGJ Nov 12, 2024
65acb33
update test for updated data
SageGJ Nov 12, 2024
2e6d51f
Revert "update test for updated data"
SageGJ Nov 12, 2024
9a00288
Revert "catch all exceptions to switch to sequential mode"
SageGJ Nov 12, 2024
2170974
catch ValueErrors as well
SageGJ Nov 12, 2024
65fb55d
[FDS-2525] Authenticated export of telemetry data (#1527)
BryanFauble Nov 13, 2024
5e891ef
update mocking for unit tests
SageGJ Nov 13, 2024
4d6fd09
Merge branch 'develop' into fds-2293-file-paths-for-manifest-gen
thomasyu888 Nov 14, 2024
5f5cc43
update test assertions for format
SageGJ Nov 14, 2024
3f90ef0
update tests assertions
SageGJ Nov 14, 2024
f4ad7a1
add mocked integration test for getting dataset files
SageGJ Nov 15, 2024
ee43ad6
use dataset clause method
SageGJ Nov 15, 2024
5a4cd90
Revert "add mocked integration test for getting dataset files"
SageGJ Nov 15, 2024
59c5a69
add mocked test for get files in dataset
SageGJ Nov 15, 2024
d2eee35
clean comments
SageGJ Nov 15, 2024
0eddd47
remove comment
SageGJ Nov 15, 2024
5fcbeb1
add test ids
SageGJ Nov 15, 2024
a1b0c90
remove unneeded param
SageGJ Nov 15, 2024
c43debc
add ids
SageGJ Nov 15, 2024
89a2b9f
use syn store fixture
SageGJ Nov 15, 2024
9cdad89
change variables
SageGJ Nov 15, 2024
51983bc
change to global var
SageGJ Nov 18, 2024
84011b0
change case
SageGJ Nov 18, 2024
3471e18
change case
SageGJ Nov 18, 2024
82082be
update test for dataset clause
SageGJ Nov 19, 2024
c542241
update use of dataset clause method
SageGJ Nov 19, 2024
8a314a7
comments
SageGJ Nov 19, 2024
bc9f479
change test name case
SageGJ Nov 19, 2024
79f5bda
update descriptions
SageGJ Nov 19, 2024
5d0599b
undo development change
SageGJ Nov 20, 2024
c1cd79a
add comment
SageGJ Nov 20, 2024
feb82e8
remove dev work
SageGJ Nov 20, 2024
c1587c6
remove temp test marks
SageGJ Nov 20, 2024
2769d6a
update test for new expected order
SageGJ Nov 20, 2024
dd9d6a9
change method for gathering files in a dataset
SageGJ Nov 20, 2024
800a4cd
update mock test
SageGJ Nov 20, 2024
95cfeed
update other mocked test
SageGJ Nov 20, 2024
9701078
change method for building dataset path
SageGJ Nov 22, 2024
7cd0ce0
wrap path in quotes
SageGJ Nov 22, 2024
65dcc35
update quotes for dataset path
GiaJordan Nov 22, 2024
f9d037d
reformat and add exception
SageGJ Nov 22, 2024
e07f973
add unit test
SageGJ Nov 22, 2024
73227f8
Merge branch 'develop' into fds-2293-file-paths-for-manifest-gen
thomasyu888 Nov 25, 2024
0a00086
fix comment typo
SageGJ Dec 2, 2024
0cca133
raise exception for empty view tables
SageGJ Dec 2, 2024
3261dcc
[SCHEMATIC-183] Update tests - Use magic mock and add parentId (#1554)
thomasyu888 Dec 3, 2024
f0968a2
remove hack related comment
SageGJ Dec 3, 2024
8229e72
add test for new exception
SageGJ Dec 3, 2024
7b0987d
add param back in
SageGJ Dec 3, 2024
9d2ea4a
add integration test
SageGJ Dec 10, 2024
f52bbb9
update var name
SageGJ Dec 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions schematic/store/synapse.py
Original file line number Diff line number Diff line change
Expand Up @@ -706,11 +706,21 @@ def getFilesInStorageDataset(
"""
file_list = []

# Get path to dataset folder from fileview to avoid building a new fileview and walking to determine folders and files within
# Get path to dataset folder by using childern to avoid cases where the dataset is the scope of the view
child_path = self.storageFileviewTable.loc[
self.storageFileviewTable["parentId"] == datasetId, "path"
][0]
parent = child_path.split("/")[0]
]
if child_path.empty:
raise LookupError(
f"Dataset {datasetId} could not be found in fileview {self.storageFileview}."
)
child_path = child_path.iloc[0]

# Get the dataset path by eliminating the child's portion of the path to account for nested datasets
parent = child_path.split("/")[:-1]
parent = "/".join(parent)

# Format dataset path to be used in table query
dataset_path = f"'{parent}/%'"

# When querying, only include files to exclude entity files and subdirectories
Expand Down
Loading