[FAI-14745] Process source airbyte connector output #102
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeniii
commented
Jan 17, 2025
| // Take the non-default value if provided with `srcOutputFile` option | ||
| srcOutputFile: cliOptions.srcOnly ? '/dev/null' : cliOptions.srcOutputFile, | ||
| srcOutputFile: cliOptions.srcOnly ? OutputStream.STDOUT : cliOptions.srcOutputFile, |
There was a problem hiding this comment.
bug. this is supposed to be stdout by default
jeniii
commented
Jan 21, 2025
| const lines = buffer.split('\n'); | ||
| buffer = lines.pop() ?? ''; | ||
| lines.forEach((line: string) => { | ||
| processSrcDataByLine(line, srcOutputStream, config); |
There was a problem hiding this comment.
processing the data and write to stdout/file instead of returning the data at the end of the function
| if (cfg.rawMessages) { | ||
| process.stdout.write(`${line}\n`); | ||
| } else { | ||
| logger.info(formatSrcMsg(data)); |
There was a problem hiding this comment.
it will look like this
| // - input stream: read from the data file user provided | ||
| // - output stream: write to an intermediate file. Overwrite the file if it exists, otherwise create a new one | ||
| const inputStream = createReadStream(cfg.srcInputFile!); | ||
| const outputStream = createWriteStream(`${tmpDir}/${SRC_OUTPUT_DATA_FILE}`); |
There was a problem hiding this comment.
if the user has srcInputFile configured, i assume they won't have src-only or src-output-file configured. but i will add this constraint in the command in another PR.
|
cjwooo
approved these changes
Jan 22, 2025
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Type of change