Allow users to specify number of docs per index when creating workloads #291
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gkamat
requested changes
May 7, 2023
|
Also consider modifying the user interface to options of this sort: |
added 4 commits
May 8, 2023 13:39
Signed-off-by: Ian Hoang <hoangia@amazon.com>
Signed-off-by: Ian Hoang <hoangia@amazon.com>
Signed-off-by: Ian Hoang <hoangia@amazon.com>
Signed-off-by: Ian Hoang <hoangia@amazon.com>
1e90e73 to
174da48
Compare
added 3 commits
May 9, 2023 12:21
Signed-off-by: Ian Hoang <hoangia@amazon.com>
… of comma separated values that need to match --indices list Signed-off-by: Ian Hoang <hoangia@amazon.com>
Signed-off-by: Ian Hoang <hoangia@amazon.com>
gkamat
approved these changes
May 11, 2023
| dump_documents(client, index, get_doc_outpath(output_path, index, "-1k"), min(total_docs, 1000), " for test mode") | ||
| dump_documents(client, index, docs_path, total_docs) | ||
| return template_vars(index, docs_path, total_docs) | ||
| dump_documents(client, index, get_doc_outpath(output_path, index, "-1k"), min(ndocs_to_extract, 1000), " for test mode") |
There was a problem hiding this comment.
IMO, the doc count specified should not be respected for test mode -- it should always be min(1000, total_docs). Users generally don't know the internals of test mode and it is unlikely they intend to specify that value.
1 task
IanHoang
commented
May 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
create-workloadcurrently fetches all documents from specified indices in--indices. Users should have the option to specify a subset of documents if they do not want to use all of the documents. In this PR, OSB now supports running--total-docsin conjunction with--indices. It takes in a comma-separated list of document counts that correspond to the respective index in the list of indices in--indices.See #289 for more details.
Issues Resolved
#289
Testing
Tested it with a few indices in private cluster. Tested the following cases:
See #289 for outputs and more details
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.