Skip to content

Commit

Permalink
improved prompt for analyze plugin
Browse files Browse the repository at this point in the history
  • Loading branch information
DavertMik committed Jan 23, 2025
1 parent 1f30058 commit 29e1be1
Showing 1 changed file with 18 additions and 10 deletions.
28 changes: 18 additions & 10 deletions lib/plugin/analyze.js
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,10 @@ const defaultConfig = {
.map((test, index) => {
if (!test || !test.err) return
return `
TEST #${index + 1}: ${serializeTest(test)}
ERROR: ${serializeError(test.err).slice(0, MAX_DATA_LENGTH / tests.length)}`
#${index + 1}: ${serializeTest(test)}
${serializeError(test.err).slice(0, MAX_DATA_LENGTH / tests.length)}`.trim()
})
.join('\n\n---\n\n')
.join('\n\n--------\n\n')

const messages = [
{
Expand All @@ -60,25 +60,29 @@ const defaultConfig = {
If there is no groups of tests, say: "No patterns found"
Preserve error messages but cut them if they are too long.
Respond clearly and directly, without introductory words or phrases like ‘Of course,’ ‘Here is the answer,’ etc.
Do not list more than 3 tests in the group.
Do not list more than 3 errors in the group.
If you identify that all tests in the group have the same tag, add this tag to the group report, otherwise ignore TAG section.
If you identify that all tests in the group have the same suite, add this suite to the group report, otherwise ignore SUITE section.
Pick different emojis for each group.
Do not include group into report if it has only one test in affected tests section.
Provide list of groups in following format:
_______________________________
## Group <group_number> <(percentage of failed tests in group compare to all failed tests)'>
## Group <group_number>
* CATEGORY <category_of_failure>
* ERROR <error_message_1>, <error_message_2>, ...
* SUMMARY <summary_of_errors>
* STEP <step_of_failure> (in format I.click(), I.see(), etc; if all failures happend on the same step)
* AFFECTED TESTS (<total number of tests>)
* SUITE <suite_title>, <suite_title> (only if all tests in the group have the same suite or suites)
* TAG <tag> (if all tags in group have the same tag)
* STEP <step_of_failure> (use CodeceptJS format I.click(), I.see(), etc; if all failures happend on the same step)
* SUITE <suite_title>, <suite_title> (if SUITE is present, and if all tests in the group have the same suite or suites)
* TAG <tag> (if TAG is present, and if all tests in the group have the same tag)
* AFFECTED TESTS (<total number of tests>):
x <test1 title>
x <test2 title>
x <test3 title>
x ...
`,
},
{
Expand Down Expand Up @@ -318,9 +322,12 @@ function serializeTest(test) {
if (test.suite) {
testMessage += '\n SUITE: ' + test.suite.title
}
if (test.parent) {
testMessage += '\n SUITE: ' + test.parent.title
}

if (test.steps?.length) {
const failedSteps = test.steps.filter(s => s.status === 'failed')
const failedSteps = test.steps
if (failedSteps.length) testMessage += '\n STEP: ' + failedSteps.map(s => s.toCode()).join('; ')
}

Expand All @@ -339,5 +346,6 @@ function formatResponse(response) {
.map(line => line.trim())
.filter(line => !/^[A-Z\s]+$/.test(line))
.map(line => markdownToAnsi(line))
.map(line => line.replace(/^x /gm, ` ${colors.red.bold('x')} `))
.join('\n')
}

0 comments on commit 29e1be1

Please sign in to comment.