Establish guidelines for usage of AI in PRs #11940

subhramit · 2024-10-12T14:04:55Z

Background

As we know, one of the ways we use JabRef is to provide software engineering training to students.
In fact, we spend time curating good first issues so that newcomers can build their skills and understand how a real project works.
Considering these goals and also the fact that the project is held in high regards at other places as well, I think it makes sense to add a check for AI-generated PRs, especially now that contributor activity will rise.

AI-generated PRs don't just take more time to review and work on feedback (as the contributor lacks context), but also snatch the opportunity from someone who would otherwise be willing to put genuine effort. Otherwise, we ourselves could have just used LLMs to solve all good first issues.

Trigger

One of the contributors had put an AI-generated PR and has used it for LinkedIn credibility:

Proposed Action

We should ensure that the efforts given by the maintainers to review code of newcomers is not wasted, and if a PR is merged into the codebase, they have learnt something while working on it.

Like OpenSUSE's kanidm project, we should have a mandatory check:

If the contributor does not adhere to this, the PR should be closed immediately.

Mandatory checks

~~Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)~~
~~Tests created for changes (if applicable)~~
~~Manually tested changed features in running JabRef (always required)~~
~~Screenshots added in PR description (for UI changes)~~
~~Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.~~
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

InAnYan · 2024-10-12T14:13:18Z

.github/PULL_REQUEST_TEMPLATE.md

@@ -16,6 +16,7 @@ Don't reference an issue in the PR title because GitHub does not support auto-li
 - [ ] Change in `CHANGELOG.md` described in a way that is understandable for the average user (if applicable)
 - [ ] Tests created for changes (if applicable)
 - [ ] Manually tested changed features in running JabRef (always required)
+- [ ] The PR contains no or minimal AI-generated code (If any present, it is trivial and I fully understand it)


What I don't like about this topic is "AI usage" == bad, which has nothing to do with goodness/effectiveness.

I like that you didn't wrote it like in that SUSE project, where no AI code is allowed (if we were to discuss this topic, then that will be too much philosophy). And because JabRef is used for teaching, we definitely need a section on AI.

But I'm not sure about that additional check. I think this is + mind effort to click that, + mind effort to think "I used copilot, is it ok or not...", "It's small, it's okay...", "No it's big, what should I do"...

I think it's best to populate some documentation page about that, where you could tell more how to use AI correctly.

Yes! I think the idea should be "the correct usage of AI" but not "disallowance of AI" or "to which extent you should use AI".

Good points, and I agree.
Yes, the usage of AI is not "bad" (we all use it), except that a new contributor can get away with a PR addressing a good first issue merged, having learnt absolutely nothing as well as taking away the opportunity from someone who would want to work on it (and not just get a PR merged).

In case of effectiveness, we can use the same argument, pass every good first issue through AI, have them solved with minimal effort and not wait for the students to do it.

Effective use of AI and the extent - yes, I like this idea too! Feel free to co-author this PR, change any text that you feel fit (or add to documentation/contributing guidelines).

@InAnYan Discussion: How does this sound:
"I fully understand the working of any code in this PR picked up from any external source/generated using AI."

calixtus · 2024-10-12T14:58:22Z

Intellectual Property is a thing in this context. We need to double check this legally I think.

koppor · 2024-10-12T18:21:45Z

.github/workflows/add-greeting-to-issue.yml

@@ -24,6 +24,8 @@ jobs:

          Newcomers, we're excited to have you on board. Start by exploring our [Contributing](https://github.com/JabRef/jabref/blob/main/CONTRIBUTING.md) guidelines, and don't forget to check out our [workspace setup guidelines](https://devdocs.jabref.org/getting-into-the-code/guidelines-for-setting-up-a-local-workspace) to get started smoothly.



we currently habe no formal code of conduct... not sure if we should invest time to create one...

HoussemNasri

LGTM!

Using AI to brainstorm ideas or increase productivity --> good 👍
Using AI to auto-generate all the code given the description of an issue without even trying to understand what is asked --> bad 👎

koppor · 2024-10-14T19:57:22Z

Related work:

Co-authored-by: Ruslan <ruslanpopov1512@gmail.com> Co-authored-by: Subhramit Basu Bhowmick <74734844+subhramit@users.noreply.github.com> Co-authored-by: Carl Christian Snethlage <50491877+calixtus@users.noreply.github.com> Co-authored-by: ThiloteE <73715071+ThiloteE@users.noreply.github.com>

* Add check for AI-generated PRs * Refine * Add to greeting * Remove reference to code of conduct * Change mandatory check * Refine condition Co-authored-by: Ruslan <ruslanpopov1512@gmail.com> Co-authored-by: Subhramit Basu Bhowmick <74734844+subhramit@users.noreply.github.com> Co-authored-by: Carl Christian Snethlage <50491877+calixtus@users.noreply.github.com> Co-authored-by: ThiloteE <73715071+ThiloteE@users.noreply.github.com> * Discard changes to .github/workflows/add-greeting-to-issue.yml * Update CONTRIBUTING.md Co-authored-by: Ruslan <ruslanpopov1512@gmail.com> Co-authored-by: Subhramit Basu Bhowmick <74734844+subhramit@users.noreply.github.com> Co-authored-by: Carl Christian Snethlage <50491877+calixtus@users.noreply.github.com> Co-authored-by: ThiloteE <73715071+ThiloteE@users.noreply.github.com> * Update PULL_REQUEST_TEMPLATE.md Co-authored-by: Ruslan <ruslanpopov1512@gmail.com> Co-authored-by: Subhramit Basu Bhowmick <74734844+subhramit@users.noreply.github.com> Co-authored-by: Carl Christian Snethlage <50491877+calixtus@users.noreply.github.com> Co-authored-by: ThiloteE <73715071+ThiloteE@users.noreply.github.com> --------- Co-authored-by: Oliver Kopp <kopp.dev@gmail.com> Co-authored-by: Ruslan <ruslanpopov1512@gmail.com> Co-authored-by: Carl Christian Snethlage <50491877+calixtus@users.noreply.github.com> Co-authored-by: ThiloteE <73715071+ThiloteE@users.noreply.github.com>

subhramit added 3 commits October 12, 2024 18:50

Add check for AI-generated PRs

63d1241

Refine

8d0073d

Add to greeting

dc68abb

subhramit requested a review from koppor October 12, 2024 14:05

InAnYan reviewed Oct 12, 2024

View reviewed changes

koppor reviewed Oct 12, 2024

View reviewed changes

HoussemNasri previously approved these changes Oct 12, 2024

View reviewed changes

Remove reference to code of conduct

1317081

subhramit dismissed HoussemNasri’s stale review via 1317081 October 12, 2024 18:59

subhramit changed the title ~~Add check for AI-generated PRs~~ Establish guidelines for AI-generated code submissions Oct 13, 2024

subhramit changed the title ~~Establish guidelines for AI-generated code submissions~~ Establish guidelines for usage of AI in PRs Oct 13, 2024

koppor added the status: devcall label Oct 14, 2024

Change mandatory check

7434a06

koppor force-pushed the checks branch from 17080c9 to 5be3612 Compare October 14, 2024 20:07

koppor and others added 3 commits October 14, 2024 22:08

Discard changes to .github/workflows/add-greeting-to-issue.yml

ce16be4

Merge branch 'main' into checks

7d3ae16

koppor previously approved these changes Oct 14, 2024

View reviewed changes

koppor removed the status: devcall label Oct 14, 2024

koppor enabled auto-merge October 14, 2024 20:19

koppor disabled auto-merge October 14, 2024 20:20

koppor dismissed their stale review via 0885afa October 14, 2024 20:23

koppor approved these changes Oct 14, 2024

View reviewed changes

koppor enabled auto-merge October 14, 2024 20:24

koppor added this pull request to the merge queue Oct 14, 2024

Merged via the queue into JabRef:main with commit 9e0343f Oct 14, 2024
23 checks passed

koppor deleted the checks branch October 14, 2024 20:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Establish guidelines for usage of AI in PRs #11940

Establish guidelines for usage of AI in PRs #11940

subhramit commented Oct 12, 2024 •

edited

Loading

InAnYan Oct 12, 2024 •

edited

Loading

subhramit Oct 12, 2024 •

edited

Loading

subhramit Oct 12, 2024 •

edited

Loading

calixtus commented Oct 12, 2024

koppor Oct 12, 2024

HoussemNasri left a comment

koppor commented Oct 14, 2024 •

edited

Loading

		@@ -24,6 +24,8 @@ jobs:

		Newcomers, we're excited to have you on board. Start by exploring our [Contributing](https://github.com/JabRef/jabref/blob/main/CONTRIBUTING.md) guidelines, and don't forget to check out our [workspace setup guidelines](https://devdocs.jabref.org/getting-into-the-code/guidelines-for-setting-up-a-local-workspace) to get started smoothly.

Establish guidelines for usage of AI in PRs #11940

Establish guidelines for usage of AI in PRs #11940

Conversation

subhramit commented Oct 12, 2024 • edited Loading

Background

Trigger

Proposed Action

Mandatory checks

InAnYan Oct 12, 2024 • edited Loading

Choose a reason for hiding this comment

subhramit Oct 12, 2024 • edited Loading

Choose a reason for hiding this comment

subhramit Oct 12, 2024 • edited Loading

Choose a reason for hiding this comment

calixtus commented Oct 12, 2024

koppor Oct 12, 2024

Choose a reason for hiding this comment

HoussemNasri left a comment

Choose a reason for hiding this comment

koppor commented Oct 14, 2024 • edited Loading

subhramit commented Oct 12, 2024 •

edited

Loading

InAnYan Oct 12, 2024 •

edited

Loading

subhramit Oct 12, 2024 •

edited

Loading

subhramit Oct 12, 2024 •

edited

Loading

koppor commented Oct 14, 2024 •

edited

Loading