[Gitlab] - Empty changes list

[a-chartier]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

This is a self hosted Gitlab enterprise cluster on 15.9.2

I am seeing an initial empty changes response from Gitlab after the web-hook triggers the GetModifiedFiles function.

The initial Query to the /changes endpoint seems to have the possibility of returning an empty list of changes once triggered after the initial web-hook. However If i add a quick sleep into this before we actually query the /changes endpoint the data is returned correctly.

To illustrate this point I have modified the Atlantis code and created a custom build that prints out the response body after the web-hook trigger event.

{
	"id": 8312,
	"iid": 102,
	"target_branch": "main",
	"source_branch": "TestBranch",
	"project_id": 3771,
	"title": "Update <somefile>.yaml",
	"state": "opened",
	"created_at": "2023-03-14T13:43:17.895Z",
	"updated_at": "2023-03-14T13:43:17.895Z",
	"upvotes": 0,
	"downvotes": 0,
	"author": {
		"id": 194,
		"username": "<Me>",
		"name": "<Me>",
		"state": "active",
		"created_at": null,
		"avatar_url": "<Generic avatar_url>",
		"web_url": "<Gitlab URL>"
	},
	"assignee": null,
	"assignees": [],
	"reviewers": [],
	"source_project_id": 3771,
	"target_project_id": 3771,
	"labels": "",
	"description": "",
	"draft": false,
	"work_in_progress": false,
	"milestone": null,
	"merge_when_pipeline_succeeds": false,
	"detailed_merge_status": "checking",
	"merge_error": "",
	"merged_by": null,
	"merged_at": null,
	"closed_by": null,
	"closed_at": null,
	"subscribed": false,
	"sha": "3f382318c8ce9b0b606682c7e5a77753b6e157d8",
	"merge_commit_sha": "",
	"squash_commit_sha": "",
	"user_notes_count": 0,
	"changes_count": "",
	"should_remove_source_branch": false,
	"force_remove_source_branch": true,
	"allow_collaboration": false,
	"web_url": "<repo-url>/102",
	"references": {
		"short": "!102",
		"relative": "!102",
		"full": "<repo-url>!102"
	},
	"discussion_locked": false,
	"changes": [],
	"user": {
		"can_merge": true
	},
	"time_stats": {
		"human_time_estimate": "",
		"human_total_time_spent": "",
		"time_estimate": 0,
		"total_time_spent": 0
	},
	"squash": false,
	"pipeline": null,
	"head_pipeline": null,
	"diff_refs": {
		"base_sha": "",
		"head_sha": "",
		"start_sha": ""
	},
	"diverged_commits_count": 0,
	"rebase_in_progress": false,
	"approvals_before_merge": 0,
	"reference": "!102",
	"first_contribution": false,
	"task_completion_status": {
		"count": 0,
		"completed_count": 0
	},
	"has_conflicts": false,
	"blocking_discussions_resolved": true,
	"overflow": false,
	"merge_status": "checking"
}

Please take note of the fact that changes is an empty slice and changes_count is actually an empty string.

Now now the I sleep for 2 seconds and try the query again.

{
	"id": 8312,
	"iid": 102,
	"project_id": 3771,
	"title": "Update <somefile>.yaml",
	"description": "",
	"state": "opened",
	"created_at": "2023-03-14T13:43:17.895Z",
	"updated_at": "2023-03-14T13:43:59.978Z",
	"merged_by": null,
	"merge_user": null,
	"merged_at": null,
	"closed_by": null,
	"closed_at": null,
	"target_branch": "main",
	"source_branch": "TestBranch",
	"user_notes_count": 0,
	"upvotes": 0,
	"downvotes": 0,
	"author": {
		"id": 194,
		"username": "<Me>",
		"name": "<Me>",
		"state": "active",
		"created_at": null,
		"avatar_url": "<Generic avatar_url>",
		"web_url": "<Gitlab URL>"
	},
	"assignees": [],
	"assignee": null,
	"reviewers": [],
	"source_project_id": 3771,
	"target_project_id": 3771,
	"labels": [],
	"draft": false,
	"work_in_progress": false,
	"milestone": null,
	"merge_when_pipeline_succeeds": false,
	"merge_status": "can_be_merged",
	"detailed_merge_status": "not_approved",
	"sha": "3f382318c8ce9b0b606682c7e5a77753b6e157d8",
	"merge_commit_sha": null,
	"squash_commit_sha": null,
	"discussion_locked": null,
	"should_remove_source_branch": null,
	"force_remove_source_branch": true,
	"reference": "!102",
	"references": {
		"short": "!102",
		"relative": "!102",
		"full": "<repo-url>!102"
	},
	"web_url": "<repo-url>/102",
	"time_stats": {
		"time_estimate": 0,
		"total_time_spent": 0,
		"human_time_estimate": null,
		"human_total_time_spent": null
	},
	"squash": false,
	"squash_on_merge": false,
	"task_completion_status": {
		"count": 0,
		"completed_count": 0
	},
	"has_conflicts": false,
	"blocking_discussions_resolved": true,
	"approvals_before_merge": null,
	"subscribed": true,
	"changes_count": "1",
	"pipeline": null,
	"head_pipeline": null,
	"diff_refs": {
		"base_sha": "3fc27008bb70a83e3e5d79fce5b6f7e0b3f02a09",
		"head_sha": "3f382318c8ce9b0b606682c7e5a77753b6e157d8",
		"start_sha": "3fc27008bb70a83e3e5d79fce5b6f7e0b3f02a09"
	},
	"merge_error": null,
	"user": {
		"can_merge": true
	},
	"changes": [{
		"diff": "<removed file diff>",
		"new_path": "<somefile>.yaml",
		"old_path": "<somefile>.yaml",
		"a_mode": "100644",
		"b_mode": "100644",
		"new_file": false,
		"renamed_file": false,
		"deleted_file": false
	}],
	"overflow": false
}

You can see that the API is returning different results for changes_count and changes. It's almost like cert parts of the change have not been fully flushed through the system before we make the initial query.

This feels like a bug with the latest Gitlab patch. We have opened a ticket with Gitlab on this. When we get a response I will update this issue accordingly.

However I think it should also be possible to try and detect against situations like this. It appears that when a change request does not have any changes the changes_count field is actually null instead of an empty string.

"changes_count": null,

Reproduction Steps

Logs

Environment details

Additional Context

[jseiser]

@a-chartier

We hit this same exact issue, with our self hosted gitlab. The MR's come in with no change which breaks everything. We also have an open issue with Gitlab.

I see you can atlantis plan to kick it off again.

[jseiser]

@a-chartier

They opened a bug for what its worth: https://gitlab.com/gitlab-org/gitlab/-/issues/397698

[a-chartier]

@jseiser
Sorry I wasn't around much this weekend, we submitted the bug report and all the logs to them once I figured out what was going on, I was hoping to avoid having other people waste time on this.

But it seems like they don't consider it a bug, but it's a breaking change. Since they don't consider this a bug I think some changes may need to be made on the Atlantis side to check until we have a change count that isn't none.

[nitrocode]

Other older issue that the author linked to

https://gitlab.com/gitlab-org/gitlab/-/issues/386562

Unsure what the easy solution is here unless we were to delay each gitlab call to the endpoint by 5 seconds... Which seems like a gross solution.

Hopefully they fix the regression in 397698 before implementing the new endpoint referenced in 386562.

[a-chartier]

@nitrocode

Yeah I don't like the idea of a hard sleep, I was thinking that in the event that a change comes back with a null or empty changes_count Atlantis could just retry the the call.

Since changes_count should either be 0 or N for it to be a valid response.

[jseiser]

@nitrocode
@a-chartier

Its pretty clear they broke their stuff during a minor version release, and are now creating MR's to reflect they did so in their docs. The current open issue to resolve this, hasn't even been assigned in 2 months. Talking to our PM, it doesnt appear that Gitlab is going to do anything to fix this on their end, since they have decided its not broken.

Unsure what the easy solution is here unless we were to delay each gitlab call to the endpoint by 5 seconds... Which seems like a gross solution.

Im not sure there is any other choice, but to query/back off/query until its not ""

We are trying to get a Pull Request together done on this, we do have a Go Lang dev, but he has no concept/idea of atlantis so trying to get this moving.