Provision of dns_resource records fails with Too many request

[hkantare]

he problem I am seeing now is that "terraform apply" or "terraform destroy" would not work when the number of resources is above certain number. Terraform would just spit out "too many requests" errors in the end. It can be when we do "apply" or "destroy". If you repeat the command again, sometimes it went through. But for "terraform destroy", repeating the commands does not seem to resolve the issue.

Here is the error reported by Terraform:

ibm_dns_resource_record.lsf-dns-worker-record-a_38: Refreshing state... [id=13546c6d-adc7-4b6a-819a-da55491f2b21/hf-lsf.com:c41d2ba1-f9c7-4e31-8cf2-ca3ff6fa9c6e/A:6948f151-0e4b-4abd-a6d3-8de604044b10]
null_resource.cluster_38: Refreshing state... [id=5378620011952565550]
Error: Too Many Requests
Error: Too Many Requests

[MalarvizhiK]

I tried earlier by placing lock in zone id. It did not work. Hence, @hkantare is suggesting to place lock in instance id and zone id and give a try for create, update, delete, read and exists.

[MalarvizhiK]

I am looking into this issue now. I will test @hkantare's file changes and see if it resolves the issue.

[hkantare]

Take latest code from the branch
and test once

[MalarvizhiK]

ok.

[MalarvizhiK]

I am unable to create 60 VSI due to over quota issue. I changed profile to bx2-2x8 and created 10 VSI and used the IP v4 address for 60 dns resource records, it worked fine. I used the latest terraform provider: https://github.com/IBM-Cloud/terraform-provider-ibm/releases/tag/v1.8.1

[MalarvizhiK]

@hkantare will send a mail to the person who sent the issue in email and follow up with him for recreation steps and terraform scripts as well. We are unable to recreate the issue.

[hfwen0502]

I think as long as there is a limit to the number of api requests within a period of time in the DNS service, users can have a chance to hit this problem, if they create a lot of VSIs with enough parallelism. I was using the DNS python SDK and it's very easy to hit this issue due to exceeding the api request within a short period of time. I am not sure if this is something that can be fixed at the terraform level. If users increases the parallelism from 10 by default to 64 for example, assuming their server has a lot of cores, users would see this problem when they are creating/destroying hundreds of VSIs.

One thing I am confused is that based on my understanding Terraform creates the dns resource records serially. Somehow I remember seeing the too many request errors using 1.7 version. I don't remember if I have tested extensively with 1.8. Maybe the problem went away in 1.8. If the dns resource record is created serially, we should have a much lower chance to hit the api limit in the DNS.

Sophia Wen hfwen@us.ibm.com

[MalarvizhiK]

@hfwen0502 Looks like the issue is not seen in terraform provider version 1.8.1. The Private dns api does have rate limiting rules. Private DNS API implemented rate limiting rules, which only allows 100 requests per 30 seconds. Excess requests will get blocked with 429 status code. Shall we close the defect ?