Fog::Compute::AWS::Error: RequestLimitExceeded when using KNIFE EC2 CREATE

[jerseyfoxcom]

I'm getting a Fog::Compute::AWS::Error: RequestLimitExceeded error intermittently when attempting to EC2 is "Waiting for Windows Admin password to be available".

Usually works fine, but 1 in 10 times i'm getting this error and it's causing us some issues in our plan to use KNIFE EC2 for automated deployment.

[jerseyfoxcom]

Having researched this - it looks like the following is happening

http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate

Is there a way to retry and perform an exponential back-off with Knife-EC2?

This block of code appears to be the culprit -

def windows_password
if not locate_config_value(:winrm_password)
if locate_config_value(:identity_file)
print "\n#{ui.color("Waiting for Windows Admin password to be available", :magenta)}"
print(".") until check_windows_password_available(@server.id) {
sleep 1000 #typically is available after 30 mins
puts("done")
}
response = connection.get_password_data(@server.id)
data = File.read(locate_config_value(:identity_file))
config[:winrm_password] = decrypt_admin_password(response.body["passwordData"], data)
else
ui.error("Cannot find SSH Identity file, required to fetch dynamically generated password")
exit 1
end
else
locate_config_value(:winrm_password)
end
end

With this I get :-

Waiting for Windows Admin password to be available..................................................................................................................................................................ERROR: Fog::Compute::AWS::Error: RequestLimitExceeded => Request limit exceeded.

The odd thing is that I get loads of those "." turn up very quickly (5 a second) and cause the RequestLimitedExceeded to throw. It's almost like the wait 1000 doesn't work correctly.

[dheerajd-msys]

@jerseyfoxcom, Since this is intermittent error, I am unable to reproduce this error. Could you please tell me which command you used and other parameters with the command?

[dheerajd-msys]

@jerseyfoxcom Hope comments added in #462 help you for the similar issue.
Please feel free to reopen if you still face issue.

[jerseyfoxcom]

Just as an FYI - I've resolve this issue my end by making the following changes to the KNIFE-EC2 Gem and recompiling it.

This is all in EC2_server_create.rb

def check_windows_password_available(server_id)
sleep 10 #TODO:// Exponential backout required to prevent connection.get_password_data being called constantly.
#Added this at the start of the loop as it doesn't fire if the return false is hit
response = connection.get_password_data(server_id)
if not response.body["passwordData"]
return false
end
response.body["passwordData"]
end
This method was firing 5 times a second for me, and moving the sleep to the top of the function rather than at the end (which wasn't being hit when a return false; was hit first) has sorted this problem out.

[jerseyfoxcom]

Just as an FYI - I've resolve this issue my end by making the following changes to the KNIFE-EC2 Gem and recompiling it.

This is all in EC2_server_create.rb

  def check_windows_password_available(server_id)
	sleep 10 #ALTIS - TODO:// Exponential backout required to prevent connection.get_password_data being called constantly.
	#Added this at the start of the loop as it doesn't fire if the return false is hit
    response = connection.get_password_data(server_id)
    if not response.body["passwordData"]
      return false
    end
    response.body["passwordData"]
  end

This method was firing 5 times a second for me, and moving the sleep to the top of the function rather than at the end (which wasn't being hit when a return false; was hit first) has sorted this problem out.

[NimishaS]

@jerseyfoxcom , this issue is resolved in knife-ec2 0.15.0.

[jerseyfoxcom]

Hey, that's brilliant - we've been using a forked version with this change for some time, but that will allow us to go back onto the main branch again.

Adam