Rebase rusoto master

AWS-CREDENTIALS.md

@@ -1,20 +1,26 @@
 ### Credentials
 
-Rusoto has the ability to source AWS access credentials in a few different ways:
+#### Best Practices
 
-1. Environment variables via `rusoto_core::EnvironmentProvider` (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
-2. AWS credentials file via `rusoto_core::ProfileProvider`
-3. IAM ECS container profile via `rusoto_core::ContainerProvider`
-4. IAM EC2 instance profile via `rusoto_core::InstanceMetadataProvider`
+Please follow the [AWS documentation on best practices](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html) for managing credentials for your account. These credentials provide anyone who has access to them access to your AWS account. You can mitigate risk that goes along with that access by following these best practices.
 
-There is also `rusoto_core::ChainProvider`, which is a convenience for attempting to source access credentials using the methods above in order.
+#### Usage
+
+Much like the [standard AWS toolchain](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html), Rusoto has the ability to source AWS access credentials from multiple sources, either independently or in a tiered fashion.
+
+1. Environment variables via [`rusoto_core::EnvironmentProvider`](https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.EnvironmentProvider.html) (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
+2. AWS credentials file via [`rusoto_core::ProfileProvider`](https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.ProfileProvider.html)
+3. IAM ECS container profile via [`rusoto_core::ContainerProvider`](https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.ContainerProvider.html)
+4. IAM EC2 instance profile via [`rusoto_core::InstanceMetadataProvider`](https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.InstanceMetadataProvider.html)
+
+The [`rusoto_core::ChainProvider`](https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.ChainProvider.html) is a convenience for attempting to source access credentials using all the methods above in that order.
 If credentials cannot be obtained through one method, it falls back to the next.
 If all possibilites are exhausted, an error will be returned.
 
 `ProfileProvider` (and `ChainProvider`) also allow you to specify a custom path to the credentials file and the name of the profile to use.
 If not explicitly provided as arguments, the values for these two parameters are computed according to the following rules:
-* **location of credentials file:** if set and not empty the value of the environment variable ```AWS_SHARED_CREDENTIALS_FILE``` otherwise ```"~/.aws/credentials"```.
-* **profile name:** if set and not empty the value of the environment variable ```AWS_PROFILE``` otherwise ```"default"```
+* **location of credentials file:** if set and not empty the value of the environment variable `AWS_SHARED_CREDENTIALS_FILE` otherwise `"~/.aws/credentials"`.
+* **profile name:** if set and not empty the value of the environment variable ```AWS_PROFILE``` otherwise `"default"`
 
 It's also possible to implement your own credentials sourcing mechanism by creating a type that implements `rusoto_core::ProvideAwsCredentials`.
 
@@ -23,11 +29,6 @@ It's also possible to implement your own credentials sourcing mechanism by creat
 If your aws account belongs to an organization and you need to use sts:AssumeRole, you're probably looking for `rusoto_sts::StsAssumeRoleSessionCredentialsProvider`. A simple program that uses sts:AssumeRole looks like this:
 
 ```rust,no_run
-extern crate env_logger;
-extern crate rusoto_core;
-extern crate rusoto_ec2;
-extern crate rusoto_sts;
-
 use std::default::Default;
 
 use rusoto_core::{Region, HttpClient};
@@ -36,7 +37,7 @@ use rusoto_ec2::{Ec2Client, Ec2, DescribeSpotInstanceRequestsRequest};
 use rusoto_sts::{StsClient, StsAssumeRoleSessionCredentialsProvider};
 
 fn main() {
-    let _ = env_logger::try_init();
+    env_logger::init();
 
     let sts = StsClient::new(Region::EuWest1);
 
@@ -56,11 +57,12 @@ fn main() {
 }
 ```
 
-### Important note about using the StsAssumeRoleSessionCredentialsProvider in the recommended way
-**Be careful** that the current behavior of `rusoto_sts::StsAssumeRoleSessionCredentialsProvider` needs to be used with `rusoto_credential::AutoRefreshingProvider` as a wrapper to get advantage of using the already cached token of AssumeRole as it lives by default for 1 hour.
-Current implementation is not using the cached token returned by the AssumeRole by default so it will be refreshed with every call to AWS resource.
+### Important note about using the StsAssumeRoleSessionCredentialsProvider
 
-This will affect the performance as well as the billing of AWS.
+**Be careful**. The current `rusoto_sts::StsAssumeRoleSessionCredentialsProvider` should be used with `rusoto_credential::AutoRefreshingProvider` as a wrapper to take advantage of using the already cached token of AssumeRole as its only valid for 1 hour by default.
+The current implementation does not reuse the cached token returned by the AssumeRole by default so it will be refreshed with every call to AWS resource.
+
+This will affect the performance as well as the size of you AWS bill.
 
 - https://docs.rs/rusoto_credential
 - https://crates.io/crates/rusoto_credential
@@ -75,21 +77,11 @@ let provider = StsAssumeRoleSessionCredentialsProvider::new(
 let auto_refreshing_provider = rusoto_credential::AutoRefreshingProvider::new(provider);
 ```
 
-#### Credential refreshing
-
-Credentials obtained from environment variables and credential files expire ten minutes after being acquired and are refreshed on subsequent calls to `credentials()` (a method from the `ProvideAwsCredentials` trait).
-
-IAM instance profile credentials are refreshed as needed.
-Upon calling `credentials()` it will see if they are expired or not.
-If expired, it attempts to get new credentials from the metadata service.
-If that fails it will return an error.
-IAM credentials expiration time comes from the IAM metadata response.
-
 #### Local integration testing of IAM credentials
 
 Edit the relevant `address`/IP locations in [credential/src/container.rs](credential/src/container.rs) and [credential/src/instance_metadata.rs](credential/src/instance_metadata.rs).
 For local testing, you can use [moe](https://github.com/matthewkmayer/moe) and set the string to this:
 
 ```rust,ignore
-let mut address: String = "http://localhost:8080/latest/meta-data/iam/security-credentials".to_owned();
+let address: String = "http://localhost:8080/latest/meta-data/iam/security-credentials".to_owned();
 ```

CHANGELOG.md

@@ -3,6 +3,42 @@
 ## [Unreleased]
 
 (Please put changes here)
+- Use static initializer for AWS profile regex
+- Add QLDB service
+- Add QLDB Session service
+- Update Skeptic tests for Rusoto v0.41
+- Don't decode query string parameters before encoding it. Results in fixing the prefix and marker
+params for s3 `list_objects` methods
+- Add Textract service
+- Update CloudDirectory API definition to `2017-01-11`
+- Add SecurityHub service
+- Add Transfer service
+- Introducing `rusoto_signature`, a standalone crate for signing HTTP requests.
+- Make static credentials into a credential provider
+- Add anonymous credentials support
+- Don't trim whitepsace when parsing xml payload. Fixes truncating of items with spaces in payloads
+such as an S3 key returned in `list_objects_v2`
+
+## [0.41.0] - 2019-10-07
+
+- Add `HttpClient::from_builder`
+- Upgrade to botocore from `1.12.163` to `1.12.230`
+- The types `rusoto_events::{CloudWatchEvents,CloudWatchEventsClient}` were renamed to `rusoto_events::{EventBridge,EventBridgeClient}`
+- Deserialize PostTextResponse correctly by allowing null values in the slots field
+- Fix Profile Config Loading: should ignore comments with '=' chars
+- Add App Mesh service
+- Fix service_crategen to parse operations with multiple static params
+- Refactor S3 integration tests - about a `#[test]` per behavior
+- Add support for non signing clients
+- Add EC2 Instance Connect service
+- Allow deserialization of regions without an endpoint specified
+- Add ApNortheast3 region
+- Add MeSouth1 region
+- Add x-amz-content-sha256 header to signed and canonical headers
+- Added `Eq` and `Hash` implementations on `Region`
+- Fixed parsing of Athena error messages
+- Fix credential_process behavior when using the non-default profile
+- Correctly read session tokens from credential_process
 
 ## [0.40.0] - 2019-06-28
 

Cargo.toml

@@ -1,8 +1,10 @@
 [workspace]
 members = [
     "mock",
+    "rusoto/credential_service_mock",
     "rusoto/core",
     "rusoto/credential",
+    "rusoto/signature",
     "rusoto/services/acm",
     "rusoto/services/acm-pca",
     "rusoto/services/alexaforbusiness",
@@ -11,6 +13,7 @@ members = [
     "rusoto/services/apigatewaymanagementapi",
     "rusoto/services/apigatewayv2",
     "rusoto/services/application-autoscaling",
+    "rusoto/services/appmesh",
     "rusoto/services/appstream",
     "rusoto/services/appsync",
     "rusoto/services/athena",
@@ -54,6 +57,7 @@ members = [
     "rusoto/services/dynamodb",
     "rusoto/services/dynamodbstreams",
     "rusoto/services/ec2",
+    "rusoto/services/ec2-instance-connect",
     "rusoto/services/ecr",
     "rusoto/services/ecs",
     "rusoto/services/elasticache",
@@ -117,6 +121,8 @@ members = [
     "rusoto/services/pi",
     "rusoto/services/polly",
     "rusoto/services/pricing",
+    "rusoto/services/qldb",
+    "rusoto/services/qldb-session",
     "rusoto/services/ram",
     "rusoto/services/rds",
     "rusoto/services/rds-data",
@@ -131,6 +137,7 @@ members = [
     "rusoto/services/sagemaker-runtime",
     "rusoto/services/sdb",
     "rusoto/services/secretsmanager",
+    "rusoto/services/securityhub",
     "rusoto/services/serverlessrepo",
     "rusoto/services/servicecatalog",
     "rusoto/services/servicediscovery",
@@ -146,7 +153,9 @@ members = [
     "rusoto/services/sts",
     "rusoto/services/support",
     "rusoto/services/swf",
+    "rusoto/services/textract",
     "rusoto/services/transcribe",
+    "rusoto/services/transfer",
     "rusoto/services/translate",
     "rusoto/services/waf",
     "rusoto/services/waf-regional",
@@ -167,4 +176,4 @@ exclude = [
 [profile.bench]
 opt-level = 3
 debug = false
-debug-assertions = false
+debug-assertions = false

Makefile

@@ -27,6 +27,11 @@ docs:
 unit_test:
 	cargo +$$RUST_VERSION test --all
 
+# Doctests can be very slow to compile and run. This option lets us skip those if needed.
+.PHONY: unit_test_no_doctests
+unit_test_no_doctests:
+	cargo +$$RUST_VERSION test --all --lib
+
 .PHONY: skeptical
 skeptical:
 	(cd skeptical && cargo +$$RUST_VERSION test)
@@ -41,7 +46,8 @@ check_integration_test:
 
 .PHONY: rustls_unit_test
 rustls_unit_test:
-	cargo +$$RUST_VERSION test --all -v --no-default-features --features=rustls
+	(cd rusoto/core && cargo +$$RUST_VERSION test --no-default-features --features=rustls)
+	(cd rusoto/services && ./rustls-unit-test.sh $$RUST_VERSION)
 
 .PHONY: check_service_defintions
 check_service_defintions:
@@ -54,3 +60,7 @@ time_credentials:
 .PHONY: bench_s3
 bench_s3:
 	(cd rusoto/services/s3 && cargo +nightly bench)
+
+.PHONY: credential_integration_test
+credential_integration_test:
+	(cd rusoto/credential_service_mock && ./run-and-test.sh )

README.md

@@ -43,9 +43,9 @@ For example, to include only S3 and SQS:
 
 ```toml
 [dependencies]
-rusoto_core = "0.40.0"
-rusoto_sqs = "0.40.0"
-rusoto_s3 = "0.40.0"
+rusoto_core = "0.41.0"
+rusoto_sqs = "0.41.0"
+rusoto_s3 = "0.41.0"
 ```
 
 ## Migration notes
@@ -110,12 +110,14 @@ Information on release schedules and procedures are in [RELEASING](RELEASING.md)
 
 See [CONTRIBUTING](CONTRIBUTING.md).
 
-## Supported OSs and Rust versions
+## Supported OSs, Rust versions and non-AWS projects
 
 Linux, OSX and Windows are supported and tested via Azure Pipelines and Appveyor.
 
 Rust stable, beta and nightly are supported.
 
+Rusoto's primary aim is to be used with AWS. Other projects that provide AWS-like APIs, such as Ceph, Minio, Yandex Object Storage, etc... are not a focus at this time. PRs to fix issues with Rusoto and AWS-like APIs are welcome but generally won't be created by Rusoto maintainers.
+
 ## License
 
 Rusoto is distributed under the terms of the MIT license.

RELEASING.md

@@ -27,18 +27,20 @@ Rusoto uses [semantic versioning 2.0.0](http://semver.org/).
 3. Publish new version of `rusoto_credential` if changes have been made to it.
 4. Publish new version of `rusoto_core` if changes have been made to it.
 5. Publish new version of `rusoto_mock` if it changes have been made to it.
-6. Run `publish-services.sh` in the `rusoto/services` dir. *Warning*: takes >2 hours on a low end Macbook. The script can be run again if an issue comes up without problems - crates.io prevents republishing.
+6. Run `publish-services.sh` in the `rusoto/services` dir. *Warning*: takes >4 hours on a low end Macbook. The script can be run again if an issue comes up without problems - crates.io prevents republishing.
 7. Tag master branch with the new version.  Example: `git tag -a rusoto-v0.21.0 -m "Rusoto 0.21.0 release."` then `git push --tags origin`.
 
 ### Git tags
 
-Due to multiple crates being in the repo, releases for each crate will be in the format `crate-vmajor.minor.patch`.
+Due to multiple crates being in the repo, releases for each crate will be in the format `crate-vmajor.minor.patch`. Rusoto core, service crates, credentials and `rusoto_mock` will all have the same versions for a new release:
 
 Examples:
 
-* `rusoto-v0.21.0`
-* `credentials-v0.3.0`
-* `mock-v0.27.0`
+* `rusoto-v0.41.0`
+* `credentials-v0.41.0`
+* `mock-v0.41.0`
+
+When bug fixes for a crate are published, all crates get a new release.
 
 ### Release notes
 

appveyor.yml

@@ -27,8 +27,9 @@ install:
 build: off
 test_script:
   - cargo update
-  - make unit_test
-  - make rustls_unit_test
+  - set RUST_VERSION=%CHANNEL%
+  - echo using rust_version %RUST_VERSION%
+  - make unit_test_no_doctests
   - make check_integration_test
 
 branches: