[storage-file] Docs updates (#6189)

* Fix failure to parse UTF8 files with BOM (#6160)

* Fix failure to parse UTF8 files with BOM

* Match frozen requirements in cosmos

* build universal wheels (#6173)

* Batch of doc

* Typo

* Update README and HISTORY of EventHub (#6151)

* Update readme

* Update sample code, obtain connection string and documentation reference in readme

* Remove import os in readme

* Added identity snippets in readme

* Update history, readme and license

* Update history

* Update history of EventHub

* Fix wrong version

* Update history of EventHub

* Update history to be more specific

* Update install command in readme for a preview version

* Update reference documentation link in readme

* don't scan anything under doc/ (#6185)

* Docs updates

* Some docs fixes

* Expand test coverage (#6170)

* Fixed model reference

* Fixed another reference

* Azure Storage Queue Track2 (#6039)

* Initial commit for Queues (#5656)

* Initial commit

* fix build

* fix build - 2

* add version

* API buildout (#5686)

* [Storage Queues] Add autorest layer to storage queues (#5687)

* add queue.json in swagger

* Add autorest layer

* Hooking up Queue client and Queue Service Client (#5817)

* Queue client

* changes

* shared access signature

* some fixes

* add tests

* Delete settings_real.py

* Refactored shared code and added tests

* Packaging update of azure-storage-queue

* Revert "Packaging update of azure-storage-queue"

This reverts commit fdc62ab.

* Add packaging overwrite

* Fixed setup

* Fixed dependency

* Storage Queues readme (#6073)

* queues readme [wip]

* small fixes

* docstrings for queues (#6066)

* Updated user-agent policy

* Updated token test

* Updated message iterator to pageable

* Renamed dequeue to receive

* Removed Py3.4 tag

* Clean pylint

* Code samples for queues (#6088)

* adds code samples for queues

* fixes to code samples and adds literalincludes for docs

* add links for TODO

* uses unicode strings for python2 compatibility

* fix readme link

* Fix CI

* [Storage Queue] pylint  + mypy pass (#6187)

* [storage-queue] Docs updates (#6186)

* Docs updates

* First round review feedback

* Adopt MSAL 0.4.1 (#6176)

* Azure Storage Blob Track2 (#5896)

* Storage blob track2 (#5894)

* Blob storage track 2

* Started pylint fixes

* Clean pylint

* Started blob walk

* Started docstrings

* Support walk blob

* Pylint

* Refactored upload

* Pylint

* Packaging update of azure-storage-blob

* Fixes to storage dependencies in CI (#5900)

* Revert "Packaging update of azure-storage-blob"

This reverts commit aeeb3ce.

* Fix Storage CI

* Fixed imports

* updating ccount_key_arg -> account_key_arg

* updating ordering

* Packaging update of azure-storage-blob

* Revert "Packaging update of azure-storage-blob"

This reverts commit 66d6877.

* reverting 66d6877. adding sdk_packaging.toml file

* docstrings for blobs (#5954)

* [storage-blob] Refactor shared code (#5960)

* Fix Storage CI

* Fixed imports

* Refactor shared storage code

* Fixed XML error

* Reverted change to models

* Merged models docstrings

* Fix setup.py version

* Missing recordings

* Fix storage SDK conflict

* Packaging update of azure-eventhubs

* Revert "Packaging update of azure-eventhubs"

This reverts commit df844f7.

* Fix EH packaging

* Updated shared code

* Review feedback

* [azure-storage-blob] Policy updates + simple upload/download (#6162)

* Last blob updates

* Azure core dependency version

* Update to user-agent policy

* Review feedback

* Removed old python tags

* Code samples for blobs (#6093)

* adds code samples for blobs

* fixes blob code samples and adds literalincludes for docs

* Storage Blobs Readme [wip] (#6011)

* blobs readme wip

* adding some samples and fixes for readme

* fix settings import

* moving code samples to a different PR

* small fixes to readme

* fill links for pypi and identity

* populating TODO links

* Fix CI

* [Storage Blob] pylint + mypy passs (#6175)

* pylint + mypy passs

* comments

* [storage-blob] Docs updates (#6184)

* Docs updates

* Some docstring fixes

* [storage-blob] Test rename (#6197)

* Test rename

* Removed test init files

* More renames

* Try flattening tests

* Blob test imports

* Flattened queue tests

* Renamed queue test case

* Updated manifest

* Renamed testcase

* Renamed samples

* Flattened tests

* Update PyPI package links on readme Key Vault (#6194)

* Renamed recordings

* fixes sample readme links (#6198)

* [azure-storage] Missing crypto dependency (#6203)

* [azure-storage] Missing crypto dependency

* Updated queue setup with crypto

* Fixed blob nspkg dependency

* Fixed queue nspkg dependency

* Added storage nspkg to shared deps

* Fixed dependencies

* Separate settings files

.docsettings.yml

@@ -5,6 +5,7 @@ omitted_paths:
   - "azure-mgmt*/*"
   - "sdk/azure-mgmt*/*"
   - "sdk/*/azure-mgmt*/*"
+  - "doc/*"
 language: python
 root_check_enabled: True
 required_readme_sections:

.gitignore

@@ -71,6 +71,7 @@ src/build
 adlEnv/
 
 code_reports
+
+sdk/storage/azure-storage-blob/tests/settings_real.py
 sdk/storage/azure-storage-queue/tests/settings_real.py
 sdk/storage/azure-storage-file/tests/settings_real.py
-sdk/storage/azure-storage-blob/tests/settings_real.py

doc/README.md

@@ -0,0 +1,5 @@
+This folder contains some documentations for this repository:
+
+The folder structure is the following
+- [sphinx](./sphinx) : contains the documentation source code for https://azure.github.io/azure-sdk-for-python/
+- [dev](./dev) : contains advanced documentation for _developers_ of SDK (not _consumers_ of SDK)

doc/dev/README.md

@@ -0,0 +1,10 @@
+# Dev documentation
+
+This folder contains documentation for developers of SDK: internal teams at Microsoft, or advanced contributors.
+
+Overview of the documents:
+- [dev setup](./dev_setup.md) : How to create a development environment for this repo
+- [release](./release.md) : How to release a package when ready
+- [packaging.md](./packaging.md) : How to organize packaging information for packages under `azure`
+
+The [mgmt](./mgmt) folder contains information specific to management packages (i.e. packages prefixed by `azure-mgmt`)

doc/dev/dev_setup.md

@@ -0,0 +1,43 @@
+# Getting the tests to run
+
+This section describes how to create a dev environment, in order to run the SDK tests,
+or execute the various commands available in the toolbox.
+
+## Setting up a virtual environment
+
+1.  If you don't already have it, install Python:
+
+    - Windows: https://www.python.org/downloads/ or from the windows store https://www.microsoft.com/p/python-37/9nj46sx7x90p
+    - Ubuntu/Debian `sudo apt-get install python3`
+    - RHEL/CentOS `sudo yum install python3`
+
+    Python is also available in Bash for Windows natively.
+
+3.  Clone the repository and go to the folder
+
+    ```
+    git clone https://github.com/Azure/azure-sdk-for-python.git
+    cd azure-sdk-for-python
+    ```
+
+2.  Create a [virtual environment](https://docs.python.org/3/tutorial/venv.html)
+    You can initialize a virtual environment this way:
+
+    ```
+    python -m venv env # Might be "python3" or "py -3.6" depending on your Python installation
+    source env/bin/activate      # Linux shell (Bash, ZSH, etc.) only
+    ./env/scripts/activate       # PowerShell only
+    ./env/scripts/activate.bat   # Windows CMD only
+    ```
+
+4.  Setup your env (installing dependencies, etc.)
+
+    ```
+    python scripts/dev_setup.py
+    ```
+
+    If you work on a particular package, you can setup only this package to save some installation time:
+
+    ```
+    python scripts/dev_setup.py -p azure-mgmt-service
+    ```

doc/dev/mgmt/README.md

@@ -1,4 +1,14 @@
 # Management and ARM SDK documentation for developers
 
-The documentation in that folder is intended for developers at Microsoft, or for advanced contributors in the ARM world.
+The documentation in that folder is intended for developers at Microsoft, or for advanced contributors, in the ARM (Azure Resource Management) world.
+
+The [cheat sheet](./cheatsheet.md) page contains a quick reminder of the basic commands if you are already familiar with this repo.
+
+In order of workflow:
+- [swagger_conf.md](./swagger_conf.md) : Describe how to configure the different Readme on https://github.com/Azure/azure-rest-api-specs for Python
+- [tests.md](./tests.md) : How to test management SDK (recordings, etc.)
+- [mgmt_release.md](./mgmt_release.md) : How to finish preapring the package, changelog, version, etc.
+
+More implementations / advanced documents:
+- [changelog_impl.md](./changelog_impl.md) : Technical details about the change log tool implementation
 

doc/dev/mgmt/changelog_impl.md

@@ -0,0 +1,107 @@
+# Automatic ChangeLog
+
+This article describes the automatic changelog system used to generated entries in the ChangeLog of every packages in this repository. Although the current implementation is specific to Autorest generated packages, the logic inside is not, and could be adapted to other type of SDK or other packages.
+
+## Overview
+
+In order to compute this ChangeLog, we do the following steps:
+- Build a JSON representation of a package. We describe the more precisely possible every methods, parameters, classes, etc. in a JSON format. We do this step for both the current candidate package and the last one released
+- We compute a JSON diff between these two outputs
+- We use a "rule" based engine to interpret all entries in the diff in terms of ChangeLog. For example, a node that disappears from the JSONs means a parameters or a methods disappeared.
+
+## Build a JSON report of a package
+
+This is an example of the current JSON format:
+
+```json
+{
+  "models": {
+    "enums": {
+      "AvailabilitySetSkuTypes": {
+        "name": "AvailabilitySetSkuTypes",
+        "type": "Enum",
+        "values": [
+          "classic",
+          "aligned"
+        ]
+      }
+    },
+    "exceptions": {},
+    "models": {
+      "AdditionalCapabilities": {
+        "name": "AdditionalCapabilities",
+        "type": "Model",
+        "parameters": {
+          "ultra_ssd_enabled": {
+            "name": "ultra_ssd_enabled",
+            "properties": {
+              "type": "bool",
+              "required": false,
+              "readonly": false
+            }
+          }
+        }
+      }
+    }
+  },
+  "operations": {
+    "AvailabilitySetsOperations": {
+      "name": "AvailabilitySetsOperations",
+      "functions": {
+        "create_or_update": {
+          "name": "create_or_update",
+          "metadata": {
+            "url": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/availabilitySets/{availabilitySetName}"
+          },
+          "parameters": [
+            {
+              "name": "resource_group_name"
+            },
+            {
+              "name": "availability_set_name"
+            },
+            {
+              "name": "parameters"
+            }
+          ]
+        }
+```
+
+The current logic to build this JSON is Python specific right now, we introspect the Python code.
+
+One important concept to allow the best possible diff, is to avoid as much as possible array in the JSON. Arrays are ordered in JSON, so if what you model does not have ordered elements, don't use arrays. For instance, "create_or_update" or "delete" are two operations that are not ordered one to the other, so we don't use an array of operations, but a dictionary where keys are operation name. This has MASSIVE impact on the processing time of the diff.
+
+## Doing the diff
+
+We tried several JSON diff libraries, and currently use JSON-Delta:
+https://json-delta.readthedocs.io/en/latest/
+
+This library provides the diff which is the most simple to parse and interpret.
+
+Roughly, this provides an array of tuple:
+- Case 1 is the JSON path to apply it on file 1
+- Case 2 is the change itself to replace in this JSON path.
+
+Then, if case 2 is empty, this means deletion.
+
+## Creating the ChangeLog
+
+Then we have a set of rules that we apply for each entry in the diff. Currently templated rules are:
+
+```python
+## Features
+_ADD_OPERATION_GROUP = "Added operation group {}"
+_ADD_OPERATION = "Added operation {}.{}"
+_MODEL_PARAM_ADD = "Model {} has a new parameter {}"
+
+## Breaking Changes
+_REMOVE_OPERATION_GROUP = "Removed operation group {}"
+_REMOVE_OPERATION = "Removed operation {}.{}"
+_SIGNATURE_CHANGE = "Operation {}.{} has a new signature"
+_MODEL_SIGNATURE_CHANGE = "Model {} has a new signature"
+_MODEL_PARAM_DELETE = "Model {} no longer has parameter {}"
+_MODEL_PARAM_ADD_REQUIRED = "Model {} has a new required parameter {}"
+_MODEL_PARAM_CHANGE_REQUIRED = "Parameter {} of model {} is now required"
+```
+
+We right now explore the JSON diff in Python, but this could be done with a set of JSON path rules to be more generic if necessary.

doc/dev/mgmt/generation.md

@@ -0,0 +1,98 @@
+# Generation of SDK
+
+Assuming your Swagger are associated with correct Readmes (otherwise see previous chapter [Swagger conf](./swagger_conf.md)), this page explains how to generate your packages.
+
+IMPORTANT NOTE: All the commands prefixed by `python` in this page assumes you have loaded the [dev_setup](../dev_setup.md) in your currently loaded virtual environment.
+
+## Building the code
+
+### Autorest versioning
+
+A few notes on [Autorest for Python versionning](https://github.com/Azure/autorest.python/blob/master/ChangeLog.md):
+- Autorest for Python v2.x is deprecated, and should not be used anymore for any generation under any circumstances.
+- Autorest for Python v3.x is the most currently used one. Should not be used, but still ok if service team are still in v3.x and they want to avoid breaking changes for a given version (rare).
+- Autorest for Python v4.x is the current recommendation. This generator can generates async code, but this should be disabled with --no-async. No package should be shipped with async based on v4
+- Autorest for Python v5.x is the work in progress based on new runtime called `azure-core` (no `msrest` anymore). To be released in November 2019 (current plan). This version will bring the official async support.
+
+#### How to recognize what version of autorest was used?
+
+Autorest doesn't write the version number in the generated code, but a few indicator will tell you what generation is used, just looking at the "models" folder
+
+- Autorest v2: One model file per model class
+- Autorest v3: Two model files per model class, the second one being suffixed by "_py3" (e.g. `vm.py` and `vm_py3.py`)
+- Autorest v4: Two gigantic model files, one called `_models.py` and the second one `_models_py3.py`
+- Autorest v5: `paged` file will import base classes from `azure.core` and not `msrest`
+
+### Basics of generation
+
+A basic autorest command line will looks like this:
+
+```shell
+autorest readme.md --python --use="@microsoft.azure/autorest.python@~4.0.71" --python-mode=update --python-sdks-folder=<root of sdk clone>/sdks/ --no-async --multiapi
+```
+
+Which means "Generate the Python code for the Swagger mentioned in this readme, using autorest for Pyton v4.0.71 or above (but not v5), do not generate async files, generate multiapi if supported (if not ignore), and assume the package was already generated and it's an update"
+
+In pratical terms, this is not necessary since the Python SDK has the necessary tooling to simplify to just specify the readme.md:
+
+- Checkout the branch
+- Checkout the RestAPI specs repo
+- Call the tool: `python -m packaging_tools.generate_sdk -v -m restapi_path/readme.md` changing the last path to the readme you want to generate.
+
+The common configuration to pass to all generation are located in the [swagger_to_sdk.json file](https://github.com/Azure/azure-sdk-for-python/blob/master/swagger_to_sdk_config.json)
+
+### Automation bot
+
+If the automation is doing its job correctly, you should not have to build the SDK, but look for an integration PR for the service in question. This link will give you for instance [the list of all integration PRs](https://github.com/Azure/azure-sdk-for-python/labels/ServicePR).
+
+## Using raw autorest
+
+If you want to use raw autorest and nothing else, not even Readme, a few tips:
+
+If you're doing basic testing and want to minimal set of parameters:
+- To call Autorest, you need the following options:
+
+  - Required parameter: `--payload-flattening-threshold=2`
+  - About the generator:
+
+     - If your endpoint is ARM, add `--python --azure-arm=true`
+     - If not, add `--python`. If your client _might_ ask authentication, add `--add-credentials`
+
+And that's it! You should now have Python code ready to test. Note that this generation is for testing only and should not be sent to a customer or published to PyPI.
+
+This command generate code only. If you want to generate a [wheel](https://pythonwheels.com/) file to share this code, add the `--basic-setup-py` option to generate a basic `setup.py` file and call `python setup.py bdist_wheel`.
+
+### Example
+
+ARM management Swagger:
+
+`autorest --version=latest --python --azure-arm=true --payload-flattening-threshold=2 --input-file=myswagger.json`
+
+Not-ARM Swagger:
+
+`autorest --version=latest --python --payload-flattening-threshold=2 --add-credentials --input-file=myswagger.json`
+
+If you want something closed to a real generation:
+
+Let's assume for now that your Swagger is in `specification/compute/resource-manager`
+
+To call Autorest, you need the following options:
+
+  - Required parameters:
+
+      `--payload-flattening-threshold=2 --license-header=MICROSOFT_MIT_NO_VERSION --namespace=azure.mgmt.compute --package-name=azure-mgmt-compute --package-version=0.1.0`
+
+  - About the generator:
+
+     - If your endpoint is ARM, add `--python --azure-arm=true`
+     - If not, add `--python`. If your client _might_ ask authentication, add `--add-credentials`
+
+## Example
+
+ARM Swagger with MD (preferred syntax):
+
+`autorest --version=latest specifications/storage/resource-manager/readme.md --python --azure-arm=true --payload-flattening-threshold=2 --license-header=MICROSOFT_MIT_NO_VERSION --namespace=azure.mgmt.storage --package-name=azure-mgmt-storage --package-version=0.1.0 `
+
+ARM Swagger without MD (if you have an excellent reason):
+
+`autorest --version=latest --python --azure-arm=true --payload-flattening-threshold=2 --license-header=MICROSOFT_MIT_NO_VERSION --namespace=azure.mgmt.storage --package-name=azure-mgmt-storage --package-version=0.1.0 --input-file=specifications/storage/resource-manager/Microsoft.Storage/2016-12-01/storage.json`

doc/dev/mgmt/mgmt_release.md

@@ -1,17 +1,8 @@
 # Doing a mgmt release
 
-IMPORTANT NOTE: All the commands in this page assumes you have loaded the [dev_setup](https://github.com/Azure/azure-sdk-for-python/wiki/Contributing-to-the-tests#setting-up-a-test-environment) in your currently loaded virtual environment.
+Once you have a PR that contains accurate with correct tests (or no tests at all, but CI is green), this page explains how to prepare for a release.
 
-## Building the code
-
-If the automation is doing its job correctly, you should not have to build the SDK, but look for an integration PR for the service in question. This link will give you for instance [the list of all integration PRs](https://github.com/Azure/azure-sdk-for-python/labels/ServicePR).
-
-If you really need to generate the code manually:
-- Checkout the branch
-- Checkout the RestAPI specs repo
-- Call the tool: `python -m packaging_tools.generate_sdk -v -m restapi_path/readme.md` changing the last path to the readme you want to generate.
-
-That's it.
+IMPORTANT NOTE: All the commands in this page assumes you have loaded the [dev_setup](../dev_setup.md) in your currently loaded virtual environment.
 
 ## Building the packaging information
 
@@ -125,3 +116,16 @@ You need to check the version in:
 ```
 /azure-mgmt-myservice/azure/mgmt/myservice/version.py
 ```
+
+Python SDK _strictly_ follows [semver](https://semver.org/). A few notes:
+
+- First release should always use 0.1.0, unless asked explicitly by service team
+- If a version is 0.5.2:
+  - Next breaking / feature version is 0.6.0
+  - Next bug fix is 0.5.3
+- If a version is 2.1.0:
+  - Next preview breaking is 3.0.0rc1
+  - Next stable breaking is 3.0.0
+  - Next preview feature is 2.2.0rc1
+  - Next stable feature is 2.2.0
+  - Next patch is  2.1.1

doc/dev/mgmt/swagger/multi_api/readme.python.md

@@ -7,11 +7,11 @@ python:
   azure-arm: true
   license-header: MICROSOFT_MIT_NO_VERSION
   payload-flattening-threshold: 2
-  package-name: azure-mgmt-network
+  package-name: azure-mgmt-servicetoreplace
   clear-output-folder: true
   no-namespace-folders: true
-  python-base-folder: network/azure-mgmt-network/azure/mgmt/network
-  python-base-namespace: azure.mgmt.network
+  python-base-folder: servicetoreplace/azure-mgmt-servicetoreplace/azure/mgmt/servicetoreplace
+  python-base-namespace: azure.mgmt.servicetoreplace
 ```
 
 ### Python multi-api

doc/dev/mgmt/swagger/single_api/readme.md

@@ -39,11 +39,6 @@ These settings apply only when `--tag=package-2019-06` is specified on the comma
 input-file:
 - Microsoft.Cdn/preview/2019-06-15-preview/cdn.json
 - Microsoft.Cdn/preview/2019-06-15-preview/cdnwebapplicationfirewall.json
-directive:
-  - where:
-      - $.paths
-    suppress:
-      - OperationsAPIImplementation
 ```
 
 

doc/dev/mgmt/swagger/single_api/readme.python.md

@@ -10,18 +10,18 @@ python:
   azure-arm: true
   license-header: MICROSOFT_MIT_NO_VERSION
   payload-flattening-threshold: 2
-  namespace: azure.mgmt.cdn
-  package-name: azure-mgmt-cdn
+  namespace: azure.mgmt.servicetoreplace
+  package-name: azure-mgmt-servicetoreplace
   package-version: 2.0.0
   clear-output-folder: true
 ```
 ``` yaml $(python) && $(python-mode) == 'update'
 python:
   no-namespace-folders: true
-  output-folder: $(python-sdks-folder)/cdn/azure-mgmt-cdn/azure/mgmt/cdn
+  output-folder: $(python-sdks-folder)/servicetoreplace/azure-mgmt-servicetoreplace/azure/mgmt/servicetoreplace
 ```
 ``` yaml $(python) && $(python-mode) == 'create'
 python:
   basic-setup-py: true
-  output-folder: $(python-sdks-folder)/cdn/azure-mgmt-cdn
+  output-folder: $(python-sdks-folder)/servicetoreplace/azure-mgmt-servicetoreplace
 ```