feat(connector):from_key parameter validation

[pallavibharadwaj]

The PR is addressing two issues:

1. Parameter validation for template params:

Parameter checking happens in dataprep connector.py where we check for keys in allowed_params. This check had to include the from_key to accommodate for the template params.

Also Fixed: Finnhub config files had the template keys mentioned differently (the fromKey was being passed as part of the query). This has been fixed now on the DataConnector repo along with the corresponding changes in the DBLP config files. [Needs merge of the corresponding PR in the Data Connector repo.]

Testing:
fromKey tested in both DBLP and Finnhub.
The following was throwing an error because first_name and last_name are not directly specified as params in the config file.
df = await dc.query("publication", first_name="Jian", last_name="Pei")

Consequence:
In the config files, the fromKey in the template param FieldDef is now to be specified as a List instead of string containing all the params the template needs.
Ex:
"fromKey": ["first_name", "last_name"]
Note: Single parameter for template should have the key specified an array of one element.

---

2. Reducing Dependency on Requests Library [#396]:

Replacing the python requests library with our own implementation serving as a wrapper around the standard http.client library

Checklist:

• My code follows the style guidelines of this project
• I have already squashed the commits and make the commit message conform to the project standard.
• I have already marked the commit with "BREAKING CHANGE" or "Fixes #" if needed.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[peiwangdb]

Is it possible for us to write our own request function in a separate utils file and make the interface similar to the original request function, instead of substituting the code in all the places? @pallavibharadwaj

[pallavibharadwaj]

Is it possible for us to write our own request function in a separate utils file and make the interface similar to the original request function, instead of substituting the code in all the places? @pallavibharadwaj

Is this for the requests library substitution? Yes, sure. I'll work on that.

[peiwangdb]

@pallavibharadwaj would you fix the check failures?

[pallavibharadwaj]

@pallavibharadwaj would you fix the check failures?

@peiwangdb The check has failed for code formatting with connector.py that does not include my changes. Re-formatting the file will result in a lot of code change in this file. I did not do it so it will be easier to review the particular changes pertaining to this issue. Do you want me to go ahead and reformat the whole file?

[peiwangdb]

@pallavibharadwaj thanks for the information. I am not sure why this formatting issue emerged. Do you have an idea who made the changes? If it is not a hard fix, could you open a new PR to solve the issue and then do a rebase? I can quickly approve of it.

[nick-zrymiak]

@pallavibharadwaj would you fix the check failures?

@peiwangdb The check has failed for code formatting with connector.py that does not include my changes. Re-formatting the file will result in a lot of code change in this file. I did not do it so it will be easier to review the particular changes pertaining to this issue. Do you want me to go ahead and reformat the whole file?

What would be the alternative to this?

[pallavibharadwaj]

@peiwangdb @nick-zrymiak

1. It's a simple fix and I have re-formatted the connector file along with all the other files that needed reformatting as a separate PR refactor(dataprep): Reformatting all dataprep files #450

2. There is one other issue that I had discussed on Discord regarding security warning with HTTPSConnection on older versions of Python (prior to 2.7.9 and 3.4.3). We have to make a choice here depending on what versions of Python we support for Dataprep. If we support later than the above-mentioned versions, we can probably ignore this warning.

[nick-zrymiak]

@peiwangdb @nick-zrymiak

1. It's a simple fix and I have re-formatted the connector file along with all the other files that needed reformatting as a separate PR refactor(dataprep): Reformatting all dataprep files #450
2. There is one other issue that I had discussed on Discord regarding security warning with HTTPSConnection on older versions of Python (prior to 2.7.9 and 3.4.3). We have to make a choice here depending on what versions of Python we support for Dataprep. If we support later than the above-mentioned versions, we can probably ignore this warning.

I've noticed the cutoff for other large Python libraries is 3.6 so may be fine.

There's also this paragraph from an article that came out Feb 6, 2019 so the numbers may be more lopsided now:

"Regarding Python 2 vs. Python 3 adoption, the survey shows 84 percent using Python 3 and 16 percent still using Python 2. Among the Python 3 users, 54 percent are using Python 3.6 and 30 percent Python 3.7, with the rest split among other versions."

On the other hand there may be a substitute for HTTPSConnection but I'm not too sure.

@peiwangdb what do you think?

[codecov]

Codecov Report

Merging #407 (a799a6b) into develop (9b16020) will decrease coverage by 0.26%.
The diff coverage is 65.85%.

@@             Coverage Diff             @@
##           develop     #407      +/-   ##
===========================================
- Coverage    81.17%   80.90%   -0.27%     
===========================================
  Files           73       74       +1     
  Lines         5535     5625      +90     
===========================================
+ Hits          4493     4551      +58     
- Misses        1042     1074      +32     

Impacted Files	Coverage Δ
dataprep/connector/schema/defs.py	52.91% <27.77%> (-4.73%)	⬇️
dataprep/connector/generator/generator.py	31.81% <37.50%> (+0.56%)	⬆️
dataprep/connector/connector.py	62.94% <52.94%> (-0.36%)	⬇️
dataprep/connector/utils.py	91.17% <91.17%> (ø)
dataprep/connector/config_manager.py	85.93% <100.00%> (+1.19%)	⬆️
dataprep/tests/connector/test_integration.py	100.00% <100.00%> (ø)
dataprep/eda/correlation/compute/nullivariate.py	98.18% <0.00%> (-0.91%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9b16020...a799a6b. Read the comment docs.

[pallavibharadwaj]

@peiwangdb @dovahcrow I have resolved the issues with the PR. Like I said before, there is still a security warning with HTTPSConnection on older versions of Python (prior to 2.7.9 and 3.4.3), which we discussed can be ignored.

[nick-zrymiak]

Very well done!