Automatic WebDriver download/alignment

[6DiegoDiego9]

I believe that making SeleniumVBA work out-of-the-box is very important to make it popular.
Automatic WebDriver download/alignment also helps in making it portable on multiple own machines.

This is why today I'm working on this:

---

Automatic WebDriver download/alignment optimizations:

• auto-proxy (if any) is now used to download WebDriver (tested working on my business PC)

• checkDriverBrowserVersionAlignment switch removed. The WebDriver download/alignment procedure is now called just when needed, inside (pre-existing) error handling, i.e. when WebDriver file doesn't exist or after it's called and an error specifically stating wrong-version occurs.
  The result is like a permanent (now removed) "checkDriverBrowserVersionAlignment = True" without the overhead of always checking in advance for driver version alignment.

• WebDriver path changed to user PC temporary folder (VBA.environ("Temp")) to avoid conflicting WebDriver files when multiple users, with possibly different browsers, work with WebDriver files stored on a network folder.
  Not user configurable.

---

@GCuser99 are you OK with this change?

I read that you didn't "checkDriverBrowserVersionAlignment = True" because of insufficient testing.
I can say that I'll commit myself to make it work flawlessy because it's important to me.

[GCuser99]

Those all sound like useful optimizations.

I'm a little concerned about a few changes though. I think the binary location should at least be user-configurable. That has been helpful to me in the past when I was testing multiple versions of the drivers (64bit vs 32bit for example). Presently user can change it through the DefaultBinaryFolder property. I don't see any compelling reason to remove that ability. But I have no problem with you changing the default to Temp folder, such as in:

Private Sub Class_Initialize()
    InitCommands
    CommandWindowStyle = vbHide
    'note that user has control through the DefaultIOFolder and DefaultBinaryFolder
    'properties so these below are just reasonable starting points...
    Me.DefaultBinaryFolder = VBA.environ("Temp") 'was ThisWorkbook.path
    Me.DefaultIOFolder = ActiveVBAProjectFolderPath
End Sub

On the removing the optionality for auto-checking driver-browser alignment - what do you mean "or after it's called and an error specifically stating wrong-version occurs." Are you referring to an error generated by the Selenium WebDriver? As it stands now, the auto-update occurs when the major version is misaligned - that seems to be the web consensus on when to update. Are you saying to instead wait until the Selenium WebDriver throws an error due to misalignment? Do you know how to trap that error for Edge, Chrome, and Firefox? And are you sure that waiting for the WebDriver to complain is the best policy for knowing when to update? Or am I misunderstanding what you are saying. Let's discuss this some more...

Thanks!

[6DiegoDiego9]

Oh I thought to not make the path user-configurable for two reasons:

• I didn't think of a reason to need to test, on the same machine and OS, different drivers than those chosen by the auto-aligner procedure, and it would still be possible to test/keep different drivers overwriting it like usual, as long as they don't raise a major version misalignment error triggering the update
• I fear that new users could choose the same path (or a subpath) of the Excel file in a network drive in multiple-users (with multiple-PC) scenarios, causing conflicting updates

Anyway, now that you made me aware of the need to test and of the existing DefaultBinaryFolder property, I'm rethinking about it.

About your last paragraph, I found that the major version misalignment is explicitly stated in the error trapped at the end (If... IsResponseError(response)) of the WebDriver.Execute method, read by GetResponseErrorMessage(response). I therefore thought to just parse this error message via InStr or a regex to trigger the alignment.

I found no consensus about the exact minor version to use because Microsoft is recommending a full match of major+minor version in the official documentation here: https://learn.microsoft.com/en-us/microsoft-edge/webdriver-chromium/
I found that the existing alignment procedure takes the last (higher) minor version of the matching major version even for Microsoft Edge too so I thought that it would likely not cause problems to use an higher minor version.
However, you made me now well aware that my logic could leave a lower minor version in place in case the user updates the browser (an it's last major version) after updating the WebDriver file, and I don't like this.
This would make me at least come back to prefer ThisWorkbook.path as DefaultBinaryFolder, so that it's easier for the user to manually update the driver, that is still possible as before.

In conclusion, I suspend this task maybe waiting to discover a faster (<0.01s) method to always check for minor version alignments too so that users will never be required to manually update the WebDriver file for normal use.

Thanks as always for your extremely useful answers :)

[GCuser99]

On the driver auto-alignment: I did some quick testing and found that the error you referred to, which happens on OpenBrowser for me, does not seem guaranteed to happen when the major version is misaligned. For example, on the current 105 version of Edge browser, the error occurs for me when I use WebDriver version 103.0.1264, but does not occur when I use WebDriver version 104.0.1293. So what does this mean? I have no idea... If the error does not occur, regardless of major version misalignment, does that mean the WebDriver is fully compatible with the browser version?

Also, since the Edge/Chrome incompatibility error occurs (at least for me) at OpenBrowser, how would you recover from that error since the WebDriver has already started? I have no doubt that you could solve that, but at what cost in terms of your time and potential complications in the code?

Furthermore, I do not get an incompatibility error for Firefox on OpenBrowser when I use a supposedly incompatible driver. It works until it hits a place where it does not. That raises the concern that even if Chrome and Edge are not throwing the OpenBrowser error with a major version mismatch, will they throw an error or misbehave somewhere else downstream?

I mention all of the above just to make the point that it may be more complicated than either one of us think or know. :-)

In case you are not aware, I had anticipated some of this complication (aka my lack of knowledge) when I designed the WebDriverManager. I set the WebDriverManager default minimum compatibility level to match just the major version so that we aren't over-updating - look at class_initialize:

Private Sub Class_Initialize()
    mMinCompatibilityLevel = svbaMajor 'this defines what it is to be "compatible" from a version comparison pov
    DefaultBinaryFolder = ThisWorkbook.path
End Sub

But user can change that to a more restrictive level through this property:

Property Let MinCompatibilityLevel(ByVal val As svbaCompatibility)
    mMinCompatibilityLevel = val
End Property

Property Get MinCompatibilityLevel() As svbaCompatibility
    MinCompatibilityLevel = mMinCompatibilityLevel
End Property

On the default binary location - I have no problem with changing that to the user temp directory if that solves or prevents a problem for you or your users. As I said before, I just don't want to get locked into that without any control, so I would push for allowing user the option to set the location to something different, and also be able to easily change it in the class_initalize subs, as user can do currently.

Were you able to make the web requests more robust with auto-proxy? I am interested to hear more...

Oh, and I am interested in getting rid of the checkDriverBrowserVersionAlignment switch, or at least setting the default to true, if we can get it to work robustly. It works fine on my system, but it sounds like you have had problems with it.

Anyway, I'm excited that someone else is looking at this with a critical and different perspective. Thanks for your enthusiasm and hard work!

[6DiegoDiego9]

Oh wow, thanks for all this information!
You saved me from bad toothaches during and after the (wasted) work :-)

About the auto-proxy, this is the info I collected so far:

• [my business (bank) PC uses a Proxy Auto-Configuration (PAC) file that is (contains) a (complex) JavaScript function the determines the proxy/ies to be used for the different destinations. (the now beheaded) IE is configured to point to this PAC file.]
• ServerXMLHTTP (currently used in WebDriverManager to download the WebDrivers) has a "SetProxy 0" feature intended "to support auto-proxy configuration features that might be included with future released products" according the the official MS page
• XMLHTTP, being based on WinINet that's the network stack provided by Internet Explorer (but you get that even when you turn the IE browser feature off), reads settings directly from the registry, the same settings we see in user's internet options in control panel
• changing "ServerXMLHTTP" to "XMLHTTP" in WebDriverManager.RequestData is sufficient to make the download work like a charm on my business (and personal) PC(s)
• WinINet is born with IE but I currently didn't find warnings from Microsoft about possible future removal after IE death
• WinINet is not supposed to be used in system service applications with multi-threads, high-frequency, etc..., like the lower-level (with many less automatic-features) WinHTTP is
• I found a couple of codes to add auto-proxy to WinHTTP:
  https://github.com/mmaysami/Winhttp-Proxy/blob/master/ConnectionProxy.bas
  https://github.com/VBA-tools/VBA-Web/blob/master/src/WebClient.cls
  However, they're long and complex scripts

Based on this info, I'd suggest to just change "ServerXMLHTTP" to "XMLHTTP" in WebDriverManager.RequestData, provided it's used only for the occasional download of WebDrivers.

[GCuser99]

@6DiegoDiego9 thanks for sharing your research.

So on your business PC, it only fails using ServerXMLHTTP on the driver download request in the DownloadAndInstallDriver method of WebDriverClass, but does not fail in the other methods that request version info, such as in GetCompatibleDriverVersion?

I'm ok with your recommendation above. I tested it and it worked on my end. Do you think we should use XMLHTTP only for the download, and ServerXMLHTTP for all other requests like the version API requests, since these will be used frequently if we turn the auto-version-alignment feature in WebDriver to True?

[6DiegoDiego9]

ServerXMLHTTP fails on my business PC when trying to reach (any) Internet urls therefore I think on both GetCompatibleDriverVersion and DownloadAndInstallDriver. XMLHTTP in WebDriverManager.RequestData solved the #problem.

I think that the best would be to use XMLHTTP for Internet urls and ServerXMLHTTP for communicating with localhost WebDriver.

[GCuser99]

BTW, I tested how long it takes to check version alignment with checkDriverBrowserVersionAlignment = True after making your suggested change above. On my system after many tests this morning I found that it always took less than .5 secs and with 90% of the time less than .1 secs. So the overhead seems tolerable to me.

One small thing that needs to be modified with auto-checking is that we should trap an error caused by being disconnected to the internet. I was doing some personal work with SeleniumVBA the other day where I was accessing and processing some local web pages when the power in our area went out for a brief period of time, which affected internet service. I could still keep working because I wasn't navigating to www. But if I had had the auto-check turned on, I would have had to manually turn it off until service came back on.

[6DiegoDiego9]

I made a fast timer check on the mngr.AlignEdgeDriverWithBrowser driverPath line, with Edge, on my Core I7 notebook, and after the first slower try (varying in the 1-4 sec range) I've got:
0,2148438 sec
0,2070313 sec
0,5527344 sec

Oh you're right about the "disconnected to the internet" case. What do we do in that case? what about just a non-stopping and non-annoying debug.print warning?

[GCuser99]

@6DiegoDiego9 - thanks for starting this discussion - you have got me rethinking a lot of this... So the general approach that I was taking to check alignment was to:

1. Get the installed browser version
2. Use the respective web api to get the compatible driver for that browser version
3. Compare the currently installed driver against the "compatible" driver for some desired threshold level.

For Firefox, that general approach still looks optimal to me. But since Edge and Chrome maintain (at least currently) synchronized version numbers between browser and driver, I realize now that it is more efficient to check the installed driver version against the installed browser version to determine whether the threshold is met, while bypassing the api web request. I'm hoping that approach should make you happy :-). To illustrate the difference, compare the revised logic below in the Start method between Edge/Chrome versus Firefox. What do you think?

I tested it with a timer on my machine and the check for Edge when driver and browser are aligned takes only an average .0564 secs (20 trials) and of course does not require a web api request.

    If checkDriverBrowserVersionAlignment Then
        Dim mngr As New WebDriverManager
        Dim bverInstalled, dverInstalled, dverCompat
        Select Case browserName_
        Case "msedge"
            bverInstalled = mngr.GetInstalledBrowserVersion(Edge)
            dverInstalled = mngr.GetInstalledDriverVersion(Edge)
            If mngr.CheckCompatibilityLevel(bverInstalled, dverInstalled) < svbaMinor Then
                dverCompat = mngr.GetCompatibleDriverVersion(Edge, bverInstalled) 'web api request
                mngr.DownloadAndInstallDriver Edge, dverCompat, driverPath
            End If
        Case "chrome"
            bverInstalled = mngr.GetInstalledBrowserVersion(Chrome)
            dverInstalled = mngr.GetInstalledDriverVersion(Chrome)
            If mngr.CheckCompatibilityLevel(bverInstalled, dverInstalled) < svbaMinor Then
                dverCompat = mngr.GetCompatibleDriverVersion(Chrome, bverInstalled) 'web api request
                mngr.DownloadAndInstallDriver Chrome, dverCompat, driverPath
            End If
        Case "firefox"
            bverInstalled = mngr.GetInstalledBrowserVersion(Firefox)
            dverInstalled = mngr.GetInstalledDriverVersion(Firefox)
            dverCompat = mngr.GetCompatibleDriverVersion(Firefox, bverInstalled)  'web api request
            If mngr.CheckCompatibilityLevel(dverCompat, dverInstalled) < svbaMinor Then
                mngr.DownloadAndInstallDriver Firefox, dverCompat, driverPath
            End If
        End Select
    End If

[6DiegoDiego9]

Hmmm with this logic in place, it would call the web api even if bverInstalled is say 105.0.1234.12 and dverInstalled is 105.1.2345.67.

I'm a little uncertain about the compatibility of such a difference in Edge and in Chrome (eg. read https://sites.google.com/chromium.org/driver/downloads and then look in the contained "Version Selection" page, even these two near pages seem to contradict each other!) but with a view to rationalizing the web calls I'd take the risk:
what about calling the web to auto-align (GetCompatibleDriverVersion+DownloadAndInstallDriver) only when ((bverInstalled's vermajor <> dverInstalled's vermajor) or (bverInstalled is higher than dverInstalled))?

Afterall, it's still possible for the user to manually update and test drivers with higher sub-major numbering. The auto-align would not be called.

There's also the case of having a very recent bverInstalled major that's higher than the highest available driver major to download but I think that in this particular case the extra time of calling the web every time to search for an new matching driver would be well spent afterall :)

[GCuser99]

Yes, that is true - if the major version is same and minor version is different, then setting the desired compatibility level=svbaMinor will trigger an update in the example that you raised. But, as far as I can tell, the minor version does not change much - in fact I have yet to see it to be other than 0!

However, I would have no problem setting the desired compatibility level=svbaMajor. That is what I have it set to now and I have yet to get compatibility "burned" using that setting.

By "bverInstalled is higher than dverInstalled" do you mean "bverInstalled's vermajor > dverInstalled's vermajor"? So if user manually installs a higher major version of the driver than the installed browser, we assume that it was intentional, and do not update? If that is what you mean, then it's an endpoint that I am indifferent about, and would be happy to try to build that in.

It should be noted that the api for getting a compatible driver version for Edge browser only takes as input the browser major version anyway, which implies to me that perhaps that's all that really matters. On the other hand, the Chrome api actually can take as input major.minor.buildmajor to look for what they call an "exact match", which implies that it might matter - so pretty much the mixed message that you got from the links above. Like I said already, I think matching major version for auto-align is good enough, and if user feels the need to optimize, they can update programmatically more frequently.

Anyway, I found some buggy WebDriverManager code and am making changes now. I will keep you posted on progress. Thanks much for your ideas.

[6DiegoDiego9]

I refactor my proposed

((bverInstalled's vermajor <> dverInstalled's vermajor) or (bverInstalled is higher than dverInstalled))

to

((bverInstalled is higher than dverInstalled) or (bverInstalled's vermajor < dverInstalled's vermajor))

to be more clear, even if the meaning/effect is exactly the same.

With "bverInstalled is higher than dverInstalled" I not only refer to the major version. I refer to the whole driver version.
Examples:

1. 105.0.1000.5 105.0.1000.4 -> call auto-align (bverInstalled is higher than dverInstalled)
2. 105.0.1000.5 105.0.1000.6 -> not call auto-align (to rationalize. User can still manually search&update it to newer submajor versions)
3. 106.0.1000.5 105.0.1000.5 -> call auto-align (bverInstalled is higher than dverInstalled)
4. 105.0.1000.5 106.0.1000.5 -> call auto-align (bverInstalled's vermajor < dverInstalled's vermajor)

[GCuser99]

@6DiegoDiego9 - I'm having trouble following your proposed checking/updating strategy. Here is the way I am thinking about this - maybe you can tell me where I am going wrong:

1. Let's say we start with an aligned driver and browser - 100% compatibility with versions being exactly the same.
2. Since my browser gets updated automatically over time, eventually my driver, that does not get updated, will not support the browser (it will be BEHIND the development of the browser).
3. So we need a way to keep the driver updated to the browser, but for efficiency reasons we want to only check for and update the driver when we think it is needed to avoid incompatibility.
4. The scheme we devise is to check/update the driver version according to the installed browser version, and we make some assumptions about when to trigger the check/update (trade-off between some small amount of possible incompatibility versus efficiency).

Under the scenario above:

I don't understand how the driver version gets AHEAD of the browser version, as in your "(bverInstalled's vermajor < dverInstalled's vermajor)". It's always either in sync because we decided to update whenever there is a browser build change, or its behind because we wait until there is a browser major/minor change.

If we use your "(bverInstalled is higher than dverInstalled)" and we assume as above that dverInstalled<=bverInstalled, that would imply that we call trigger the check/update routine as often as possible, which would be equivalent to:

'below evaluates to true if the browser and driver versions are not exactly the same
If mngr.CheckCompatibilityLevel(bverInstalled, dverInstalled) < svbaBuildMinor then

Is that what you want? I thought originally you were concerned about minimizing web requests from an efficiency perspective, and my proposal above was meant to address your concern.

I suppose in rare cases a user manually downloads a driver that happens to be ahead of their installed browser. In that case using my original proposal, the auto-align would update the driver in the backwards direction. I think that would be a good thing personally since I'm not sure that drivers are backward compatible with older versions of browsers.

BTW, I looked at Chrome version numbers back to below major version 90. Out of the nearly 4000 not one had a minor version <> 0. So it appears that using either svbaMajor and svbaMinor for desired compatibility level is effectively the same in terms of frequency of checks/updates.

Below is a slightly updated version with comments...

    If checkDriverBrowserVersionAlignment Then
        Dim mngr As New WebDriverManager
        Dim bverInstalled, dverInstalled, dverCompat
        Select Case browserName_
        Case "msedge"
            bverInstalled = mngr.GetInstalledBrowserVersion(Edge)
            dverInstalled = mngr.GetInstalledDriverVersion(Edge)
            If mngr.CheckCompatibilityLevel(bverInstalled, dverInstalled) < svbaMinor Then
                'the installed driver major/minor version is different than the installed browser major/minor version
                'so find the most compatible driver version for installed browser
                dverCompat = mngr.GetCompatibleDriverVersion(Edge, bverInstalled)
                'its possible that the browser version is a dev version, and there is not yet
                'an updated driver version, so check to make sure dverCompat is not already installed
                'if they are different, then install the compatible driver
                If dverCompat <> dverInstalled Then mngr.DownloadAndInstallDriver Edge, dverCompat, driverPath
            End If
        Case "chrome"
            bverInstalled = mngr.GetInstalledBrowserVersion(Chrome)
            dverInstalled = mngr.GetInstalledDriverVersion(Chrome)
            If mngr.CheckCompatibilityLevel(bverInstalled, dverInstalled) < svbaMinor Then
                'the installed driver major/minor version is different than the installed browser major/minor version
                'so find the most compatible driver version for installed browser
                dverCompat = mngr.GetCompatibleDriverVersion(Chrome, bverInstalled)
                'its possible that the browser version is a dev version, and there is not yet
                'an updated driver version, so check to make sure dverCompat is not already installed
                If dverCompat <> dverInstalled Then mngr.DownloadAndInstallDriver Chrome, dverCompat, driverPath
            End If
        Case "firefox"
            bverInstalled = mngr.GetInstalledBrowserVersion(Firefox)
            dverInstalled = mngr.GetInstalledDriverVersion(Firefox)
            dverCompat = mngr.GetCompatibleDriverVersion(Firefox, bverInstalled)
            If mngr.CheckCompatibilityLevel(dverCompat, dverInstalled) < svbaMinor Then
                'the installed driver major/minor version is different than the most compatible driver major/minor version
                'so install it
                mngr.DownloadAndInstallDriver Firefox, dverCompat, driverPath
            End If
        End Select
    End If

[6DiegoDiego9]

1+2+3+4: yes we're aligned until here :)

I don't understand how the driver version gets AHEAD of the browser version,

Assuming that you refer to the major versions, otherwise the cases are too obvious, I thought to some real and possible scenarios:

• I often put my Excel files on a network shared folder for use by multiple users or multiple PC of my own, for example on a OneDrive path used by my multiple PCs.
  If I put the webdriver on that shared path too, for example because in some cases I don't expect simultaneous use by multiple PCs, it could happen that some accessing clients have an older major version of the original browser
• I (really) use to have multiple Chrome browser versions installed, including old versions. I could confuse (having the same chromedriver.exe name) or forget a newer driver in place while I test with an older Chrome version, or I just prefer to have the auto-align doing the work (switching webdrivers) for me
• manually as in your supposition: for example today my Chrome version is 105.0.5195.127 while the latest downloadable version seems to be 106.0.5249.21. 106 is likely a beta. If I wrongly download and put it, the auto-align will correct the error (unless turned to "False")

imply that we call trigger the check/update routine as often as possible

Not often, according to Google "Chrome releases a full OS update about every 4 weeks. Minor updates, such as security fixes and software updates, happen every 2–3 weeks."
In my opinion, even if the auto-align will be called once per day to align a driver's buildminor that went behind the browser, it would still be the best for me.

which would be equivalent to
If mngr.CheckCompatibilityLevel(bverInstalled, dverInstalled) < svbaBuildMinor 'this is true if the browser and driver versions are not exactly the same

My proposal is not to trigger the auto-align on any not exact full match.

If browser version is 105.0.1000.5 and driver version is equal or higher (inside the same major), for example 105.0.1000.5 or 105.0.1000.6 or 105.0.1001.1 or 105.9.9999.9, it should always not call the web auto-align method.
while it should call the web auto-align method if driver is 105.0.1000.4 or lower (including lower majors).

[GCuser99]

Ok I think I'm at least understand what your preference is...

A couple of comments:

If we set it to check/update as I think you are saying, I think we will get a web request a lot more frequently than every 2-3 weeks. Maybe you meant "a new install every 2-3 weeks"? For example, I am letting my Chrome browser update automatically. Setting the check/update to make a web request with any installed driver/browser mismatch means that, at least currently, the web request happens every time I start the driver. That is because my browser version is 105.0.5195.127, and the most compatible driver version for that browser version is 105.0.5195.52, which is already installed. But that installed driver version is not an exact match to the browser version, and of course is less than the installed browser version. The last driver update for build 105.0.5195 was on Aug. 23 which is the one I currently have installed.

But I'm not oppose to the frequent web requests - for some reason I misunderstood that it was a concern of yours - sorry for that misunderstanding.

I now understand that you do not want to do the check/update if the browser version is behind the driver version, unless it's a major version mismatch. Correct? But I'm having problems with the overall logic - if you think the minor build version mismatch (the last part of the version number) is important enough to check/update when the driver version is behind the browser version, then why would you not also think it's important to check/update when the driver is ahead of the browser version? Is it because you believe that the drivers are backward compatible with past browsers (with same major version), and not with future browsers (with same major version)?

Anyway, I'll give this some more thought... I try to make design choices on what I can imagine the majority of "potential" users would understand and agree to, irrespective of my own circumstances. If the cost of doing a web request each time the driver is started is of little concern to these "potential" users (1/5 of a second by your measure), then I certainly have no problem with setting the driver to check/update every time there is a mismatch, although I personally suspect that that it is overkill (I have not run into a compatibility problem using the consensus "make sure the major versions are aligned" in the 9 months I've been using the software). And of course, I can tweak my copy of the software to suit my needs, without imposing my particular preferences on others. But on your conditions to restrict updating to only if the (driver is behind the browser) or (there is a major version difference) - I'm wondering if a majority of potential users not in your particular circumstance would agree to and even understand that? This is what I am struggling with now...

Anyway, good discussion. Thanks for your patience.

[6DiegoDiego9]

I now understand that you do not want to do the check/update if the browser version is behind the driver version, unless it's a major version mismatch. Correct?

Correct! :)

if you think the minor build version mismatch (the last part of the version number) is important enough to check/update when the driver version is behind the browser version, then why would you not also think it's important to check/update when the driver is ahead of the browser version? Is it because you believe that the drivers are backward compatible with past browsers (with same major version), and not with future browsers (with same major version)?

I honestly didn't know that the last buildminor of Chrome driver for current release 105.0.5195.127 is so much behind (52).
Since Edge current last (stable release) driver is fully aligned to last Edge browser version 105.0.1343.42 and MS page seems to instruct to get the driver with full match including the buildminor, I thought that the recommendation was the same for Chrome too.
However, I now see that the buildminor is officially called "Patch" by Chromium (link) and this together with the fact that Chromium instructs to search a driver removing the buildminor (link), moves a bit my preference toward your setting of ignoring at least the buildminor.

Is it because you believe that the drivers are backward compatible with past browsers (with same major version), and not with future browsers (with same major version)

I just thought that drivers, within same major version, are very likely backward compatible with past browsers and less likely forward compatible.

Anyway, if I remember correctly my time measurements were of AlignDriverWithBrowser and this procedure includes the GetInstalledBrowserVersion and perhaps the check of the installed driver version. This means that the running time of the mere web call (DownloadAndInstallDriver) is even lower than those ~0.2sec numbers.

At this point, I'd be OK with all the possible discussed rationalizing-web-calls logics from my most far extreme to your most far extreme :)
I'd give more weight, like you wrote, to code easiness and readability too.

[GCuser99]

Ok I'll default it to ignoring the build minor, but add a constant in the declare section to make it easy to change. We can try that for a while and if in future we run into compatibility issues we can change it to match exactly. I also like this compromise because its driver-server responsible in terms of minimizing the number of web requests. At the same time, it mostly takes advantage of the auto-check ability (better than the manual "match major/minor" practice) - drawing a good balance I think. Thanks again for the collaboration!

[6DiegoDiego9]

100% ok with all your points

[6DiegoDiego9]

I had a new idea that could change all: we could store the last-web-check timestamp in the webdriver file name (as suffix) or in one of its hidden properties!

So that we could simply do a new web-check every once in a while, I'd say "always unless recently checked"
or when "(bver-dver incompatibility) OR (NOT recently checked)".

Recently = maybe <1hr?

[GCuser99]

That's an interesting idea... But I think we should go with the current compromise solution first. If we find problems with it, then work on solutions to fix. I honestly feel that the current solution is going to work great. Let's give it a try. :-)

[GCuser99]

@6DiegoDiego9, I need your help... With the change to using XMLHTTP60, we broke the geckodriver download. Below is a test sub - do you know of a way to fix? I'm pretty ignorant in this area... the only solution I can think of is instantiating client using late binding as in the commented-out section below, but hoping you know of a cleaner solution...?

Sub test_geckdriver_download()
    'this works using ServerXMLHTTP60, but does not using XMLHTTP60
    'Dim client As New MSXML2.ServerXMLHTTP60
    Dim client As New MSXML2.XMLHTTP60
    
    'Dim client As Object
    'If True Then 'we are doing something with geckodriver
        'Set client = New MSXML2.ServerXMLHTTP60
    'Else 'we are doing something with chrome/edge drivers
        'Set client = New MSXML2.XMLHTTP60
    'End If

    url = "https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-win64.zip"
    
    On Error Resume Next
    client.Open "GET", url, False
    
    client.send
    
    While client.readyState <> 4
        DoEvents
    Wend
    
    On Error GoTo 0
    
    If client.Status = 200 Then
        Dim payload      
        payload = client.responseBody
        Debug.Print "download successful"
    Else
        Err.raise 1, , "Error: " & client.Status
    End If
    
End Sub

[GCuser99]

I can get the download to work using:

Sub test_geckdriver_download()
    'this works using ServerXMLHTTP60, but does not using XMLHTTP60
    'Dim client As New MSXML2.ServerXMLHTTP60
    'Dim client As New MSXML2.XMLHTTP60
    Dim client As New WinHttp.WinHttpRequest
    
    'Dim client As Object
    'If True Then
    '    Set client = New MSXML2.ServerXMLHTTP60
    'Else
    '    Set client = New MSXML2.XMLHTTP60
    'End If

    url = "https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-win64.zip"
    
    On Error Resume Next
    client.Open "GET", url, False
    
    client.send
    
    client.waitForResponse 1000

    On Error GoTo 0
    
    If client.Status = 200 Then
        Dim payload
        payload = client.responseBody
        Debug.Print "download successful"
    Else
        Err.raise 1, , "Error: " & client.Status
    End If
    
End Sub

But wondering whether that breaks on your business pc?

Also had luck using this one...

Private Declare PtrSafe Function URLDownloadToFile Lib "urlmon" _
    Alias "URLDownloadToFileA" (ByVal pCaller As Long, _
    ByVal szURL As String, ByVal szFileName As String, _
    ByVal dwReserved As Long, ByVal lpfnCB As Long) As Long

status = URLDownloadToFile(0, url, filepath, 0, 0)

[6DiegoDiego9]

I found that the reason is that XMLHTTP denies access to sites that aren't trusted by IE. ServerXMLHTTP doesn't perform those checks.

I was able to make it work with XMLHTTP by adding "https://github.com" in trusted sites and setting the Security level for trusted sites to "Low". However this is not a viable option for us.

ServerXMLHTTP60 and WinHTTP don't work on my business PC unless proper proxy settings are configured.
As wrote in one of my messages some days ago I found a couple of scripts to configure auto-proxy on WinHTTP but they're complex.

URLDownloadToFile was the solution! :-)
It worked with your syntax on my business PC where all the other failed. Thanks for the finding! :-)

[GCuser99]

Awesome - thanks for your help!

[GCuser99]

@6DiegoDiego9 - I refactored the WebDriverManager class for improved logic and clarity. I set the default minimum compatibility for Align*DriverWithBrowser to update to the latest and greatest compatible driver so that when user updates programmatically, as in test_Update_Driver module they get the most current. However, as we discussed, in order to minimize what are probably frequent and unnecessary web requests, the setting in the WebDriver class Start method is update only when there is a new build major. The auto-check-and-align in WebDriver class is set to True, so that the software functions out-of-the-box as per your encouragement at the start of this thread. I implemented a runSilent mode for the auto-check-and-align in WebDriver Start method so that the process is less-likely to interrupt browser automation in case of an updating error. Thanks for your help on this!

[6DiegoDiego9]

Thanks for another exciting update! :)

BTW I just read that you changed ServerXMLHTTP60 to XMLHTTP60 while after this message I expected it to be changed to URLDownloadToFile for complete compatibility with all possible URLs including "https://github.com/mozilla/..."
What about it?

[GCuser99]

We are using XMLHTTP60 for getting the compatible driver version using the respective api's, and URLDownloadToFile for the actual install download for all three browser drivers. Do you see a problem with that?

Thanks...

[6DiegoDiego9]

Ah you're right!
I tried these on my behind-proxy PC:

1. RequestData("https://msedgedriver.azureedge.net/104.0.1293.81/edgedriver_win64.zip")
2. RequestData("https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-win64.zip")
3. RequestData("https://api.github.com/repos/mozilla/geckodriver/releases/latest")

Results: 1 ok, 2 fail, then I expected 3 to fail too but it succeded! (I don't know why)

[6DiegoDiego9]

After many uses during last weeks on both my business and personal PCs, including a couple of other users trying it, I must say that I'm happy with the new automatic download. It always worked seamlessly.

I still prefer to always set the binary into the Downloads* or Temp** folder mainly for three reasons:

• supporting multi-user (on network path or with different users on my Windows Server VPS)
• multiple Excel apps in different folders can share the same binary
• keeping the Excel app folder clean

• CreateObject("Shell.Application").Namespace("shell:Downloads").self.Path
  ** VBA.Environ$("Temp")

I'd vote to set the Downloads folder as default (still leaving the DefaultBinaryFolder property Let procedure), or at least as default (of optional) parameter of the DefaultBinaryFolder property Let.

What do you think?

[GCuser99]

Yes @6DiegoDiego9, I agree with you that defaulting to a central location would be an improvement.

As far as implementation is concerned - is this what you are thinking of doing in the WebDriver class?

Private Sub Class_Initialize()
    InitCommands
    Me.CommandWindowStyle = vbHide
    'note that user has control through the DefaultIOFolder and DefaultBinaryFolder
    'properties so these below are just reasonable starting points...
    'ThisWorkbook.Path and ActiveVBAProjectFolderPath will be different for a project
    'referencing the SeleniumVBA Add-in, but same otherwise
    Dim oShell As New Shell
    Me.DefaultBinaryFolder = oShell.Namespace("shell:Downloads").Self.Path 'ThisWorkbook.Path
    Me.DefaultIOFolder = ActiveVBAProjectFolderPath
End Sub

And similarly, in WebDriverManager class?

Private Sub Class_Initialize()
    Dim oShell As New Shell
    Me.DefaultBinaryFolder = oShell.Namespace("shell:Downloads").Self.Path 'ThisWorkbook.Path
End Sub

If so, I would be ok with that...

Should we rename "DefaultBinaryFolder" to "DefaultDriverFolder" to clarify that we are referring to the driver and not the browser?

On the options for the default, what do you think the pros/cons are for Downloads vs Temp?

Thanks again for your ideas, effort and teamwork!

[6DiegoDiego9]

• WebDriver.cls and WebDriverManager.cls mods:
  yes 100%.
  I'd better express the Dim line like you already did for the DownloadAndInstallDriver method:
  Dim oShell As New Shell32.Shell 'Microsoft Shell Controls and Automation (Shell32) 'CreateObject("Shell.Application")

• DefaultBinaryFolder vs DefaultDriverFolder: yes +1 for DefaultDriverFolder. I already wanted to write to you this in my prev. message...
  Furthermore, assuming that the 2nd most common pref. could be the current dir, I also tend (50-60%) for:
  Public Property Let DefaultDriverFolder(Optional folderPath As String= ".\")
'Note: default folder if this property is not set is Downloads dir
  for "suggesting" the ".\" syntax and choice when user is in the act of changing it.

• Downloads vs Temp: on my business PC, Temp failed as destination at the fso.CopyFile line because Temp is set to "C:\Users\myUsername\AppData\Local\Temp" and as most of C: drive it's (partially) unaccessible. For example it's unaccessible to File Explorer (and to the fso.CopyFile).
  Downloads is set to "E:\Download" instead, where (in E: drive) we're free to put our files, and it works 100%.
  Aside from that, I'd be 50%/50%.

[GCuser99]

Great - sounds like a plan!

[6DiegoDiego9]

Done: #35