Suggestions about UrlDownloadToFile #175
Replies: 33 comments 13 replies
-
|
It's ok for me. However, since it's used two times in SeleniumVBA, and in separate classes, we need it as a Friend Function. |
Beta Was this translation helpful? Give feedback.
-
|
As I recall, I think MSXML2.XMLHTTP60 worked for @6DiegoDiego9 back when we discussed this issue, but as you point out in your excellent post above, it did not work for Firefox - thus the reason to go with UrlDownloadToFile. I'm unable to test on a work network (only home use), but your suggestion looks reasonable to me. It does raise the tangential issue of whether to continue to support Firefox. The code base contains a number of hacks to facilitate that support, several of which I think raise our TotalVirus score (UrlDownloadToFile being one of them). The only other place that we use UrlDownloadToFile is in the DownloadResource method of the WebDriver class. I think I can refactor that method using pure CDP while maintaining most of the functionality. On the other hand, despite the low percentage of Windows Firefox users worldwide, we have had a surprising number of SeleniumVBA users post issues related to Firefox. So I'm reluctant to eliminate that without very good reason. When I have time, I think I'm going to produce another test without Firefox and UrlDownloadToFile support to see if that significantly reduces false-positives. Anyway, I'm good with making your suggested change if it works for @6DiegoDiego9. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not personally a Firefox user; however, I acknowledge all your worries, Mike. Regarding the change, I propose placing it in WebShared, just under |
Beta Was this translation helpful? Give feedback.
-
|
@hanamichi77777, can you add one more technique to your test matrix (WinHttp.WinHttpRequest.5.1)? As I understand it, it handles redirects elegantly... Public Function downloadToFile(ByVal Url As String, ByVal fileFullPath As String) As Boolean
Dim client As Object
Dim bytes() As Byte
' Create WinHTTP client
Set client = CreateObject("WinHttp.WinHttpRequest.5.1")
' Enable automatic redirect handling
client.Option(6) = True ' WinHttpRequestOption_EnableRedirects
' Open and send request
client.Open "GET", Url, False
client.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
client.send
' Check for success
If client.Status = 200 Then
bytes = client.ResponseBody
saveByteArrayToFile bytes, fileFullPath
downloadToFile = True
End If
End Function |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for @GCuser99
Details are in the image below.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks @hanamichi77777. Bummer... The reason why Firefox is not working for XMLHTTP60 when Edge/Chrome do is because the geckodriver download Url is redirected, and XMLHTTP60 does not support redirection. Another example of where XMLHTTP60 fails is in the DownloadResource method, when downloading from a redirected Url such as "https://github.com/GCuser99/SeleniumVBA/raw/main/dev/logo/logo.png" which gets redirected to "https://raw.githubusercontent.com/GCuser99/SeleniumVBA/main/dev/logo/logo.png". I wonder why your Office network 1 does not like UrlDownloadToFile? Do you think it is a security policy? So unfortunately, I don't believe your proposed solution will work on your Office network 1. Can you try this "double fallback" solution on your Office network 1? If the MSXML2.ServerXMLHTTP60 fallback does not work on that network, then please try the WinHttp.WinHttpRequest.5.1 method I shared above. Public Function downloadToFile(ByVal Url As String, ByVal fileFullPath As String) As Boolean
Dim client As New MSXML2.XMLHTTP60
On Error GoTo FallbackMethod1
'redirects not supported with this method
client.Open "GET", Url, False
client.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
client.send
' Check for success and save byte array to file
If client.Status = 200 Then
saveByteArrayToFile client.ResponseBody, fileFullPath
downloadToFile = True: Exit Function
End If
FallbackMethod1:
'redirects supported here
Dim client2 As New MSXML2.ServerXMLHTTP60
On Error GoTo FallbackMethod2
client2.Open "GET", Url, False
client2.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
client2.send
' Check for success and save byte array to file
If client2.Status = 200 Then
Debug.Print "ServerXMLHTTP60 fallback used for: " & Url
saveByteArrayToFile client2.ResponseBody, fileFullPath
downloadToFile = True: Exit Function
End If
FallbackMethod2:
On Error GoTo 0
Debug.Print "UrlDownloadToFile fallback used for: " & Url
downloadToFile = (UrlDownloadToFile(0&, StrPtr(Url), StrPtr(fileFullPath), 0&, 0&) = 0)
End FunctionHere is a test sub for your convenience: Sub test_DownloadResource()
Dim driver As SeleniumVBA.WebDriver
Dim element As SeleniumVBA.WebElement
Dim fso As New FileSystemObject
Set driver = SeleniumVBA.New_WebDriver
driver.StartEdge
driver.OpenBrowser
driver.ImplicitMaxWait = 2000
'img element with only src attribute
driver.NavigateTo "https://github.com/GCuser99/SeleniumVBA/wiki"
Set element = driver.FindElement(By.CssSelector, "img[alt='SeleniumVBA'")
element.DownloadResource ("src")
Debug.Assert fso.FileExists(driver.ResolvePath("logo.png"))
driver.DeleteFiles "logo.png"
driver.CloseBrowser
driver.Shutdown
End Sub |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @GCuser99
If I use MSXML2.XMLHTTP60 in a My Office network1 environment, I will receive a message saying "This site's security certificate revocation information cannot be used. Do you want to continue?" and you can download it by simply setting it to "Yes." If there is a problem with the certificate in MSXML2.XMLHTTP60, it appears that the user can display a dialog and choose to continue The code presented by GCuser99 will work. |
Beta Was this translation helpful? Give feedback.
-
|
I created the code with the help of copilot. The following code goes in the declaration section(83Lines) The following code is the procedure(116Lines) (10/6/2025) |
Beta Was this translation helpful? Give feedback.
-
|
I was wondering how to get the proxy settings, but copilot's answer is as follows Why is WinHttpGetIEProxyConfigForCurrentUser the only one that works? |
Beta Was this translation helpful? Give feedback.
-
|
@hanamichi77777, thanks for the code suggestion and all of your testing - very excellent. I took your code and asked Claude AI to simplify it (while honoring functionality) using a hybrid approach with WinHttp.WinHttpRequest.5.1 and proxy related API. Below is what was produced after a couple of iterations (<80 lines). Provided this still works in all of your environments, and it does not break in @6DiegoDiego9's and anyone else's who is paying attention, and does not significantly increase our Total Virus score, then I'm ok with replacing UrlDownloadToFile with this "lite" version. 'WinDevLib declares
Private Declare PtrSafe Function WinHttpGetIEProxyConfigForCurrentUser Lib "winhttp.dll" (ByRef pProxyConfig As WINHTTP_CURRENT_USER_IE_PROXY_CONFIG) As Long
Private Declare PtrSafe Function GlobalFree Lib "kernel32" (ByVal hMem As LongPtr) As LongPtr
Private Declare PtrSafe Function lstrlenW Lib "kernel32" (lpString As Any) As Long
Private Declare PtrSafe Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (Destination As Any, Source As Any, ByVal Length As LongPtr)
Private Type WINHTTP_CURRENT_USER_IE_PROXY_CONFIG
fAutoDetect As Long
lpszAutoConfigUrl As LongPtr
lpszProxy As LongPtr
lpszProxyBypass As LongPtr
End Type
Private Const HTTPREQUEST_PROXYSETTING_PROXY As Long = 2
' Convert pointer to string
Private Function ptrToStr(ByVal lpsz As LongPtr) As String
Dim Length As Long
If lpsz = 0 Then Exit Function
Length = lstrlenW(ByVal lpsz)
If Length > 0 Then
ptrToStr = Space$(Length)
CopyMemory ByVal StrPtr(ptrToStr), ByVal lpsz, Length * 2 ' Unicode = 2 bytes/char
End If
End Function
Public Function downloadToFile(ByVal Url As String, ByVal filePath As String) As Boolean
Dim http As Object
Dim bytes() As Byte
Dim ieProxy As WINHTTP_CURRENT_USER_IE_PROXY_CONFIG
On Error GoTo ErrHandler
Set http = CreateObject("WinHttp.WinHttpRequest.5.1")
http.setTimeouts 15000, 15000, 15000, 15000
' Apply IE proxy settings if found
If WinHttpGetIEProxyConfigForCurrentUser(ieProxy) <> 0 Then
If ieProxy.lpszProxy <> 0 Then
Dim proxyStr As String
proxyStr = ptrToStr(ieProxy.lpszProxy)
Dim bypassStr As String
If ieProxy.lpszProxyBypass <> 0 Then bypassStr = ptrToStr(ieProxy.lpszProxyBypass)
http.setProxy HTTPREQUEST_PROXYSETTING_PROXY, proxyStr, bypassStr
End If
' Free allocated strings
If ieProxy.lpszProxy <> 0 Then GlobalFree ieProxy.lpszProxy
If ieProxy.lpszProxyBypass <> 0 Then GlobalFree ieProxy.lpszProxyBypass
If ieProxy.lpszAutoConfigUrl <> 0 Then GlobalFree ieProxy.lpszAutoConfigUrl
End If
' Send request
http.Open "GET", Url, False
http.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
http.send
' Check status and save response to file
If http.Status >= 200 And http.Status < 300 Then
bytes = http.ResponseBody
saveByteArrayToFile bytes, filePath
downloadToFile = True
Else
Debug.Print "HTTP error: " & http.Status
End If
Exit Function
ErrHandler:
Debug.Print "Error: " & Err.Description
End Function |
Beta Was this translation helpful? Give feedback.
-
|
I was concerned about security issues with ignore all common SSL errors. However, in My Office (network1), I can download using the a hybrid On the other hand, I tried three times but can't download using the UrlDownloadToFile, I recommend removing "http.Option(4) = &H3300". |
Beta Was this translation helpful? Give feedback.
-
|
Awesome - thanks for checking! If/when you have time, can you also try it with the following line commented out? http.setProxy 0 ' HTTPREQUEST_PROXYSETTING_DEFAULT (WinHTTP default) = 0 |
Beta Was this translation helpful? Give feedback.
-
|
No problem and thanks! I edited the hybrid version of your code above to take that line out. It is also not necessary to enable the redirects as that is enabled by default. I changed the CopyMemory declaration to be compatible with WinDevLib, which I use on all API declarations in the twinBASIC DLL project. Anyway, it's now a fairly standard WinHttp downloader, but with the addition of @hanamichi77777's proxy stuff. It solves an issue on one of @hanamichi77777's corporate networks that does not seem to allow UrlDownloadToFile. @6DiegoDiego9, what do you think of this as a replacement for UrlDownloadToFile? While we wait to hear back from Diego, I'll do a VirusTotal check to see how the replacement compares on false-positives. |
Beta Was this translation helpful? Give feedback.
-
|
Here are my test results: Home PC:URLDownloadToFile: success Business PC:URLDownloadToFile: success |
Beta Was this translation helpful? Give feedback.
-
|
Code using low level WinINet based API. Redirection is supported by default Since it is a code that is not used much, I think it is unlikely that it will be falsely detected by Defender. P.S. I thought about whether I could solve this using the WinHttp API, but I couldn't find a generic solution. Fixing Certificate Error 12057 via Internet Options |
Beta Was this translation helpful? Give feedback.
-
|
Here is an extension of the winhttp hybrid that supposedly handles auto-proxy for PAC and WPAD but of course I have no way of testing this on my local machine. It is <140 lines. But honestly, I'll let you guys decide since you can test under a number of different network situations. Private Declare PtrSafe Function WinHttpGetProxyForUrl Lib "winhttp" (ByVal hSession As LongPtr, ByVal lpcwszUrl As LongPtr, pAutoProxyOptions As WINHTTP_AUTOPROXY_OPTIONS, pProxyInfo As WINHTTP_PROXY_INFO) As Long
Private Declare PtrSafe Function WinHttpOpen Lib "winhttp" (ByVal pszAgentW As LongPtr, ByVal dwAccessType As Long, ByVal pszProxyW As LongPtr, ByVal pszProxyBypassW As LongPtr, ByVal dwFlags As Long) As LongPtr
Private Declare PtrSafe Function WinHttpCloseHandle Lib "winhttp" (ByVal hInternet As LongPtr) As Long
Private Declare PtrSafe Function WinHttpGetIEProxyConfigForCurrentUser Lib "winhttp.dll" (ByRef pProxyConfig As WINHTTP_CURRENT_USER_IE_PROXY_CONFIG) As Long
Private Declare PtrSafe Function GlobalFree Lib "kernel32" (ByVal hMem As LongPtr) As LongPtr
Private Declare PtrSafe Function lstrlenW Lib "kernel32" (lpString As Any) As Long
Private Declare PtrSafe Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (Destination As Any, Source As Any, ByVal Length As LongPtr)
Private Type WINHTTP_AUTOPROXY_OPTIONS
dwFlags As Long
dwAutoDetectFlags As Long
lpszAutoConfigUrl As LongPtr
lpvReserved As LongPtr
dwReserved As Long
fAutoLogonIfChallenged As Long
End Type
Private Type WINHTTP_PROXY_INFO
dwAccessType As Long
lpszProxy As LongPtr
lpszProxyBypass As LongPtr
End Type
Private Type WINHTTP_CURRENT_USER_IE_PROXY_CONFIG
fAutoDetect As Long
lpszAutoConfigUrl As LongPtr
lpszProxy As LongPtr
lpszProxyBypass As LongPtr
End Type
Private Const WINHTTP_AUTOPROXY_AUTO_DETECT As Long = &H1
Private Const WINHTTP_AUTOPROXY_CONFIG_URL As Long = &H2
Private Const WINHTTP_AUTO_DETECT_TYPE_DHCP As Long = &H1
Private Const WINHTTP_AUTO_DETECT_TYPE_DNS_A As Long = &H2
Private Const WINHTTP_ACCESS_TYPE_NO_PROXY As Long = 1
Private Const HTTPREQUEST_PROXYSETTING_PROXY As Long = 2
' Convert pointer to string
Private Function ptrToStr(ByVal lpsz As LongPtr) As String
Dim Length As Long
If lpsz = 0 Then Exit Function
Length = lstrlenW(ByVal lpsz)
If Length > 0 Then
ptrToStr = Space$(Length)
CopyMemory ByVal StrPtr(ptrToStr), ByVal lpsz, Length * 2 ' Unicode = 2 bytes/char
End If
End Function
Public Function downloadToFile(ByVal Url As String, ByVal filePath As String) As Boolean
Dim http As Object
Dim bytes() As Byte
Dim ieProxy As WINHTTP_CURRENT_USER_IE_PROXY_CONFIG
Dim autoProxyOptions As WINHTTP_AUTOPROXY_OPTIONS
Dim proxyInfo As WINHTTP_PROXY_INFO
Dim hSession As LongPtr
Dim proxyStr As String
Dim bypassStr As String
On Error GoTo ErrHandler
Set http = CreateObject("WinHttp.WinHttpRequest.5.1")
http.setTimeouts 15000, 15000, 15000, 15000
http.Option(4) = &H3300 ' Ignore common SSL errors (Optional)
' Try IE proxy settings first
If WinHttpGetIEProxyConfigForCurrentUser(ieProxy) <> 0 Then
' Handle static proxy
If ieProxy.lpszProxy <> 0 Then
proxyStr = ptrToStr(ieProxy.lpszProxy)
If ieProxy.lpszProxyBypass <> 0 Then bypassStr = ptrToStr(ieProxy.lpszProxyBypass)
http.setProxy HTTPREQUEST_PROXYSETTING_PROXY, proxyStr, bypassStr
GoTo ProxyConfigured
End If
' Handle PAC file or auto-detect
If ieProxy.lpszAutoConfigUrl <> 0 Or ieProxy.fAutoDetect <> 0 Then
hSession = WinHttpOpen(0, WINHTTP_ACCESS_TYPE_NO_PROXY, 0, 0, 0)
If hSession <> 0 Then
' Configure auto-proxy options
If ieProxy.fAutoDetect <> 0 Then
autoProxyOptions.dwFlags = WINHTTP_AUTOPROXY_AUTO_DETECT
autoProxyOptions.dwAutoDetectFlags = WINHTTP_AUTO_DETECT_TYPE_DHCP Or WINHTTP_AUTO_DETECT_TYPE_DNS_A
End If
If ieProxy.lpszAutoConfigUrl <> 0 Then
autoProxyOptions.dwFlags = autoProxyOptions.dwFlags Or WINHTTP_AUTOPROXY_CONFIG_URL
autoProxyOptions.lpszAutoConfigUrl = ieProxy.lpszAutoConfigUrl
End If
autoProxyOptions.fAutoLogonIfChallenged = 1
' Get proxy for this specific URL
If WinHttpGetProxyForUrl(hSession, StrPtr(Url), autoProxyOptions, proxyInfo) <> 0 Then
If proxyInfo.lpszProxy <> 0 Then
proxyStr = ptrToStr(proxyInfo.lpszProxy)
If proxyInfo.lpszProxyBypass <> 0 Then bypassStr = ptrToStr(proxyInfo.lpszProxyBypass)
http.setProxy HTTPREQUEST_PROXYSETTING_PROXY, proxyStr, bypassStr
' Free proxy info strings
If proxyInfo.lpszProxy <> 0 Then GlobalFree proxyInfo.lpszProxy
If proxyInfo.lpszProxyBypass <> 0 Then GlobalFree proxyInfo.lpszProxyBypass
End If
End If
WinHttpCloseHandle hSession
End If
End If
ProxyConfigured:
' Free IE proxy config strings
If ieProxy.lpszProxy <> 0 Then GlobalFree ieProxy.lpszProxy
If ieProxy.lpszProxyBypass <> 0 Then GlobalFree ieProxy.lpszProxyBypass
If ieProxy.lpszAutoConfigUrl <> 0 Then GlobalFree ieProxy.lpszAutoConfigUrl
End If
' Send request
http.Open "GET", Url, False
http.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
http.send
' Check status and save response to file
If http.Status >= 200 And http.Status < 300 Then
bytes = http.ResponseBody
saveByteArrayToFile bytes, filePath
downloadToFile = True
Else
Debug.Print "HTTP error: " & http.Status
End If
Exit Function
ErrHandler:
Debug.Print "Error: " & Err.Description
End Function |
Beta Was this translation helpful? Give feedback.
-
|
I created this using Copilot to meet all four of the following conditions. ✅ Condition Check ② Achieve proxy connectivity equivalent to UrlDownloadToFile ③ Support redirects equivalent to HttpRequest ④ Handle certificate errors equivalent to HttpRequest |
Beta Was this translation helpful? Give feedback.
-
|
Thanks guys for your efforts! |
Beta Was this translation helpful? Give feedback.
-
|
Good news! I finally tested both on your last solutions on all my 4 environments, using a test module with 7 test cases generated by Sonnet-4.5-high. More specifically:
You can download and run my test here. Just switch the access modifier (Private/Public) of the function DownloadToFile in either the GCuser99 or hanamichi77777 module, depending on which one you want to test, and launch Tests.TestDownloadToFile, which calls the only DownloadToFile left Public. Results:GCuser99 solution: hanamichi77777 solution: My final though, considering the 7/7 passes (tie), the VirusTotal reports (hanamichi77777 wins), and the fact that WinINet is discouraged (GCuser99 wins), is that I'm fine with both solutions, with no preference. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for the very interesting experiment. |
Beta Was this translation helpful? Give feedback.
-
|
@6DiegoDiego9 thanks for the test suite! For a base case compare, I also added a test suite for UrlToDownloadFile API. I am not on a business pc and I'm getting very strange and inconsistent results for each test - fail/pass, sometimes long delays, and sometimes garbage download results. I'm going to have to take a closer look to see what is going on with my system! |
Beta Was this translation helpful? Give feedback.
-
|
Oh, it looks like there is currently a problem with https://httpbin.org (503 Service Temporarily Unavailable). What is interesting is that WinInet version passes all the tests with that website down but the downloads are (of course) garbage. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe my IP address is blocked. Using a VPN seems to have corrected the situation for now. The base case API fails on bad url test. One thing I noticed Is that the WinInet version, even though it passes the redirect test, the test_redirect.png result is not readable - anyone else see this? |
Beta Was this translation helpful? Give feedback.
-
|
Occasional Certificate Errors occurred when using UrlDownLoadToFile or WinInetAPI on NetWork1.
|
Beta Was this translation helpful? Give feedback.
-
|
Great teamwork @6DiegoDiego9 and @hanamichi77777 - thanks! |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
test1 is as follows. |
Beta Was this translation helpful? Give feedback.
-
|
Linking files to GitHub might raise some concerns. Although it is the same code, it is a newly created file, so there will be no false positives. (Update) test9 remains a false positive.
|
Beta Was this translation helpful? Give feedback.
-
|
SeleniumBasic with WebDriver automatic update tool applied is showing false positives.
|
Beta Was this translation helpful? Give feedback.
















Uh oh!
There was an error while loading. Please reload this page.
-
Depending on my company's network, UrlDownloadToFile may not work and may not be able to download.

Using MSXML2.XMLHTTP60 allows for stable downloads, except for FireFox.
So, in order to support FireFox, I first tried running MSXML2.XMLHTTP60, and if there is an error, I suggest that UrlDownloadToFile be used.
Beta Was this translation helpful? Give feedback.
All reactions