Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bloomberg.com bot detection #18

Closed
mccutchen opened this issue Jan 3, 2023 · 1 comment · Fixed by #19
Closed

bloomberg.com bot detection #18

mccutchen opened this issue Jan 3, 2023 · 1 comment · Fixed by #19
Labels
needs-workaround A common upstream domain requires a special workaround

Comments

@mccutchen
Copy link
Owner

It looks like bloomberg.com is now detecting urlresolver.com as a bot, but an example failed resolution points at a possible path forward:

image

Dumb fix: detect their bot landing page and fall back on the previous hop (which, in this case, is the real URL we want).

Clickable example URL, for posterity:

https://link.mail.bloombergbusiness.com/click/30059877.127888/aHR0cHM6Ly93d3cuYmxvb21iZXJnLmNvbS9uZXdzL2FydGljbGVzLzIwMjItMTItMjIvZmlyZWQtdHdpdHRlci1tYW5hZ2VyLXN1ZXMtb3Zlci1jYW5jZWxsYXRpb24tb2Ytc3RvY2stb3B0aW9ucz9jbXBpZD1CQkQxMjIyMjJfTU9ORVlTVFVGRiZ1dG1fbWVkaXVtPWVtYWlsJnV0bV9zb3VyY2U9bmV3c2xldHRlciZ1dG1fdGVybT0yMjEyMjImdXRtX2NhbXBhaWduPW1vbmV5c3R1ZmY/5f15b2f8375b191df5003b21B03bb0f6b
@mccutchen mccutchen added the needs-workaround A common upstream domain requires a special workaround label Jan 3, 2023
@mccutchen
Copy link
Owner Author

Even dumber fix: It turns out the destination URL (plus some junk) is encoded directly in that link.mail.bloombergbusiness.com wrapper URL's penultimate path component:

$ echo "aHR0cHM6Ly93d3cuYmxvb21iZXJnLmNvbS9uZXdzL2FydGljbGVzLzIwMjItMTItMjIvZmlyZWQtdHdpdHRlci1tYW5hZ2VyLXN1ZXMtb3Zlci1jYW5jZWxsYXRpb24tb2Ytc3RvY2stb3B0aW9ucz9jbXBpZD1CQkQxMjIyMjJfTU9ORVlTVFVGRiZ1dG1fbWVkaXVtPWVtYWlsJnV0bV9zb3VyY2U9bmV3c2xldHRlciZ1dG1fdGVybT0yMjEyMjImdXRtX2NhbXBhaWduPW1vbmV5c3R1ZmY" | base64 -D
https://www.bloomberg.com/news/articles/2022-12-22/fired-twitter-manager-sues-over-cancellation-of-stock-options?cmpid=BBD122222_MONEYSTUFF&utm_medium=email&utm_source=newsletter&utm_term=221222&utm_campaign=moneystu

@mccutchen mccutchen mentioned this issue Jan 5, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs-workaround A common upstream domain requires a special workaround
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: interstitial handling improvements mccutchen/urlresolver
1 participant
@mccutchen