If anyone wants to see that, try launching the browser via Selenium. I used to do that to partially automate some activities, such as download bank statements. I'd have my Selenium using script open a browser and go to the bank, then wait for me to login and get to the account page.
I'd login, dismiss any popup or interstitial promotions the bank decided to give me, get to the account page, and tell my script to continue.
My script would then use Selenium to click the download button, click the "custom date range" radio button on download popup, fill in the range fields to cover the last 60 days, pick OFX for the download format, and start the download, prompting me to let it know when the download is finished.
When the download finished, I could then go to one of my other accounts at that bank, tell the script I'm there, and that one gets downloaded, and so on.
My bank isn't giving CAPTCHAs so that would still work if I were to get around to updating my script to deal with some redesigns they did of their pages which broke finding the relevant elements on the page.
But I've found that if I do visit a site that uses hCaptcha while using the Selenium launched browser, it seems to get stuck. Click to tell it I'm not a bot. Then get an image test. Answer that correctly and get another image test. Answer that correctly. Then it goes back to the click if you are not a bot thing, and repeats--two more image tests and back to the beginning.
Here's a program if anyone wants to try this and has the Selenium Webdriver package for Python3 installed. This will open a browser and take you to fanfiction.net. Trying to actually read any story will bring up the CAPTCHA.
#!/usr/bin/env python3
from selenium.webdriver import Chrome
driver = Chrome()
driver.get("https://www.fanfiction.net")
input("press enter when done")
driver.close()
driver.quit()
I'm not sure if the looping is a Cloudflare thing or a fanfiction.net thing, because the latter is the only site I use that has Cloudflare's CAPTCHA.
you could get past the CAPTCHA, but that stopped working a while ago.
There's this project to provide a Selenium Chrome driver that is supposed to not trigger anti-bot detectors [1], but it still hit the CAPTCHA loop when I tried it.
fanfiction.net has also simply broken the Calibre FanFicFare integration thanks to their CloudFlare shenanigans.
The workaround is to simply visit all chapters separately and then point Calibre at the Google Chrome cache folder.
So nice going there, fanfiction.net. Instead of offering a 1-click .epub download like AO3 (which is completely CDN-able with a very long TTL), you now had to serve 50 individual requests. Great engineering work there.
(Obviously they do this to serve ads on every request)
AO3 is OSS and vastly understaffed. Having worked on some of their tickets, IMO they could use 20 contributors working part-time for a year or two to stabilize it until the idea of useful new features becomes viable.
I strongly encourage anyone with Rails experience to contribute [0]. There is a giant test suite which definitely helps with stability. The ticket time-to-resolve is simply quite slow due to the above-mentioned understaffing, so don't be discouraged!
Yeah there’s lots of detect and anti detect stuff going back and forth. It’s pretty silly and frustrating for situations like yours. Doing things for yourself to speed up mundane life things.
There’s so many anti-detect libraries on GitHub these days. Wonder how many work well.
I'd login, dismiss any popup or interstitial promotions the bank decided to give me, get to the account page, and tell my script to continue.
My script would then use Selenium to click the download button, click the "custom date range" radio button on download popup, fill in the range fields to cover the last 60 days, pick OFX for the download format, and start the download, prompting me to let it know when the download is finished.
When the download finished, I could then go to one of my other accounts at that bank, tell the script I'm there, and that one gets downloaded, and so on.
My bank isn't giving CAPTCHAs so that would still work if I were to get around to updating my script to deal with some redesigns they did of their pages which broke finding the relevant elements on the page.
But I've found that if I do visit a site that uses hCaptcha while using the Selenium launched browser, it seems to get stuck. Click to tell it I'm not a bot. Then get an image test. Answer that correctly and get another image test. Answer that correctly. Then it goes back to the click if you are not a bot thing, and repeats--two more image tests and back to the beginning.
Here's a program if anyone wants to try this and has the Selenium Webdriver package for Python3 installed. This will open a browser and take you to fanfiction.net. Trying to actually read any story will bring up the CAPTCHA.
I'm not sure if the looping is a Cloudflare thing or a fanfiction.net thing, because the latter is the only site I use that has Cloudflare's CAPTCHA.It used to be that if you added
and changed opening the driver to you could get past the CAPTCHA, but that stopped working a while ago.There's this project to provide a Selenium Chrome driver that is supposed to not trigger anti-bot detectors [1], but it still hit the CAPTCHA loop when I tried it.
[1] https://github.com/ultrafunkamsterdam/undetected-chromedrive...