Hacking around the Twitter login thing

[kordlessagain]

  from flask import Flask
  import requests

  app = Flask(__name__)

  @app.route('/<path:url>')
  def proxy(url):
      return requests.get(f'https://twitter.com/{url}', headers={'User-Agent': 'Googlebot'}).text

  if __name__ == '__main__':
      app.run()

[_Algernon_]

Or alternatively, stop supporting user hostile sites and take this opportunity to break your addiction. Touch some grass, so to speak.

[Pxtl → _Algernon_]

Yup. I think more people need to admit to themselves that this is an addiction. When Twitter started going sideways in ways I find politically unacceptable (I have trans people in my family) I started using Mastodon. But of course, on Mastodon people would still link back to "the birdsite" and I'd gradually gravitate back to it more and more until I'd let my Twitter usage go back to normal, because realistically that's still where all the action is. Just out of habit.

It's really easy to let these habits control you.

The recent mayhem has given me another opportunity to try at a clean break.

[wiskinator → Pxtl]

I know what you mean. I fully deleted my 11 year old twitter account, and now I get almost frustrated enough to make a new one whenever I click on a HN news web article that links to twitter for context :/

[panarky → wiskinator]

I've been wondering if it's better to delete my account or just stop using it.

Deleting my account doesn't destroy my data, everything still exists, it's just not visible to me anymore. But deletion allows someone else to use my handle.

[Pxtl → panarky]

I've put mine private. Then nobody can steal the name, and the lack of engagement eliminates the temptation to use it much.

[throwawayadvsec → Pxtl]

I'm not addicted to twitter at all, my last tweet was a decade ago, but yeah a lot of people link interesting stuff there, doesn't mean I'm addicted to it

[sys_64738 → _Algernon_]

This. These hacks only last until they're plugged and are generally a waste of time publicizing.

[throwawaymobule → sys_64738]

That may be the idea here, if one wanted to bamboozle the site into stopping search engines from indexing them.

I heard yesterday that they already blocked googlebot and got delisted, but I didn't confirm, and I guess that was false.

[wunderland]

That it’s this easy to get around the login ban shows how poorly executed and hastily planned this was.

I imagine it will be rolled-back in the next week as it becomes clear to advertisers how much lower value tweets are

[samwillis → wunderland]

It was announced as an anti scraping "feature" but the reality is it's about encouraging signups, tracking for ads and starting the sales funnel for upgrading to Twitter blue. Every other social media site has done the same at some point to increase logged in users.

[leros → samwillis]

It's probably also about getting people to use the API.

With Twitter's data being public, bots and data analyzers can just scrape the site instead of paying for the now extremely expensive API. Not only does this mean Twitter isn't getting paid, but scraping puts a lot more load on Twitter servers than equivalent API usage.

[samwillis → leros]

Anyone doing industrial scraping already knows all the tricks for user agent swapping and making headers, including their order, look like "real" browsers or bots. It's not about pushing people to the API.

[jdiff → samwillis]

Combined with the other changes that were also hastily and sloppily implemented and are now getting slowly rolled back, I think that strengthens the theory that Twitter was facing extreme capacity crunch and threw a whole bunch of slapstick measures to reduce traffic at the wall to keep their remaining servers from toppling over.

[dgrin91 → wunderland]

To play devil's advocate a bit - you can call their solution good from an engineering standpoint because they were able to do it quickly and cover the vast majority of cases. Yes you can seemingly easily get around this with a quick user agent switch, but 99.9999% of users won't do that, so this solution is good enough.

[3cats-in-a-coat → dgrin91]

To play the... angel's prosecutor (?), but how can it be good from engineering standpoint when they're not trying to block users without login, supposedly, but this army of scraping AI firms and botnets?

Who ever scraped a page and doesn't know to try referrer, user-agent and all other 101 tricks in the book?

[rsynnott → 3cats-in-a-coat]

God's Advocate, in fact (https://en.wikipedia.org/wiki/Devil's_advocate). These used to be _official posts_ in the Catholic Church; the devil's advocate's role was to argue against the canonisation of a prospective saint.

(I can't, offhand, figure out if they got a special uniform, but I do hope so.)

They now use a more civil law-ish approach to this.

[proamdev123 → rsynnott]

You’re right, but OP’s “angel’s prosecutor” has a nice ring to it. I love it!

[wunderland → dgrin91]

It immediately broke embedded tweets all over the web, something which was unique to Twitter. This lowers Twitter’s overall value in hopes of reaching some short term sign-up goal.

[addisonl → dgrin91]

Didn’t they ddos their own website on Saturday when they rolled this out?

[kordlessagain]

Install Flask:

  pip install Flask

Save the code to `elon.py` and then run it:

  python elon.py

Use http://localhost:5000/<twitter_handle> to view a user's page without being forced to login.

[babypuncher → kordlessagain]

I have no interest in actually viewing twitter, but I might use this to build a scraping tool that dumps everything to /dev/null just because I can.

[rajivharlalka → babypuncher]

Wouldn't that be just wasting resources more useful that the thing being dumped.

[rgbrenner]

Twitter already reverted it this morning:

https://techcrunch.com/2023/07/05/twitter-silently-removes-l...

[runjake → rgbrenner]

Still in effect here, as of 2:39PM Pacific. No cache to clear.

[rgbrenner → runjake]

I just checked again and it's back for me too.

[izoow]

Hah, interesting. Wouldn't a user agent switcher extension accomplish the same though?

[gsanderson → izoow]

You don't even need an extension. Open Developer Tools and find the appropriate option. For example in Chrome, there's a "Network Conditions" panel. Change to e.g "Googlebot" in the dropdown. Seems to work, at least for now.

[kordlessagain → izoow]

Yes, in theory that should also work: https://chrome.google.com/webstore/detail/user-agent-switche...

[SCdF]

Weirdly viewing tweets without being logged in started working for me again today. Who really knows though: I feel like every time I get linked a tweet I'll get a different weird experience when I try to view it, so who knows what will happen tomorrow.

[ezfe]

No need to use Flask, just use the browser dev tools to set the value to Googlebot and it works directly in the browser

[pmontra]

I was able to view a Twitter page today, from a link on HN. I thought that they had lifted the ban because I'm not taking any countermeasures. Firefox on Android.

[ChrisArchitect]

Tell HN: