

Tell HN: How to ban url shorteners from linking to your site - pj

This is just an idea I'm going to throw out there.  It seems a bit odd to suggest it because any link is a good link, right?  Maybe not.<p>Anyway, something like this in a bit of javascript would prevent people from linking to your site using a url shortener:<p><pre><code>  &#60;script&#62;
  if (document.referrer.indexOf('bit.ly') &#62; 0 
    || document.referrer.indexOf('someothershortener')&#62;0)
   document.location.href='http://yoursite.com/noshortener.html';
  &#60;/script&#62;
</code></pre>
Put in that noshortener.html some content that will tell people why you don't allow shorteners to your content, here are some suggestions:<p><pre><code>  1) You want to know where your visitors are coming 
  from and shorteners steal the real referrer.
  2) Shorteners are a layer of indirection that could 
  be modified beyond the user of the shortener's 
  discretion to do something like digg is doing now
  3) If the shortener goes away, the link to the 
  content is no longer valid
  4) It slows down the web due to one more DNS lookup, 
  one more server redirect, etc...
</code></pre>
Most users of shorteners, when they use one to link to a site are going to click the link just to make sure it works.  When they do, they'll see your banned url shorteners page and it will help all of us "experts" in the web who understand why url shorteners are bad educate the lay users of the web as to why those url shorteners are bad.<p>Yes, we take a hit, but we are sacrificing a little traffic for a better web experience in the future.
======
zain
None of the reasons you listed are significant to the person reading them. It
is not the visitor's problem that you can't track referrers or that it slows
down the web, and the fact that they're seeing this page means the shortened
is working as intended. Furthermore, you're basically making it impossible to
share links to your site on Twitter (where they get auto-shortened).

All in all, you're doing little more than pissing off visitors you've probably
worked pretty hard to get.

~~~
pj
_the fact that they're seeing this page means the shortened is working as
intended_

They won't actually see the page they used the URL shortener to create a link
to, because the browser will automatically redirect to another page where you
have explained why you are disallowing url shortened links to your content.

I also disagree with your premise. I think the reasons I mentioned are
significant to all users of the web.

I do understand the issue you have presented though, that the short term
benefits to the user of using a shortened url, may outweigh the long term
ramifications I have listed, but perhaps as a group here, we can come up with
some arguments that may help us content producers take back the web from the
url shorteners.

Really, I just think when taken to the logical conclusion, a web saturated
with url shortened links is really a not very good web.

I'm trying to propose a solution here, it may not be the best one and it will
take a coordinated effort among content producers. I suppose those who ignore
the call to arms are going to "please" url shortener users more than those who
don't, but with a little effort and a genuine interest in a sustainable web, I
think we can win this war.

~~~
rex
_...the browser will automatically redirect to another page where you have
explained why you are disallowing url shortened links to your content._

If you're going to the trouble of serving up content from your site, why would
you disallow the content they came to see anyway? Maybe the concept could work
if you had support from higher-up in the redirection process (from url
shortening services themselves), but to effectively re-direct from your own
page to another "you an't access this content" page on your site will only
make end users confused and disinterested, driving traffic away.

I agree that URL shorteners are bad and acknowledge you're trying to do
something about it, but this isn't the way to solve the problem.

~~~
calambrac
Because it's more important that the shorteners die before they become more
entrenched than that I get their traffic. The fact that I can seriously say
this right now is something worth fighting to preserve.

~~~
ubernostrum
TinyURL has lasted longer than many of the sites its links point to.

~~~
calambrac
The fact that for however many years TinyURL served a niche purpose hiding
goatse links somehow means the recent explosion in the use of these services
is positive for the web? And: what good is a shortened link to a dead site?

~~~
ubernostrum
"And: what good is a shortened link to a dead site?"

About as much good as an un-shortened link to a dead site, which is what it
turns into after the redirect.

I guess my problem here is I still haven't seen an actual _argument_ to back
up the sorts of "behold a pale horse: and his name that sat on him was Digg,
and Twitter followed with him" hysteria that's been going around. Yeah,
sometimes links die of old age; we've been dealing with that for years and
it's not a new or unique problem. And yeah, sometimes people hide nasty things
behind redirects; again, we've been dealing with that for years and it's not a
new or unique problem.

So what's with the gloom and doom?

~~~
calambrac
_About as much good as an un-shortened link to a dead site, which is what it
turns into after the redirect._

That's exactly my point; you were the one who pointed to TinyURL outlasting
some of the sites it stores links to as though that were a good thing.

 _Yeah, sometimes links die of old age; we've been dealing with that for years
and it's not a new or unique problem._

Yes, but this introduces another failure mode, where the end target could be
fine, but it could be the shortener service that dies. That sure is awesome if
it's your content being linked, and you have no way to fix it, huh? At least
with regular links, if a link goes dead, it was your own fault, and you can
fix it if you want.

 _And yeah, sometimes people hide nasty things behind redirects_

So why would you want to create more opportunities for this to happen?

 _We've been dealing with that for years and it's not a new or unique problem_

But not on the scale that we're starting to see now. You seem to be operating
on the premise that things we've seen used in moderation without too much
trouble are automatically safe for heavy use, but the Digg thing shows how
ripe for abuse these services really are.

------
callmeed
I'm all for blocking framing sites like the Diggbar, but killing all URL
shorteners is a bad idea, IMO.

With Twitter growing so fast and _automatically_ shortening pasted URLs, you
could actually be losing some quality traffic if people start talking about
your business/app/site.

~~~
calambrac
Someone with a big vested interest in seeing the internet not break needs to
step up and provide a shortening service that says "if you own the target, you
own the shortened link, period".

~~~
barredo
like <http://awe.sm> ?

------
jerf
Honest question: Why do you care?

Consider this like an impromptu poll about why you care about not getting
linked via a shortener. (If you _don't_ care, I'd ask that you not post a
reply to my message here, though I obviously can't enforce that.) I ask
because I wouldn't care, but I am interested with an open mind in why someone
would.

~~~
pj
I think really, because I care about the future of the web. I know there are
myriad problems with url shorteners and they are jeopardizing the future of
the web. My website is just one little piece of it, I know, but education is
grand. I think people, when presented with rational reasons that benefit us
all, will listen to them.

The person linking to your content is interested more than anything in
bringing users to your content. If you put up a page that prevents them from
doing that, then they'll use another link, or not link to you at all.

Really, it's about education and education comes at a cost. If you are
educating, then you aren't focusing on coding or something else.

On the page where the javascript redirects, you could say, "Thank you very
much for linking to my content. I think it's great that you appreciate what I
have written, but please understand that the method you are using to link to
it is not good and here's why: yada yada yada..."

~~~
ubernostrum
You really think that being able to get a shortened URL to put in a post on
Twitter is such a threat to the future of the Web that you need to,
essentially, implement a huge "fuck you for trying to visit my site" message?

If that's the case (and believe me, I've read Schachter's bit, and many of the
zillions of "me too" posts following it, and so know the arguments inside and
out at this point), I really don't have anything to say to you except
"congratulations on your new job in the newspaper industry, I'm sure you'll
fit right in".

~~~
joshu
Indeed.

I guess the main points were:

1) The shortener ought to be owned by the publisher; at least then the
document and the identifier are connected within the same
organization/entity/whatever. At least that way if the URL is lost, the
document probably is too.

2) The shortening is necessary due to design decisions by twitter (and poor
ones, at that.) Changing these in the right way could make the entire problem
dissapear.

Punishing users is probably not the right way.

~~~
ubernostrum
Well, URL shortening has value outside of Twitter. There are plenty of
services which generate absolutely enormous URLs which would be a pain to
sling around on their own, and being able to get a quick, short version to
refer to is awfully handy (think "directions on Google Maps").

~~~
calambrac
And before Twitter, that's the only kind of thing these services were used
for. They were hardly prevalent, and they certainly weren't "necessary" in any
sense. The downside just didn't really matter that much.

Now that's starting to change. By analogy: there's a legitimate use for DDT,
and when I go camping once a year I happily buy a bottle of it, but that
doesn't make spraying it out of a firehose around cities a good idea.

~~~
ubernostrum
You're blowing this so far out of proportion that I'm not sure it's possible
to have a reasonable discussion with you.

~~~
calambrac
You really believe that people who think heavy widespread use of url
shorteners is bad for the web are too unreasonable to even speak to? Dude...
alright, whatever.

~~~
ubernostrum
You compared the use of shortened URLs to spraying poison out of a firehose.
Can't quite put my finger on it, but yeah, I think that's a bit unreasonable.

~~~
calambrac
You do know that they did actually used to spray DDT out of a firehose, right?
That that wasn't just some random bullshit I made up on the spot, it's
actually a case where we did wrong by taking something that's okay in
moderation and trying to use it in large quantities? Do I really think url
shorteners are as bad as spraying poison out of a firehose? No, but _it was a
fucking analogy_.

------
a2tech
I tend to think that any link and visit is a good one. What I would prefer to
do is using the above javascript, detect incoming traffic from URL shorteners
and display my own informational bar across the top with the pages real URL
and perhaps simple stats of how many people were referred to that page using a
shortened link.

~~~
pj
That's a good idea, instead of

    
    
      document.location.href="...";
    

You could use:

    
    
      document.getElementById('urlshorteners').style.display='block';
    

Where originally, the div is like this

    
    
      <body>
      <div id='urlshorteners' style='display:none;'>
            Url shorteners information...
      </div>
    

I like that idea.

------
jasonkester
While we're talking about it, here's a feature I'd like to see these URL
Shortening services impliment:

\- If the "Shortened" URL you are about to give me is actually _longer_ than
the URL I gave you, just give me back that URL.

I see this sort of thing on Twitter all the time, and it drives me nuts. "hey,
check out:

<http://tinyurl.com/mweytn>

... which redirects you to...

<http://twiddla.com/>

Uh... What exactly have you shortened again?

------
keyist
The reasons you give are all valid reasons. However, you miss the reason most
likely to sway end-users: a shortener service can relate every page a user has
seen to that user (or at least, to his ip). The privacy angle is the biggest
factor imo.

In your info page you may also want to promote Untiny
(<http://untiny.com/extra/>) as a service which helps bypass shorteners with
minimum fuss.

------
jrockway
This is exactly why I use noscript.

But anyway, why not a standard API, similar to robots.txt or favicon.ico that
will let each site control shortening? Requesting
<http://foo.com/shorten?url=http://foo.com/long/url/goes/here> will return
<http://foo.com/lkjdhf> or whatever.

------
philfreo
I think if you're going to do this it'd be better done as a server side
redirect, like through .htaccess

However even though I'm, in general, against URL shorteners, I wouldn't do
this because I think there are some situations where they are helpful.

~~~
pj
That's true. I just threw this together. I think you could even create a
database table and generate the redirect on the server side or dynamically
create the javascript. Most browsers have javascript enabled. Maybe a hacker
with some time (maybe I'll create one) could create a site with a dynamically
created .js file and a remote link like this:

    
    
       <script src="http://takebackthewebfromurlshorteners.com/bannedreferrerslist.js">
      </script>
    

This .js could be dynamically generated from a database table of known url
shorteners. Such a service could even get really fancy and show stats from all
the known shorteners, etc.

You could add parameters like this:

    
    
      <script src=".../bannedlist.aspx?allow=digg.com,bit.ly,etc..."></script>
    

Then the user could customize exactly which url shorteners they want to allow.

All kinds of things a service like this could do!

~~~
nir
<http://www.listable.org/json/url-shortening-sites> :)

~~~
pj
Yeah! That's what i'm talking about!!

now it's just a FOR loop!

~~~
bhseo
<http://www.blackhat-seo.com/2009/short-url-services/>

A bigger list.

------
ErrantX
As a user if you did that to me I would just close the link and lose interest
in your site....

Yes, agreed, the DiggBar style thing IS annoying and we should stop that. But
short links are _important_.

Several forums I frequent insist you use shortners for URL's longer than 25
characters (you get a warning if you dont!) to keep things neat. Twitter -
your isntantly killing traffic from that crowd.

Now, showing a box on the page with some of that info would be useful /
understandable.

------
throw_away
it would be nice if url shorteners did a double redirect so that the following
happened:

1) user clicks on <http://sho.rt/abcdef> located on <http://original-
referrer.com>

2) sho.rt redirects to [http://sho.rt/abcdef?original-
referrer=http://original-refer...](http://sho.rt/abcdef?original-
referrer=http://original-referrer.com)

3) second url redirects to the content.

in this way, content owners will know where their traffic is coming from,
people who place shortened urls don't have to do anything special and the
pasted urls remain short. the only downside is a slight increase to latency to
retrieve the second redirect.

------
tallanvor
Why not pass the url the user was trying to visit to your noshortener page?
Then you can have a brief blurb and let them click the link to continue. You
still don't get full referrer information, but it's better than nothing and
the user gets to view the content.

------
nir
Might find this list of URL shorteners useful:
<http://www.listable.org/show/url-shortening-sites> (can be exported as
JSON/SQL/plaintext)

------
tokenadult
How long are the URLs on your site, that they are getting shortened so often?

~~~
andrewf
Putting as much content in the URL as possible is a common SEO tactic.

A bit silly, really - we're producing readable English links targeted at the
machines, and short strings of unintelligible gibberish for humans to look at
and click on.

~~~
jrockway
_Putting as much content in the URL as possible is a common SEO tactic._

Are there any controlled studies that shows that this does anything useful? Or
is it an imagined effect? ("We think it should work this way, so it does!")

------
paul9290
ALl links will die someday! Not every website is going to be around forever!

The only ppl who hate on URL shorteners is this crowd here while millions of
others usual them daily!

We are hackers and thus if this is such a big issue and change, one of us
should do something about it. Something that works for the users and solves
the issue about 3% of all users rant about!

------
skwaddar
I have no idea what your beef is, I think you have a stick up your ass about
something.

Why you think you're some sort of web "expert" is beyond me. A real expert
wouldn't be using javascript for this for a start.

Punishing third parties because of some moral crusade is dumb.

Slows down the web hahahah jeesh.

