
Stunned by Go - squidsoup
http://how-bazaar.blogspot.co.nz/2013/07/stunned-by-go.html
======
rsc
Go developer here. I posted this on the blog too.

""" This is a mistake, plain and simple. It was pointed out on the Go
developer mailing list a few days before your post, and you can see my reply
there.

NewRequest should take an io.ReadCloser. Unfortunately, due to backwards
compatibility, a long standing semantic mistake like this is not something we
can just fix, but we've at least documented it.

The tweet you quoted is wrong, and I want to explain why. If an interface
value v is passed to a function f, then what does happen from time to time is
that f will look for custom methods on v that are at least logically
equivalent to the static interface type f is declared to expect. For example,
io.Copy takes an io.Reader and an io.Writer and does the obvious thing,
Reading into a buffer and then passing that buffer to Write. Of course, it
would be nice to bypass the buffer when possible, and so there is a standard
WriterTo interface that an io.Reader can also implement that, in contrast to
Read, which copies into a buffer, says "read the data out of yourself and
directly into this Writer". If io.Copy finds that method, it will use it, but
the operation - reading - is the same. There is an analogous case for the
Writer too. If you pass a value v to json.Marshal, Marshal will check to see
if v implements json.Marshaler, meaning v knows how to marshal itself as JSON,
in which case v is given that opportunity. Again, same operation you
requested.

In contrast, if you have an io.Reader, while it might be okay to look for
other Read-like methods, it is certainly not okay to look for Close. That's a
very different operation, it's surprising, and it shouldn't be there. FWIW,
I'm not aware of any other instances of this kind of semantic mixup in the
standard library. But again, unfortunately, we can't change it until Go 2. All
we can do for now is document it and move on. """

~~~
manojlds
I know of atleast one instance in C# where something like this happens. The
IEnumberable<T>.Count() extension sees if the input is actually a ICollection
and uses the count on that to get the count directly, rather than iterating
over it.

I remember writing extension methods in C# for IEnumerable<T>, which would of
course take in an IEnumerable<T>, but saw if the actual input was a
ICollection<T>, IList<T> etc to optimize the operations.

~~~
benhoyt
I'm not a heavy C# dev, but I don't understand why your first paragraph would
be so: isn't that the point of implementing an interface like IEnumerable<T>
\-- so that you can implement Count() in an optimized, specialized way that
"gets the count directly"?

~~~
rat87
(I'm not a c# expert and I may be wrong) IEnumerable really only requires a

IEnumerator GetEnumerator()

method most of the real "meat" of ienumerable is in the static
System.Linq.Enumerable class. You see c# doesn't have multiple inheritence or
scala like traits or type classes but it does have a cool compiler hack called
Extension methods([http://msdn.microsoft.com/en-
us/library/vstudio/bb383977.asp...](http://msdn.microsoft.com/en-
us/library/vstudio/bb383977.aspx)). You take a static class with static helper
functions like you might make in java and you add a this keyword to the first
argument then you can call it with a syntax that makes it look like it was a
method on that class. For example something like

    
    
        public static class Enumerable
        {
            public static int Count<TSource>(this IEnumerable<TSource> source
            {
                int i=0;
                for(TSource t in source)
                {
                    i++;
                }
                return i;
            )
        }
    

And if you use Extensions methods on an interface type you get all of the
extension methods _for free_ once you implement the minimal interface. One of
the downsides is that you can only really define it in one place and can't
override it and if you want even somewhat efficient type specialized versions
you need to cast.

I believe in scala you can use traits in a similar manner for default
implementation and specialize them for collections but traits are more
complicated/powerfull feature.

~~~
pjmlp
This is not a compiler hack, there are quite a few languages out there that
support this concept.

As for your Scala remark, extension methods in Scala are done via implicits.

~~~
rat87
I didn't mean hack in a derogatory way, I think its a fairly elegant solution
in many ways just that to my knowledge it doesn't really change the
static/dynamic type and is just a simple compiler transformation.

I should have mentioned that I know even less about scala then c#. I/m not
quite sure how implicits are implemented/optimized but I'm pretty sure you can
have inheritence/overiding traits as implicits.

------
tikhonj
Wow, this is pretty unfortunate. Also a very good example of why so many
people maintain that Go is not nearly as statically typed as languages like
Haskell or OCaml: casting (essentially subverting the type system) is very
common, and limits most of the type system's advantages.

In large part this is because Go's type system is not very expressive. I would
just like to point out that OCaml did the same things as Go (i.e. structural
sub-typing). But with (global) type inference. And variants (including
structural subtyping for those too). And generics (parametric polymorphism).
Years before Go was developed.

In OCaml, this sort of thing probably would be _much_ less likely to come up.
After having used the OCaml object system a bit (but still more than most
people!), I am actually very happy with it. It certainly has issues--oh boy,
does it ever--but the issues are superficial. They are the result of very
limited manpower in the project, not fundamental shortcomings of the design.

In OCaml, the compiler can figure out what methods you used without your
telling it. So it would automatically realize that the function called the
close method, and infer the appropriate type. You wouldn't even have to write
an interface (called a class type in OCaml) for it. If you _did_ write an
interface, but it did not expose a close method, the code would give you a
type error.

I suppose you _could_ try to do some sort of runtime casting and duck-typing,
but I am not sure how. It would certainly not be a common thing to do, at all!
Instead, the type would reflect exactly how the object is used in the
function. It would be more explicit, more predictable and safer.

And, harping back on the same theme, it would be fully inferred for you.

The best of both worlds, really.

~~~
epidemian
Thanks for the informative comment.

> So it would automatically realize that the function called the close method,
> and infer the appropriate type.

So, if i understand this correctly, OCaml would infer that the function takes
"an object with write() and close() methods" because it uses both methods in
its definition. That's reasonable. But wouldn't that prevent it from accepting
objects that don't have a close() method?

Wouldn't the proper type for the parameter of this http library be something
like "an object with a write() method or (an object with write() and close()
methods)" (i should have used another notation for this hehe)?

I think the biggest problem in this case is no so much the weakness of Go's
type system, but the design decision of mixing two different behaviours (that
require two different type signatures) in the same method in the http library.
It seems there should be one method that takes a Reader and only calls Read()
and another method that takes a ReaderCloser and calls both Read() and
Close(), it can even delegate to the first one (assuming, of course, that Go
accepts Reader as a valid subtype of ReaderCloser (i would hope so!)).

~~~
andolanra
You're right; OCaml would have to supply two different functions, one that
takes an object that exposes _< close : (); write : string -> (); .. >_ and
another that takes _< write : string -> (); .. >_. Reproducing the Go system
would require the ability to, at runtime, see whether an object of type _<
write : string -> (); .. >_ can be downcasted to _< close : (); write : string
-> (); .. >_, which would require more runtime type information than OCaml
currently supplies. Some people have made proposals to implement such
downcasting (throwing an exception if the downcast fails), which would permit
the Go-like implementation, but as far as I know, such proposals haven't been
met with much enthusiasm.

~~~
epidemian
Thanks for the examples of record types in OCaml.

> but as far as I know, such proposals haven't been met with much enthusiasm.

Well, if the proposed use cases are similar to this http Go library, then i
think i can see why that's the case. As just adding a _close: ()_ method to a
type could mean that code that uses that type now behaves differently.

------
tptacek
The Golang team (or at least, Russ Cox) agrees with you:

[https://groups.google.com/d/msg/golang-
dev/o9i_SbGRrzI/SSuqW...](https://groups.google.com/d/msg/golang-
dev/o9i_SbGRrzI/SSuqWmdQxpIJ)

------
BarkMore
This issue was recently discussed on the development mail list:
[https://groups.google.com/d/msg/golang-
dev/o9i_SbGRrzI/gWvnB...](https://groups.google.com/d/msg/golang-
dev/o9i_SbGRrzI/gWvnBnORqRYJ)

The fix is to document that Close is called:
[https://codereview.appspot.com/11432044/](https://codereview.appspot.com/11432044/)

~~~
tux21b
Yes, Roger Peppe, one of the Go contributors, had exactly the same issue, but
luckily Brad and Russ, two core devs, were able to solve the mystery
immediately and updated the documentation.

It's now quite funny to read a blog post from someone completely different
(Tim Penhey) who had encountered and solved the same issue with exactly the
same usecase on the very same day, complaining about Go without even
mentioning the discussion [1] or the latest changeset [2] that solved the
problem.

[1]: [https://groups.google.com/d/topic/golang-
dev/o9i_SbGRrzI/dis...](https://groups.google.com/d/topic/golang-
dev/o9i_SbGRrzI/discussion) [2]:
[https://codereview.appspot.com/11432044](https://codereview.appspot.com/11432044)

~~~
thirsteh
That is pretty funny, but it doesn't really change anything. It's unfortunate
that it called Close() without at least giving proper notice, and most people
will probably agree that in a perfect world where we could start over without
breaking APIs, the implementation would be different.

~~~
tux21b
I was just criticizing the publishing skills of some people... The problem
itself is unfortunate and I am personally not a huge fan of the "no API
changes" philosophy participated by the Go team. I can understand the reasons
behind it, but it's still hard to read several CLs per day that are abandoned
or done differently (in a non-optimal way) because of this contract.

~~~
enneff
It's much better for the core team to compromise a little than force everyone
to update their code all the time.

In particular, small semantic changes that can't be caught statically will
just break people's code with no warning whatsoever. That's bad.

~~~
bluehex
The end user gets to decide when they upgrade the toolchain so it wouldn't be
"no warning" if they wrote about the change in the release notes. I'd much
rather they break code relying on undocumented weird behavior, than document
the weird behavior and keep it alive indefinitely.

~~~
enneff
If we let small semantic changes like this in, it would be too onerous to
upgrade, and so nobody would upgrade. Go users should not have to read and
understand all these tiny changes, look through their code to see if anything
breaks, and finally make the correct fix.

It's just not tenable, and we know this from experience (before Go 1 we
introduced breaking changes with each stable release).

FWIW, this is the first time we've encountered this bug, and it's been part of
Go since it was released. Hardly worth breaking people's code for.

------
georgemcbay
This is caused by an unfortunate bit of code in the standard library, but is
hardly representative of a fundamental problem in Go as the article implies
(unless you view casting and interface coercion as a fundamental problem, and
I know some people do, but that's basically a different debate since Go does
not pretend to be a purely statically typed language by any stretch of the
imagination).

I've been using Go regularly for nearly two years now and haven't run into an
issue quite like this either with the http package (what he's doing that
causes this to be a problem is legal and should work, but is not a common use
case) nor with any other bit of the standard library.

~~~
thirsteh
It depends on what your definition of "fundamental problem" is. If being able
to treat a value differently than it was received, or to perform computations
that you shouldn't be able to, e.g. impure actions inside a pure function,
then very few languages satisfy that property. Even Haskell isn't safe because
of unsafePerformIO--you'd need Safe Haskell, at the very least.

Go has never tried to be, and the authors have never tried to give people the
impression that it is that kind of language. Some of the safety comes from the
language e.g. doing array bounds checking, not letting you manipulate memory
manually, etc., but the rest comes from discipline and idiomatic code. (Same
thing as not using 'undefined', 'fromJust', 'unsafePerformIO', etc. in Haskell
unless you're absolutely sure they won't do harm--they are too useful to
remove, even if they are the sources of many headaches.)

------
bicknergseng
I feel like the author missed a great opportunity. "Stopped by Go"

~~~
gfodor
Stop, Go

~~~
f2f
or, in "explosive tabloid headline style":

Go: Stop!

------
codezero
This is pretty interesting, but I wonder if it's just a one-off problem with
the http library.

It'll be interesting to see if someone familiar with Go has a response to
this, but from the blog post it doesn't appear that this contract is being
habitually violated, only that it can be, which is still a problem in
practice.

~~~
frou_dh
It's a bit insidious since even if that situation is rare, just knowing that
it's possible introduces a new dimension of doubt next time things go awry
with any code you didn't write. Enter a _" Seriously, No Groping"_ type
modifier ;)

------
marshray
> The standard http library was calling Close on my io.Reader. This is not
> expected behaviour when the interface clearly just takes an io.Reader (which
> exposes one and only one method Read).

So if I implemented my own object with a Reader interface and passed it to
this HTTP library, could it find some other method on my object named Close()
and it could call it?

Is it pure duck typing happening here, or is the HTTP library looking for a
commonly-defined interface with documented semantics for Close()?

~~~
usea
In Go, a type doesn't have to declare that it implements an interface; it
automatically satisfies an interface if it implements the appropriate methods.
Most interfaces in Go only contain one method.

If a type has a Close() method, then it is a Closer. The code that tries to
cast it to a Closer will succeed. So to answer your question, it doesn't
matter what the semantics of your Close() method are, it will succeed solely
based on the fact that your type has a method with that name.

~~~
melvinmt
> your type has a method with that name.

I would say with the same function signature, not just the name.

~~~
usea
Go doesn't support overloading. Method dispatch is done by name only.

[http://golang.org/doc/faq#overloading](http://golang.org/doc/faq#overloading)

~~~
dragonwriter
But interface satisfaction is by signature, not name, and that is what is
being discussed, not method dispatch.

~~~
usea
Sorry, I didn't mean to conflate the two. Thanks for the clarification.

------
GhotiFish
Even if there is agreement that this is a problem, it strikes me that it can
_become_ a problem.

The standard library should be exemplary, what do you think would happen in
the hands of a developer who's "not as smart as he thinks he is?"

Wasn't that part of go's pitch? I never really considered that libraries would
go so far as to inspect the types for methods rather than just make those
methods a requirement. The fact that they are able means someone else is going
to use it, and it's going to bite me. Don't like.

~~~
nknighthb
What was part of Go's pitch, exactly? And where did you see it pitched?

People outside the Go team have tried to pitch Go as a lot of things, most of
them having no relationship to what Go was actually intended to accomplish.

The FAQ[1] may be enlightening. Nowhere does it say Go is supposed to be a
straightjacket that makes bad programmers into good ones.

[1]
[http://golang.org/doc/faq#What_is_the_purpose_of_the_project](http://golang.org/doc/faq#What_is_the_purpose_of_the_project)

~~~
GhotiFish
You'll have to forgive me, I'm at work right now so I can't pull up the talk
Rob Pike did on Go, I think it was 2011.

Around the point where he explained Go was not a language to expand the
boundaries of computer science. It's for programmers who "often are not as
smart as they think they are"

I'll look it up when I get the chance. That might be later tonight.

~~~
nknighthb
I'm reasonably confident you're assigning too much significance to what was
probably an offhand remark in a particular context. I just looked at slides
from a 2010 talk[1] by Pike which was pretty explicit about what they were
doing in this area:

"It's a form of duck typing, but (usually) checkable at compile time. It's
also another form of orthogonality."

And this one from Russ Cox in 2010[2]:

"Compiled, statically-typed languages (C, C++, Java) require too much typing
and too much typing:

* verbose, lots of repetition

* too much focus on type hierarchy

* types get in the way as much as they help

* compiles take far too long"

A common theme for Go from the beginning has been the balance between type
safety and not getting in the way of the programmer. The analogy to duck
typing has been a consistent observation throughout Go's public existence.
This is incompatible with the sentiment you took from whatever talk you saw.

Go's architects didn't set out to make a Java clone. It's more like Java's
evil twin.

[1]
[http://talks.golang.org/2010/ExpressivenessOfGo-2010.pdf](http://talks.golang.org/2010/ExpressivenessOfGo-2010.pdf)

[2]
[http://talks.golang.org/2010/go_talk-20100112.html](http://talks.golang.org/2010/go_talk-20100112.html)

------
bsaul
Anyone proficient in GO would care to provide the minimal code that illustrate
that extremely weird behavior ? I find that post so disturbing that i still
believe i've missed something.

~~~
thirsteh
An example:

[http://golang.org/src/pkg/io/io.go?s=11741:11801#L336](http://golang.org/src/pkg/io/io.go?s=11741:11801#L336)

Line 336. io.Copy takes a writer and a reader, but actually asserts whether
they implement ReaderFrom/WriterTo, then use their methods, ReadFrom and
WriteTo, if they do. (Same thing with the http package checking if the reader
implements Closer, and calling Close if it does, though admittedly 'we should
be able to call Close' is a much more unsafe assumption than 'if it implements
ReadFrom, and it has the right type signature, we can just call that function
instead'.)

The reason it doesn't take values implementing the interfaces ReaderFrom and
WriterTo is that io.Copy works even if you don't implement those interfaces.

It's an unfortunate and regrettable predicament, but it's not something that
comes up a lot. As others have mentioned in this thread, Go certainly isn't a
language with referential transparency by any means--the type system provides
minimal sanity checking, and the language itself is memory-safe, but it's not
trying to be a "high-assurance language."

~~~
MetaCosm
This io.Copy behavior seems like an "optimization"... less explicit than I
would like but sticks to the idea of an io.Copy.

Having a resource you own Closed on you isn't an optimization, it is a
behavior change.

~~~
thirsteh
I agree. I think I edited my comment to specify that it's a much more unsafe
assumption just before you posted your comment.

------
rjempson
I am just wondering whether you can write your own wrapper class that
implements (exposes) the read method and doesn't expose a close method (or
expose a close method that does nothing) and pass that in?

I am not suggesting this is a good solution, just curious.

~~~
enneff
Yes, easily:

    
    
        var rc io.ReadCloser // a file, whatever
        readerOnly := struct{io.Reader}{rc} // only a Reader

~~~
frou_dh
That's cool. The io.Reader is an unnamed field, so its method is effectively
pushed to the surface of the wrapping struct instead of needing to be
explicitly forwarded to.

------
pedromorgan
Why dont the Go team do golang2 now.. There are a few issues.. already
mentioned in open fire chat this year 2013.. Would be nicer to actually
recongise flaws now.. and offer alternative.. even faster tht python 3000
could ?

Thats whats in my mind.. and fix the the documentation so the classes
interlink and it a bit of a java/md/rst/doc style..

~~~
dragonwriter
> Why dont the Go team do golang2 now.. There are a few issues..

I suspect that it is because Go 1 (issues and all) is useful enough that
people are making productive use of it, and they'd like to flush out more of
the issues that may exist -- and give more time for different approaches to
those issues to be tried in experimental forks and, where possible, libraries,
preprocessers, etc.

> and fix the the documentation so the classes interlink

Not sure what this means, and Go doesn't have classes. (Its not just an issue
of calling classes different names, it divides up functionality differently
than class-based OO systems -- packages are in a sense like classes [they are
the closest equivalent to the function of classes as containers of static
methods], structs are a bit like classes [in the role of containing members,
and in that anonymous embedding approximates inheritance], every type is a bit
like classes [in that any type can have methods attached to it, so long as
they are attached in the same package as the type is defined]; but all these
things are also unlike classes in various ways, too.)

------
f2f
for those not familiar with Go, here's the problem translated in C terms: "I
passed a pointer to your function and it called free() on it. how dare you!"

~~~
nostrademons
More "I passed a pointer to your function and it called free() on it, _without
documenting that it was taking ownership of the memory_ ". Ownership contracts
are an important part of the API of any function that does stuff with
resources, whether memory, file descriptions, network connections, etc. If the
API implies that it takes a pointer to a resource but will only read from or
write to it, and then disposes of it, that's a bug.

~~~
marshray
I think that's a good point. It's often said that garbage collection helps
prevent memory leaks, and it's also said often that memory isn't the only kind
of resource which could be leaked.

Here is another case where weak- and dynamic-typing language features have
come up in the context of resource management.

I'm not saying this one example proves anything, just that it might be
interesting to be on the lookout for other events of this pattern.

------
xinn
This behavior is explicitly defined in the godoc for io. I don’t see why the
author should be "stunned" by a function doing exactly what it says it does.

~~~
nwp90
That'll be because it's only been in the docs since they reported their issue.

~~~
thirsteh
You're both correct. It's been in the docs for io.Copy for a long time, and
was only recently added to the http docs.

------
Groxx
So what's the fix for this? Wrap the object in another that only defines Read?
Or is there another way?

------
cmccabe
Supporting typecasting is hardly earth-shattering. Pretty much every
statically typed programming language in actual use supports typecasts. That
includes Scala, Java, C++, C, C#, Pascal, Algol, and even Ada.

This title should be "stunned by shitty poorly documented library function"
not "stunned by Go."

There are a few languages out there that don't support typecasting. I think
SML was one, for example. But they made a lot of other design choices to be
able to support that choice, and so far that set of choices has not gained
popularity. At minimum, you would need a fairly complex type system with
generics to be able to completely forbid typecasting. You also run into
problems where stable APIs completely block progress in a particular area.

~~~
pcwalton
> There are a few languages out there that don't support typecasting. I think
> SML was one, for example.

Haskell too, unless you explicitly opt in with Data.Typeable.

It's arguably true in a certainly commonly-used subset of C++ as well: this
type of typecast doesn't work in C++ unless you use dynamic_cast or another
RTTI system (like COM). dynamic_cast only works on classes with a vtable. Many
C++ projects don't use any form of RTTI. (Note that the Go code does not
assert that the Reader is a ReadClose; rather it depends on RTTI to determine
whether the reader is a ReadClose.)

~~~
nly
dynamic_cast downcasts in C++ only work across public inheritance. In
particular a object that inheritance a Closable interface privately and a
Reader interface publicly wouldn't allow a cross cast.

dynamic_cast isn't a loophole in the type system, it's there to strengthen it.

~~~
marshray
I don't think I've ever seen C++ that inherited an interface privately.

That said, it's extremely common for classes to lack any virtual functions and
thus be unusable with dynamic_cast.

~~~
nly
Well protected is the default.

~~~
pjmlp
The default is _private_ for classes and _public_ for structs.

Section 11.2 of the C++ standard.

------
MatthewPhillips
Go didn't fail you, OOP failed you.

~~~
marshray
There's nothing about OOP that requires weak typing.

~~~
MatthewPhillips
I was referring to the fact that this function mutates the state of one of its
arguments.

~~~
marshray
But so does Read(), right?

Even in Haskell I'd be calling the HTTP thingy in the context of the IO monad.

------
tantalor
<tldr>App author found a nasty example of a common antipattern in library code
which caused a bug in his app. Documentation was updated to point out dragons.

