Hacker News new | comments | show | ask | jobs | submit login
Common Gotchas in Go (deadbeef.me)
229 points by omn1 11 months ago | hide | past | web | favorite | 72 comments



I think only #1 really counts as a gotchya. #3 is literally just "spend at least 2 seconds knowing what a slice is" (which you should do! there's only like 4 data structures in Go built in, so get to know 'em all). #2 is similar.

My favorite gotchya is that assigning a nil pointer to an interface will not give you a nil interface.

Once you know what's going on, it totally makes sense, but usually you don't start out with that deep an understanding of how interfaces work, and it violates natural-looking assumptions about algebraic equality, which makes for very easy counter-intuitive behavior.

Example:

    package main

    import "fmt"

    func main() {
        var x *ErrorX = nil
        var err error = x
        if err == nil {
            fmt.Println("nope, this won't happen ")
        } else {
            fmt.Println("here's my non-nil error:", x)
        }
    }

    type ErrorX struct{}

    func (x *ErrorX) Error() string {
        return "hi i'm ErrorX"
    }
https://play.golang.org/p/BB1b2_HDb6I


Covered in the FAQ, but really I keep forgetting all the time. https://golang.org/doc/faq#nil_error

Under the covers, interfaces are implemented as two elements, a type and a value. The value, called the interface's dynamic value, is an arbitrary concrete value and the type is that of the value. For the int value 3, an interface value contains, schematically, (int, 3).

An interface value is nil only if the inner value and type are both unset, (nil, nil). In particular, a nil interface will always hold a nil type. If we store a nil pointer of type * int inside an interface value, the inner type will be * int regardless of the value of the pointer: (* int, nil). Such an interface value will therefore be non-nil even when the pointer inside is nil.


> Once you know what's going on, it totally makes sense

Anything “makes sense” from an operational point of view, so long as you internalize the rules. But does Go's handling of `nil` denote anything you ever actually want to express, regardless of language?

> it violates natural-looking assumptions about algebraic equality

No, Go's handling of `nil` doesn't violate anything about equality: everything (besides floats, but you can't fault Go for those) is equal to itself and to nothing else. But the syntax (i.e. using a single unadorned token `nil`) does a very poor job of reflecting the language's semantics (the existence fo multiple `nil` values, all different from one another).


I'm sure there are uses for it; in this case, since making nil pointers not work would require extra rules, it doesn't seem worth doing. A suggestion I saw on reddit was to use a different word for "nil" for interfaces ("unset" or something), which would make the gotchya have less bite.

And I wasn't trying to say nil-to-interface violated anything real, just that it violated natural-to-make-but-actually-untrue assumptions (which is I guess pretty much the definition of a gotchya):

If we have

    a = nil
    b = a
it's natural (but incorrect) to assume that we have

    b == nil
Basically, I agree with you that the syntax is being poorly suggestive of the semantics. I like the "unset" suggestion because then we'd have:

    a = nil
    b = a
which wouldn't falsely suggest

    b == unset
Too late now, though.


The design for #1 is good and consistent. There are not reference values in Go. All assignments in Go are value copies.

BTW, in fact, the ", _" in the author's code can be omitted.

     for idx := range zoo {
       zoo[idx].legs = 999
     }


Hypothetically, if the language defined

    for i, x := range xs 
to have x repeatedly be a new reference to each of xs members, it seems like we'd avoid 2 gotchyas (capture of x by a closure inside the for loop not doing what we want, and #1 in the original article).

I think the downside would be the concept of references doesn't exist anywhere else in Go (so the "for ... range" loop would have to be its own thing, rather than explainable as a transformation to simpler code) & maybe also implementation concerns I don't know about.

But if we were willing to pay the complexity cost it would make the language nicer, IMO. (I'm not saying we should be willing to pay the complexity cost though -- a lot of Go's charm is how much the language leaves out.)


> Hypothetically, if the language defined for i, x := range xs to have x repeatedly be a new reference to each of xs members, it seems like we'd avoid 2 gotchyas

I'm not convinced. I believe what you'd end up with is that someone will write a "Common Gotchas in Go" article about how, if you think you are operating on a copy in the loop, you are actually operating on a reference.

Really, it seems very non-obvious to me, why one would be a less surprising behavior than the other.

(The fact that the loop variables are not scoped to the loop body - i.e. closures will share the references - is another issue and that I would pretty unambiguously call a gotcha that should be fixed…)


You think so? It just feels like I basically never want it to be a copy, but you do frequently want to modify the element you're iterating over.

It actually pushes you to design things in ways you might otherwise not. E.g., instead of []X you'll have []*X just so it'll be easier to modify from inside a loop.


> It just feels like I basically never want it to be a copy, but you do frequently want to modify the element you're iterating over.

Feels differently to me :)

> It actually pushes you to design things in ways you might otherwise not. E.g., instead of []X you'll have []* X just so it'll be easier to modify from inside a loop.

Never do that. Instead I use indexes when I actually want to access the element.

(Where I do do that is in maps, but not because of range, but because index-expressions over maps are not addressable)


> Never do that. Instead I use indexes when I actually want to access the element.

Yeah, it's pretty clumsy though.


I agree on all of these. #1 would even count as gotcha when it would the loop would yield a reference instead of the a copy, since lots of code might expect a copy. I think in some other languages (e.g. C#) there was also some confusion and change about that.

There are of course also the usual gotchas when capturing things in a closure (is it by value or by reference?). But this is not Go specific.

You explained the main Go-specific Gotcha in my opinion: The dual-nullability of interface types, and the fact that non-null interface objects might contain null pointers to structs that implement the interface.


> I think in some other languages (e.g. C#) there was also some confusion and change about that.

Nah. They wanted references, they got references. The issues were an oversight related to scope.


I think only #1 really counts as a gotchya. #3 is literally just "spend at least 2 seconds knowing what a slice is"

Most programming language gotchas are literally, "just spend {small time increment} learning about {thing}." Those things build up and get lost/misplaced. Programming "nickels and dimes you to death." Design is about the emergent interaction of myriad details. Programming language design is exactly that as well.


i'd say a "gotchya" is something in a language where there's a subtlety that confounds what you would reasonably expect to happen, after obtaining a casual famiarity with the language.

the slice examples doesn't qualify because it's integral to the fundamental idea of what a slice is. they only way this could confound your expectation is if you'd literally spent zero time learning what a go slice is.

(so i guess it is a gotchya if you see "slice", know python slices, and assume the concepts are the same and don't bother to investigate any further.)

by the way, i agree with you that "Design is about the emergent interaction of myriad details"; by that criteria go is very well-designed...


How do Go programmers deal with the fact that it is hard to reason about whether slices alias?

Go lacks const, so there's no way to prevent your caller from modifying the slice you return, or the callee from modifying the slice you pass it. Also the result of append() may or may not alias its parameter, depending on the allocator's mood that day.

Do Go APIs just defensively copy, or pepper APIs with "do not modify" warnings, or rely on runtime tests? Given the fact that append() may mask aliasing bugs, is there a way to make append() alias maximally?


It's rarely a concern in practice. Public APIs should document if a slice may or may not be modified, for example: `io.Writer` says ` Write must not modify the slice data, even temporarily`, and `bufio.Scanner.Bytes()` says `The underlying array may point to data that will be overwritten by a subsequent call to Scan`.

The `append` builtin may allocate a new array and copy, or reuse the same array, but the result should almost always be assigned back to the original variable. The only time you wouldn't assign it back to the same variable is when you are carefully managing the slices yourself for some specific reason, which should never be exposed via an API.


Do you also rely on function documentation telling you whether it modifies its arguments, “even temporarily”? Or do you just read the source for every function you use, transitively?


It definitely can lead to hard to find bugs. I even made an issue out of one case where it could happen in very unexpected ways (which is now fixed on the go master branch): https://github.com/golang/go/issues/21149

Apart from the issue above I've found that it's usually quite easy to reason about though. Usually when you take a slice you either use it temporarily without modifying or you throw away the original one. In the rare case where you do still need both slices to outlive the same function you just copy one of them manually defensively.


I make defensive copies when I'm worried about it.

Append will reuse the current backing array when it has enough capacity to hold whatever you're appending. So to alias maximally like you suggested, you could try creating all your arrays/slices with lots of extra capacity. That would be pretty expensive, but you could try doing it one-off just to see what happens in your test suite?


On #3 “slicing”, on a related note for Python, remember after a new list is created from slicing, each element/item in the new list is the exact reference to the corresponding element/item in the original list. This is espeically problematic when you try to do things on a mutuable object in either list, but expect the other list remains completely shield from change. You have to do deepcopy (but for immutable object deepcopy has no effect).

Makes sense and probably is a given for most programming languages.


My newest two gotchas:

1. Don't use private attributes for struct types if you ever want to serialize/encode them (exception: Mutex and Channel Attributes). Just had to refactor a whole lot of code just, because I decided I wanted to save the struct to disc. Btw. the type itself can and probably should be private.

2. Don't use Pointers in map key structs. As nesting maps is a pain the simple solution is to use structs as keys. But if you do so please remember not to use any pointer fields within those structs. Again when you encode and decode maps with pointers within the key structs those pointers will bite you.


Re: #1 - You can always make your type implement the marshal/unmarshal method(s) so that it knows how to [de]serialize itself.


For #1, the "internal" package magic name allows you to have "private public" types; types that are public in terms of what reflect thinks about them, but are private in the sense that external packages still can't get at them (which is the operational definition of "private" in Go) so it's safe to have all of their attributes be "public", i.e., start with capital letters.

I often use the "internal" when I have a package with a lot of internal-only marshaling behaviors, so that the godoc for the package isn't made up of 80% internal implementation details and 20% payload.

"As nesting maps is a pain"

Spend a few years in Perl with its autovivication and it'll seem a lot less painful. However, this is just an observation, not an actual recommendation.


The private attributes thing has also but me in the ass...encode/decide functions won’t touch private variables and won’t throw an error if all variables in a struct are private. What kind of issues did you run into with pointers inside of a struct in a map?


In my case it wasn't just that I used pointers which might work in some cases. Instead I used a pointer to a global variable containing a function reference (directly using a function reference was not possible as those are not 'compareable'). Therefore, whenever I started the programm the global variable had a different location and therefore it didn't macht the location it had when the file was saved (during an earlier execution).


Not GP: I'd guess if you briefly forget that pointers are memory values you'll run into some Fun when trying to re-create a struct and use it as a key.


Tl;dr: Understand the difference between values and pointers.


There are lots of languages in which values and pointers are different that do not have these gotchas.


I liked how this was written. The way this was written sort of reminds me of go. It's as simple as approriate, and written in a way to be memorable. Nice UX trick deadbeef!



In fact, `gofmt` can help simply this. `gomft -s` would simply

    for idx, _ := range zoo {
        zoo[idx].legs = 999
    }
to

    for idx := range zoo {
        zoo[idx].legs = 999
    }


A language with a linter, testing, and everything built in? Sign me up!


You may want also to watch a talk by Alan Donovan, co-author of The Go Programming Language [0], on static analysis tools [1].

[0] http://www.gopl.io/

[1] https://talks.golang.org/2014/static-analysis.slide#1


Also check out Elixir. Testing as first-class baked in, data randomizer for tests baked in, formatter baked in, BEAM VM, pattern matching, the list goes on and on and on.


This is one of Go's best features. Static binaries is the other killer feature.


For #2, the "if len(a) == 0" clause is totally non-sense.

I hope Go supports "append(nil, data[:2]...)", which will make code much cleaner.


Not sure what you mean but it seems to work for me

https://play.golang.org/p/U9Uuy-_sdjn


"[]string(nil)" is more verbose than "nil".


Seriously? That's your problem with this construct that otherwise does exactly what you asked for?


It is easy for compilers to deduce the type of the nil in "append(nil, data[:2]...)".


Sure, and it might just in a future version of the spec if there isn't a good reasoning for not doing that.

Doesn't mean that "it's more characters!" isn't still a really silly complaint.


The last gotcha sort of leads to a different gotcha. That copy statement will only work as expected if the destination slice is large enough - thus always initialise it with the length of the (sub)slice you want to copy over. You won't get any errors if it's smaller, but you will surely not get what you desired.


the append way is much efficient than the make+copy way: https://github.com/go101/go-benchmarks/tree/master/append-vs...


Not sure it holds any more, I got these results

  $ go test -bench=.
  goos: darwin
  goarch: amd64
  Benchmark_AllocWithMake-4     	    1000	   1626664 ns/op
  Benchmark_AllocWithAppend-4   	    1000	   1574720 ns/op


I got nearly the same ratio as the github example. (CPU is overclock to ~4 GHz)

  $ cat /proc/cpuinfo | grep 'model name' | uniq
  model name: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
  
  $ go version
  go version go1.9.1 windows/amd64
  
  $ go test -bench . --benchtime 10s
  goos: windows
  goarch: amd64
  Benchmark_AllocWithMake-12    20000  870349 ns/op
  Benchmark_AllocWithAppend-12  30000  561120 ns/op


Please note that the Benchmark_AllocWithMake function even doesn't call "copy(y, x)".


I am surprised that make would be slower than append! What's the explanation?


My Go gotcha: x = append(y, ...) is usually unsafe for x != y.

I want to find/build a linter that catches such cases...


Can you explain this in more detail?

I've never used this construct, but I was unaware of its consequences. You've piqued my interest.


They may be referring to the idea that the result of that operation is defined based on the y's capacity, cap(y). (Capacity is the total amount of space in the slice backing y whereas length is the number of entries used in y; append will not allocate until the new length > capacity)

If the capacity is less than the new length, a new backing array will be be allocated. This results in x != y

However, if the capacity is sufficient to contain the new values then x == y.

Normal usage of append is overwriting the initial slice variable so you don't need to worry about this, x = append(x, ...). If you use two variables, as here, you can potentially have two different slices used as though they were equivalent in later code.


Exactly. The inadvertent aliasing can produce situations like this: https://play.golang.org/p/XUtyQ6ShYaz , or more plausibly: https://play.golang.org/p/Xx8lZWVWgIu


Thanks for this. It gets weirder! The slice grows/doubles if capacity is insufficient. So if you initialize the original slice all the way to capacity (len=4, cap=4), it works correctly. But if len=3 and cap=4, it shows bad behavior. https://play.golang.org/p/qf3lAoin4dM


Thank you, to both of you, for the explanation.


The comparison "x != y" is illegal for slices.


Sure, i meant that x and y are different identifiers.


There was a good post about many gotchas that was on HN a little while ago. Just to share :

http://devs.cloudimmunity.com/gotchas-and-common-mistakes-in...


it also takes some figuring that the first character of anything needs to be uppercase if 1) you try to a function outside of the main source file, and 2) use an element from a struct in a template, i.e. {{.element}} does not work...


The first one could be mitigated by providing an array of article pointers ([]*Article). That way, article.legs = 99 would work.

Else, the way mentioned in the article, or (&article).legs = 99 (ugly) would work. Take your pick.


  (&article).legs = 99
This would not work actually because it would only be modifying the copy that exists in the inner loop of the for loop. You'd be taking a pointer to the for loop stack variable and modifying it.

If they were already references as you first mention that way simply article.legs = 99 would work yes.


Thanks for pointing it out.


Looks like a newbie article. The gotcha that got had me a few times is closure within a loop:

for _, ele := range eles { go func() { // use ele } }

Since ele is a loop variable it is not captured as you might think it is. You need to copy it before the closure.


Or pass it into the func, especially if its running in a goroutine.

    for _, ele := range eles { go func(ele string) { // use ele }(ele) }


Also the other "fun" approach:

  for _, elem := range elems { elem := elem; go func() { /* use elem */ }() }


It's a shame to have code end up like this when Go is supposed to be anti- "stutter".


This might have been done by Go for perf reasons. Why should the Go compiler copy a new variable on the stack every loop iteration?


No, not really, it's a natural and unintended consequence of how the spec scopes variables in loops/switches/conditionals:

https://golang.org/ref/spec#Blocks

The problem you are trying to solve is, that with a statement like

    for i := 0; i < n; i++ {
        doAThing(i)
    }
you want `i` to be valid inside the loop body, in the for-clauses but not outside the for-statement. Go solves this by saying that an if/for/switch statement has an implicit block surrounding them and that block is what scopes the loop variables. AFAIK no one considered, that this would have this consequence in relation to closures.

Performance wouldn't really matter, because compilers tend to be pretty good at optimizing these kinds of things. They already reorder when they check for the condition and how they jump non-intuitively and the naive instruction sequence would involve freeing some stack-space at the end of the loop, reallocating it in the next iteration and then writing the new value to it. Figuring out that you can save the actual stack-pointer operations isn't that hard.


Can the compiler implicitly copy i, basically convert it to:

  for i := 0; i < n; i++ {
    i := i
    doAThing(i)
  }


Sure. It's an easy problem to fix, now that the team is aware it. I'm pretty sure they'll solve it for Go 2. In the meantime, it would break the compatibility promise, so it's just a kludge we have to live with.

I basically just wanted to point out that the problem isn't so much an implementation question (or about performance). But that no one thought of it and it's now mainly a question of how to best phrase the spec for this :)


Yeah the range thing got me too.

At some point I tend to just ignore the second part of the range and only use the index...

and then after that I just switch back to a damn for loop like god intended.


Didn't read the article yet but congrats on the awesome domain name!


I seriously need a browser extension to not render "characters" in color. How the hell did this happen anyway?


I look at stuff like this and wonder why I'm reading it because I'll never use this language.. but i'm glad it's here for people who do!




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: