Hacker News new | past | comments | ask | show | jobs | submit login
Avoiding Reflection (And Such) in Go (jerf.org)
116 points by ngrilly on Dec 5, 2015 | hide | past | web | favorite | 36 comments



Maybe I am misreading, but the Little Types section I think recommends an ugly habit, especially for libraries. When calling this "func Set(ID, Width, Height, Color)", assuming I acquire integers through user input, this means my code would then look like - extLibrary.Set(extLibrary.ID(id), extLibrary.Width(width), extLibrary.Height(height), extLibrary.Color(color)), which frankly is a nightmare. I'd expect the function to take 4 integers, then convert internally if needed.

Further, it makes Godoc more confusing, since now I have to search around to figure out what each of these custom types is.


As a general rule, convert incoming input (user or otherwise) to the correct type as quickly as possible, and convert it to the outgoing type as late as possible. When programming in this style, for you to reach the point in the code where the Set call is being made and for you to still have 4 "int"s almost certainly means you should have converted them earlier. Generally you've got a validation phase somewhere, which is where this should be converted, which means that your "Width" isn't just "what happended to be in the input" but is a "validated Width" as well. Since it floats on the validation code it's often not even 1 extra line of code, just another token that I mentally model as sort of "sealing" the fact that this is a good value.

This rule goes way beyond "tiny types"; for instance you will drive yourself insane in the web world if you don't do this, with double-decodings and double-encodings quite challenging to prevent with any other policy. [1]

"Further, it makes Godoc more confusing, since now I have to search around to figure out what each of these custom types is."

Absolutely yes. In any language where this is possible and easy, I do wish the auto-doc systems either automatically, or with some metadata (preferably a label on the type somehow), would make it clear when something is just a one-layer wrapper around a simple type in all the places where it appears in the docs, without having to click through. I consider it worth the cost, but this is definitely an entry on the cost column that is accidental, not essential. This is especially true when the base type is numeric, which means that Go will let you pass in a literal number in the source code, but that is not always your first expectation.

[1]: In a statically-typed language you can also play games with having the type label what "layer" of decoding the value is currently in, which works but gets more tedious. This is theoretically possible in a dynamic language, but goes against the grain, since generally people program with just 'strings' in those languages.


Hi, I enjoyed your article.

You have some good points about tiny types, but if you read through the code in the Go standard library, you won't find this style applied in very many cases. Typically, only if one wishes to attach a method to a built-in.

As another poster rightly pointed out, it complicates the documentation quite a bit, and I think a lot of Go programmers would consider it pointless abstraction.

    // NewBox creates a new Box. It returns an error if length
    // or width are less than or equal to zero.
    func NewBox(length, width float64) (Box, error) {
        ...
    }
Consider the simplicitly of NewBox. We may omit a Length and Width type, and any validation of these by stating the expectations of the inputs in the documentation. So, the API is dramatically simpler. The expectation is that the programmer will not pass invalid arguments to NewBox, by way of common sense or reading the documentation. Programmers who pass in positive numeric literals may even opt to ignore the error returned, since the documentation states the the only reason an error would be returned is if the input parameters were <=0.

Go puts a big emphasis on documentation, and this often compensates for lack of power in its type system.


To me the perfect example of when you want to use tiny-types is when dealing with values in different units, say, temperatures in Celsius and F. They're both floats, but mean very different things. The extra typecast you have to do before calling a function with a float is a good sanity check.


> Go puts a big emphasis on documentation, and this often compensates for lack of power in its type system.

This could be reworded to:

> Any language X that puts a big emphasis on documentation compensates for lack of power in its Y.

I disagree this is true for those languages including Go.


In this example, Set() is in the public API, so it is exposed to the end user. I am interested both in your approach and in axaxs's concerns, and I'm wondering if there is resolution to be had.


Using little types is fine, but I prefer to not use them nor do I recommend them. I have dealt with several large C/C++ codebases which used them extensively. It often took too much time to find the declaration to see the real type. Then I would forget and repeat the process again and again and again. Overall they wasted more time than they saved.

Some of the fancy IDEs help with this problem, but not everyone likes using them so you can't count on that as a solution.


I suppose it depends on what you're using the tiny types for. For example, if you have a unique ID type, you might have a function called nextId(), and it gives you a new UniqueId. But it really doesn't matter what the "real" type is. It could be an int, or maybe a UUID, or maybe generated string.


It could matter, depends what you are doing. But yes your larger point that they can be useful sometimes is true. I would generally recommend that tiny types should be used sparingly when there is a good reason rather than most or all of the time like the OP seemed to be suggesting.


This is an excellent, well researched and well written article, that shows a lot of great things can be programmed in Go, despite the apparent simplicity of the language, while staying honest about the limits of the language for some use cases.


"despite the apparent simplicity of the language"

Simplicity is not a weakness or failing of a programming language, it is something that every language designer should strive for. To imply otherwise is to say (non-essential) complexity is a good thing.


I think the contrast drawn isn't with a simple language, but with a too-simple one. As Einstein was paraphrased, "Everything should be made as simple as possible, but no simpler."


Agreed. I feel like I'm never quite sure where Go falls on this line.


Where Go falls on this line depends on your conception of a programming language. If you look at a language as static definition that doesn't change with time, it would be easy to see Go as too simple. But looking at a language as an organic thing that grows over time, I think Go's simplicity and resistance to change generally gets it about right. To many modern languages suffer a complexity problem from years of accrued features.


I fully agree. Bad choice of words on my part.


I'm going to get stabbed in a dark alleyway for this, but I would argue that most Go projects don't benefit from being perfectly go-ish.

It's a small, simple, language that is terrifically pragmatic. Write code that's clear, self-documenting, and works. Then go back and fix up performance, convention, and style, as needed.

I wonder how many thousands of hours has been spent talking back and forth over an ultimately trivial part of Go.

......

Solid articlie though. As someone who writes a lot of Go, is sick of hearing about Go, I enjoyed it.


Excellent explanation of composition over inheritance in Go. As someone new to Go, this was invaluable.


> Because composition doesn't intermingle objects the way inheritance does, it allows much more interesting "chains" of objects to be created that extend each other, without the complexity exploding.

Go's (really Plan 9 C's) composition support does "intermingle" methods, which has pretty much the same effect as intermingling fields. To see this, note that Go needs rules about who wins if methods for an outer object and an inner object have the same name (and they have to consider diamonds, which you can have with composition!)

> So, "Go doesn't have generics" is only half true.

I know this is going to be me annoying about terminology, but I really disagree. :) Go has polymorphism. It doesn't have generics, at all. "Generics" has a specific meaning: the ability to take types as parameters.

> It can be profitably argued that the primary virtue of "generics" is generic algorithms rather than generic types.

I don't agree. Generic types are necessary, because you often can't express generic algorithms naturally without generic types. Just as an example of where I needed generics in my work most recently, I wrote a polygon clipper over the past couple of weeks that has special fast paths when it clips rectangles and falls back to Sutherland-Hodgman clipping for complex polygons. This was natural to express by making the clipping result generic over the type of polygon (rect or edge list). I could maybe have done it in Go by using the "container-as-interface" trick that the sort package uses, but that would introduce a lot of boilerplate.

There's also the issue that all these interfaces make it awfully hard for the optimizer to produce as good code as it would if you actually had generics. Unless you inline everything away (in which case SROA and constant propagation can mostly get you there), you'd need a nontrivial interprocedural analysis. Performance issues around generic programming tend to have outsized effects, because functions that tend to be generic are precisely those functions that tend to be called a lot (because they're generic!) and shoot to the top of the profile.

> As a side note, if Go ever does grow full generics, I expect it to do so by extending the half it has, rather than adding a new concept of generics on the side. It's easy to imagine a generic BalancedBinaryTree that requires its nodes to conform to a LessThan interface.

This I agree on entirely. Note that this is pretty much exactly what Swift did :)

> If you have a constrained set of messages, for instance, you can use the type system to more carefully declare them as coming from a particular set of types, not just interface{}.

But you lose half of the point of sum types, in that the compiler can check exhaustiveness for you. It's worse, because since you're doing a Go type switch, you can match against cases that aren't even in the sum type and the compiler won't complain! It's also an awful lot of boilerplate to create a half-approximation-of-a-sum-type, meaning that the ergonomics favor people not using it. interface{} is just plain easier to use compared to the sum type pattern, which is why it gets so much use.

> Go still manages to cover so much with its meager feature set. There's a lesson here for language nerds. I don't fully know what it is yet, but I'm working on it.

I think the lesson is "people will figure out design patterns to approximate things that your language doesn't support". This is no surprise: people write OO systems in the C preprocessor. But I think it would be a mistake to conclude "omit useful features from the language if there's any way people can possibly approximate them". This was the approach that Java famously took, and people rightfully soured on the extreme emphasis on "design patterns". And what you've described in this post is nothing more than a set of design patterns. Design patterns have their place and are very useful, but they're also the sincerest form of feature request.


> It doesn't have generics, at all.

The provided tooling provides templating, which isn't dissimilar to where many other languages stop in their generics journey. The interface does leave a lot to be desired, granted.

      func max(a, b T) T {
      		if a > b {
      			return a
      		}
      		return b
      	}

      $ gofmt -r 'T -> int' -w max.go


No, that's just a very basic implementation of macros. Even C can implement max that way.

What distinguishes templates from macros is that templates deeply interact with and are instiantiated at the time of typechecking. Macros, on the other hand, are expanded at parse time (or, in more sophisticated implementations, at name resolution time).


Okay, macros. Either way, it provides what most people are asking for in generics (not you, perhaps), even if the implementation isn't exactly pleasant. From a practical standpoint, there really isn't a whole lot of difference in utilizing max (other than the ugliness):

    # Common pattern in other languages
    max<int>(1, 2)

    # Similar pattern in Go
    #go:generate gofmt -r 'T -> int' -w max.go
    max(1, 2)
But yes, "true" generics are much more complicated (and arguably impossible in Go without major changes to the language), I realize. A number of languages that even claim to have generics fall short on that front, which is a path the Go authors have said they do not want to go down.

Although you do have me curious how you'd implement max in C in this type-safe way using standard tooling:

    #define max(a, b) (a > b ? a : b)
...obviously wouldn't fit the bill since it doesn't enforce types like to Go version does.


> Either way, it provides what most people are asking for in generics (not you, perhaps), even if the implementation isn't exactly pleasant.

No, it doesn't. Not even close.

Even something like generic List<T> and sort<T>(List<T>) (separately compiled) will fall down because that simplistic scheme doesn't give each generic type and function a different name, and there's no way for sort<T> to look up whatever name List<T> got instantiated into.

There are also all sorts of hairy issues you will get into when you have name resolution and separate compilation: how do you make sure the macro got instantiated into the same lexical environment that it got declared in? This matters a lot when you have package A exporting a bunch of generic types and functions for package B to use, and it's going to be B doing the expansion using B's types. Even if you set up things just right to resolve the names in the proper lexical scope, how can B's expansion of A's templates use private symbols from A?

Finally, you have the dreaded C++ template error message problems, except they're going to be even worse with this scheme because the compiler is going to complain about code post-textual-replacement, with no way to interpret the chain of events that led to the error. At least clang and GCC have lots of semantic information that they can use, since templates are expanded during semantic analysis.


   #define defmax(name, type) type name(type a, type b) { return a > b ? a : b; }
This seems to be exactly how you would define a "generic" max function in Go with the current tooling, too. Which is a shame because C macros just aren't good enough.


Good example. Thanks.

> Which is a shame because C macros just aren't good enough.

No disagreement here. However, that still perfectly satisfies what may people claim they want in generics, especially those who most frequently trumpet that Go is lacking them. If we took that exact macro and added a little hypothetical syntactical sugar:

    template<type> type max(type a, type b) {
        return a > b ? a : b;
    }

    max<int>(1, 2);
a lot of people (not everyone) would be very happy, even though there is no theoretical difference at all. I've seen countless proposals for Go generics that do nothing more than that, but have been rejected for obvious reasons. That's where I'm coming from.


There's a huge theoretical difference. To name one obvious example, the template is hygienic with respect to name collisions, while the macro isn't.


I think you may be reading too much into the hypothetical implementation. It was highlighted as nothing more than syntax sugar, after all. If all you do is translate:

    template<type> max(...) into #define defmax(type) ...
and

    max<int>(...) into defmax(int); max(...)
  
The resultant code will be identical once it has gone through the preprocessor. All you are gaining is prettier looking code, which shouldn't be discounted, but generics are about more than that. Yet this example would be quite satisfactory to many regardless; curiously even you seemed excited about it.


> The resultant code will be identical once it has gone through the preprocessor.

Well, sure. Hand-written assembly is also identical to compiled code, but we still have compilers :)

> All you are gaining is prettier looking code, which shouldn't be discounted, but generics are about more than that.

Yes.

> Yet this example would be quite satisfactory to many regardless; curiously even you seemed excited about it.

I don't think macros are a satisfactory replacement for generics, no. I don't know how I seemed "excited" about it.


> Well, sure. Hand-written assembly is also identical to compiled code

I mean when you convert the template<type> statement to the #define statement with the hypothetical preprocessor, it would be the same as if you had written the same code using #defines originally. They are functionally interchangable. If this satisfies generics in your mind (I don't think this describes you, for what it is worth), then standard C macros are more than sufficient to cover all the cases you would want, albeit less pleasant to type.

> I don't think macros are a satisfactory replacement for generics, no. I don't know how I seemed "excited" about it.

Consider your previous post misinterpreted then. It read to me like you thought the hypothetical code was theoretically different and solved problems with the previously provided macro, even though it was the exact same macro in both cases. The only difference was the slightly different syntax. Which, I might add, anyone can fairly easily add something similar to their Go code if they really wanted to.


"Go's (really Plan 9 C's) composition support does "intermingle" methods,"

Just for clarity, what I mean is that when those methods are called, the get the specialized type, as opposed to multiple inheritance where the two base classes can end up calling into each other in a mutually-derived child because the methods still receive an object of the full derived type. You certainly can end up with conflicting method names. I have to admit I don't dock Go much on that, because as we all know, there are two hard problems in computer science, caching and naming things.

"I don't agree. Generic types are necessary, because you often can't express generic algorithms naturally without generic types."

I'm pretty sure I basically said that latter part.

"...polygon clipper..."

I didn't repeat it in this post, but in other places I actively volunteer my belief that anything that involves "doing lots of math" (in the full sense of the term, rather than "a lot of arithmetic") is a bad place to use Go. But I'd observe that every time someone reaches for a reason why they really badly needed generics, it always seems to involve math. I expect this means something.

(It's OK, you don't need to go searching for one that doesn't. I fully believe they exist, and I occasionally encounter them.)

"I think the lesson is "people will figure out design patterns to approximate things that your language doesn't support"... And what you've described in this post is nothing more than a set of design patterns."

I'm not sure that's supported by the text very well. The only thing in there that probably qualifies as a "design pattern" is the sum type approximation; everything else is too small. The only "design pattern" I find myself frequently reusing in Go is external API + "sum type" protocol + goroutine server receiving that protocol. (The boilerplate in that case is the external API implementation, and the use of a channel to return a value; the message types and the server loop themselves are generally not redundant.)

I cut this off the end, but maybe I shouldn't. More what I'm going for here is that I've got a simple language that does a good job with X% of the problems of the world, and over there, I've got a language with a laundry list of complicated features that are much harder to understand and work with, and it goes a good job with Y% of the problems of the world, and I think there's a lot of people who master one of those hard languages and come away thinking X is roughly 5% and Y is roughly 100%. But especially from an engineering-in-a-large-team perspective, it's worth being reminded that that's a cynical estimate, not a reasonable one. It varies from place to place, of course, but X can easily be 80% and Y merely 95%, and might want to try to be sure that you're only paying the price for Y when you're in that other 15%.

I'm curious whether it's possible to build a better language that covers more of the space between Go & Rust, but with a more Go-like cognitive footprint, and I'd like to poke interested people to consider if there might be an answer to that. Again, especially from a programming-in-a-large-group perspective, I still don't really love any of my options. I realize that from your POV (no sarcasm) Go seems like a terrible downgrade, but from my POV it has been an immense upgrade from a world of dynamic scripting languages, in terms of what I can guarantee and provide in my APIs without writing in a language that virtually nobody understands. There's no way I could put Rust or Haskell or anything else that strong and expensive into where I'm putting Go. Go's in a sweet spot that happens to be poorly occupied right now, and I would love nothing more than for someone to improve on it. This is especially true because other than perhaps getting real generics added on later (which I still think is at least conceivable) there's a lot of other places where Go is already very near its own local optima; it's not going to improve much, I think. It will be pretty much what it is.


I'm with you on the last bit, honestly. There is room for some statically-typed language to occupy the network-services niche where performance needs to be good—maybe 2x slower than optimal (note that this is not acceptable for all network services)—and speed of development trumps most other considerations. Some sort of hybrid of Swift, Rust, and Go, perhaps, as none of them are 100% ideal for this space.

It's no secret at this point that I disagree with lots of Go's language design decisions, but it's undeniably an excellent tool, whatever its strengths or weaknesses as a language are (hat tip to tptacek for this phrase). The Go team and community did a great job of making it very practical.


It seems like a lot of dynamically typed language programmers are catching on to Go and it's the best thing they've ever known. Generics are just not that hard. They are not the difference between a comprehensible codebase and an incomprehensible one. The only reason they're not in the language along with higher-order list manipulation among other things is because of the idiosyncrasies of Go's creator.


Bear in mind that while my job is in dynamically-typed languages, I'm fluent in Haskell, and have done non-trivial work in it. Go wouldn't be as large an upgrade for me if Haskell hadn't taught me so thoroughly what static type systems can do. I'm not just "a dynamically-typed programmer who just discovered Go and thinks its the best thing since sliced bread" or something.


I'm very thankful for your work in figuring out how to make more static guarantees in Go code.


Here's one way you could do the clipping generically in Go[1]. This has a Clip function that uses a type assertion. Another (perhaps better) implementation would add Clip methods to both Rect and Poly.

Either way, I don't think this example proves the necessity of generics. Some algorithms are certainly more difficult to express in Go (graph data structures/algorithms in particular in my personal experience).

[1]: https://play.golang.org/p/xYQWDS41A0


> Here's one way you could do the clipping generically in Go[1]. This has a Clip function that uses a type assertion.

Well, sure. But the title of the article is "Avoiding Reflection in Go". :)


You Go people really likes the pain.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: