
Grappling with Go - dcu
https://blog.ntpsec.org/2017/02/07/grappling-with-go.html
======
pjmlp
> It’s unprecedented for a language to be both garbage-collected and as
> squarely aimed at systems programming as Go is, because systems programmers
> have trouble tolerating the stop-the-world pauses that happen when the GC
> runs.

Only for those that never heard of Mesa/Cedar, Algol-68, CPL, Ada, Modula-2+,
Modula-3, Oberon, Oberon-2, Active Oberon, Component Pascal, Oberon-07, C+@,
Sing#, System C#, D.

~~~
mmarx
Ada is (usually) not garbage-collected, precisely due to the unpredictable
pauses.

~~~
pjmlp
Ada was actually designed with support for GC, which was dropped in Ada 2012
due to lack of adoption across all Ada vendors.

[http://www.ada-europe.org/manuals/Rationale_2012.pdf](http://www.ada-
europe.org/manuals/Rationale_2012.pdf)

------
stevekemp
I continue to be surprised by the sheer amount of writing we see about ntpsec.
The project was started over two years ago, and yet progress reports seem very
optimistic and self-congratulatory, despite an obvious lack of progress.

Sure rewriting "bits" of things, with more features, and adding my python
sounds useful. But here we are 500+ days since the project started and there
are only minor experiments in moving from C.

I get the feeling this is one of those projects which will never be complete.
Perhaps not a bad thing, providing those involved are having fun, but so much
verbiage to say so.

~~~
jerf
What progress are you looking for? There seem to be downloads here, posted
pretty regularly: ftp://ftp.ntpsec.org/pub/releases/

I don't know if they build and work, but I'd suspect they do, based on the
refactorings being discussed. The commits seem to be clicking along. A
diffstat of roughly where I think they branched off of ntp vs. master takes a
moment to poke through, because the raw stats are "1032 files changed, 126489
insertions(+), 143789 deletions(-)", but there's a lot of trading C code for
test code and more docs in there.

I don't know exactly what metric you're using but this doesn't match my
"obvious lack of progress".

~~~
tptacek
They started with Mills ntpd, so there have been downloads available since t0.
There's a significant amount of code you can lose from a large C project just
by getting rid of ifdefs that nobody uses, so the +/\- line count isn't a
great metric either.

The true metric for success for something like ntpsec is the number of
meaningful security problems ntpd has been vulnerable to since ntpsec's
inception that ntpsec hasn't been.

At the point where they rewrite in a new language, they're embarking on a
fundamentally different project with a different value proposition, which sort
of moots the progress they've made (or not made) on hardening ntpd.

(Fair warning: I am both an ESR skeptic and an ntpsec skeptic; I genuinely do
not like the idea behind the ntpsec project).

~~~
groks
> The true metric for success for something like ntpsec is the number of
> meaningful security problems ntpd has been vulnerable to since ntpsec's
> inception that ntpsec hasn't been.

They talk about 3 areas where bugs have been removed, including:

    
    
      Much of ntpd’s most convoluted code lives in ntp_proto.c, which
      implements the state machine central to the protocol. Of the 29
      vulnerabilities that have received CVEs so far in 2016, a couple
      of multi-KLOC functions in ntp_proto.c are responsible for 15 of
      them — just over half. Of course, "just rip it out" wouldn’t
      suffice in this case: this is core business logic, not junk code.
      So we rewrote those functions from scratch, cutting line count
      considerably and yielding a far more readable result.
    

[https://blog.ntpsec.org/2016/12/13/fantastic-bugs-and-
where-...](https://blog.ntpsec.org/2016/12/13/fantastic-bugs-and-where-to-
find-them.html)

The other two areas are guaranteed bug-free because they removed the code.

~~~
tptacek
From this paragraph you might get the impression that ntpsec had dodged 29
CVEs in 2016. But that's not the impression I have: I think it has been
vulnerable to many of the bugs reported in ntpd.

Further: not all CVEs are equivalent, so in addition to wanting to know how
many vulnerabilities ntpsec was also vulnerable to, you also want to know what
the distribution both of severity and of exposure in the default configuration
those bugs had.

Ultimately: I feel about ntpsec the way I would have felt if, in 1997, someone
had proposed SendmailSec. The answer to the Sendmail security problem wasn't a
"hardened" Sendmail (though the Sendmail team sure tried); it was Postfix and
qmail.

------
ridiculous_fish
> I checked this by writing loccount as a parallelized tree traversal. Each
> file in the tree gets a goroutine spawned to count its lines; the threads
> run in parallel and as each finishes its work it stuffs a statistics block
> into a channel.

In the past, this approach would blow up because Go would spawn a kernel
thread for each blocking system call that was performed. I had to use a rate
limiting channel to constrain the number of kernel threads. Is that still the
state of affairs?

~~~
StreamBright
Isnt there a way to limit the number of threads? Starting N threads each reads
the same queue and executes the operation. The queue gets full when there are
no workers are available. I though you can implement something like this in
Go...

~~~
pcwalton
You risk deadlocks if you have a limit on the number of system threads.

Think about something like goroutines reading and writing from pipes.

------
perlpimp
"I think writing toy programs - conscious finger exercises - are a terrible
way to learn a new language." whoa hold on there. everyone's learning style is
different. Doing "dumb simple" things first and extracting reward from it is
essential to growing progress, even for advanced programmers imo.

------
ENTP
For me, the "learning curve" of Go was very small and that's the kicker. You
can get up to speed quickly and get a 'feel' for the language quickly. It's
been a great learning experience and I recommend it thoroughly.

~~~
fauigerzigerk
I continue to be surprised by this argument. It seems to only matter if your
job is to learn one language per week.

I don't need to get up to speed in a weekend. It's the speed that I'm getting
up to that matters. Of course there is a limit to this principle, I grant you
that.

What matters as well is how readable other people's code is, or even my own
code after not touching it for a year. And I mean readable for someone as
proficient in the language as the original author. This aspect sometimes seems
underappreciated by proponents of "powerful/expressive" languages.

~~~
weberc2
Learning curve is important because it means you can bring a new team member
up to speed quickly--you don't need to _find_ Go developers, you can make
them. The learning investment will pay dividends very quickly. Also, Go can be
_mastered_ in 1-3 months, whereas other languages take years. Being able to
quickly (i.e., cheaply) produce experts in the language seems advantageous.

------
tormeh
>It’s nice to see CSP in a language I can use for production, and very
gratifying to find that it really is an effective tool for carving up
concurrency problems. If there is one pitch the Go designers have knocked
right out of the park, this is it. I’m already sure I’m going to miss it a lot
in Rust.

Wait until he realizes that Rust has channels, too. It's not as pure as Go's
implementation, but it's a superset.

------
IshKebab
So I have to wonder about the performance difference between his sloccount and
loc ([https://github.com/cgag/loc](https://github.com/cgag/loc)) which is so
fast it always makes me think that it hasn't done anything (same as ripgrep).

I'm like 98% sure loc will be faster, but it would be interesting to know by
how much.

~~~
dcu
so I ran this test using the linux source code. I warmed everything to prevent
caching/buffering advantages.

For loc the best time was: 55.425s and the worst was: 1m30.557s

For loccount the best time was: 42.345s and the worst was: 56.396s

loc supports more languages but I also believe it has more false positives.
loc reporting seems nicer as well.

~~~
j1f4
Pass the release flag when you build loc; `cargo build --release`.

I see ~5 seconds for loc and ~34 seconds for loccount to count a freshly
cloned linux repository.

edit: woops, that was user time. These both spin all the cores. loc takes ~0.7
secs and loccount ~6.6 secs real time.

~~~
dcu
I'm not seeing any changes in my results. I'm testing in an old laptop, I
guess that's why it's so slow.

My rustc version is 1.15.0 and go 1.8.

~~~
j1f4
If you are running the program with `cargo run` that takes the same --release
flag. Could be the machine difference though.

~~~
dcu
I'm not. Are you testing with go 1.8 btw?

~~~
j1f4
Go 1.7.4 and rust 1.14. Interesting discrepancy in the relative speeds of the
tools we are seeing.

------
akerro
Rust and Go are not competing with each other. Why do we see so many
comparisons of them?

~~~
blub
Because they are competing with eachother!

Unless Rust wants to position itself as embedded-only it will compete with Go
for higher level tasks.

~~~
koffiezet
No they're not. Go is very easy to pick up, Rust is not. That point alone
should be enough. Without prior knowledge of both, no sane person would
consider writing a browser-engine in Go, and yet that's exactly what they're
doing with Rust. Just as no sane person would pick up Rust to write generic
web services in that scenario.

Put Rust and Go in front of a python coder who wrote dozens of web services
using flask - and I am pretty sure I can predict what tool he'll pick. Sure
you could write web services in Rust, that's not what I'm arguing, learning
Rust for writing stuff like that is simply not worth it for people having to
write such services.

Rust's complexity is comparable with C++, and the main reason for this
complexity is to address problems that are mainly seen in C/C++ applications.
Rust's reason for existing is simple: modern low-level language with built-in
(memory) safety with no performance trade-offs (implying concurrency support
etc). That makes it squarely aimed at C++, where developers accept the
complexity as a trade-off for performance.

Go's reason of existence and goals are completely different, it addresses the
gap that existed between compiled and interpreted languages. It wants to be a
simple, safe, easy to deploy language that offers more performance over
classic scripting languages and thus is able to address more performance-
critical issues. It's main target is clear: servers and networking services,
where a GC's impact less important due to the I/O bottlenecks in most
scenario's.

~~~
nicoburns
The question of Go or Rust for something like web services is not as clear cut
in my opinion. I have JavaScript/PHP background, and have recently learnt Rust
for just this use case. Rust is slightly more complex (but really not that
much, nothing close to as bad as C++), but also offers more correctness
guarantees. Both are good choices, hence these discussions.

~~~
problems
> but really not that much, nothing close to as bad as C++

C++ dev here. Rust's borrow checking system is crazy and even after a week of
trying I couldn't understand it in a way that I could reliably use the
language.

In C++ everything is fairly straight forward and well documented, in Rust it
feels like I'm inserting random symbols in hopes that something will compile.
The syntax is verbose and ugly, maybe not as bad as C++ template code can get
sometimes, but it's nasty in its own right.

I want to like Rust, I really do, but this just really turned me off it.

~~~
dbaupp
> fairly straight forward

This is a rosy-eyed view of C++ if I ever saw one. C++ has a lot of accidental
complexity from layers and layers of features and backwards compatibility
concerns: maybe some of the features are "straight forward" in isolation, but
its interactions with the rest of the language rarely are.

C++ is deceptive, in that the compiler barely keeps track of anything for the
programmer, so there are many cases when code should have "random symbols" (or
something equivalent) added, but isn't, just being broken instead. The Rust
compiler front-loads this: trying to turn runtime brokenness into compile-time
errors, which does make it annoying to get code that runs, but it makes it
less annoying to get code that actually works correctly.

------
Ygg2
This has surprisingly little to do with Rust. Maybe change title to original?

~~~
bluejekyll
I had the exact same thought. On top of that, the fact that he did this
_before_ the post on Rust is strange. Why publish them out of order? Reading
this first would have made it clear that he already had a strong bias towards
Go, and also would have showed the Rust community what it was in detail that
he liked in Go, so as not to have inflamed so many who love Rust.

It seems at this point clear to me, that the previous Rust piece was written
out of sheer frustration, and that the entire period while "learning" Rust he
really wanted to be writing in Go, which he had already found to enjoy. This
is understandable, but creates a clear bias; which is completely fine, but
would have been nicer to see this for that reason.

Everyone's experiences are going to lead them in different directions. I
learned Go, decided it was missing features that I really wanted, and that's
when I decided to learn Rust. That was a decision driven as much from comfort
with a language as it was technical. Which brings me back to his original Rust
post, it was written as though comparing two languages on their technical
merit, when in reality (after reading this post) it was really about comfort.
Again this is fine, and would have been clearer if posted in the original
order.

------
bborud
I've changed my main language for doing work only a handful of times and it
usually takes a few years of doing side-projects in a new language before I
switch.

For instance it took about 7-8 years before I switched to Java (and then only
after establishing a programming style that doesn't rely on frameworks at
all). The language wasn't hard to learn, but it took a while for the JVM and
the standard library to get to where it needed to be.

C++ never made the cut. Quite possibly because C++ is not really _one_
language. There were at least as many styles of C++ as there had been projects
I worked on (except perhaps at Google, which is the only place I've worked
where I've consistently seen good quality C++ code).

I'm in the process of evaluating Go as my main language and I have to say that
this is the fastest I have gotten comfortable with a new language. I've been
noodling around for a few months, doing side-projects, but my experiences so
far have been very positive. Unlike most languages I haven't come across any
uncomfortable stuff that would be hard to overcome. Just the usual confusion
about what the idiomatic ways to do things are.

What is striking is that it is equally comfortable for doing relatively low-
level (my first Go program talked to a chipset over SPI on RPi, doing
something that is slightly timing sensitive) stuff and stuff that needs you to
build abstractions.

I haven't used Rust yet. Mostly because I don't have the time. And I chose Go
since some of my friends use it for software that you and I touch every day,
thus providing me with access to people who have used it "for real".

Go just might be my next "main language". (Actually, right now it feels like
I'll switch to it and never go back, but the tiny rational bit of my brain
says "oh yeah, here's a list of other times you had a good feeling about
something that turned to crap" :-))

~~~
sanjayts
Sorry if you saw this coming but do you _really_ not miss generics and
exceptions? I like Go given that it's a lean language and the compiles down to
a self contained binary which doesn't consume gobs of CPU or memory and is
pretty quick to start-up and run. Perfect language for writing command line
tools and one-off network clients.

But I dread the feeling of writing repetitive error handling code and copy-
pasta to get a simple generic functionality to work across data types. Maybe I
should just consider it a better C and move on...

~~~
bborud
I haven't wrapped my head around how I want to do error handling in Go yet, so
it is a bit early to say. I think I really need to write more code that deals
with abstraction layers before I can have a useful opinion on the matter. Not
that I'm too worried about not having figured out that bit yet since error
handling took me a while in all other languages as well. (I think my first
approach was to use panic() where I used exceptions in Java. But that doesn't
really tell you anything very useful unless you also know how I use exceptions
in Java :-))

I don't mind a bit of boilerplate as long as it produces understandable code.
Writing boilerplate can be automated. Reading and understanding code can't.

I do understand where you are coming from though. Perhaps we're getting a bit
spoilt as programmers in feeling that error handling should be ... more
elegant for a language designed within the last decade? :-)

I think the same goes for generics, though I have a hunch that I'm not going
to miss it. And I'm not sure why, but that's how it feels now. Whenever I do
design classes that are genericized in Java I do so sparingly. This is mostly
to ensure that the code is easy to reason about. I'm not fond of "clever" uses
of generics because while it may feel satisfactory to have things click
together in a beautifully complex manner, it can make code hard to read.

------
saywatnow
DNS lookup stalls as motivating concurrency .. messy, complicated code with
known bugs .. seriously? It's not that hard to call getaddrinfo() in a thread
and feed results back (atomically) over a pipe. No shared memory or mutexes
needed.

Looking at Android as a representative of "embedded" also seems a bit
fallacious. The lowest-end android device I can buy has a lot more ram,
storage and clocks than an average home router.

EDIT: this comes across pretty snarky - more so than is warranted. I'm
surprised by the first part, but have no right to impugn the whole project
based on that. The surprise remains, but I apologise for my tone.

~~~
pjmlp
> Looking at Android as a representative of "embedded" also seems a bit
> fallacious. The lowest-end android device I can buy has a lot more ram,
> storage and clocks than an average home router.

Well you can look at ARM Cortex-M3 running MicroEJ instead.

[http://www.microej.com/resources/supported-
platforms/](http://www.microej.com/resources/supported-platforms/)

~~~
saywatnow
I don't understand the relevance?

~~~
pjmlp
MicroEJ OS, a Java based OS for micro-controlers like the ARM Cortex-M3, with
128 MB NAND FLASH + 512 kB internal for storage and 32 MB SDRAM + 96 KB
internal for program data, running at 12 MHz is far away from the average
Android device configuration, yet can still run a mix of Java/C stack.

~~~
saywatnow
OP: go has an interest in suiting embedded because Android.

Me: Android doesn't need to meet constraints anywhere near a home router
(which needs to run ntpd)

You: there's a proprietary Java-based OS for Cortex-M3

.. still don't get it. Are you trying to sell me something?

~~~
pjmlp
That there are Android like OSes that run in hardware less powerful than home
routers.

~~~
saywatnow
The relevance of Android here is that Go has a reason to target it. Not that
it's a Java/C hybrid.

------
w8rbt
What does he mean by this?

 __ _" The language does have annoyances. The absence of const is a minor
one."_ __

[https://gobyexample.com/constants](https://gobyexample.com/constants)

    
    
        const s string = "constant"

~~~
peterwaller
He's referring to it as a type qualifier, I would assume.

[https://en.wikipedia.org/wiki/Const_(computer_programming)](https://en.wikipedia.org/wiki/Const_\(computer_programming\))

~~~
w8rbt
Got it! Thanks for the clarification.

------
edibleEnergy
My first major project with Go was BugReplay[1], and I had pretty much the
same experience as the author. As someone with a lot of perl and python
experience the idea of how restricted the language is stylistically initially
seemed like a stumbling block but quickly turned into a huge asset.

Now when I go back to python or perl (or javascript) I'm super conscious of
the choices I make regarding type checking and parameter checking.

[1]: [https://www.bugreplay.com](https://www.bugreplay.com)

------
gamesbrainiac
Why is a 62kb binary a problem for a tool like sloccount? For me 62 KB is darn
small.

~~~
LukeShu
The "62K" was lines of code, not bytes. He didn't actually give a binary size.

So I downloaded the code and compiled it. The binary is 2.5MB; or 1.6MB after
running it through `strip -s`.

