
Go command support for embedded static assets (files) - eberkund
https://go.googlesource.com/proposal/+/master/design/draft-embed.md
======
breakingcups
Again, more magic comments.

The proposed feature is great, but the unwillingness of the Go team to use a
separate, clearly defined project file or at the very least a separate syntax
in your code file leads them to stuff every additional feature into comments,
a space shared by human notetaking.

Let's have a look: * Build constraints (// +build linux) * Code generation
(//go:generate <command> <arguments>) * Cgo, you can even stuff entire C
programs in the comments (// #include <stdio.h>) * Cgo flags (// #cgo CFLAGS:
-DPNG_DEBUG=1) * and now this, file embedding (//go:embed html/index.html)

Most novices would assume the commented out code does nothing, and rightly so
in my opinion. Half of these features aren't even code-file specific but
project-wide, making deciding which file to put them in hard and looking them
up even harder.

~~~
Asooka
These should all just have been pragmas with their own syntax. I would even
accept something as heretical as "@" for the pragma command, e.g.

    
    
      @build linux
      @generate <cmd> <args>
      @C #include <stdio.h>
      @cgo CFLAGS: -etc
      @embed path/to/file
    

Maybe these could even be come kind of code that you can write yourself and
define your own pragmas and code transformations by passing some lib folder to
the compiler with prebuilt compiler extensions... Ah well.

~~~
yiyus
Besides the precious four bytes we would save for each pragma, what's the
advantage of using "@" instead of "//go:"?

~~~
MaxBarraclough
1\. It doesn't look like a comment, so it's less confusing to people new to
the language

2\. It allows the programmer to skip comments entirely, when reasoning about
program behaviour

3\. You can write comments without fear of the comment changing program
behaviour (probably of little concern in practice, but still)

Putting directives in comments is always a hack. C and C++ get this right,
with _#pragma_.

~~~
saturn_vk
You also get no completion help from the language server since you are in a
damn comment

~~~
MaxBarraclough
That's another good point.

------
leetrout
That’s showing some real support to the community IMO.

There are plenty of purists that argue both for and against the concept but
one cannot deny it fits very well in the sales pitch for Go. You can have a
simple tool chain to ship a binary for any number of systems and that support
for embedded assets would be a first class citizen.

I would also love to see them address more robust plugin options or officially
adopt / endorse the pattern HashiCorp uses as they’re doing here with the
bindata prior art.

~~~
api
Go is not a purist language. It's pretty relentlessly designed for maximum
productivity with minimum complexity. This sort of feature is right in line
with its mission.

~~~
Filligree
I keep hearing that, but Go is one of the languages I'm least productive in,
and I know I'm not alone in that.

You can pick it up in a week, sure. That's not the same as productivity.

~~~
change442200
I'm a dev for more than 10 years. Much Python, C#, Java, JS/TS, Kotlin, a bit
of Rust and was recently forced to use Go.

Writing out source code by hand: Nothing beats TS or Kotlin. Not even Python.

But what surprised me with Go after a couple of month (and please, nobody
should judge a language until having done a serious project): Taking
everything into account, from installing to looking for libs, writing code,
testing, compiling, revisiting code after a couple of month ... Go is an
absolute dream in terms of overall efficiency.

Writing out error-checking or for-loops is a one shortcut in GoLand. Getting a
table-test template for a function as well. People complaining about having to
type this stuff out should learn a proper editor.

~~~
Cthulhu_
Whilst I agree with your comment, I would like to point out / express my
personal opinion that having to rely on your editor to churn out boilerplate
like error checking is a smell; it reminds me of ye olde Java where Eclipse
could generate hashCode and equals. Later on I always used Lombok, and if I
were to work in Java again I'd push hard for Kotlin, which has stuff like that
built in. I know Scala does as well but it's a can of worms, it has too many
options and no two developers will write the same code.

~~~
api
I've never needed to have an editor churn out Go error handling, and I don't
find it that hard. It could be better, but not much better.

Exceptions are not less complex. They just move the complexity around.
Exceptions as typically implemented also don't play well with event driven or
concurrent models.

The thing I actually like about Go errors is that it makes you think about
them right when they happen, while an exception encourages dealing with errors
"later" which often means "never" or "as an afterthought."

Go _kind of_ has exceptions, but panics are intended only for very extreme
cases like out of memory errors that crash most applications and are often not
recoverable. Using panics for non-fatal errors is bad Go code.

One reason I think people don't get Go is that it occupies a language niche
that formerly was not occupied by any language. It's like Python meets C, a
"low level scripting language." It's designed to be productive and pragmatic
but fast and compiled and capable of dealing with pointers or even embedded
ASM.

------
AnonC
I love this idea of making executables contain assets, and it fits well with
the Go ethos of a single (statically built) executable. This makes
distribution simpler for applications that do need to deal with assets of
different kinds.

A tangential comment: is anyone else concerned a bit that discussions and
debates happen on platforms like GitHub and reddit? These two platforms are
large enough to be around for quite sometime, but are we making it easier to
lose historical context because platform creators/designers are choosing third
party platforms that they don’t host or control (or in some instances don’t or
can’t pay for)?

~~~
aspenmayer
It’s concerning that the debate that formed and forms open source is not
itself available on open source platforms and/or archived and licensed and
available for access in the same manner and license as the code itself. Good
spot, quite apropos!

------
atombender
Video presentation: [https://golang.org/s/draft-embed-
video](https://golang.org/s/draft-embed-video)

~~~
brown9-2
I hope this catches on as a trend, it’s really nice to see the process being
so open to the community to give feedback.

------
nicoburns
Seems pretty complicated compared to Rust's `include_bytes!("path/to/file")`.
Sounds like it might have a compile time advantage though, seeing as it
produces a seperate package.

~~~
justasitsounds
I know it's been mentioned before, but I really feel there is a need to codify
an 'internet lore law' about how many comments can appear on a Go themed HN
post before someone brings up how superior Rust is

Something like: Pikes law of unfavourable comparison OR HN Golang oxidation
rate

~~~
papaf
I suspect that, programming in Rust is so onerous, that Rust programmers must
take lots of breaks and troll Go threads so they feel less bad about
themselves.

Or maybe they're just waiting for something to compile?

~~~
no_wizard
To be fair, I remember thinking the same thing when Go was new. HN was pretty
dominated by everyone apparently switching to Go (if you were to believe the
general attitude anyway) and how Go was the clear path in the future

Not that it is or isn’t and clearly Go does things really well in some problem
spaces but yeah, this seems to be a common theme for all young languages

I do think some members of the Rust community Take it waaay too personally,
which I never saw in general when Go was the hip thing. That I agree can be
very obnoxious

~~~
pantulis
Same with Nodejs, and same with Ruby when the Rails craze. That's us devs.

------
banana_giraffe
I'm happy to see this, I've always liked using embedded assets in executables
to make single file distributions for tools.

That said, I really don't like the further overloading of comments. Comments
are for the human, not the compiler.

~~~
kstenerud
It's a consequence of go's simplicity by design. You can't actually destroy
natural complexity; only move it around. And if you're unwilling to design a
proper container around that slowly increasing complexity, it just squeezes
out between the cracks in odd ways.

In go's case, it's manifest primarily through magic (magic files, directories,
comments, names, env vars, switches, etc that you just have to know how to
invoke). Another warning sign is the need for an external build tool like
makefiles in order to build-in-one-command because you need extra steps beyond
"go build" (such as "go generate").

------
mjibson
I wrote one of the listed tools (github.com/mjibson/esc) and am thrilled about
this proposal. I think it's great and solves all the problems in a great way.

~~~
stevekemp
I wrote an unlisted tool too, and I am also a fan of this proposal.

The fact that there are so many of these "embed file in binary" tools suggests
that it really is a problem that could be usefully solved once, in a
consistent and reliable fashion.

------
lsllc
I think it's a well thought out design, particularly the implementation of
some common Go interfaces (such as fs.FS) that will allow this to be
transparently used with existing Go code.

The only thing I'd want is to allow environment variables in the "go:embed"
statements, while my assets might be in the same repo and thus relative, they
may also be in a different asset repo (if I'm using git I could use git
submodule, but if I'm using something else that might not be possible).

~~~
yiyus
The best way to do this with the current proposal is to write a simple go
package in your assets repository and import it. Something like:

    
    
        package assets // import "github.com/me/myproject/assets"
        //go:embed *
        var FS embed.Files
    
        package mypackage
        import "github.com/me/myproject/assets"
        //use assets.FS
    

Would that cover your use case?

~~~
lsllc
Yeah! (as long as the UI/UX folks don't mind a bit of Go code in their repo)

I was kind of hoping for:

    
    
      package foo
    
      //go:embed $ASSETS/*
      var assets embed.Files
    

But one issue with this is that it might make a dogs breakfast of the
filenames within the `assets` object -- it sort of needs something like this:

    
    
      //go:embed * from $ASSETS
    

(and it would not include the $ASSETS path in the object).

~~~
yiyus
An interesting option that I've not seen proposed yet would be to accept
go:embed comments before importing a package (obviating the need to write the
assets package in my previous example).

So, assuming your assets are in the repository github.com/me/myproject/assets,
you could just do:

    
    
        package foo
    
        //go:embed */*
        import "github.com/me/myproject/assets"
    

And the go tool would take charge of generating an assets package with
assets.Files.

Then we don't even need a magic embed package at all. And generating a package
would give us more flexibility, for example:

    
    
        package foo // import "foo.io/foo"
    
        //go:embed Name string "name.txt"
        //go:embed Data []byte "bin.dat"
        //go:embed "images" "templates" "html/index.html"
        import embedded "foo.io/foo"
    
        //use embedded.Name, with type string
        //use embedded.Data, with type []byte
        //use embedded.Files, with type fs.FS

------
ceocoder
My former coworker Miki Tebeka wrote nrcs[0], this was specifically for static
files - css, JS, png etc - for shipping a self contained web server binary. I
still use it now and then because it is well designed and does one thing and
that one thing really well.

[0] [https://github.com/tebeka/nrsc](https://github.com/tebeka/nrsc)

~~~
joice
I am using stuffbin to embed files. github.com/knadh/stuffbin

------
axaxs
Can someone explain please the point of embedding bin data into a binary, vs
just bundling those assets and opening as a file at runtime?

~~~
zelly
Performance. No additional I/O at startup and guaranteed to be in memory.

~~~
jeffbee
I don't know what platform you had in mind, but this isn't generally true for
Linux. No part of a Linux executable is in memory except the page containing
the main entry point, initially. You'd have to pre-fault the sections of
interest and protect them with mlock to make sure they stay in memory. If you
don't their cached pages may be dropped, which is essentially the same thing
as swapping.

In short, these embedded assets are no more likely to be in memory than any
other file-backed data. If you want to guarantee they are in memory, it is up
to you to make that happen.

~~~
yencabulator
Even then, it avoids open, path traversal, read read read, close.

More important is avoiding having to answer the question "where are my
assets".

------
AlexMax
Regardless of the bike-shedding over the exact syntax, this would be really
cool to see. I have no clue how C/C++ made it into 2020 without this language
feature - and yes, I know about incbin and xxd, and Go having it ahead of
those languages would be ironic given how new and conservative of a language
it is.

------
tmaly
This would be huge. I have had to deal with 3rd party libraries to try to
embed assets for a web server.

~~~
marcrosoft
Which is the way it should be. Not everything needs to be in std lib.

~~~
donatj
While I agree that "not everything needs to be in std lib", having worked with
other languages that support embedding out of the box, let me tell you it's
handy.

It also seems like an area where it'd be nice for there to be a single uniform
way to do it.

~~~
Gibbon1
Of all things C is missing the ability to import binary assets. Which is kinda
deranged when you think about it.

~~~
shakna
For runtime, you can certainly read any arbitrary binary file in C without
problems.

For compile time, I make heavy use of xxd -i to generate header files in a
range of projects. It is no longer installed by default on most +nix systems,
but it is generally still in your package manager (often installed alongside
vim), and has been around since 1990.

~~~
anitil
Oh I wish I'd known about 'xxd -i' ! When I saw it in the video linked above I
was so annoyed that I hadn't used it before

------
ocdtrekkie
This is funny to me to see on HN tonight because I spent a bunch of time
packaging a Go app on Sandstorm this evening, and one of the listed packages
apparently was pitching a fit about some static assets not being pulled into
the Sandstorm package correctly...

I resolved it, checked HN, and saw this was a peeve they wanted to solve. Hah.

------
malandrew
I would love if gob encoded files were supported natively with some special
sugar syntax.

I have a project I work on where I need to package a bunch of data that won’t
be available locally. Given that I’m loading data, it seems most optimal to
just package it as encoded go data from the get-go.

Right now I’m using go-bindata to embed gob files.

------
skocznymroczny
D has the import statement for that:
[https://dlang.org/spec/expression.html#import_expressions](https://dlang.org/spec/expression.html#import_expressions)

------
lenkite
Go needs proper typed annotations like Java

~~~
pjmlp
Go started being a Java 1.0, instead of learning what a modern language should
be like, so every missing feature gets eventually added with a couple of
hacks.

Go 15 is going to be a pleasure to use.....

~~~
jjice
Is 15 the version that generics are going to be implemented in? Or are
generics still in the proposal stage?

~~~
pjmlp
Hopefully by then they should be available, we will be discussing how they are
unsound and how it was a mistake to add them in like that, while everyone will
be discussing this new language that is going to be so much better and fix all
the problems.

------
jedisct1
Something I'd love to see in Zig.

------
jhardy54
Can you run one of these files as an executable?

~~~
zelly
It looks like this is just for static data files. To run a piece of memory as
executable you'd have to explictly mark the page as executable with e.g.
mprotect(2) on Linux. I guess there's nothing stopping you from doing that
from Go also.

~~~
q3k
You still need an ELF loader, unless you're shipping shellcode. As far as I
know, there's currently no great way to load an arbitrary segment of memory as
an ELF, without doing some sort of copy (eg. memfd_create, write, exec).

------
marcrosoft
No mention of go-bindata which was one of the originals :(

I think this problem is best solved by a library.

Leave the damn language alone!

~~~
coder543
> No mention of go-bindata which was one of the originals :(

There’s literally an entire section in the appendix dedicated to `go-bindata`
and explaining what it generates. It was also first in the list of libraries
mentioned.

I also disagree with you entirely. The document does a great job explaining
the problems with having this “solved by a library”. I’m excited about this
draft design.

------
bww
I’m aware there is demand for this sort of thing but I’d suggest that almost
nobody actually needs it. This strikes me as a feature in search of a problem
– especially considering how many of the examples in this document relate to
static web assets.

If you’re considering bundling static assets into your binary for a service,
you would almost certainly be better served by containerizing your service and
copying those assets into the image.

~~~
anonfunction
I disagree, one amazing aspect of Go is the static binary that can easily be
distributed. I embed templates and other text files in my binaries and it
makes the installation so simple and without extra dependencies or steps. Even
though I'm a huge proponent of containers I don't think it is needed for
things like CLI tools.

~~~
pjmlp
I was quite amazed doing static compilation with Turbo Basic, Turbo Pascal,
Turbo C, Turbo C++,....

~~~
anonfunction
When I see all the CLI tools that require NPM / PIP to install hundreds of
dependencies I am quite amazed when one requires a single binary. I'm not
saying it's a unique feature with Go but that it is nonetheless a feature.

~~~
pjmlp
What amazes me is that installing hundreds of dependencies is acceptable, and
that modern generations don't get that static compilation goes back to the
first compiler, while dynamic compilation only went mainstream around
mid-90's.

