
File System Interfaces for Go – Draft Design - networkimprov
https://go.googlesource.com/proposal/+/master/design/draft-iofs.md
======
didip
I am surprised that it doesn't have public interfaces with context.

Network file system could use the same interfaces + timeout settings.

~~~
infogulch
Oh god no, this draft is actually pretty good, don't ruin it by infecting it
with context. Context is the worst library in Go: it's a pile of hacks to get
cancellation and goroutine-local storage by _manually passing new heap-
allocated objects through every function interface in every library between
low-level IO up to task management_. It infects everything, obliterating
separation of concerns making even libraries that should have nothing at all
do do with timeouts include code for it. And it includes an untyped key-value
store implemented as a linked-list of pairs (!!!!), because why not?

If you can't tell, I don't like context. I've said before [0] I really hope
that Go 2 comes up with an actual solution to the cancellation and task-local
storage problems and deprecates context. Some comments in that thread pointed
to alternatives that looked pretty decent, I wonder what state they're in
these days.

[0]:
[https://news.ycombinator.com/item?id=18561884](https://news.ycombinator.com/item?id=18561884)

~~~
solatic
Everything old is new again. This challenge is basically why monads exist,
because the monad allows you to cleanly separate between the state in which
the algorithm is running and the algorithm itself. Reminds me of:
[https://philipnilsson.github.io/Badness10k/posts/2017-05-07-...](https://philipnilsson.github.io/Badness10k/posts/2017-05-07-escaping-
hell-with-monads.html)

I wonder what poor abstraction the Go authors will come up with instead?

~~~
nyanpasu64
>
> [https://philipnilsson.github.io/Badness10k/posts/2017-05-07-...](https://philipnilsson.github.io/Badness10k/posts/2017-05-07-escaping-
> hell-with-monads.html)

Every "monadic solution" prints the same code block without explaining how it
would work, the types of the various variables, the semantics of the <\-
operator. I didn't leave the page with an understanding of how monads achieve
these tasks.

~~~
virtue3
The article is literally: "Hell thing" Step 1 -> monads step 2 -> ???? step 3
-> profit?

This is apparently the monads curse where people that get them can't effing
explain them to save their life.

~~~
strbean
Monads are just monoids in the category of endofunctors! /s

The issue is pretty much a language barrier. All these articles talking about
benefits of / interesting ways to use monads are written by people who speak
the language, assuming the reader speaks the language as well. As with many
functional programming topics, the fundamentals aren't incredibly easy to wrap
your head around. But if you already understood monads, can you imagine how
annoying it would be if every resource, discussion, article, etc. relating to
them started with a pages-long introduction on What is a Monad?

In Haskell, this is a fundamental topic. Monads are used everywhere. If people
always explained how they worked when discussing them, it would be like
looking up sorting algorithms and having every algorithm description start
with a long-winded explanation of how for loops work.

~~~
infogulch
> But if you already understood monads, can you imagine how annoying it would
> be if every resource, discussion, article, etc. relating to them started
> with a pages-long introduction on What is a Monad?

I agree that it would be unreasonable for every article that uses monads to
describe what they are. But I don't think it would be unreasonable for every
one of them to link to _another_ article that does explain them.

~~~
dllthomas
Would you say the same about for loops? Depending on your audience, it's
totally reasonable to expect they know certain things. Further, figuring out
what to recommend isn't always easy, and I'd usually rather authors put their
efforts into presenting the material that they have to share.

------
suessflorian
Awesome to see some work on filesystem API's but oh boy... you can really see
the challenge provided by the backwards compatibility promise. Alias's in os
package to definitions in io/fs, the duplication of the http.FileServer.

------
shawnz
Much of the criticism I have seen towards Go has been in regard to the poor
design of its filesystem APIs (example:
[https://news.ycombinator.com/item?id=22443363](https://news.ycombinator.com/item?id=22443363)).
So rethinking this might make the language significantly more attractive for
some. Especially considering the addition of generics, I am becoming much more
interested in the language again.

------
SEJeff
I’m kind of surprised they don’t mention afero at all, who was one of the
first big packages to abstract filesystem access. Shoutout as they make unit
testing filesystem ops a piece of cake!

[https://github.com/spf13/afero](https://github.com/spf13/afero)

~~~
kyrra
They are likely aware of it, as spf13 works on golang at Google. :)

~~~
SEJeff
Yeah I realize that. It is just surprising is all. It is a very nice to use
interface. Getting an equivalent in stdlib would be great.

------
LTClipp
I would like to put it out there that I'm working on a pathlib library that is
attempting to solve a lot of the problems that this design draft is
addressing.

[https://github.com/chigopher/pathlib](https://github.com/chigopher/pathlib)

------
networkimprov
This (recent but buried) comment suggests replacing FileInfo in the ReadDirFS
interface with a DirEntry type, for reasons of performance and future
extensibility:

[https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draf...](https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/g0hwjij/)

It was prompted by this proposal for os.Readdirentries():

[https://github.com/golang/go/issues/40352](https://github.com/golang/go/issues/40352)

It seems to me that FileInfo isn't a suitable interface for either a general
filesystem construct, or a performant implementation.

------
bfuclusion
Is stat really something you want as an abstraction over all filesystems? You
might not even have that information available in all of them.

~~~
heinrichhartman
Calling stat on all the files is just really expensive if your directly is
large.

~~~
tptacek
FileInfo is an interface, isn't it? You could populate fields other than
"name" lazily, right?

~~~
TheDong
You can't because the interface doesn't allow any errors. Lazily populating
data is usually fallible, and the interface's methods are all infallible.

~~~
tptacek
OK, I'm convinced. Not a great interface, should just return an iterator.

------
sanxiyn
This is very valuable for languages to get right. Tcl had it for a long time
and I envy them: [https://wiki.tcl-lang.org/page/VFS](https://wiki.tcl-
lang.org/page/VFS)

------
parhamn
Maybe they can just put in double-glob (recursive) support in one Go as the
embedded files stuff needs it too. Wonder how that will work from a backwards
compatibility perspective? Do asterisks have to be escaped to begin with?

------
boring_twenties
os.Readdir has to be one of the stupidest, most broken interfaces I've ever
seen.

Reading the whole dir into an array is bad enough as it is, but they even call
stat() on every file too.

~~~
mpfundstein
i like the function. saved me a lot of manual work already ;-)

just use it when you need it and be aware of the downsides...

~~~
Varriount
And what if I'm searching through a large directory? Your response doesn't
address the argument being made.

Creating an API where the entire set of data is returned from a data source,
without an option to limit the result, is bad. It means that, if I have to
read a directory with 2000 files (and this is quite possible! - think /bin,
etc.) the function will have to allocate a slice with 2000 entries, and then
call stat on all of them.

Stat is not free, especially on certain filesystems (NFS!), nor is reading
directory entries.

~~~
mpfundstein
that's why I wrote: be aware of the downside...

It's a mega convenient function if you want to slurp in a few files.

~~~
ikiris
"ORM is great because it means I have all the data avaliable!"

------
joshuak
This is great. I've had challenges related to filesystem abstraction for
years. I scoured the go packages on github for a solution, and found afero[1],
but at the time that package had been languishing for years without anyone
merging PRs, many of which were critical fixes. It's been picking up again
recently so maybe it's better now, but at the time I needed a solution so I
built abfs[2].

Abfs is conceptually identical to the go draft design, including the concept
of "extension interfaces", although at the fs interface level not the file
interface level. One of the problems I ran into early was the need for an easy
to use file handle. The `net/http` FileSystem interface shows that you must
always implement a custom Open function and a custom File. An object that
implements `net/http` File is not adequate, it must actually be cast to
http.File. Because of this I've found that it is burdensome to allow
flexibility in the definition of the File interface because on the one hand
specific functionality not obviously available to a generic File interface
must be looked for using type assertion and the consequences of not finding it
are poorly defined (should we return an error, should we proceed with some
work around, etc), and on the other functions that return a file handle have
many possible choices making un-anticipated interoperability unlikely. So
instead I opted for always returning a `absfs/absfs` File interface, but
return a ErrNotImplemented for functions that are not supported by an
implementation. I don't like this, but I like it better then having to do a
lot of interface unwrapping to get to the same place, and having it be an
error makes the consequences more concrete.

Nevertheless, I'm a million percent behind having a filesystem abstraction in
the go standard library. It is immediately useful in testing to redirect
potentially costly io operations. It is composable, allowing you to do very
little work to wrap a file system with mutexes, timouts, and other gating
mechanisms. It allows you to support a transactional file system by spawning a
FileSystem from another FileSystem. Caching, copy on write, layering, and
arbitrary data transformations are all much easier to reason about and
implement using a prototypical filesystem as the model. Cheers, thanks for
this Russ and Rob! From where I'm standing it would be one of the most
valuable improvements to the go ecosystem possible at this point!

[1]: [https://github.com/spf13/afero](https://github.com/spf13/afero)

[2]: [https://github.com/absfs/absfs](https://github.com/absfs/absfs)

------
henvic
The best thing about Go is how things are designed thinking ahead in the
future and avoiding hypes.

Instead of a language with more features than one can count, we have something
really concise, yet powerful. This is something I miss a lot about Go and I
think can perhaps even have a greater impact than generics for most people (at
least I feel like this is my case), yet this is only being considered now
after ideas matured in the community and people developed ways around it (see
[https://github.com/golang/go/issues/35950](https://github.com/golang/go/issues/35950)).

The outcome will probably be something that will be robust and stable and once
in the language will likely last a long time unchanged without feeling
awkward.

~~~
danudey
I'm viewing this from outside of the ecosystem, but nothing I've seen from Go
externally shows any sort of "thinking ahead", so much as "building what
Google needs and wants to have".

This blog post is just one example of those kinds of issues, and it's just the
most recent of such that I've read: [https://fasterthanli.me/articles/i-want-
off-mr-golangs-wild-...](https://fasterthanli.me/articles/i-want-off-mr-
golangs-wild-ride)

Things like no package management, no vendoring, importing modules directly
from GitHub without any concept of versioning, the confusing mess that is
$GOPATH, the awkward handling of errors, and so on. Lots of things that were
fixed, but shouldn't have been broken in the first place.

Go certainly has use and functionality and benefits (goroutines sound
_awesome_ ), but Go itself seems like kind of an awkward mess when viewed
outside of the context of "internal Google tool".

~~~
Sean-Der
Maybe not relevant to your actual point! But most of the things you mentioned
aren't a problem anymore.

* no package management

I use Go Modules for Pion and really love them
[https://github.com/pion/webrtc/blob/master/go.mod](https://github.com/pion/webrtc/blob/master/go.mod)
I select the versions I want and everyone that downloads my code uses the
proper version.

* No vendoring

Just do `go mod vendor` and you are done

* importing modules directly from GitHub without any concept of versioning

You set the versions in your `go.mod` file.

> Go itself seems like kind of an awkward mess when viewed outside of the
> context of "internal Google tool".

Go is actually really nice from a community standpoint. WebRTC is driven by
Google and is magnitudes worse. I deal with a wonderful mix of arrogance and
ignorance. I appreciate the great tech they bankroll and try not to let the
other stuff bother me.

~~~
codys
The fact that go eventually fixed things doesn't really speak to the top-level
comment on this thread that "The best thing about Go is how things are
designed thinking ahead in the future and avoiding hypes."

That the things needed fixing (and especially in cases where there were
previous ways of doing things that had to be entirely discarded) seems to go
against the idea that there was thinking ahead in those areas.

~~~
Sean-Der
I think the 'In favor of Go' argument is that the Go authors defer decisions
they don't know the answer for yet. They would rather leave something
unimplemented, then do it the wrong way.

But maybe you are right and they didn't anticipate these issues! I have no
idea either way :)

~~~
infogulch
I appreciate the answer "I don't know" when it's the right answer. And I think
the Go team is an excellent example of holding steady on "I don't know" until
the answer arrives, despite all the flak they've received for taking their
time. I also think the way they've decided to go about announcing/framing
these recent drafts is smart; they're learning from the past.

------
zemnmez
I really like the way these proposals preserve file metadata

------
justicezyx
One thing I guess people dont realize is that a lot of Go design is closest
resembles Google's internal C++ APIs.

There is no exception for this API either.

I kind of feel weird when there has been loud praise on Go team, while in
general it was no mention of the decade of hard work of evolving C++ APIs...

~~~
jeffbee
Do people really sing the praises of File? Or, do they curse its many and
dangerous rough edges?

Like you I read this proposal through that lens, but I actually don't see a
line from Google's File abstraction to this. Is this Go abstraction really
forward-compatible with cloud-native filesystems? And when I say forward
compatible I mean would it work with Google's decade-old Colossus? I think it
isn't because it puts Stat in the required interface and Stat returns
FileInfo, many of the fields of which might be meaningless in a cloud
filesystem (like mode, which is a unixism, and IsDir, which doesn't make sense
for non-hierarchical filesystems).

If you can't tell I'm not much of a fan of trying to abstract over
filesystems. Mostly they don't resemble each other at all, except in some
extremely high level concepts.

~~~
justicezyx
So first, what exactly I was meant to say, is that: Go design resembles
closest to Google's internal C++ APIs.

Then, when I saw people praises Go APIs, I feel that some people worked on C++
APIs at Google, were missing some credits. That's a derived feeling from the
above.

That is not to say that this File design was already being praised (not to say
that it does not deserve praise); or that it should be criticized (not to say
that it does not have problems).

As for your example to contradict my claim: "Stat returns FileInfo, many of
the fields of which might be meaningless in a cloud filesystem": 1\. This has
to be there because a File API has to be compatible with OS files (Unixy). And
Google's API also does the same thing. 2\. Incompatibility derives from
enforcing information that not present in another scenario. Clearly, the
presence of such attributes does not prevent them to be applied on cloud files
systems.

And further, Cloud file systems do have mode, and directory...

~~~
athrun
> And further, Cloud file systems do have mode, and directory...

The most popular cloud file system, S3, doesn't have folders. It has prefixes,
but if you treat them like folders, you're in for a lot of pain.

~~~
jeffbee
Colossus doesn't have directories either.

[http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-
Ke...](http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf)

Page 9

