
My God, it's full of yaks - ingve
http://ronjeffries.com/articles/016-0607/yaks/
======
wheaties
The problem here is MacOS. The amount of work around and hacker I've had to do
on Mac far exceeds anything on Linux and more surprisingly Windows. Let that
sink in, it's easier to get things to work on Windows than Mac.

~~~
__jal
Not my experience at all.

Aside from some expected path issues (builds too stupid to actually look
rather than assuming a disk layout, etc) and some BSD <-> Linux issues (builds
that assume all the world runs linux), I've had very little problem getting
most things working on OS X.

It does help to learn what Apple has, uh, improved. This point is related to
what an awful time I have getting anything to work on Windows - I hate it and
don't want to learn about it.

At least in the article, there are hints that the author hasn't taken the time
to learn about the platform they want to run code on. Which is fine - I feel
that way about Windows[1]. But it is also a good idea to understand why you're
having the problems you're having.

[1] Which is why I don't touch Win at work and the VM I run at home is a hacky
mess that works for what it does, but only after a few aircraft carriers-worth
of cursing and slash-and-burn hacking that I know can't be the 'right'
windows-land way to do things. (The abomination is completely firewalled off
from the outside world - I try not to be a disease vector.)

~~~
douche
It's really not _that_ hard to setup a Windows workstation...

~~~
weberc2
I can vouch that it was at least as of Windows 7. But I hear the Windows folks
now have a proper shell and some scriptable package management!

------
cwp
This is why I love nix so much.

Getting a dev environment set up with nix is the usual amount of hassle. The
difference is that nothing ever works by accident. Therefore, once it does
work, I know I've captured all the dependencies and configuration, and I can
reproduce the environment wherever I want.

~~~
optforfon
I'm trying to read the docs and understand how nix works. It seems really
fantastic but at the same time so radically different that it seems like it
would collide with a lot of built in assumptions of Linux development.

For example, does nix play nice with CMake or config files? ie. I can just
point them at the packages in /nix/ and it'll work?

Or for example, I often find myself wanting to have a few different version of
OpenCV around (with different build flags) - but all version 3.0. Nix seems to
make a new hash/folder when the dependencies change. Can you make make a new
build/folder when the dependencies stay the same? (or specify to NOT make a
new folder and to overwrite the previous compiled version - ex: you fix a bug)

I have a ton of overlapping dependencies between contracts and projects and
this would be a cool tool to keep it all organized (right now in my workflow
each project/contract would have it's own copy of OpenCV.. which is a bit of a
mess)

(I'm totally cheering for this - hope it becomes mainstream, just wondering
how "prime-time" is it)

~~~
cwp
Yeah, there's a learning curve. No doubt about that.

It does sound like nix is a good fit for your use case. You could certainly
create separate environments each with a slightly different build of OpenCV.
They'd share dependencies where possible, but not step on each other where
they differ.

Nix provides a tool called the standard environment (stdenv), which makes
compiling classical unix software pretty easy. It assumes the typical
download, configure, compile, install sequence and provides the right
environment to make that work. It's really flexible, with lots of hooks that
let you override the default behaviour and customize the build. The Nix manual
has a pretty good overview of it. [https://nixos.org/nixpkgs/manual/#chap-
stdenv](https://nixos.org/nixpkgs/manual/#chap-stdenv)

~~~
optforfon
oh, very interesting! I'm definitely going to look into this further. Thanks
for the pointers

------
bduerst
While this isn't a solution for many, I usually do development on a Linux VM
for these exact reasons. Something not working? Boot a new VM from scratch,
install necessary packages, and run it.

Usually faster than a couple days of internet searching.

~~~
dman
That involves compromises on battery life. Why not just run Linux instead?

~~~
superswordfish
This involves compromises on battery life.

~~~
dman
Well played :)

------
nickbauman
Ron is right: it's a mess. But I have a tried and true recipe for app engine
python and java both. First, download the Linux cloud SDK (works fine on a
mac). Then download the python google appengine SDK. Put the appengine SDK
inside the _platform_ directory of the cloud SDK. Add this to your python
path.

I use this framework for Python:
[https://github.com/agostodev/substrate](https://github.com/agostodev/substrate)

I use this Clojure template for Java: [https://github.com/nickbauman/cljgae-
template](https://github.com/nickbauman/cljgae-template)

------
userbinator
I think the real problem here is the rise of huge libraries/frameworks/SDKs
along with all the dependencies they bring, and the dependencies _those_
dependencies need, etc. The problem isn't in code you wrote and should easily
be able to debug, it's in the orders of magnitude more code you _didn 't_
write but depend on. There's too much complexity, too many moving parts, and
it creates a fragile system in total.

But I think it doesn't really need to be that complex and involve so much yak-
shaving. Around 10 years ago, when I wanted a personal web app of the similar
CRUD type, I opened a text editor, browser, and the PHP documentation, and it
was done in a few hours. Uploading a single file to the server was all that I
needed to get it working and test it.

 _I thought I’d write this article about my experience, to try to draw some
learning from it. I created the folder and the file, typed in the YAML and the
first paragraph, and started Jekyll to build the html as usual._

Would it have been easier and faster to just write the HTML directly and
upload it to the server instead of requiring an elaborate (despite hidden
behind one command) "build" process that again depends on a large
infrastructure of software?

 _Me, I just want to write programs, mostly small, that do things that may be
worth writing about. And I want to write articles like this one, convert them
to HTML, add them to a few indexes, and put them on my site_

My advice is to avoid all the complexity and big systems that claim to do just
about everything while requiring plenty of configuration and dependencies, if
all you need is something that could be much simpler. Otherwise you are
"making more yaks to shave".

------
twic
One lesson i had beaten into me at an old job is that software should come
with a self-contained reproducible build with an absolute minimum of external
dependencies (which might be none at all). If GAE provided a zip file with
everything in it, all set up and ready to go, you could download that and get
working.

But making a self-contained build is _hard_ , so nobody ever does it.

Java has got usefully close in recent years. You can download and build a
Gradle project with only a JDK and a shell installed, plus HTTP out to the
internet; the project can (should!) contain a standard wrapper script which
can download the right version of the build tool, and the build tool can then
download the right versions of the dependencies and do the build. The
downloads are cached locally so you don't download the internet on every
build. The wrapper script only does straightforward stuff, so it doesn't
require a particular shell or version, and the JVM changes slowly and safely
enough that you can get away with being pretty sloppy about versions (ie any
Java 8, 2014 - 2017+, will do).

There's no dependency on having some SDK installed, or some version of the
runtime, or some build tool, or some package manager, or any other bits and
bobs. Just a shell and a JDK. You don't have the problems listed in the
article, and you don't have to invest time in avoiding them.

The next step would be for the wrapper to download the JDK itself, but i think
that's unlikely to happen, due to bandwidth and licensing.

Once you're in the build tool, there are strong conventions about what's
where. The 'build' task should do everything; 'check' should run every kind of
validation possible; 'assemble' should build every artifact. Ideally, none of
them should require any setup, although that requires some discipline on the
developer's part.

I don't know of any good way to set up things MySQL or RabbitMQ if you need
those for integration tests, which is definitely a problem. Perhaps we need a
Docker plugin, so in my build i can say "to run the integration tests,
download and start the following images, and inject the container IP addresses
into the tests like so". The usual way to get around this is to use pure Java
in-memory options for integration tests (eg H2 for a database, maybe something
like HornetQ for a message queue), and you can get those like any other
dependency, with no manual setup needed.

Sorry for the ramble. tl;dr yaks are bad, but there is hope for canned pre-
shaved yaks.

~~~
jwhite
Haskell's stack tool is aiming for something like this. All you need is the
stack binary. stack will download GHC for you at a given version, and fetch
your dependencies for you. The dependencies are locked to a fixed snapshot of
versions. There are curated snaphots, with an LTS model that guarantees all
packages in the snapshot work together, so you're insulated from package
authors randomly breaking API compatibility on you, and you can upgrade to
newer LTS snaphots when you're good and ready.

[http://docs.haskellstack.org/en/stable/README/](http://docs.haskellstack.org/en/stable/README/)

~~~
rspeer
I'm starting to use stack for the one thing I do in Haskell. Is it really true
that all you need is the stack binary? No shared libraries or anything?

When I set it up starting from Ubuntu packages, the process was still
convoluted and involved bootstrapping from an older version of ghc.

~~~
jwhite
It seems true on OS X and Linux. I'm not sure if it's 100% statically linked,
or if it depends on a particular version of glibc on Linux. Edit: I would
check, but I'm not close to a linux box.

I haven't tried it on Windows.

The download instructions say it depends on libffi, libgmp and zlib, which I
didn't realise, plus some other tools (gcc, etc.) which I assume are for
building packages that wrap C libraries.

[http://docs.haskellstack.org/en/stable/install_and_upgrade/#...](http://docs.haskellstack.org/en/stable/install_and_upgrade/#linux)

------
robohamburger
This is why I have always been scared of GAE. Seems like the only thing that
works precisely like GAE is GAE :)

True unit tests though ought be runnable outside GAE since your code should
not be coupled with GAE if it is unit testable. You still need to test your
code correctly integrates with google app engine.

I would take a look at tox and py.test for testing in python, these are what I
use in my normal linux environment.

~~~
nickbauman
GAE has some rough edges, to be sure. But once you know about them, the really
hard stuff is dead easy to test. The official test stubs are awesome to work
with. I live or die by them: they've never let me down. If it works with the
stubs, it has always worked with the real thing.

~~~
robohamburger
That is good to hear! I will check out GAE next time.

------
pnathan
I have things to do that don't include wasting days of my life because Bozo
the Developer updated something and expected downstream to just cope with the
issues. Certain communities live on that behavior; its a waste of my time
churning.

I would like to point to the Rust developers as ones fostering a more
deliberate culture with tools and practices that manage change without
exploding downstream.

~~~
reitanqild
Java is quite nice this way as well if you can get away with it.

------
phlakaton
I don't want to derail a splendid rant on yaks, but for GAE users running into
this little problem of how to test and run:

We use OS X for development, for the most part, but Linux works well too.

We use nose2 with the following plugin to run unit tests:
[https://github.com/udacity/nose2-gae](https://github.com/udacity/nose2-gae) .
It assumes your GAE SDK is installed at /usr/local/google_appengine.

For API tests, we generally create a webapp2.WSGIApplication object during
setup, configured with the routes we want to test, and we send webapp2.Request
objects to that application using request.get_response(wsgi_app).

Hope that helps!

------
SolarNet
One of my jobs as the person usually responsible for setting up and writing
the core bits of new projects at my company is to make a build that _just
works_. All most all of our core (e.g. serious) projects consist of the
following steps:

* git clone

* setup.{bat, sh)

* Open IDE of choice.

* Click build button. (or run make)

The only dependencies are git, a default PC OS install (Linux, Windows, Mac),
and your IDE of choice. Any project or system that can't provide that easily
is too complicated by far, our non-technical CEO and sales staff can build and
run our code without any help.

------
mikelward
I tried to reproduce to file some bugs, but the docs Ron linked to and the
datastore_test.py[1] they in turn link to don't do "import webtest", so I
guess the docs have been updated recently.

A quick search suggests webtest is part of Pylons[2]. But Pylons isn't listed
as a built-in third-party library[3]. So I'm not sure if there's anything to
fix.

1\. [https://github.com/GoogleCloudPlatform/python-docs-
samples/b...](https://github.com/GoogleCloudPlatform/python-docs-
samples/blob/master/appengine/standard/localtesting/datastore_test.py) 2\.
[http://docs.pylonsproject.org/projects/webtest/en/latest/](http://docs.pylonsproject.org/projects/webtest/en/latest/)
3\.
[https://cloud.google.com/appengine/docs/python/tools/built-i...](https://cloud.google.com/appengine/docs/python/tools/built-
in-libraries-27)

~~~
X-Istence
The Pylons Project is a larger organisation that maintains a lot of open
source software, including webtest, Pylons is ALSO a web framework, but that
is also unrelated to webtest.

In the authors case, webtest should have been installed separately, rather
than using it from within the cherrypy source code, which seems to vendor
webtest.

------
swagtricker
Yep. There's a reason after 20+ years that I bought this T-Shirt:

[https://teespring.com/maksesoftwarebetter2](https://teespring.com/maksesoftwarebetter2)

~~~
RowanH
Great find. Okay so if 5 other people who feel this aptly visually depicts
their daily lives and could click reserve on the hoodie version I would be
stoked ! :)

~~~
epimetheus
I signed up for notifications, it's available now, for 3 more days.

------
mcguire
Before you start trying to draw conclusions about the general state of
development, consider that Mr. Jeffries...

" _Some further very simple tweak, which I no longer remember, and the test
ran … and did nothing. That’s not too surprising, I guess, because there’s no_

    
    
        if __name__ == '__main__':
    

" _which, I believe, means that no one calls the testrunner._

" _You’re supposed to know that!_ "

...is also the fellow who had some trouble with Sudoku[1].

I say this not as an attack on Mr. Jeffries, but as a way of pointing out that
shaving yaks is easier if you know to start with shears and not a hatchet.

[1]
[http://ronjeffries.com/xprog/articles/oksudoku/](http://ronjeffries.com/xprog/articles/oksudoku/)

~~~
tamana
If I read that right... that Sudoku sllver only got as far as becoming a
Sudoku board API, aka a object wrapping an array?

And the article spends more bytes talking about whether some code is YAGNI,
than the amount of bytes in the code itself?

~~~
mcguire
As far as I know, yes, that's where he gave up.

------
Tyr42
I don't want to be harsh on the buy, but some of those dark corridors were in
fact marked as not where you wanted to go.

>To eliminate this warning, please install libyaml and reinstall your ruby.

This line should make you think of using brew or apt-get or whatever to grab
libyaml first. A few clues, the lib prefix is much more common for native
libraries rather than interpereted ones. Second, the request to re-install
Ruby means that this is a dependency that is required at compile time for
Ruby, which rules out anything you'd install as a gem.

But maybe I just have shaved too many yaks and know what what already.

~~~
tamana
I wouldn't expect anyone to know those details, but I would expect someone to
follow the directions if they want it to work.

------
justinlardinois
I can echo GAE being hard to work with at times.

I did an app for a school project and was using Flask. At the time App Engine
only had webapp and web2py available as frameworks (maybe it's different now)
so a bit of shoehorning was necessary to get Flask working, and Flask was
relatively young at the time so there weren't a lot of examples out there of
the two technologies being used together. I recall having a lot of trouble
working with GAE's file upload API.

------
X-Istence
As the current maintainer of WebOb it pains me to see Google App Engine
shipping 1.1.1 by default, and 1.2.3 as one that the user can select.

------
pcr0
I'm glad Docker solves this for me, at least at the company I just joined. One
of their microservices has tons of dependencies, including npm modules,
databases, testing frameworks, and so on. Most people would scoff at using
Docker as a glorified VM, but it honestly does the job, with much less
overhead vs VMs.

------
iLemming
yeah it's a mess. Modern developer has to know and understand language
semantics and ecosystems of Python, Ruby, Java, C, Lua, Javascript and
sometimes these are not the main languages of her work. Add to it things like
Clojure, Erlang, C++, C#,F#, Purescript, Typescript, Elm, Mathlab... you get
the idea

------
pbackx
You Python developers have it easy :) Developing in Java on AppEngine adds an
additional herd of yaks to the meadow.

~~~
skybrian
The Go variation seems pretty good, though. (Or maybe I've just gotten used to
it?)

~~~
rspeer
I just recently installed a piece of software written in Go, from source, on
Ubuntu 14.04. It wasn't so nice for a non-Go-programmer.

The instructions wanted me to download a binary tarball of Go and untar it
directly into /usr/local. As if I would never need to uninstall it. Apparently
even though the version of Go has changed several times since Ubuntu packaged
it, they think this will be the last version ever, or something?

I ended up hunting for an Ubuntu PPA that had it instead. Of course all the
accepted answers on Stack Overflow point to PPAs that don't exist anymore. A
comment sitting at some tragically low score saying "hey, try this one
instead?" pointed me to something that worked.

I think you dealt with it the first time and you've gotten used to it. (And
maybe there's some nice way to get a newer version of Go when you already have
Go.)

~~~
skybrian
I'm not sure I understand the problem. Renaming the old directory to go.old
and installing a new one seems pretty easy to me, but apparently not?

BTW, you can put the Go SDK anywhere you like, as long as you set GOROOT. See
"custom location" on
[https://golang.org/doc/install](https://golang.org/doc/install)

~~~
rspeer
For reference, I'm referring to the directions you get at the top of the
search results when you type "installing go on ubuntu" into Google.

The directions link to Stack Overflow, though I don't know how the search
result picked out these steps, as Stack Overflow has a different accepted
answer that doesn't work. Intervention by Google, possibly?

I didn't realize it would only make one directory. That is reassuring. I would
still prefer packages as a way to install, upgrade, and remove Go, instead of
remembering months from now that Go is installed differently from everything
else and there's this directory I need to mess with.

Now, that all would have been more intuitive as something to do in my home
directory. If it only needs one directory and it can go anywhere, why would
these Google-blessed directions tell me to mess with /usr/local? I believe
you, I just assumed that people would be averse to messing with system files
without a package if there were any other option.

(I think of Python, where the one thing you're supposed to do with your system
Python is use virtualenv to install a non-system Python. Which is another
example of something that's only clear to veterans of the language.)

Now... this is a problem with a whole lot of programming languages.
Programming languages change. Their libraries change way too fast for an OS-
approved package manager to keep up, so you need a package manager for your
language. And that changes too, and nobody designs their packaging system to
smoothly update to a better packaging system designed by someone else.

Maybe one day every programming language and OS will unite under one purely
functional packaging system and never replace it... haha, no.

------
digi_owl
Seems to me the problem is layers of "automagical" systems for locating
dependencies. That in turn end up clobbering each other in various ways.

------
dkarapetyan
This is nothing. Ramp-up time at any software shop with more than 2 engineers
is months. He had a really easy time from where I stand.

~~~
BurningFrog
You need to get into better software shops!

~~~
dkarapetyan
They don't exist. Software is an awful business.

------
hellofunk
It really is full of yaks! [0]

[0]
[https://github.com/search?q=yak&type=Code&utf8=%E2%9C%93](https://github.com/search?q=yak&type=Code&utf8=%E2%9C%93)

------
mkesper
MacOS does not seem to be very developer friendly.

~~~
pyre
I'm curious what the failure of MacOS was in this instance. This comment at
face value seems like you're trolling.

~~~
x0x0
The intermixing of macos python and brew installed python has been painful for
me in the past.

Specifically, I spent hours and hours trying to fix

    
    
       import matplotlib.pyplot
    

producing

    
    
       ImportError: cannot import name _thread
    

but only in ipython, not the interpreter. (HAHAHAHA fuck you developers).

A ton of googling and swearing produced the fact that a package named six that
does god knows what was being read out of

    
    
       /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc
    

instead of

    
    
       /Library/Python/2.7/site-packages
    

rm -rf'ing the former path fixed things, but fuck only knows what else I may
have broken. Just because I want to plot things from ipython.

ps -- 6 months ago the above used to work. Why did it stop? I have no bloody
idea. But this consumed multiple hours of my life.

~~~
mason55
Seems like this would have been straight forward with a virtualenv?

~~~
geofft
Yup. Really, no OS is developer-friendly if you're going to mess with the
system Python interpreter, because the system Python interpreter exists to
serve system software written in Python. You can easily get yourself into very
similar problems with Debian or Fedora or whatever else.

Install a virtualenv, and on OS X, maybe install Python from python.org, and
then everything works amazingly.

~~~
x0x0
After I learned more that was probably the right thing to do, but I'm not a
python dev. I just want to use sklearn, pandas, and associated tools. I'm not
sure what the smooth path to doing that is. At various times people recommend
macports, brew, anaconda, and so on... Note that compiling numpy from source
sucks, and I'd much rather someone else do so.

~~~
geofft
There are now wheels (binary distributions of Python modules) of numpy, scipy,
and scikit-learn for both Linux and Mac OS X, so it should just be a matter of
`virtualenv /tmp/v` (or wherever), `source /tmp/v/bin/activate`, `pip install
-U pip` to make sure your pip is new enough to handle wheels, and `pip install
numpy scipy scikit-learn`. Nothing gets built from source. Unfortunately the
same can't (yet) be said for pandas; `pip install pandas` at this point will
try to compile it.

Actually _acquiring_ virtualenv on OS X seems not to be super straightforward,
especially if you want to use the system Python interpreter. The best option
seems to be to `brew install python`, and then `pip install virtualenv`. Or
download Python from python.org, and maybe get Python 3 while you're at it.

~~~
solotronics
one way is to get a standalone python and git clone virtualenv, then use the
standalone python to make virtualenv for each project.. this leaves out
anything to do with the system python

