
Replacing Python - antics
http://roscidus.com/blog/blog/2013/06/20/replacing-python-round-2/
======
tikhonj
I think the important insight from these articles is not which language the
author ends up using (OCaml[1]), but rather which languages he's managed to
rule out by now. In particular, I think it's good advice to avoid both ATS
(however much I like dependent types) and Go, both for completely different
reasons. You probably wouldn't want to use Rust in the short term either.

[1]: [http://roscidus.com/blog/blog/2013/09/28/ocaml-
objects/](http://roscidus.com/blog/blog/2013/09/28/ocaml-objects/)

ATS makes everything more difficult than it's worth except for some very
specific use cases. Unlike some of the other languages like Haskell and OCaml,
I don't think this is just because it's different; rather, it's because ATS is
simply so much more demanding. It gives you extremely solid static guarantees
and high performance, but it's a tool that really sacrifices expressiveness
and programmability to get there. Haskell and OCaml may be difficult to
_learn_ , but ATS is actually difficult to _use_ \--a very different concept.

Go, on the other hand, is neither difficult to learn nor difficult to use.
But, more generally, Go has very little to commend itself and--compared to the
other options--quite a few shortcomings. If you're willing to put in a little
bit of effort to learn something new, there are a whole bunch of options which
are simply better. And you _should_ be willing to put in the effort: your
programming language is your single most important tool; it affects not only
how you write your code and how you maintain it but even how you _think_. So
it seems extremely shortsighted to choose a language because it's easy to
learn and similar to what you already know!

Rust is awesome, but it's simply not ready yet. This is widely acknowledged by
everyone in the project, and is part of their policy of open development. Once
it _is_ ready, it will be a very good choice for certain domains.

~~~
pron
I think you may be overestimating the importance of the programming _language_
over the whole programming _environment_. I think it is the programming
environment that is the most important tool – not just the language. There are
often many requirements that might lead you to choose a possibly inferior
language because it lives in an altogether superior environment for your
needs.

I have been writing large, server side software for many years. These are
long-running applications that require superb performance as well as first-
class monitoring tools. So, for me, one requirement for a language is that it
runs on the JVM. The JVM gives you superb performance as well as unparalleled
monitoring tools (made even better by the inclusion of the flight-recorder and
Java Mission Control in the latest version of the JVM); you also get a vast
ecosystem with a huge selection of very high quality libraries (and some other
important features like dynamic linking) – all that before you even choose a
particular language. You can also buy commercial support for every single
component you're using if you need it.

On the other hand, if you're writing a command-line tool that needs a very
fast startup time, or if your RAM is very constrained, then the JVM might be a
bad choice for you, which would rule out the JVM languages no matter how good
they are.

My point is that there are more important issues to consider than the mere
merits of the language itself.

Also, you're putting a lot of emphasis on expressivity, while downplaying the
importance of a short learning curve. I think the two are of similar
importance. It's been my experience that programs written in some of the more
expressive languages are actually harder to maintain than code written in the
easy-to-learn ones. The reason seems to be that programmers use the
expressivity of the former (which seems to be of a particular nature or
natures: functional construct, higher-order types, meta-programming) to model
their own thought-process, which may be hard to replicate for someone else
maintaining the code. The easier-to-learn languages usually work at a lower
level of abstraction, which can sometimes be easier for others to follow
because its a well-understood common denominator.

Now, I'm not saying that worse is better, or that more "primitive" languages
are better, or even that one should pick them over more expressive ones. I'm
just saying that, especially when working in a large team or writing code that
would need to be maintained for a long time, there are other issues to
consider.

~~~
fauigerzigerk
_> The reason seems to be that programmers use the expressivity of the former
(which seems to be of a particular nature or natures: functional construct,
higher-order types, meta-programming) to model their own thought-process,
which may be hard to replicate for someone else maintaining the code._

I agree with that. Working on code in a large and ever changing team is very
difficult, and it seems to benefit from less abstraction, even if it leads to
more verbose code. That's the whole secret of Java. I've been implementing a
small part of my current project in Go and it has similar benefits.

That said, when working alone or with one or two others, I feel kind of dumb
repeating some things over and over at the same level of abstraction. There
are lots of things like this little piece of C++ code that I tried to
translate into reasonably efficient Go:

    
    
      template<typename V, typename R>
      R defaulted(const V& val, const R& default_val) {
        return val == V() ? default_val : val;
      }
    

I came away scratching my head a little, probably because I'm a Go newbie.

~~~
pron
As an aside, Go is also "a Java" in the same sense that Clojure is a lisp
(similar syntax and concepts, same level of abstraction, same philosophy). In
fact, Google seems to be turning out many of these "Javas" lately (Android,
Go, Dart).

While Java and its variants' appeal at most large organizations is considered
by some a sign of their conservatism, Google is anything but when it comes to
picking the best tools. Google's fondness of Java is evidence of the merits of
the Java philosophy when it comes to maintaining large codebases by large
teams.

Creating a language that follows this philosophy while still boosting
productivity (and maybe providing other benefits as well) is an interesting
challenge. I think Go falls short. Kotlin looks interesting.

~~~
berntb
I always thought the Java love at Google came from needing _speed_ more than a
startup -- they automatically will get factors of ten more users directly for
most everything they do. (Edit: Point is, "Java-like" is Google's optimum for
other reasons than being conservative.)

~~~
pron
Sure, performance is a big requirement for them, but there are other languages
with good performance and better expressivity (like Haskell and Scala), yet
they are not used at Google, and Google's new languages did not adopt their
philosophies.

~~~
nostrademons
Google uses Java because they were able to pick up a lot of really skilled
Java developers during the 2001-2004 recession, and those devs built many of
the products that were introduced in 2004-2007. Once a product's been built
and adopted by the marketplace it's _very_ difficult to change the
implementation language.

Most of the devs who were hired at Google from 1999 - 2002 still prefer C++,
and products built in that era (Search and much of the infrastructure) are
still in C++. In general rewrites from C++ -> Java have not gone well; I know
at least one such frontend that was rewritten back in C++ a year later.

~~~
pron
I'm not sure I get your point. Are you saying that Google's new programming
languages adopt the Java philosophy because of developers they hired over ten
years ago? Also, I'm not sure I understood what you were trying to say about
C++ at Google.

~~~
xerophtye
I think his point is that the choice of language at google isn't exactly a
managerial or strategic decision. They just happen to have a lot of java devs
(probably at senior positions now, since they were hired a long time ago).Even
if they have gone out, if most of your stuff is made in a language, it would
be really difficult for new comers to shift that paradigm to some other
language.

The part about C++ was to describe a similar scene. When they had mostly C++
devs, their language of choice was C++. and now they have mostly java devs, so
java it is!

~~~
berntb
pron's point was [~ mainly] about the _new languages_ from Google.
nostrademons wrote mostly regarding what I wrote (and why it was
wrong/simplified).

------
jzwinck
Perhaps the title should be "Why Python is good enough." Between this post and
the preceding one ([http://roscidus.com/blog/blog/2013/06/09/choosing-a-
python-r...](http://roscidus.com/blog/blog/2013/06/09/choosing-a-python-
replacement-for-0install/)), the takeaway for me is that the author didn't
find sufficient benefit in any of the candidate languages to justify switching
an existing product--perhaps not even a new product.

It's true that Python can leave you with hidden crash bugs in unusual code
paths, but the language brings so many benefits that it's easy to forgive
this. And there seemed to be a vague concern about performance at the
beginning of the article, but I didn't see anything substantive in that
regard.

So yay, let's all just keep using Python. Back to work.

~~~
zanny
The other thing is his only complaint (slowness) isn't really a problem. If
you have a critical hot spot in your server code, extract it, implement it in
C / C++, and use ctypes to call it.

~~~
thinkpad20
Having recently started doing python full-time, I would say that by far the
biggest problem with python is its propensity to explode in your face. Code
written in python seems so fragile; except for the most blatant bugs like
syntax errors or too few arguments, nothing gets caught. You end up having to
write tons of unit tests, which often expose bugs that would be trivially
caught with any kind of typing. Performance is great, but code reliability is
much more important. With python you get neither - just rapid development
(slowed down by having to write a ton of tests) and expressiveness (which is
excellent).

~~~
gtaylor
Don't let its ease of use fool you, there are a lot of ways to shoot your foot
off in Python. Typically, it's kind of like pulling the trigger (make the
mistakes), only to see your foot get blown off a few months/years down the
road. I find that your luck with larger Python projects is largely determined
by two things:

* Experience with Python, and general experience in duck typed environments.

* Discipline. Not just your own, but anyone else working on your project.

There are certain mistakes that will make your life miserable down the road.
These could be bad organizational conventions, inconsistent exception
raising/handling policies, lacking a strategy for documentation, etc. If you
make mistakes like this, you probably won't get bitten by it until later.

Writing good Python can be harder than writing good <X Language with more
rigidity>. With experience and time, this gets a lot easier.

------
tghw
The first Python example throws up a big red flag: manually parsing command
line arguments instead of using argparse. Why reinvent the wheel when there is
a standard library to handle it?

Likewise, asserting that "For storing general records, Python provides a
choice of classes and tuples" completely ignores one of Python's fastest and
most powerful data types: dictionaries.

Lastly, in get_value, the nest of ifs is unnecessary and makes the code seem
more complex than it really is.

~~~
Erwin
Over the last decade, there's probably been half a dozen "standard" command
line parsing tools. getopt being the classic, then we had optparse and that
got deprecated, then we got argparse. And there there's e.g. twisted.usage and
the pretty cool docopt.

In this case the author wants to parse "program <foo | bar | baz> [optional
args]" and does that in a few lines of code. You seriously think that's a "big
red flag" ?

Do you also believe the author does not (despite having created the zero
install tool 10 years ago) know Python has dictionaries?

This is a blog post where the author evaluates a number of
languages/environment to replace his particular Python needs -- do you think
that him not using a new standard library module invalidates his finding?

~~~
jzwinck
We've had optparse since at least 2003; programs that don't use standard tools
to parse their command lines usually have lame bugs such as not responding
properly to "\--help" (must do nothing but write to stdout and return 0 on
*nix).

I'm not the one who first mentioned that hand-written option parsing is a red
flag. But it is one--especially in Python, which has nice things built in.

~~~
talex5
The actual 0install Python code does use optparse.

It wasn't stated in the post, but the next step after writing this code was to
use it as a front-end to the real Python version. If invoked as "0install run
NAME ARGS..." exactly then we handle it (the fast path), otherwise we fall
back to the Python version. In particular, that means that the Haskell/OCaml
version must NOT try to handle --help, etc. I wrote the comparison Python code
to be similar to the other languages.

This OCaml front-end appeared in 0install 2.3. For 0install 2.4 there is a
full option parser written in OCaml. I didn't use a library for this because
a) it needs to be 100% compatible with the Python parsing and b) it needs to
handle tab-completion too.

------
njharman
I do not understand at all author's statement that OCaml datastructures are
big win over Python's

    
    
      let {cache; config} = b in
      print_endline (String.concat "," cache);
      print_endline (String.concat "," config)
      ;;
    

vs, actually I'm not even sure what the ocaml is attempting, some variation of

    
    
      print '%s,' % b.cache
      print '%s,' % b.config
    

Which, in Python, if your printing more than a few should be

    
    
      print ',\n'.join((b.config, b.cache, b.data))
    

Or if want all fields and they are in correct order

    
    
      print ',\n'.join(b)
    

>> The syntax [of NamedTuples] isn’t great, though, and you can’t do pattern
matching on the names, only by remembering the order:

Other than misuse of term "pattern matching", that statement is true and is
trivially overcome with two line function, one line lambda, or once and for
all by subclassing NamedTuple. Here's the function variant:

    
    
      def GimmieThing(keyword args in any order)
         return ThingNamedTuple(args in correct order)

~~~
anaphor
How is it a misuse of the term "pattern matching" ?

~~~
njharman
Generically
[http://en.wikipedia.org/wiki/Pattern_matching](http://en.wikipedia.org/wiki/Pattern_matching)

And as for a language / call semantic look at Erlang for what actual pattern
matching looks like.

~~~
anaphor
That doesn't contradict the use of pattern matching in OCaml.

~~~
njharman
Both the OA and myself are talking about Python.

------
hartror
> A major benefit of OCaml and Haskell is the ease of refactoring. In Python,
> once the code is working it’s best not to change things, in case you break
> something that isn’t covered by the unit-tests. In OCaml and Haskell, you
> can rename a function, delete old code or change a data structure and rely
> on the compiler to check that everything’s still OK.

These sort of statement always bothers me when I encounter it. If you're not
testing the code you are refactoring how do you know it still works or even
worked in the first place?

Static typing gives you useful things (with a trade off), not having to write
tests isn't one of them.

~~~
tikhonj
He never said anything about _not_ writing unit tests. Rather, his claim is
that--even _with_ unit tests--refactoring is not safe in Python et al; in
Haskell and OCaml, by contrast, many of the same actions are guaranteed safe
by the type system.

Static types do not mean you don't have to write unit tests. But they _do_
mean you can write _fewer_. (And, looking at libraries like QuickCheck, a
static type system can make writing tests _easier_.)

Quite a bit of the safety you get in Haskell especially comes down to
controlling effects. Mutable state implicitly couples all the code in a given
scope--unless you've read and understood all of it, distinct parts might have
effects on each other that you're unaware of. In Haskell, on the other hand,
this is impossible, so refactoring like extracting variables and reordering
your code can be entirely safe even if you _haven 't_ looked at exactly what
the code does. The refactoring actions are guaranteed not to change the
semantics of the code by the language itself.

The best way to think about it is that refactoring becomes a purely syntactic
action. You just remember a few rules akin to algebra, and you can rewrite
Haskell code in a bunch of different ways all preserving the original meaning.
Regardless of what that meaning really is. This also extends to library code--
most good libraries come with algebraic _laws_ , which ensure you can rewrite
their code in logical ways. Once you learn these laws, you can start doing
things like changing multiple passes over a datastructure into one with the
same confidence.

If you make a change that _could_ break things, the type system will often
help you find every place that _does_ break. This extends beyond just type
mismatches: Haskell also ensures that you consider every possible case in your
functions; if you forget an option, it will give you a warning. This means
it's safe to add a new alternative to an existing type: you will get a warning
everywhere you haven't considered this new option.

I've found this to have a profound effect on how I program. With Haskell, I
actually follow the rule of making any code I visit look better than before,
simply because refactoring has very little mental overhead. I can move things
around liberally, break them into multiple modules, condense them into fewer
functions and even change the types, knowing that any mistakes I make will be
caught by the type system.

~~~
njharman
> his claim is that--even with unit tests--refactoring is not safe in Python
> et al;

Well that claim is False. I and thousands of other Python developers disprove
it daily. Just because they can't refactor dynamic languages doesn't me we
can't. Something not-dynamic is probably the language for them. And Python is
the language for us.

In other words there is far more variability between developers than there is
between languages. Find the language(s) that work for you and quit believing
they are the languages for everyone.

------
gmantastic
Cython might also be worth considering. It lets you add static type
declarations to Python and produces compiled code. You could avoid a big-bang
rewrite, migrating the slower operations first. I don't know whether it would
meet all your cross-platform requirements though.

[http://cython.org/](http://cython.org/)

~~~
wikiburner
Isn't Cython a solution that frequently requires significant code changes
(refactoring to C-like code) beyond adding typing?

~~~
gmantastic
It's selectively adding C-like type declarations to Python code speed up the
slow bits. C code is then generated and compiled into a Python extension
module. This is similar to adding type information to Common Lisp code to
allow the compiler to optimise it.

[http://docs.cython.org/src/quickstart/cythonize.html](http://docs.cython.org/src/quickstart/cythonize.html)

~~~
wikiburner
My impression was that you sometimes have to refactor your Python to make the
code more C-like. From what I have heard generators aren't supported, and
there are other situations where you need to make your loops (and other
elements of your code) more C-like.

That's not the case?

You just add typing to your python and you're good to go?

~~~
gmantastic
There's some info on semantic differences here:
[http://docs.cython.org/src/userguide/limitations.html](http://docs.cython.org/src/userguide/limitations.html)
\- it doesn't bring up much, and the design goal is full language
compatibility. I've only played with it so far, but it's case of optionally
adding type info, and also writing some distutils scaffolding to say how to
build the module as a cython extension. This is then callable from Python via
the regular import mechanism.

~~~
wikiburner
Thanks for the link. I was just poking around the Cython site myself, and it
looks like they've come a long way since I last looked into it. It seems that
they're getting much closer to full compatibility.

It makes me wonder if we're closing in on the day when Cython gets rolled into
Python proper, and you can just _" import static-types"_ to activate optional
typing, then add some static typing to your code, and you would get Java
performance from your Python code.

~~~
ris
"and you would get Java performance from your Python code."

Nah, I doubt it would ever use _that_ much memory.

~~~
wikiburner
:) I guess what I meant to say was Java speed (or greater) Python.

------
ssadler
I find it suspicious that Java or Scala didn't make the list. Java may not be
sexy but it checks many boxes... And I suspect that Scala would have been a
serious contender in brevity too.

~~~
mynameisme
Yeah, it's a joke that Rust (which I think will be a great language, but all
of the syntax hasn't even been decided yet) is being considered, when Java and
friends are not.

Really, this article is about "I want to rewrite my program in the newest,
coolest language", not about which is the best tool for the job. And that's
fine, but the author should present it that way.

~~~
mjhoy
> newest, coolest language

I'm sure you meant "newest and/or coolest", or "recently become coolest", but
it's worth noting that Haskell is about as old as Python, and OCaml is not
much younger.

------
abraxasz
A quick comment concerning Haskell. I've been learning it on and off, with
some help from my roommate who's a Haskell genius. The way he writes Haskell
always amazes me. He starts with a very straightforward verbose version, then
constantly refactors it (on the fly, it's not a separate step), abstracting
stuff out in typeclasses and monads until it seems that most of the code is
just monads and typeclasses definitions, and only a couple lines that seem to
actually do something.

I always joke around saying that if your Haskell code doesn't have typeclasses
and monads, you're doing something wrong. By which I mean that unlike other
languages, there's a huge, huge, huge gap between writing great haskell, and
regular haskell, with a very steep learning curve.

~~~
ijl
Interesting. I wonder about maintainability if the writer's "obvious" version
is only there to be refactored out. Can you still read the intent when he's
done?

~~~
abraxasz
I can't, by a long shot: I don't know Haskell deeply enough. But I think he
can read his code without problem.

------
dscrd
I like articles like this one, but this doesn't really bode well for 0install.

~~~
talex5
Any particular reason why? We just released 0install 2.4, which has around
10,000 lines of OCaml. It seems to be working well so far, but there are bound
to be a few bugs... we can always use more testers!

------
dchichkov
> You still have classes, objects, functions, mutable data and low-level
> access to the OS, all with an easy and concise syntax, but you gain type
> checking, much better data structures and a huge amount of speed for no
> additional effort. Why aren’t more people using it?

I think it is a community issue actually. Not the language. It is just not
converging. Just think of how many distinct OCaml code styles you have seen.
Some use hardcore math notation, some not; some use recursions everywhere;
some use imperative syntax; some use pseudo-OOP approach. Some even try to
fork the language syntax. And a result. Well. Even with the language itself
providing easy and concise syntax, ocaml _code_ is not very readable. And not
very popular.

------
nothingspecial
The Xen hypervisor relies on a curious mix of Python and OCaml.

------
brandonbloom
It seems pretty clear to me that Julia is gunning for Python's sweet spot.
Worth checking out.

[http://julialang.org/](http://julialang.org/)

------
_random_
Plus he will be able to use F# easily since it is derived from OCaml.

------
IgorPartola
Wonder if he looked at Nimrod. On paper that is a language that would both fit
his needs and would not be a far leap from his current Python implementation.

~~~
dmytrish
When I looked at Nimrod it seemed very raw to me: REPL malfunctions and
crashes, the whole language seems to be more an enthusiast effort than a
production-quality tool. Rust is very raw too, but it already looks pretty
solid.

~~~
dom96
Nimrod by default compiles to C. Its REPL is just an extra feature which as of
right now is not stable because few people use it. It for example doesn't even
support the FFI currently. As for whether Nimrod is production quality, I
would say yes. The compiler is written in Nimrod which is impressive in
itself. I have written many projects in it already and even though the
compiler is still not at 1.0 the projects continue to build with no (or very
little) errors between new compiler versions. Examples of my nimrod projects:
an IDE ([https://github.com/nimrod-code/aporia](https://github.com/nimrod-
code/aporia)), a build farm ([https://github.com/nimrod-
code/nimbuild](https://github.com/nimrod-code/nimbuild)) and a web framework
([https://github.com/dom96/jester](https://github.com/dom96/jester)). The
Nimrod forum ([http://forum.nimrod-code.org](http://forum.nimrod-code.org)) is
also written in Nimrod.

------
hershel
Anybody knows or have any reasonable guesses when Rust will be ready?

~~~
chrismorgan
I hope some time next year, but see recent discussion of it at
[https://news.ycombinator.com/item?id=6454455](https://news.ycombinator.com/item?id=6454455).

------
vph
Why is it necessary to replace Python for this project? And what are the
objectives for the replacement? It's not clear what the author's problems and
objectives are.

~~~
__float
This is all explained in the first part he so kindly links to in the third
word ;)

[http://roscidus.com/blog/blog/2013/06/09/choosing-a-
python-r...](http://roscidus.com/blog/blog/2013/06/09/choosing-a-python-
replacement-for-0install/)

------
anaphor
You could've avoided a lot of the "case of blah" nonsense in the Haskell code
by using Data.Maybe and Data.Either, just sayin'.

------
dancecodes
python is nice

