
Postmodern Error Handling in Python 3.6 - knowsuchagency
http://journalpanic.com/post/postmodern-error-handling/
======
jstimpfle
It's interesting to hear that Guido is now excited by types. A few years ago
his opinion was still that types can't catch most problems. I think I will
listen to that interview.

Personally I think types are over-hyped for most (profane) code.

The problem with _complicated_ and/or _specific_ types (which promise to catch
more errors) is, in one word, coupling. They create serious dependencies
across the whole project or even across project boundaries.

The other problem is: one functions-guy's URL is the next function-girl's
string. And all this up- and downcasting usually creates considerable noise.
Often overzealous typing is painting oneself in a corner, creating more pain
then relief. It's like creating these totally arbitrary object-oriented
inheritance-based taxonomies which might make some sense in one place of the
code but totally break down in the next place.

But speaking about _primitive_ types like _int_ and _str_ , or maybe _list of
int_ \- most function arguments can accept only one of those for the code to
make sense. So strategically placed type annotations can help reduce some
boilerplate and put some useful barriers for error search there. While the
simple-types cases are also the easy ones - the first thing you'll notice
while testing is usually that you put a _str_ where an _int_ was expected.

~~~
adwn
> _The problem with complicated and /or specific types (which promise to catch
> more errors) is, in one word, coupling. They create serious dependencies
> across the whole project or even across project boundaries._

With dynamic typing, you'll still have this coupling, but now it's unchecked
and (in practice) undocumented.

~~~
jstimpfle
Most parts of the code don't actually care about all aspects of the type (or
the value), but only that it's an object that they can hand to the next
function. Or they only care about a subset of the value's properties (like it
being a string, not necessarily a URL).

For example, in Haskell there is the filter function:

    
    
        filter :: (a -> Bool) -> [a] -> [a]
        filter _ [] = []
        filter pred (x:xs)
            | pred x = x : filter pred xs
            | otherwise = filter pred xs
    

It doesn't actually care what (the type of) a is. This is one of the showcase
examples of the power of Haskell's typesystem. (And admittedly it works nicely
in this case).

But you can have that easily with no typing at all.

    
    
        def filter(pred, xs):
            return [x for x in xs if pred(x)]
    

So, dependencies avoided and it's not a huge source of bugs. While in the
typed case, a system is needed to prove that the properties that get input
come out again (or possibly a function of these properties). That quickly gets
quite involved for more advanced cases.

Just look what the average Haskell code looks like, how many language
extensions are typically needed, how crippled the code typically is (libraries
dictating exactly what properties the client must be able to statically
precompute). Most of the Haskell community seems to agree that the end-goal
would be a practical system for dependent types (the unification of types and
values, i.e. no types at all ;->).

For example, there are some libraries trying to capture the construction and
execution of valid SQL queries in the type system. To understand these
libraries without doubt a genius-level IQ is required. But the client code is
still basically an unreadable mess. And the libraries are not reusable for
cases where the database schema isn't known at compile time. The static type
system actually _prevents code reuse_ \-- it makes the code really inflexible.

~~~
adwn
That's actually a great argument _for_ static typing! Let's say your distant
input component is changed and in some cases doesn't send lists of lists, but
one flattened list. With Haskell, the compiler will immediately notify you of
this bug, while with Python, you have to hope that your integration tests are
sufficiently exhaustive to catch this, or you'll catch it in production.
Whoops.

Like I said, the coupling is still there, but it's hidden by dynamic typing.

~~~
jstimpfle
In most cases you will not be able to write your code in Haskell in the first
place. Do try to write a database query construction kit that deals with
schemata read at runtime (a very reasonable requirement).

Even for a non-genius it's pretty easy to make something that works, with
dynamic types. [http://jstimpfle.de/projects/python-
wsl/main.html](http://jstimpfle.de/projects/python-wsl/main.html)

I don't think something like that can be created in Haskell (without resorting
to dynamic types).

~~~
Filligree
You always have dynamic typing as an escape. If that's what it takes, then
that's an option, but I think in practice you'd be able to keep that sort of
purity to a small portion of the program.

Haskell is a research language, which sometimes shows in the libraries. I'm
not sure if I'd recommend it for general use; I'm more partial to Kotlin these
days. It's definitely an interesting approach, though, and anyone who's
designing languages ought to make themselves familiar.

------
St-Clock
For our small team, we found that type annotations are useful as a substitute
of documentation and it cut our docstrings at least by half. For most small
and well-named API functions, we don't need docstrings anymore.

We're big fans of type aliases also. For example, API functions do not return
an int, they return a PK. They do not return a str (or Text), they return a
URL.

We're also heavy users of typing.Optional to mark parameters or return values
that can accept and return None.

We tried mypy but it failed miserably on our large codebase and frankly, with
our large test suite, we never had a typing error found in production so I
think it's not a good investment of our time.

I'm eager to test type annotations with our next intern to see if it speed up
the ramp up process.

~~~
lsh
Python type hinting and aliases in Python 3:
[https://docs.python.org/3/library/typing.html](https://docs.python.org/3/library/typing.html)

I've just finished porting a small application to Python 3 just because 2020
and deprecation is in sight already - there are so many things that have
passed me by in the Py3 world.

~~~
kzrdude
What I missed first was this deal with provisional API. Nice that it's widely
tested, but it should be clear that typing is provisional: Your program is not
future compatible if you use it.

Asyncio was provisional until Python 3.6, so it is effectively a Py 3.6
feature that has been backported to 3.5 in time and space.

------
demetri
I am mostly confused by this. The author started out with a friend's question
about an Either implementation and ended up with an enumeration and an option.

Enumerations are useful for a number of reasons in typed, compiled languages
(I guess they provide readability in Python), but they are not the same as
monads (of which Either is one type). I am not a Python developer, but as far
as I can tell, this implementation gets you none of the real benefits that an
a true Either implementation would get you in terms of functional goodness.

Am I missing something here? Is there some kind of Python magic that makes
this better than just propagating the exception?

~~~
kortex
I thought the same thing. There are so many other options in the Python
toolbox, Enum is far from the first I would reach for. Not sure what the
benefit is.

------
chimeracoder
> I’ll be the first person to admit I have no idea what postmodernism actually
> means

Well, if we want to take the title literally:

Error handling is a fundamentally Modernist idea - shoring up errors and
"correcting" or at least containing them.

Postmodern error handling would be anything that reasonably counts as a
reaction against Modernist error handling. The most basic would be not
handling errors at all, and instead embracing the chaos of an error-prone
system, and designing a system in which errors propagate but converge to a
desirable value.

~~~
adamcharnock
To add to this, AFAIK postmodernism is actually a critique of modernism rather
than a being a substitute for it. This critique may/will eventually produce a
concept which can replace modernism.

I found that enlightening, but it is probably also splitting hairs with
regards to the blog post.

~~~
bandrami
It's also important to remember that "postmodernism" is at this point about
130 years old.

~~~
chimeracoder
> It's also important to remember that "postmodernism" is at this point about
> 130 years old.

You're thinking of Modernism. Postmodernism is more like 60 years old,
depending on how you draw the line, and it's still an ongoing and developing
set of philosophies.

~~~
bandrami
The term was used in the 1880s in painting circles; it probably varies by
field

~~~
coldtea
> _The term was used in the 1880s in painting circles_

Not in any great capacity that culture in general was aware of.

Only in the mid-20th century, and especially post 60s, it became an actual
thing with the meaning we associate with it.

------
rspeer
There are cases where you could justify writing "except Exception as e" and
having it return a value, like if you're the author of Flask or Raven or
something that includes handling arbitrary errors in other people's code in
its job description.

But within your own code, you would not want to represent "an unforeseen error
occurred" with a value, no matter how much you like enums.

If you were parsing JSON and you got an unexpected error that isn't about
parsing JSON, logging the error and continuing is not the right option. There
is probably nothing reasonable your program can do. Raise the error so your
broken code stops running.

~~~
nicolaslem
I often find myself using "except Exception", while I don't like it, I cannot
find another way.

For instance when making an HTTP request. The requests library can throw
thousands of different errors (SSLError, BrokenPipe, socket errors, errors
from urllib, errors from requests itself...).

As far as I know, it is the only choice if you want to know if a request
succeeded or not so you can deal with it without your view returning a 500.

I would really appreciate if someone has another solution to this.

~~~
nitely
Kennethreitz requests lib? Just use requests.exceptions.RequestException, that
should handle every error, if not then report it.

For python urllib IOError and OSError should suffice. At least in py3.

~~~
jwilk

      import urllib.request
      try:
          urllib.request.urlopen('https://wrong.host.badssl.com/')
      except (IOError, OSError):
          pass
    

causes:

    
    
      ssl.CertificateError: hostname 'wrong.host.badssl.com' doesn't match either of '*.badssl.com', 'badssl.com'

------
Animats
This looks like an attempt to use Rust-type error handling in Python. This is
pounding a screw. Rust does it that way because Rust doesn't have exceptions.
(Although, as with Go, all the heavy machinery for exception unwinding has
been retrofitted so "recover" will work. Rust and Go just don't have the
syntax for exceptions.) Python doesn't need to become Rust, or Haskell.

Writing

    
    
        except Exception as e:
    

is clueless Python. If you're checking for something that went wrong in the
outside world, you catch EnvironmentError. Catching Exception catches far too
much, including syntax errors. It's sometimes useful to catch Exception at the
top level of a program, log an error, and restart, but not in interior code.

Note how useless the examples in the article are. It's not like someone is
trying to handle all the things that can go wrong in a network connection, or
in nested code where you have a network connection and a database transaction
open at the same time, either can fail, and you have to unwind properly.

Typed language are fine, but this semi-checked addition of types to Python is
a mess. Python doesn't need this. I'm in favor of design by contract, typing,
and assertions, but not in this broken semi-optional way.

 _Stop Guido before he kills again._

~~~
quotemstr
When you have a generation of people brought up to believe that exceptions are
somehow evil, what do you expect?

~~~
Animats
I know. Python is one of the few languages to get exceptions right. There's a
reasonably sane exception hierarchy, and if you catch something, you get
everything in its subtree. The "with" clause and exceptions interact properly,
so if you hit a problem closing a resource in a "with", the right stuff
happens. Since Python is reference counted/garbage collected, you don't have
the ownership problem of exception objects you have in C++. Python isn't big
on RAII, so you don't have the exception-in-a-destructor problem (or worse,
the exception during GC problem) unless you do something to force that.

Python is probably the best type declaration free language around. Trying to
make it into something else damages the language. Python has become uncool,
though, and so "cool features" are being added that don't quite fit.

These changes to Python are so big that this should have been called Python 4.
But if Guido had done that, the reply from the big users would have been
"Hell, no, we won't go!".

------
dbrgn
If you just need a Rust-like Result type for Python:
[https://github.com/dbrgn/result](https://github.com/dbrgn/result)

------
leshow
I'm a bit confused by this post, the 'enums' in rust are actually 'algebraic
data types' and not enumerations like what the author is showing in python.
the code here isn't really very equivalent and if anything just shows how much
less expressive python is than a language which has proper algebraic types.

------
nicolaslem
Type annotations are also helpful without mypy. PyCharm uses them to provide
hints and warnings during development.

At first I was reluctant of the syntax, after a while I got used to it.
Function definitions look a lot like Rust ones and do not require the endless
docstrings just to document the type of the expected parameters anymore.

~~~
insertnickname
My question is this: What happened to all the duck-typing evangelism? "If it
walks like a duck and quacks like a duck, it is a duck", "static typing only
catches trivial errors", and so on.

~~~
nicolaslem
Not a definitive answer to the problem but mypy supports a limited sets of
common "interfaces", for instance dict-like or list-like.

[http://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html#st...](http://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html#standard-
duck-types)

~~~
throwaway_374
That's right, mirroring the built in (3.x) abc collections heirarchy. Which
then allows you to do interesting things like delegation based polymorphism by
registering with the appropriate metaclass so - just to come back to your
parent's point - interrogating the type reveals it to be list-like or dict-
like.

------
mythrwy
There must be something I don't get. What is the advantage of creating an Enum
class to store relationships rather than just using a dictionary or
namedtuple? (Arguably) readability? I mean, I see it works but fail to grasp
the advantage.

------
gaius
_I’ll be the first person to admit I have no idea what postmodernism actually
means_

It means it means there's no right or wrong way to do it, everything's
subjective. Which is actually the exact opposite of the Python philosophy!

------
adamcharnock
Am I correct in thinking that duck tying and type hinting are not mutually
exclusive? Type hinting is not enforced at runtime, so we are still free to
pass around whatever we like as long as it behaves as expected.

~~~
shawabawa3
Correct, although it kind of defeats the point of typing if you ignore type
errors.

There is however an Any type which you can use for things you don't need
enforced

------
raverbashing
"Mypy allows you to add type annotations and enforce them prior to running
your program"

Yes, and if I wanted type annotations to stop my program from working I
wouldn't be using Python

Enforcing types is exactly what Python _IS NOT_ about. Because of Duck Typing
and everything else

So this is not "postmodern error handling" this is "let's code Java in
something else and pat ourselves in the back"

Do you want to check errors in "compile time"? Use Pylint. It does the right
thing

~~~
uweschmitt
I'm not a big fan of type annotations because the code easily turns unreadable
which is not what I expect from Python code.

But for larger projects type checking is valuable. Types introduce additional
contracts between components to manage and control the overall complexity of
the system.

For small scripts and self contained applications I regard type checking as a
burden. For larger projects not.

Pylint is a valuable tool to detect typical errors but does not check types.
Regard a function which returns the sum of two given arguments. It will work
with both arguments being integers and for both being strings. PyLint will not
complain here, but if your code uses this result you might trigger errors much
later during progam execution which are hard to detect or understand without
type checking.

~~~
njharman
> But for larger projects type checking is valuable.

Wrong solution to the wrong problem. Break that too large for duck typing
fucking mess into smaller independent components with well defined and
enforced interfaces.

~~~
adwn
> _components with well defined and enforced interfaces_

"Well defined and enforced interfaces"? That sounds like a job for static
typing!

But seriously, that's exactly what types are good for: defining and enforcing
an interface.

------
mixedCase
Postmodern error handling would probably be treating any error as a successful
result.

------
diminoten
What is a "maybe implementation" and an "either"?

------
gcb0
kinda of a bad pattern.

lots of abuse of error handling apparently just to have a fetch and parser in
the same method?

why not have one promise of a fetch and then a promise of a successful parse?

------
miloshadzic
Another reading of this paragraph is that it's essentially douchebaggery.

~~~
minikites
I'll never understand the techie disdain for the humanities and liberal arts.

~~~
xkxx
Maybe it's partly because of anti-intellectualism, the idea that being very
smart is bad. I had never heard about the idea, until recently. It was some
article about American anti-intellectualism and why it is so popular. Reading
that article was a very weird experience for me. I believe this concept
doesn't exist in the culture of the country where I am from.

~~~
gbog
That's a very strange thing for us Europeans. When I meet a new guy I will
scan some cultural fields and see if we have some common ground where we can
sit the discussion for a while, it could be post-modernist philosophy,
classical music, anime, or porn even, but we need to find place to settle.
Younger, I did the same with some people from the US and noticed big "!name
dropping!" flags. Right, I was name dropping, just to try to find a common
topic. I wasn't trying to shower anyone with "intellectualism", though: it is
perfectly ok to not have read Proust. (But it would be weird and a no-go to
have familiarity of no parts of cultural life except baseball and "football")

~~~
coldtea
> _That 's a very strange thing for us Europeans._

It's even worse in Britain. Anything remotely smart or cultured is considered
"pretentious" etc.

