Hacker News new | past | comments | ask | show | jobs | submit login
Python exceptions considered an anti-pattern (sobolevn.me)
47 points by sobolevn 35 days ago | hide | past | web | favorite | 65 comments



It amazes me how enduring formulaic it is to single out some particular design tradeoff of a language, draw up some examples of expressing something where that tradeoff creates worse code, and then act like it's some mortal flaw in the language.

Python chose untyped exceptions, period. How is this surprising, given that its basis is untyped parameters?

If you don't like that, use Java with its checked exceptions. Or remove exceptions from the implicit monad altogether and use C or Rust. Just don't then go on to write some lengthy post about how any one of those is too explicit.


Do you mean unchecked exceptions? Python exceptions are strictly typed, that's core to how they work. An except clause will catch subclasses of the target, so you need to have a custom type for each exception you want to raise in your code.

The mistake I see people make is to use built-in exceptions without subclassing, making it impossible to explicitly catch specific errors. Or the opposite, catching Exception, which will catch everything indiscriminately.


No. I mean checked exceptions. At the language level, Java checked exceptions are typed, and unchecked exceptions are untyped.

The Python runtime is dynamically typed. The Python language is untyped.


Both checked and unchecked exceptions are typed in Java. Unchecked exceptions are not included in the method signature, but that doesn't mean that the exceptions are untyped.


The exceptions, meaning the runtime objects themselves, are indeed typed.

But at the language level, unchecked exceptions are not part of method signatures (as you said), therefore not type- checked/declared/inferred. Therefore appropriately described as untyped -

When calling a method, you have no (formal) list of unchecked exceptions that might be raised.

If you think of exceptions as being an implicit union type around every function return (ie monad), analogous to how you have to explicitly check for errors in C/Rust, you'll see what I mean. Java's unchecked exceptions are akin to calling a function in Python that you expect to return objects of only one type, but not being "sure" that it can't return something else.


The author puts an exception in a didactic example of a "division" function. Then upon finding that the exception cannot easily generalize to all possible use-cases of the didactic example the author concludes:

> So, the sad conclusion is: all problems must be resolved individually depending on a specific usage context.

Why is that sad/revelatory? If the wrapper simply reproduces the behavior of a single infix operator for floating point numbers, then almost by definition one would need to reimplement NaN in order to get a general purpose exception handler.


Exactly. Besides, I fail to see how exceptions are the problem in there.

> Exceptions are hard to notice

No. In Java they are easy to notice. In Python we have a dynamic language, so there is no safety net for that. It's a limit of the language that we chose, by design, with pros and cons. It's not exception related.

> So, the sad conclusion is: all problems must be resolved individually depending on a specific usage context. There’s no silver bullet to resolve all ZeroDivisionErrors once and for all

That's a good thing. There are many reasons to have a ZeroDivisionErrors, and they beg all for a different solution. Sometime it's you, sometimes it's the user, sometime it's just how things are. Again, it's not related to exceptions in any way.

> For example, the system might notify the user to change the input, because we can not divide by 0. Which is clearly not a responsibility of the divide function

Indeed, you are supposed to sanitize your inputs at the intersection between your program and the rest of the world. No surprise, this has nothing to do with exceptions.

> Now we just need to check where this exception is actually caught. By the way, how can we tell where exactly it will be handled?

Because you write the handling code. If you don't write error code handling, how is that the exception system fault ?

> There’s no way to tell which line of code will be executed after the exception is thrown.

Yes, there is. Either you handle it, and the next step is in the try/except, or you don't, in which case your program crashed. Same with any way of dealing with errors. It's not about exceptions.

> We have two independent flows in our app: regular flow that goes from top to bottom and exceptional one that goes however it wants.

Nope, the exception flow goes from the bottom of the stack to the top. It's well defined, we even have a beautiful stack trace for that. Don't blame exceptions if you don't even know the basics.

> Exceptions are not exceptional

Compared to the rest of the lines of your program, they are. Exceptions are called that way because they represent a special case. The fact computing is full of them is, you know it now, not related to exceptions.

> How to be safe?

Handle the error. Like in any language, with any error handling tool. There is a difficulty in Python: the exceptions are not listed in the function signature. That's hardly the fault of the concept of exception, and is just a design compromise.

> Now you can easily spot them! The rule is: if you see a Result it means that this function can throw an exception. And you even know its type in advance.

So basically you rewrite Java in Python. Again, not a problem with the exceptions. And oh, no, don't do that. If you want that, use Java, not Python. It's perfectly reasonable, but don't turn Python into Java. The fact exceptions are not written in the function signature is, I repeat, a design decision. It has pros. It has cons. But don't use a screw driver like a hammer, that's bad.

> ow to work with wrapped values?

Ok, now you are trying to implement Haskell in Python.

Use Haskell then.

Python is not made for this. It's an easy to write and read language. You are supposed to be able to edit Python code with notepad if you have to. Every line should be short and to the point, with a well defined role.

And we expect exceptions to bubble, that's how we like it.

> But how to unwrap values from containers?

See what I mean ? The author just opened the pandora box. Now he or she has to write long chunks of text just to explain the basic of error handling.

Without this system, I can just pdb.set_trace() in there, now it's full of inlines, chaining, anonymous callbacks and wrappers.

That's just... no.


> Python is not made for this.

Maybe, but there are ungodly big and crufty codebases in Python all around the world, and a lot of stuff depends on them. And it turns out that Python was never meant to be the thing that is being depended on. That's unfortunate. Hence the whole gradual typing (mypy) and other kinds of safety efforts.

You're not forced to use this. But it seems pretty useful, even for scripts.


I see a lot of commenters here are as perplexed as I was when I first read this post.

The key thing to understand here is that post is really about an attempt to implement the "Railway Oriented Programming" paradigm [1] by Scott Wlaschin, but in Python. I would suggest reading the Scott's post on ROP and at least skimming the video before going on.

So ROP is, as Scott himself states, a way to take all those nice Haskell concepts and techniques and apply them in F# in a way that won't overwhelm those who are new to them.

The problem with the original post is that it presents two problems with exception in Python and then offers the "returns" library as a solution which, ultimately, doesn't end up solving either of those problems.

The first problem the post describes is that the exceptions are not part of the function signature. The second problem is that exceptions are essentially gotos and that this makes reasoning about the execution flow very difficulty.

To tackle both problems, the returns library offers its own implementation of the Either monad in form of the Result container. Having presented that solution, the post promptly decides that monadic code in Python is unreadable and offers the @pipeline decorator which allows you to write code that looks imperative, which partially defeats its purpose as a solution to the problem about the flow reasoning. I say partially, because it replaces implicit gotos with implicit returns, which is a marginal improvement over exceptions.

The post then decides that using the Result class in the signature is also ugly and offers the @safe decorator which allows you to write the same code you would write without ROP, but now it wraps everything in Result behind the scenes. Even worse than that, this produces the return type of Result[whatever_your_success_type_is, Exception]. For those familiar with Java, this is very similar to putting "throws Exception" in your method signature, except that it's hidden and implicit.

I'll end this with a bit of advice for the author, preceded by an apology, because I'm not sure if I have managed to find a way of phrasing it that doesn't sound harsh. You might want to do some serious reading about functional programming and monads and how these concepts have been carried over into and grafted onto mixed-paradigm languages like Java. I mention Java specifically, because one of the complaints in your post is that Python won't be supporting checked exceptions in the nearest future. A lot has been written about checked exceptions in Java -- on both sides of the discussion.

[1]: https://fsharpforfunandprofit.com/rop/ [2]: https://en.wikipedia.org/wiki/Tagged_union


Well or solve the perceived problem and use the solution (their "returns" library)...


The interesting bit starts at https://sobolevn.me/2019/02/python-exceptions-considered-an-..., where it introduces a library for wrapping errors in return types, even if it got little to do with the headline. IMHO the article would be stronger with a title about what it presents and without the first half.


Agreed, I read the first half thinking You know, I kind of agree, but it would 'trigger' most of my colleagues - then I started thinking Oh I wonder if you could build something into mypy like rust's Result<Ok, Err> (I realise that's not rust's invention, probably haskell's?, but it's rust that made me more familiar with the concept) - then I got there in the article, great!

"Typing failure modes, not just successful returns" or something would sound like a much more appealing article to a much wider audience.


Thanks, my idea was to state the problem and propose a solution. Hope, that you will find `returns` helpful.


One risk with the title is that it doesn't convey that it's actually proposing a solution instead of being one of many "$languagefeature is bad" articles. I personally also found the problem statement presentation not all that convincing and almost closed the tab before reaching the interesting section, which is why I made the comment above.


I'm sorry, but from my perspective - a java developer that has moved to scala - this "let's drop checked exceptions for Either[Exception, T]" stuff... stinks.

And for one specific reasons: Exceptions are not important per se, but also for the stacktrace they take with them, and that is given to you when you need to handle it.

I lost too much time in this last month trying to decipher where the hell stuff happened looking at the stacktraces in the logs.

Coming back to the python approach: I like python :) and it's philosophy it's to have unchecked exception, with all what it concerns. The idea is that if you don't know an exception, you really don't know what to do. So let the code blows up - ideally in tests :) - and understand what realistically can happen and how to deal with it.


Thanks for the solution! It's elegant and is a great fit for my current work which involves retrofitting my codebase with type annotations.


Counterargument: Exceptions are Pythonic https://jeffknupp.com/blog/2013/02/06/write-cleaner-python-u...


I'm soundly in that camp. For instance, we write API endpoints that look a lot like this:

  def update_password_view(session_cookie):
      values = request_params(['old_password', 'new_password'])
      user = get_user(session_cookie)
      verify_password(user, values['old_password'])
      update_password(user, values['new_password'])
      return 200
  
  def request_params(param_names):
      values = {}
      for key in param in param_names:
          try:
              values[key] = request.params[key]
          except KeyError:
              raise BadRequestError('Missing parameter', key)
      return values
  
  def get_user(session_cookie):
      users = db.get_user_with_session(session_cookie)
      if len(users) != 1:
          raise NotFoundError('No user with that session')
  
      return users[0]
  
  def verify_password(user, old_password):
      if hash(user.old_password) != hash(old_password):
          raise BadRequestError('Bad password')
  
  def update_password(user, new_password):
      user.password = hash(new_password)
      db.update_user(user)
Notice that each function raises an HTTP-ready exception, so update_password_view has no explicit error handling of its own. You can look at that function and read the intent of how it actually works, as each line is only reachable if the one before it 100% succeeded. After `user = get_user(...)`, you know that `user` will have valid data and not some sentinel value you have to check for.

Our actual implementations are more subtle. We have our own exception hierarchy with classes like `UserNotFoundError` or `BadPasswordError` that subclass the corresponding HTTP error classes, so you can still write code like:

  def upsert_user(data):
      try:
          user = get_user_by_email(data.email_address)
      except UserNotFoundError:
          user = User(data)
          db.save(user)
      do_something_with(user)
in the cases where that exception isn't fatal.

In practice, we've found this coding style to be much easier maintain than idioms like `if not_found(user): return None` where you spend half your lines of code explicitly checking return values for error sentinels. Life's too short to live like that.


The two articles together present an interesting perspective on python programming philosophy. I'm far more inclined towards the exception approach. And while I like that type annotations exist, using them in examples convinces me that the examples don't apply to all python


I never understood, that people who don't understand Python at all, or don't like essential Python constructs, why they bother using it? Why not use a different language in the first place? If you are using this library, you are not writing Python anymore and you lost the biggest advantage of the language: simplicity.


Often the language is chosen because it is the only option like JS for web, or it is the best option for the project. For example, you are doing physics and you need a general purpose language that is easy to write and handles gigantic numbers correctly so you pick Python.


>Often the language is chosen because it is the only option like JS for web, or it is the best option for the project. For example, you are doing physics and you need a general purpose language that is easy to write and handles gigantic numbers correctly so you pick Python.

You could pick Julia, or R. A community could create their own tools. It's not like JS, which has an absolute monopoly and people were forced to use it.

Python __became__ popular for that purpose.

Not Java. Not Haskell.

So don't try to implement Java or Haskell error handling in Python if you are using Python in a niche where it shines: it shines here because people working in that field decided it fits well.


JS interpreter, thanks to asm.js, can be used to run any language. Knock yourself out.

Of course you get to deal with a bit of FFI when calling built-in functionality of the browser.


Thanks to adapters, you can plug an american fridge in a french socket. I'll still buy the one made for the local electrical system.


So, you'd write in an inferior language when even web framework authors don't want to?

(See CoffeeScript, TypeScript, Elm, etc. Or Transcrypt if you want pythonic semantics.)

I understand this being a business requirement made by a business droid. Other than that, there is really no reason to be married to JavaScript.


"X considered an anti-pattern" article titles considered an anti-pattern


I'm bothered by how frequently people paraphrase this quote Dijkstra to suit their own purposes. Borrowing a famous quote makes me think that they need the help sounding credible.


Title should be "another person on internet considers it an antipattern".

I'm part of the people who advocate automated testing and insanely high coverage. Does that mean I want to write all tests manually ? No, I use fuzzers, light test code against many autogenerated fixtures and the like, and get no runtime exceptions.

Duck typing means "bad language" for some people, for me it means "freedom".


The library described in this blog post looks kind of interesting as an implementation of a Maybe monad in python, but the example case is pretty silly.

It re-implements a 4 line function as a 13 line class, but the logic at the caller doesn't get any simpler:

  try:
     result = get_user_profile(id)
  except:
     # handle any exceptions...
vs. with the library:

  result = FetchUserProfile(id)
  if (result is a Failure):
    # handle the failure
  # do something with the result


I'm glad this issue is getting some attention. I found the "Exceptions are not exceptional" section to capture something I've noticed several times — that code in Python can fail at a huge number of points even in a small function, nevermind the many exceptions that might happen in nested functions, or code we're using from libraries.

We should have language-level mechanisms for being explicit about what's supposed to happen when unexpected situations happen.


> that code in Python can fail at a huge number of points even in a small function

This is why unit-tests are really important in Python.

> We should have language-level mechanisms for being explicit about what's supposed to happen when unexpected situations happen.

That's implemented as a try-except block. You're not supposed to catch errors at the low level unless you're explicitly handling them. If there's an error, bubble it up to the main().

Python's try-except can catch classes, sub-classes, and/or groups (tuples) of classes. It's important to categorize your errors.


Any sufficiently typed programming language will grow a vocal subcommunity that will try writing Haskell in it.


Since using Rust, I've longed for this in Python but

- imo it effectively requires using type checking (mypy)

- Too out of place with the rest of the python ecosystem (even if I like something, I'd rather not force non-standard practices on others dealing with my code)


I don't know, for things where you can get team buy-in, this looks like a great way to have more solid contracts. I've always disliked how you can specify accept/return types in Python but then there's this whole side-channel of exceptions that you can't check or document well.


How familiar are you with Java?


I was about to post something similar to this.

In java you need to specify what your function can throw and if it tries to throw something that it has not specified as being able to throw, then your program won't compile.


Agreed, checked exceptions solve the same problem, but having separate control flow mechanism for certain kinds of data still seems subpar. Return values can solve the same problems and are more general.


Not if you cannot enforce them to be checked.


That's an acceptable solution as well.


Not enough to have PTSD from it, why?


Maybe you should be familiar with it.


Cool, I'll file it under "to maybe be familiar with, things", thanks.


Or mention "PTSD" when somebody brings up Java to signal that you're above such foolishness, and rely on nobody seeking an explanation for your aversion.


Here is the greatest inconvenience of Python exceptions for me. Say you have to try 10 different methods, and you only need one to work. Then you have to write 10 try...except blocks so that each next block is indented relative to previous. This creates unreadable code and does not scale for say 100 methods. The solution that came to mind is labeling try blocks and referring them in except blocks. For example:

  try as method1:
      method1()
  except@method1 try as method2:
      method2()
  except@method2 try as method3:
      method3()
  except@method3:
      raise NoMethodWorked()


There are several solutions that occured to me off the top of my head, here's the simplest one:

    try:
        method1()
        return
    except Exception: # or a more specific exception type
        pass # try the next method
    try:
        method2()
        return
    except Exception:
        pass
    # ...
    try:
        method10()
    except Exception:
        raise NoMethodWorked()
Seriously, there are so many ways to solve this in a readable, maintainable way, without resorting to introducing weird new syntax.


I can't think of a single time I've needed a large collection of "backup" methods in case of a chain of failures; I can't even think of an example which could scale to 100. Anyways, this solution does scale:

    methods = [method1, method2, method3, method4]
    for method in methods:
        try:
            method()
        except Exception:
            pass
        else:
            break
    else:
        raise NoMethodWorked()
You can even pair specific exceptions to each method:

    methods = [(method1, TypeError), (method2, KeyError)]
    for m, e in methods:
       try: m()
       except e: ...
But the whole thing really sounds like you're trying to do too much with one function and you really should rethink the whole structure of your code.


This is really helpful. Thank you.


Who.....codes like this? Use a method dispatch instead, and have the dispatcher worry about throwing the exception:

    try:
        DispatchController.dispatch()
    except:
        raise DispatchFailed()
And/or use a lookup table. Especially since methods, classes, decorators, etc are all first class citizens in Python.


You probably want to look in to try-except-else patterns.


> There are so maybe potential problems with these three lines of code, that it is easier to say that it only accidentally works.

A very strong and emotional point. Accidentally working code will have exceptional handling because of problems during development.

The whole article is depreciated by naive library implementation because unwrap() hides source of original exception. Sadly it's not even a POC.

Edit: Failure doesn't capture trace information at all. Library users will get unusable error in a wrong place.


> So, the sad conclusion is: all problems must be resolved individually depending on a specific usage context.

...and that's a good thing.

Recognising that specific usage contexts require specific recovery strategies is a key part of effective program design.

Division by zero is an exception, yes. That's basic math.

Getting to the point where one of your inputs is "bad" (zero in this case) shows that you have a problem. You should either catch zero up front with a conditional or catch it when it blows up, with an exception. Python favors the latter.

But without context you don't know what that bad input means and so you cannot "fix" division by zero in the general case because it isn't the division by zero that is the cause of the problem.

Asking "what should a division by zero actually return?" is the wrong question asked at the wrong point with the wrong information. Does the zero indicate lack of initialization? Does it indicate an empty container or volume? Does it indicate absence? How much of a problem to the logic of the program is this particular zero? How much of a problem is it for the person running it?

So while I personally dislike exceptions and prefer return codes, exceptions are just the messenger here and they are an effective messenger. Don't shoot them.


Interesting fact of life: floating point returns NaN or Infinity for certain operations and it propagates. Real pain in the rear to debug. You cannot even reliably trap on it most of the time because there are no such options in the compilers.


It all feels very Go-ish/Java-ish to me vs Pythonic.


I was thinking Rust-ish. Java's error handling is similar to Python's.


Putting the exception handling inside the divide is foolish. Put the handler outside in the calling function where decisions about why it happened can be implemented.

Python could use a "raises" keyword though. Whether that has real benefits is debatable.


news flash - there is no perfect system for this.. Its a feature of Python IMO that a coder can just write a few lines and run.. some architectures may not need good error handling, except for development e.g. ephemeral data flows

Lazy is good (sometimes)


I have a question. How proposed solution can solve problem with exceptional control flow? Monadic approach skips calculations under the cover and one need to track branching in his head.


If you're not Dijkstra, Hoare, Knuth or a principal contributor to the language's design, I would refrain from "considered (harmful|anti-pattern|…)."

It's hyperbolic and frames your position in a way that's likely to create poison. The people without enough experience & knowledge to evaluate your argument critically are likely to end up parroting it and looking like fools. The people who can evaluate your argument critically are going to scrutinize it much more carefully because you've taken an absolutist position.

Unless you have the knowledge & authority to really make such an absolute statement you do yourself and your readers a disservice.


Calling a personal opinion an anti-pattern is an anti-pattern.


It reminds me of the Principal Skinner meme. "Maybe I'm out of touch... No, it's the Python programmers who are wrong."


Yet another Monad tutorial in disguise. Good review of exceptions, etc., and how to handle them. tl;dr: Use Maybe and Either instances (or equivalent GADTs) for better control flow.


And here I am using exceptions to do HTTP redirects with CherryPy. I haven't had major issues with exception handling in Python yet.


Lets all agree that it is impossible to write code that will never have an unexpected outcome. Imagine that we somehow write a function that is totally bullet-proof. It can't fail, it will always do precisely what it was intended to do. Further, lets say whenever we run this function we run it on N computers and take the consensus result if any of the computers disagree. No matter how large N is, if we run the function enough times eventually we will get a majority of the computers to agree, return the same result, and that result will be wrong. Whether it's from cosmic rays resetting bits in memory, or multiple cosmic ray strikes resetting multiple bits and thus defeating ECC, sooner or later things will break no matter what you do. Even if you shield all the computers with 5 meters of pre-nuclear age lead, eventually it will break.

The point is that it is just not possible to get to perfect reliability. Your actual reliability is always going to be less than 100%. You can invest money and effort to get closer to 100%, but obviously you are going to get diminishing returns.

The correct analysis is to decide where the optimal trade-off is between investing in reliability and the return on that investment.

Example 1: You are calling a web service that checks the weather. The service might be down. If it is down you wait a few seconds and try it again. The 'cost' of it being down is that a user doesn't see the current weather. Is it worthwhile to carefully try to determine whether the error when calling the service is due to a server returning a 500 status code versus invalid json?

No, it's not worth it. Either way the client can't use the response. In fact, it doesn't matter what causes the exception, since anything that goes wrong can't be corrected by the client. Whether it's bad json, a network failure, dns failure, the server is being rebooted, the webserver is misconfigured, or the device is in airplane mode, the resolution is always the same, wait and try again in a few seconds. Exceptions work pretty much ideally in this case, you only have to code the 'happy' path and handle all exceptions the same way generically.

Example 2: You are writing code to update a database containing financial transactions. If something goes wrong in an unexpected way you need to make sure the financial data isn't updated or left partially updated.

Again, you don't care about unexpected exceptions. For failures you expect and are coding to work around them, possibly by catching the generic exception where it happens deep in the call stack, and then raising your own exception class which properly identifies the error and contains the context necessary to perform the recovery. For example, if you need to send an email via receipt for the transaction, you call some function which formats and sends the email. That function fails due to the email server being unreachable. The network exception is caught and an EmailCantBeSent exception is raised with the relevant details in it (the user_id you were emailing, the transaction_id the email is for). The resolution is to log a critical error and insert a record to the database with the relevant details of the email so that someone can make sure it is sent later. Then it continues with the transaction. If something unexpected happens the database transaction is never committed.

My point is that there are two kinds of exceptions you run into, the ones you are being careful to trap and resolve as part of your applications design, and the ones that you aren't trying to resolve and so result in just a generic 'this failed' situation.

So finally getting back to finding the optimal tradeoff between investment to improve reliability and payback on that investment, you just need to make sure your generic failures are rare enough that you aren't pushed far from that optimal point, which is almost always going to be the case, even if you basically don't ever handle any exceptions and only code for the happy path. Obviously there are tons of counter-examples and sometimes you need to make sure things work even when something goes wrong (if you are working on an autopilot you will require much higher reliability and so much more careful planning to reach it compared to a twitter client, where you just need to not lose what the person typed).

Ok that's a lot longer than I intended.

TLDR; If you do any kind of analysis on why code fails and what you should do about that, you quickly realize that this library doesn't help at all. This library isn't even bad, the problem it is meant to solve is not well posed.


"X considered harmful"

"X" implies "X considered harmful"

Thus

"X considered harmful" considered harmful

"'X considered harmful' considered harmful" considered harmful

...

QED


argumentum ab auctoritate




Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: