Hacker News new | past | comments | ask | show | jobs | submit login
Positional-only Parameters for Python (lwn.net)
76 points by l2dy 6 months ago | hide | past | web | favorite | 67 comments



> Over time, we’ve had a trend of adding unnecessary, low-payoff complexity to the language. Cumulatively, it has greatly increased the mental load for newcomers and for occasional users.

I couldn't agree more with this sentiment from Raymond Hettinger. I first learned Python starting with 1.5.2, and coming from a background of C and Perl. One of the things that really attracted me to the language was how compact it was. It took no more than an hour to learn most of the language, a day to read the entire spec, and maybe a second day to read the entire standard library documentation. It suffered from virtually no gotchas... nothing crazy like Perl's scalar vs list context, etc.

I haven't had trouble keeping pace with Python and I even probably agree with most of its changes. But this change really seems superfluous. How does it fit into the zen?

Can a language ever reach a point of being done?


You're not alone. My goto example of this is the pathlib module added in 3.4. It has this feature for making it easier to concat strings representing file system paths...

    >>> from pathlib import Path
    >>> Path("foo/bar") / "baz"
    PosixPath('foo/bar/baz')
Don't get me wrong, the existing solutions until pathlib (os.path.join or string concatenation) left a lot to be desired. And there are certain tricky path-parsing things this module does that are wonderful. But overloading the __div__ operator for a "cute" API just feels... wrong...

Of course, after my initial disgust, I now use it everywhere :)


I think this is fascinating because I generally agree that Python is seeing a bunch of features it doesn't need. But then your go-to example is one of my all time favourite feature adds. I deal in path concatenation all the time.

Maybe the answer is that none of us work in all the domains where Python is used, so we are all speaking with ignorance/elitism when we decide which list of features are superfluous.


I'm not sure why it's wrong? Python isn't Haskell. If you're dealing with path manipulation a lot, it makes for much more readable code.


"Python isn't Haskell."

Not sure exactly what you're saying there? Haskell strikes me as far more likely to define an operator for that.

Although Haskell will define something other than /, which unless you jump through some hoops is going to be the fractional division operator [1]. You'd have to be crazy to try to give a "path" an instance of Fractional, though I'd be intrigued to hear someone's definition of the "reciprocal" of a path.

[1]: https://hackage.haskell.org/package/base-4.12.0.0/docs/Prelu...


> I'd be intrigued to hear someone's definition of the "reciprocal" of a path.

  ..


Reciprocal is its own inverse, so the reciprocal of a/b/c and d/e/f can't both be "..".

The closest you could come is an anti-path, where if you are at a/b/c, you could have c'/b'/a' that when combined together produced the root path of both these things. You could then have things like b'/a, which represents "rise up to the previous directory, which is named 'b', and descend into a" where the reciprocal is a'/b. I'm done fiddling with this for now, but it's possible you could work this into a sensible scheme with some concept of "reciprocal". It's useless in practice because who wants a .. operator that asserts the name of the directory it is rising from? (And presumably, what, errors if it isn't right? Refuses to rise like .. does at the root?)

However since Fractional instances also have to be Num instances [1], and there's no practical way to fill in most of those, it's not going to get you very far. And the Fractional requirement of fromRational, while you can try to write something that turns a Rational (which in Haskell is specifically two Integers being used as a ratio) into a path, it's going to be very silly.

[1]: https://hackage.haskell.org/package/base-4.12.0.0/docs/Prelu...


yeah but haskell would use ///

at least


I'm curious, why not override the usual + operator, which is far more idiomatic, instead of trying to get cute by using using an operating which generally implies division. Yes, it's fun once you understand it, but at a first look, it's more confusing than useful, whereas + would be fairly self explanatory.


Well, / is path traversal; I think + would be too easily conflated with the “similar but importantly different” string concatenation, especially since « + "/" + » is an idiom in string-based path manipulation. For instance:

  p1 = "/a/b/c"
  x = "d"
  p1 + "/" + x == "/a/b/c/d"
  p2 = Path("/a/b/c")
  # If + replaced / for Path:
  p2 + "/" + x == Path("/d")
Now if you've just reflexively written the concatenation the way you did before, the type signatures match up but the result is completely wrong. The current situation where / doesn't exist for strings and + doesn't exist for Paths is much safer versus that.


You have a point, and I can’t tell you why they decided to go with /usr/bin/bash instead of usr+bin+bash back in the dawn of computing. I suppose you could write you own set of tools to change the entire ecosystem to use that convention, even providing new URL definitions.

But either way, I can’t see the problem with python defining the exact same operator to mean the exact same thing as you’d expect in that specific scenario.


The plus operator is traditionally expected to be symmetrical, i.e. a + b == b + a. The division operator, on the other hand, is not: a / b != b / a. The fact that a slash is used as a separator in paths is only a welcome coincidence.


In the context of strings, that's not true, and very commonly accepted:

"a" + "b" == "ab"

"b" + "a" == "ba"


Using + for string concatenation was an error introduced early in Python's design, and is to difficult to repair now.


And if you are not dealing with this all the time, it makes the code completely unreadable.


Note that you don't have to use `/` to do this, you can use Path.joinpath or call Path/PurePath with multiple path segments.


This is very much TIMTOWTDI. Larry Wall would be proud.


The first time I used Python on a team, I learned that "One Obvious Way" was a lie.

In order to achieve OOW, you need really high-level features (assembly language programmers have every possible way to accomplish a task!), but also a very limited set of them (so nobody can pick the 'wrong' one). But as a general-purpose programming language, users also need power and flexibility.

It's an unstable tripod where as soon as one starts getting higher, you mess up the others. I can tell they've been trying to slowly raise all three (and deprecate/remove old and lower-level pieces from the bottom), but it's a delicate balancing act, and it's easy to run into other constraints, like complexity.


I think part of the reason Wall adopted TIMTOWTDI as a design is because he observed that it's what happens anyway, so you may as well embrace it.


I don't think that's out of line for Python. I mean, it overloads * for string and collection repetition...


I won't show you my pipe library then


I've stopped recommending Python as a new language for people. It was my go-to for a long time, but after sitting with a couple of people trying to learn it, I discovered the hard way that it has passed the "easy-to-learn" horizon a long time ago.

I am deeply unconvinced the proliferation of features is even remotely necessary or helpful.

But then, I admit I've become deeply skeptical of the whole "language as a proliferation of features" model of language development in general. Python is just a particularly frustrating example because it's a clear case of a useful simple language becoming a huge, complex language that isn't really that much more useful. I'm coming around to what doesn't go into a language being more important, because there's always another feature the language needs and always some reason to put it in. If you don't have a clear thesis on why and how you're going to reject things, you won't often enough.


I couldn't agree more. All of these people teaching python to children is so crazy to me when the language is so BIG!!


Can you expand on what you think has become difficult and what you recommend now?


The big problem is when the beginner first encounters some external code. One particular case was trying to teach a QA automation engineer about what was going on. First of all, I was having trouble tracing back all the definitions involved due to the excessive "cleverness" involved in assembling the API, but we'll put that down to the testing API doing things it shouldn't.

The problem I had was that I had to explain "oh, well, this is a generator because of the yield, and the arguments don't show up in the documentation for the function because it's introspecting on the keywords you pass in dynamically, and this here is a generator comprehension with its own syntax, and this is what the decorator on top is doing on this code, which is why this is breaking" and honestly it kept going like that for a while. When it all worked it was fine, but when anything broke, it's become far too easy in Python, and arguably idiomatic Python, to erect a rather intimidating wall of features and automatic rewirings and fancy decorators and suddenly to fix their problem the novice is swallowing all of twenty years of language development in one shot... or they give up and do something else.

I don't have a current go-to recommendation, though I've found myself recommending "Go but stay away from any concurrency features for a good long time" a few times. Ignoring all the other controversies that HN likes to indulge in, that's not a half-bad intro to "real programming" for a modern language. It's not the only good choice, of course.


Ah interesting. I totally hear that.

Your idea of a beginner is someone in a mature environment though, I guess my thought when I read your previous comment was a total beginner starting from a blank slate.


> I haven't had trouble keeping pace with Python and I even probably agree with most of its changes. But this change really seems superfluous. How does it fit into the zen?

Explicit is better than implicit, Simple is better than complex, Readability counts, I'm sure others would match.

Positional-only parameters are useful in general and even necessary to implement / override APIs originally defined in / for C (which has always had access to positional-only parameters). Currently, defining an optional positional-only parameter is:

    def foo(*args):
        if len(args) > 1:
            raise TypeError(f"Expected at most 1 argument, got {len(args)}")

        if args:
            arg = args[0]
         […]
* the number (and optionality) of positional parameters is completely invisible and implicit rather than readable and explicit

* it requires various non-trivial assertions repeated every time

By contrast, with PEP 570 this becomes:

    def foo(arg=None, /):
        […]
It does require the knowledge of "/" but spells out exactly what the contract is clearly, cleanly, and without requiring additional developer legwork.


A huge perk of doing it at the function definition level is that IDEs can give you a more reasonable function signature.

Seeing "foo(args)" as an autocomplete necessitates looking at the docs to find out what "args" gets used for.


I read the PEP and understand the argument (no pun intended). I'm just not convinced. I don't find the justifications given in the PEP at all persuasive. Rarely have I ever needed to enforce positional over keyword arguments and the rare times I've needed to, I can use:

    *args, **kwargs
as needed, and again, don't find the PEP's arguments against doing so convincing.

Also, I mean, come on, if a function signature is:

   ord(c)
It's pretty damn obvious not to use "c" as a keyword argument. And as hard as it is for programmers to name things, I can't recall ever struggling to name a function parameter.

If you need to rename a parameter that it's likely someone might be calling as a keyword, then bite the bullet and break backwards compatibility or just introduce a new function name if backwards compat is truly that important. (And besides, this has to be one of the more trivial checks for a static analyzer to find.)

> the number (and optionality) of positional parameters is completely invisible and implicit rather than readable and explicit

It's invisible in the function signature, but it should be in the function documentation or as a last resort, read the source. I find myself reading Python library source code not infrequently because it's not unusual to need to understand a function's behavior. Getting the function call correct is like the least hard part? But again, this really feels like an edge case.

> it requires various non-trivial assertions repeated every time.

In fact, that's a pretty trivial assertion if you ask me. ¯\_(ツ)_/¯.

FWIW, I don't find:

    (arg=None, /)
to be a legible syntax for declaring "this function takes a single optional positional argument". And in its full form, this is truly ugly:

   def f(pos1, pos2, /, pos_or_kwd, *, kwd1, kwd2):
Seriously, WTF. The library author couldn't make up their mind about pos_or_kwd? Why even allow that? This is no better:

   def f(pos1, pos2, /, *, kwd1, kwd2):
Just no.

Maybe I'd be more convinced with a cleaner syntax. I think the Per-Argument Marker option proposed in the PEP is a lot more obvious (not sure about using a '.' though) and think the argument against it is pretty weak: "While this approach may be easier to read, it has been rejected because / as an explicit marker is congruent with ∗∗ for keyword-only arguments and is less error-prone."

I note that "less error-prone" is stated without evidence and that "is congruent with ∗∗" is definitely against the zen.

But generally, as I said, the entire PEP is not convincing to me. Anyway, I'm sure I'm repeating the arguments against the PEP others have made and Guido has ruled in favor anyway, so I'm just screaming at the ocean.


On one of the threads, they suggest destructuring syntax.

Taking that a bit further and ignoring backwards compatibility entirely:

    def foo([po1, po2], pk1, pk2, *args, {kw1, kw2, kwd1=default, kwd2=default, **kw}):
       pass
And it implies some obvious shortcuts:

    def all_pos_only[po1, po2, po3=default, *args]:
       pass

    def all_kw_only{kw1, kw2, kw3=default, **kw}:
       pass


As a fan of Clojure, I would love for Python to get rid of most of strange and archaic (IMO) unpacking syntax and just switch wholesale to Clojure-style destructuring. In my opinion, it's a single solution for all of those problems and it ends up looking a lot cleaner. It's also just a lot easier to work with if you're trying to unpack a single value from a nested structure or define a flexible API.


If I ever saw this in real life:

    foo([po1, po2], pk1, pk2, *args, {kw1, kw2, kwd1=default, kwd2=default, **kw})
I'd think... that function is trying to do too much with its parameters and should be multiple functions.

Now yes, I realize an author could already do that with ∗args and ∗∗kwds, but the fact that the author would need to handle the argument parsing is sufficient disincentive to prevent most people from doing so.


But if you had to refactor the above function, you wouldn't have to reverse-engineer some argument parsing. You'd know how it must be called anywhere in the project, just by looking.

And generally, I think the disincentive rationale is self-defeating:

* the premise behind disincentive is you're dealing with someone who is (at least in the moment) careless

* now they're going to have to write more code to do a thing they've set out to do

* but we've already selected for an individual who is writing carelessly

In this case, it can only work if extra argument checks are the last straw that causes this individual to say "enough" and refactor that function. For what my experience is worth, I think this is unlikely.


> The library author couldn't make up their mind about pos_or_kwd?

I really like the possibility of optionally calling a positional-argument function with named keyword arguments, which can be more legible in some calling contexts. I would argue that in most cases most function arguments should be of the pos_or_kwd type. YMMV.

The reason Python added keyword-only arguments was so that library authors could freely reorder them (e.g. add a new keyword argument before another, or add another positional argument before keyword arguments) without breaking backwards compatibility.

The only reason to have positional-only arguments is to allow libraries to rename their arguments (without changing the semantic meaning) and guarantee that calling code doesn’t break. This doesn’t seem like a very strong need.


> would argue that in most cases most function arguments should be of the pos_or_kwd type.

Oh, that’s definitely the case. My point was: why would you have a function that had three different kinds of params: pos only, keyword only, and pos_or_kwd. That would be insane. You probably wouldn’t. That it’s allowed (after this PEP) is an consequence of this change which is just more evidence to me that the PEP isn’t needed.


I've asked this question too and what I think what is going on is that the python dev community is still pretty active - and people love to change things, get their code and ideas into the language. I think they've been pretty good about rejecting a lot of the really awful stuff, but not as good maybe at thinking about what is really necessary.

And this type of discussion makes me think about Django - it used to be a nice little simple framework, then it got a big community around it and grew every nature of features to such a point that I think nobody should actually use it. The code ends up being pretty awful and I think probably the rise of Flask coincides with Django getting too big and too complex over a period of years.


Jose Valim has said that there is no plan for an Elixir 2.0, because he feels it is pretty much "done" in terms of major features. It's a refreshing change.


I feel the same but about Java. I feel that with Java, however, it is much, much worse because of its enterprise friendliness and thus being much more reluctant about breaking changes to the language. I would love to see certain features getting dropped from languages as new ones are added. I think Python has done an okay job of keeping the language consistent and simple and cleaning up the libraries, etc.


Funny enough, I first used Java in 1995 or so. What a nice little language I thought then. I then managed to avoid it through its enterprise madness of the 2000s and finally started using it again around 5 years ago. It's still bureaucratic but I don't find the language itself complex.


> Can a language ever reach a point of being done?

TeX? Yes, I know there are still new releases occasionally, but as far as I know, it's feature-complete, and only occasionally gets bug fixes. I wish the developers of more languages, and software in general, would recognize when their projects were done. Unfortunately, that doesn't seem to be the way things are headed.


Unchecked type annotations were when change for the sake of change seemed to take over in Python.


perl is hard only if you come from a certain point

trust me, if you have no apriori regarding what a programming language is, perl will make a lot of sense


Well if you want my path it was something like: BASIC, 6502 assembly, Fortran, Pascal, C, C++, Scheme, Perl, shell (sh, tcsh, sed, awk, etc), Java, Python. A bit of PHP in there somewhere. Some TCL and lua and ruby too.

I may have the order a bit mixed up. I started with perl 4 and got to be reasonably adept, using it as my primary language from maybe 96-2000. I've used Python as my primary language since 2000. I'm pretty fluent in C, Java, Objective C and shell.

Anyway, I had to fix a couple hundred line Perl script the other day and it had me scratching my head for long enough that I was pretty close to rewriting the whole thing in Python. One thing I really love about Python is that it's trivial to test bits and pieces of code via the REPL.


you have a point, perl might give too much linguistic rope for people to hang themselves and future maintainers, but I'd slightly argue that it's because people don't grasp the nature of contexts


python lost the 'only one obvious right way' zen long ago sadly.


Keyword arguments are safer and more explicit/self-documenting. It's not clear that further accommodations for positional arguments are needed, except in variadic situations as described in https://www.python.org/dev/peps/pep-3102/. I would argue the use case described in the "background" section constitutes an unsafe level of uncertainty.

One important use case for the existing facilities is to take a function whose signature used to contain positionals, and migrate a positional to be a keyword argument without breaking existing uses of the API.

I think `/` is a poor choice of symbol, too. If such a separator were to be a good idea, it should be * * by analogy with `*` for separating keyword-only args.


> Keyword arguments are safer and more explicit/self-documenting.

Keyword arguments are also more redundant (a function taking a single argument usually does not need that argument to be named, quite the opposite, even for some multi-parameter functions it makes no sense e.g. `max(a=1, b=2)` is downright no matter how you name the arguments)), and the lack of positional-only parameters means any old name to what was intended as a positional effectively becomes part of the API.

I think Swift has the better pick there, an argument can be either positional or named, not both, and positional-only parameters are used to great effect. Sadly Python's history of positional-or-named can't really be fixed.


> a function taking a single argument usually does not need that argument to be named

Would you rather have:

thread_pool.shutdown(True)

or

thread_pool.shutdown(wait=True)

Even with single argument, it's not always clear what the argument means from the function name, so being explicit makes the code more readable.

EDIT: I realize you said "usually", but my point still stands.


You can maintain clarity with your positional arguments with just:

wait = True

thread_pool.shutdown(wait)


I am sorry but this looks too much like Java to me.


Keyword arguments also encourage overly general functions that try to do everything, and there's a performance cost to that (such functions tend to include a mini arg parser and dispatch mechanism), and it's a wash on documentation, since looking up how to use such a function really isn't any harder than looking up which of a number of functions to use.


I think these arguments usually come down to the domain of problems you work with. Some people simply don’t write code where it makes sense to have positional arguments. Some domains have arguments where positional arguments are more natural/self-documenting.

For example, concatenation, mathematical operations.


Some people are saying the syntax is ugly, and I agree. But it addresses a real problem in Python: the names of the arguments of your public functions are also part of your public API (since they can be called as kwargs). In my experience, this is rarely desirable and makes fixing bad names unnecessarily hard.


And worse, they cause problems in some APIs e.g. defining / overriding MutableMapping.update(), it should take a single optional positional-only parameter and any number of keyword arguments, currently this requires using *args and hand-rolled assertions, otherwise you can get kwarg conflicts and other such issues.


Function names and class names are also part of your API and there's no facility to allow those to be arbitrarily renamed. Being able to rename positional parameter names is a very weak argument in favor of this feature.

The only argument I'm even remotely sympathetic to is "If a function uses a name for a parameter, that precludes callers from using it as a keyword argument elsewhere in the argument list." But the workaround for that rare case is to use:

   *args, **kwargs


This seems pretty nuts. I'm with Steve Dower here - make all parameters allowed as keyword arguments.

Guido's objection that "writing len(obj=configurations) is not something we want to encourage" makes not one iota of sense. Allowing it is not encouraging it.

There's mention of one bug which allegedly could have been fixed with positional-only parameters:

https://bugs.python.org/issue9137

The bug here is that dicts have an update method which you can call two ways, by passing a single dict, or with keyword arguments, and if you try to call the single-dict version using a named parameter, it's not clear what should happen, and at one point, for OrderedDict, it could end up throwing an exception.

But the mistake here isn't confusing parameters, it's having a single method that you can call in two different ways. That sort of thing might fly in Perl, but it's a source of ambiguity and confusion that, IMHO, is not up to Python's standards.


The arguments for this change undervalue one of my favorite things about Python: developer freedom. The language is versatile to the point of being inefficient, but that lets me adapt it as necessary. Just because they can't imagine a situation where this change would be a roadblock doesn't mean it won't happen. What about building an argument list with a dictionary and then passing it as an expanded dictionary?

Also, I don't like the idea that package developers' opinions on what's "readable code" outrank mine, when we're talking about my code.


The more and more I see the direction python is taking, the more it feels like python will eventually become the new Perl. I honestly feel, if Python 2.7+ had "fixed" unicode and included orderedDict (which I guess came after 3.5) I would happily stay with it.


Python 2.7 has it... "from collections import OrderedDict"


If backward compatibility is a goal, one can't make capricious changes to the order of positional args. It seems pointless to make special provisions to prevent named args for the sake of "more flexibility" when positional is decidedly less flexible.

You can always opt for dict args if you want to change a signature and maintain backwards compatibility with obsolescent arg names.

If I was BDFL this would be a dead issue.


I do find having keyword parameters for free, as Ada does, to be an interesting feature, but the need to make the parameter names part of the interface is a drawback. Common Lisp separates positional parameters and keywords, but at the cost of some efficiency and yet other drawbacks.

In general, though, I don't see the issue with always needing to pick good names. It's very simple to use single-character names if one needs to, such as for a mathematical function, or simply a procedure where the letters have clear meanings or will likely never be used to start with.

Asides from the design features I thought were jokes when I first learned of them, such as one-line lambdas and whatnot, this ''beginner's language'' certainly has a great deal of complicated syntax sugar and other things being added to it.


In Raku, parameters can be either positional or named parameters (declared with the colon-pair syntax). However, from what I understand, only positional parameters can be passed as positional and named parameters as named. The closest Raku subroutine to Python's `def fun(a, b, c=None)` would be:

    sub fun($a, $b, :$c) { }

    fun(1, 2)                # This works!
    fun(1, 2, 3)             # Error: Too many positionals passed...
    fun(:a(1), :b(2), :c(3)) # Error: Too few positionals passed...
    fun(:c(3), :a(1), :b(2)) # Same as before
    fun(:c(3), 1, 2)         # This works!
   
Positional parameters are required by default but they can be made optional by providing a default value in the subroutine's signature or by placing a `?` right after the parameter. However, much like Python, positional parameters must appear before optional parameters:

    sub fun($b, $a?, :$c) { }
    
    fun(1) # This works!
On the other hand, named parameter, as evidenced by the first code snippet, are optional. However, they can be made required by placing a `!` right after the parameter:

    sub fun($a, $b, :$c!) { }
    
    fun(1, 2)        # Error: Required named parameter 'c' not passed
    fun(1, 2, :c(3)) # This works!

Remaining arguments can be slurped by using slurpy parameters. In its most basic form, slurpy positional parameters are indicated with `A` followed by an array parameter:

    sub fun($a, $b, A@rest) { }
However, regardless of the structure of the arguments, they are flatten. This automatic flattening can be avoided by using `AA` instead. Or it can be done conditionally by using `+`: If there's a single argument, flaten it. Else, leave them alone.

Slurpy named parameters are indicated also with `A` followed by a hash parameter:

     sub fun(:$c, :$d, A%rest) { }
Both can be used in conjunction:

     sub fun($a, $b, :$c, :$d, A@rest, A%rest) { } # Or:
     sub fun($a, $b, A@rest, :$c, :$d, A%rest) { }
Edit: HN eats up the asterisks so using `A` to stand for asterisk.


There is a simple solution to this problem. Just name your parameters like this:

    def fun(parameter1, parameter2, parameter3):
        a = parameter1
        b = parameter2
        c = parameter3
        ... (rest of function)
This also helps the user to see in an eyeblink which position corresponds to which parameter.


Is this a new thing? I thought that was possible in Python already. Look at the numpy documentation: https://docs.scipy.org/doc/numpy/reference/generated/numpy.m.... There, they already use the / for positional-only arguments.


> Is this a new thing? I thought that was possible in Python already.

Yes and no: the syntax was used in documentation (that's the "argument clinic" mentioned in the article) and Python functions defined in C were able to define positional-only parameters, but it's a new thing for Python code: before 3.8, it's not possible to define positional-only parameters in pure Python[0], with it the "argument clinic" syntax becomes actual Python syntax (rather than pseudo-python).

[0] well you can use *args and a bunch of hand-rolled assertions


You can already do:

  def show((a, b, c)):
    print 'a = %s, b = %s, c = %s' % (a, b, c)
And there's no way to pass a, b, and c other than positionally. You can write:

  show((1, 2, 3))
But not:

  show((a=1, b=2, c=3))


That feature was removed in python 3:

    $ python2
    >>> def f((a, b)):
    ...   return a + b
    ...·
    >>> f((1, 2))
    3

    $ python3
    >>> def f((a, b)):
       File "<stdin>", line 1
        def f((a, b)):
              ^
    SyntaxError: invalid syntax




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: