My issue with this that is will eventually sneak into libraries and the users of that library would be expected to use these tag strings all over the place to utilize the library. This prevents people from having a uniform coding style and make code harder to read.
The concern isn't having features that will make it easier to write DSLs, my problem is that people will misuse it in regular Python projects.
I know that one of the authors are Guido, but I'm not buying the motivation. Jinja2 and Django template are pretty much just using Python, it's not really much of an issue, and I don't believe that business logic should exist in your templates anyway. As for the SQL argument, it will still be possible for people to mess it up, even with Tag Strings, unless you completely remove all legacy code. The issue here isn't that the existing facilities aren't good enough, it's that many developers aren't aware of the concepts, like prepared statements. If developers aren't reading the docs to learn about prepared statements, why would they do so for some DSL developed using tag strings?
Obviously Guido is a better developer than me any day, so I might be completely wrong, but this doesn't feel right. I've seen tools developed to avoid just doing the proper training, and the result is always worse.
There’s Python the scripting language, replacement for bash scripts, R and Lua.
Then there’s Python the serious software development language. Where people have style guides, code review, test coverage, and projects with more than one directory.
I understand why people in the second group are fearful of this feature and DSLs in general. I’m in the first group and I’m quite excited for it.
I'm in both groups and I'm getting tired of how Python's syntax additions increasingly turn it into a mechanism for mid-level engineers to assert their dominance over colleagues by writing code that only people who have been deeply immersed in Python for years can understand.
Our Python users' Slack channel at work is already overcrowded with messages to the effect of, "halp what's this syntax how does this code work."
This is a problem with our industry that people get paid $$$ to use a language but at no point are they compelled to read the manual for that language from cover to cover.
Hi, I'm a PEP author. The hope was that users would say "I know f-string syntax so I know this syntax." Similar to JS and its template literals -> tagged template literal jump.
That would be an easy mental jump for skilled python users, but I think that it would be surprising for many less experienced users. Just giving it a function call syntax would be more in line with the principles established in PEP-20 IMO.
I'm in the second group and I'm very excited for it, getting the linting right to prevent people doing wild formatting into dangerous string-based APIs is not easy, this provides an opportunity to make it much easier and safer.
I can get behind that. I see some of the same issue in regards to type annotation. I'm heavily leaning into the dynamic / duck-typing aspects of Python, so type annotation is often complicated or very broad, to the point where it's a little redundant. If you're not really writing code like that, type annotation is an awesome addition to the language.
I'd be very interested in seeing where this goes, it certainly has it's uses, but there's also a ton of projects where it really doesn't belong.
I'd be interested in chatting with you sometime on duck typing (protocols) and how it (along with this PEP) could bring component-driven development to Python.
It was a reference to PyTorch, admittedly somewhat oblique.
Python is very popular for machine learning, in days gone past ML researchers used the Torch library in Lua. TensorFlow and PyTorch made Python the dominant language in that field.
Hi, I'm one of the PEP authors. Your point covers one of my motivations, so here's a bit about what interests me. Mainly, "Jinja2 and Django templates are pretty much just using Python."
I see things differently. In those, nearly everything is a parallel universe where things aren't Python. Scope rules. Imports. Control flow. Calling things. Navigation. Refactoring. Linting. Therefore, Python tooling can't really help much. Tools have to write special rules to help on each template language and framework.
Instead, imagine if Black, Ruff, mypy, pyupgrade, and IDEs could treat the template part as software.
I realize I'm describing "people who want Python tooling", which might be a minority. I also realize there's pushback about additions to Python. That said, I think TSX and friends have shown -- there's lots we can do to improve Python web development.
But you could do that already, could you not? Django does. Its just not really SQL anymore then.
Someone else also pointed out that you could just do this with functions. It seems like a very fancy way of avoiding using (). I don't know, maybe show me how that would solve the issue of unsafe SQL and I'd be more easily convinced.
{ query: 'select * from users where id = $1', values: [id] }
So if you tried an injection like this:
sql`select * from users ${'where id = 3'}`
It turns into an invalid statement since "where id = 3" cannot exist as a parameterized value for the same reason this doesn't work:
{ query: 'select * from users $1', values: ['where id = 3'] }
Where you go from here is to offer a query(statement) function that requires the use of the tag string so that you can't accidentally pass in a normal string-interpolated string.
bash -- DSL for invoking subprocesses :: what you rather see in a code review: one-line shell pipeline or equivalent Python code? https://stackoverflow.com/questions/295459/how-do-i-use-subp...
sh-like DSL on top of Python literal strings that doesn't invoke shell might be interesting (pep may help)
regex -- DSL for search/replace in text. Useful in moderation
jq -- DSL for search/replace in json. Useful on the command-line
xpath -- DSL for searching trees (hierarchy)
sql/xslt -- I would rather read Python most of the time instead but sometimes it is helpful to have an option of writing SQL directly (pep may help)
toml/json -- writing Python directly is preferable (in the context of .py file)
markdown -- DSL for writing markup (instead of html/word). I wouldn't say no to inline docstring rendering in Python code (pep may help).
The same for (subset of) latex -- for math formulas `$e^{i \pi} -1 = 0$`
dot/plantuml (ebnf, sequence diagrams, etc) could be useful for literate programming-heavy style.
I have to admit that at first glance I don’t like this. These seem to be essentially normal str -> Any functions, with some naming limitations due to the existing string prefixes special-cased in the language. I don’t feel like adding this additional complexity is worth being able to save two parentheses per function call.
I think at this point Python really needs to just settle down. I don't like this not because it's an intrinsically bad idea, but adding another thing to the already fairly large pile of things a Python user needs to know in order to read somebody else's code needs to be something that brings more benefits to the table than just "it slightly improves a particular type of function call".
At the risk of riling some people up, this smells like Perl. Some poor Python user comes across
greet"Hello {user}"
and there isn't even so much as a symbol they can search for on the internet, just an identifier smashed into a string.
But I guess Python and I parted ways on this matter quite a while ago. I like to joke about Katamari Dama-C++ but Python is starting to give it a run for its money. C++ is still in the lead, but Python is arguably sustainably moving more quickly on the "add more features" front.
My guess at the challenge is that the community who maintain and develop a language are by that very nature not in touch with what the complexity feels like for the average user.
Also it’s harder to do nothing than something.
That being said, I think this is partly abstract. I’ve just ignored a lot of new Python features without issue. And while I worried that they’d force me to learn new things to understand others’ code, that’s not really materialized.
That is probably a much better example than any of those present in the PEP. I quite like your example. I'm not sure I'd want to write code like that, but it shows the usefulness much more clearly.
Doing `html('<div>Hello {name}</div>')` would be possible. I have a system that's based on it. Two issues:
- No tooling in Python will do anything with the string, so DX suffers.
- Evaluating the variable requires frame evaluation, which is...problematic.
You could do `html(f'<div>Hello {name}</div>')` and get f-string coding assistance. But you'd also get immediate evaluation. There's nothing the `html` function can do with the `{name}` part.
This was my first thought as well. But an important difference is that the arguments are not eagerly evaluated, but they are passed as lambdas which can be evaluated if desired. This means that it can be used for example in log messages (if you don't want to evaluate the string at the wrong log levels). But is it worth it for that? Idk.
Even if eager evaluation it's already a very compelling way of managing basically every lightweight "templating" for safety: e.g. embedded dynamic HTML or SQL. `markupsafe` is great, but it's way too easy to perform formatting before calling it, especially with f-strings.
That f-strings were "static" was by far my biggest criticism of it, given how useful I find JS's template strings.
And this proposal seems like a straight up better version of template strings:
- the static strings and interpolations are not split and don't have to be awkwardly re-interpsersed which I've never found 100% trouble and 0% utility
- the lazy evaluation means it can be used for things like logging (which really want lazy evaluation), or meta-programmation (because you can introspect the callables, and you get the expression text)
The expressions for the data you want to log out can be expensive, so ideally you only want to compute them after you’ve checked if the logger was enabled for the level you need.
In most APIs this requires an explicit conditional check and the average developer will not think of it. This allows said check to be performed internally.
Actually, the % syntax eagerly evaluates the log string, you need to pass the variables as arguments to the logging function like this: logger.debug("bla: %s", myvar).
It's such a subtle difference that I only notice it when my IDE underlines it :/
But in Python I could also imagine YET ANOTHER constant prefix, like t"", that returns a "template" object of some sort, and then you could do html(t"") or whatever. That is, it would be just like f"" but return an object and not a string so you could get at the underlying values. Like in JavaScript the ability to see the original backslash escaping would be nice, and as an improvement over JavaScript the ability to see the expressions as text would also be nice.
But the deferred evaluation seems iffy to me. Like I can see all the cool things one might do with it, but it also means you can't understand the evaluation without understanding the tag implementation.
Also I don't think deferred evaluation is enough to make this an opportunity for a "full" DSL. Something like a for loop requires introducing new variables local to the template/language, and that's really beyond what this should be, or what deferred evaluation would allow.
I don't particularly mind the prefix thing. It came up in the PEP discussion, as did choice of backticks to indicate this is different. But JS template literals -> tagged template literals shows, you can get from A to B without a fundamental change.
I'm very interested though in the deferred part. I agree that there is complexity. I weigh that, though, against the complexity of existing Python HTML templating, where finding out what just happened is...harder.
I think we can get a TSX-level of DX out of this. And maybe a Lit level of composition. Agree that it is non-zero complexity.
Which isn't really templating at all, but just the ability to use inline expressions and easily construct objects does get the job done.
Or in a SQL builder with JavaScript tagged templates, I do:
exec(sql`
SELECT * FROM a_table
WHERE category = ${category}
${subcategory ? sql`AND subcategory=${subcategory}` : sql``}
`)
That is, I nest tagged templates to handle different logic conditions and loops.
If there's deferred execution, it's done with ?: and .map() – though these very long expressions don't work nearly as well in Python. (List comprehension is in some ways better than .map()/.filter(), but not for very large expressions like in a JSX template.)
This like a bad idea on the first glance? Maybe I don't get the whole pitch here?
It just doesn't seem worth it to define a whole new thing just to abstract over a format() function call. The laziness might be interesting, but I feel like "lazy strings" might be all that's needed here. Laziness and validation (or custom string formatting logic) are separate concerns and should be separated.
> It just doesn't seem worth it to define a whole new thing just to abstract over a format() function call.
That could also be leveraged at f-strings themselves.
> Laziness and validation (or custom string formatting logic) are separate concerns and should be separated.
In which case the one to move out is the laziness not the customised interpolation. Because the latter is the one that's necessary for safer dynamic SQL or HTML or whatever.
If a syntax such as f"{variable}" is already a feature - and turned out to be a popular one - why shouldn't we be able to add our own custom "f"s? Because that is what this is about. It might make generating output even simpler.
I applaud the idea and am pleased to see that Python keeps innovating!
What was wrong with the standard way to write function application?
Python is sufficiently dynamic that an implementation of greet(...) can look up one level to resolve {name}, right? That's why Python will forever run like a dog. Might as take advantage of it to build such capabilities in user space.
This crap is going to end up inside f-strings inside tag-strings inside f-strings inside... We have a language. Don't extend it to express what it's perfectly capable of expressing already.
Your reply appears to indicate that you do not properly understand the new proposed feature. It is most certainly not just about dropping two parentheses.
> Tag strings extract more than just a callable from the Interpolation. They also provide Python string formatting info, as well as the original text.
The feature is akin to moving print from a keyword to a function. That change also made a huge difference in that it unified the output stream and avoided having undefined objects like a "print" keyword.
Here, you can think of the feature as moving an "f" string from a hardcoded, predetermined definition to a generalizable and programmable behavior.
If "f" strings have become so popular so quickly it means they addressed a pressing need. It is logical to assume that a programmable version of an "f" string would be even more useful.
(PEP co-author here.) You've described it well. As the "How to teach it section" emphasizes, we'd like consumers of tag functions to just think of it as an f-string with other stuff that happens before evaluation.
From their POV, inside the quotes, what you know about f-strings, you know here as well.
Why could you not know these things without a language feature?
> ...other stuff that happens before evaluation...
A greet(string) function could parse the string and resolve the names itself:
parsed = parser(string)
resolved = resolver(parsed)
return formatter(resolved)
If you hate boilerplate, make the first two steps into a decorator.
A PEP introducing a grand unified theory of magic (tag strings) isn't inherently better than the status quo of some (f-string) magic. Less magic is better.
Frame hacks with sys._getframe necessarily imply dynamic scope not lexical scope. Dynamic scope does not work with nested functions, including comprehensions. See this issue with the htm library, https://github.com/jviide/htm.py/issues/11
I hate the idea of reusing the existing string/bytes prefixes for something that is completely different. How is someone expected to know that br"" is inherent Python syntax and my"" is essentially an user-defined function? And the only way to ever add a new prefix into the language (like f"" was added quite recently) is to wait until Python 4, at which point we'll need 3to4 to automatically rename all of your old tag strings that are now conflicting and people will bitch about how badly major Python upgrades suck.
It seems the purpose of this proposal is to have a way to essentially have custom string interpolation. I don't think that's necessarily a bad idea on its own, but this syntax feels out of place to me.
Instead, why not add a single new string prefix, like "l" for "lazy"? So, f"hello {name}" would immediately format it while l"hello {name}" would produce an object which contains a template and the captured variables. Then their example would be called like: greet(l"hello {name}").
I can't help but believe that this is introducing more spooky action at a distance and is bound to be abused. Is it really more usable this way? Do they have any concrete and practical examples where this improves readability?
I would have loved to see Java introduce something similar to the IntelliJ @Language-annotation in the standard library but maybe they'll figure out the sweet spot in a future String Templating JEP.
Yikes. Don't get me wrong, I totally understand the reasoning why this would be useful (though I violently disagree with the idea of deferring the evaluation of the contained expressions), but it's also so very kitchensinky and adds so little over just calling a function (which doesn't require a 20-page explainer, as everyone already knows how function calls work). It also promotes using what looks like string interpolation (and what might be string interpolation, you can't tell at the "call site") for things which we know string interpolation is the wrong tool. The API also seems really, I dunno, weird to me. The string is split around interpolations and verbatim portions result in one argument, which is "string-like", while interpolations become four-tuple-like (one of which is a lambda, which you call to perform the deferred interpolation). This seems really awkward to me for building stuff like the suggested use cases of XML/HTML or SQL templating.
Also the scoping rules of this are a special case which doesn't appear in regular Python code so far: "The use of annotation scope means it’s not possible to fully desugar interpolations into Python code. Instead it’s as if one is writing interpolation_lambda: tag, not lambda: tag, where a hypothetical interpolation_lambda keyword variant uses annotation scope instead of the standard function scope." -- i.e. it's "as if you wrapped all interpolation expressions in a lambda: <expr>, except it uses different scoping rules".
> This seems really awkward to me for building stuff like the suggested use cases of XML/HTML or SQL templating.
Compared to what?
At the end of the day you're still doing string formatting, if you want parsing, then you'd feed the item into a parser, which this doesn't preclude.
The interface sounds a lot better than JS's anyway, as that completely separates the literal strings and the interpolations so you have to re-intersperse them which is muggy.
> interpolations become four-tuple-like
They become an Interpolation object, which can be unpacked if you find that more convenient, but you can access the members if you prefer:
- 0 is getvalue is the callable to retrieve the evaluated expression
- 1 is expr is the raw text form of the expression
The intent here is to support the following approach for tag function authors:
1. Parse to an AST, generally using an off-the-shelf parser. In practice, it's possible to rewrite interpolations with a placeholder suitable for a given language, eg x$Nx for HTML. Of course if that doesn't actually work, you might have to write/modify an existing parser. Hopefully we can cleverly avoid this extra work.
2. Walk/compile the AST, filling interpolations, but taking in account the context. This can for example take the form of building appropriate query strings that avoid Bobby Tables SQL injection, whether by mapping to SQL placeholders or with appropriate quoting (such as for a column or table name).
3. Memoize these steps, much as we see with builtin DSLs in Python, like the re module; see https://github.com/python/cpython/blob/3.12/Lib/re/__init__.... We do plan to make this easier/faster by supporting getting the original source of the template string (Template.source), vs the *args approach we show in the PEP at the start of this discussion (this will become Template.args instead; Template here is the proposed protocol of the object passed in).
This seems very unpythonic in the way that it breaks the one best way to do things adage. It’s syntax sugar for a function call. Just keep it a function call if needed.
The much bigger feature here is buried under the DSL stuff. Python is effectively implementing a method of lazy evaluation of function arguments! I thought I would never see the day! It's crazy that if this PEP is accepted, functions in Python will actually be a special case of f-strings.
I hope they eventually grant this power to regular functions because otherwise I know folks will end up myfunc"{arg1},{arg2}" to get that feature.
Off-topic, but when did Python become so... verbose? From the PEP:
def greet(*args: Decoded | Interpolation) -> str:
result = []
for arg in args:
match arg:
case Decoded() as decoded:
result.append(decoded)
case Interpolation() as interpolation:
value = interpolation.getvalue()
result.append(value.upper())
return f"{''.join(result)}!"
Isn't that just this?
def greet(*args):
def convert(arg):
return arg.getvalue().upper() if hasattr(arg, getvalue) else arg
return ''.join(convert(arg) for arg in args) + '!'
Guido has been pretty vocal about his preference for loops over comprehensions, so it's just a matter of personal preference. But they're also trying to use the newer pattern matching rather than duck-typing, and comprehensions don't support pattern matching (isinstance doesn't count).
The pattern matching stuff is neat but seems pretty half-baked. Only available to imperative code and restricted in what can be matched. I wish they'd finish fleshing it out because right now it feels tacky.
Actually my number one wish forever has been that functions would become actual first-class objects, able to be defined and manipulated during runtime at the syntax level. Would have been great for so much of my code, and the original decorator module. Things are better with the new inspect module but it's still ridiculously clunky.
iirc I wanted to encode state in a recursive-descent parser without additional complexity to the parser.
Similar in purpose not design; protostrings provides lazy and context-sensitive strings from the bottom up rather than this top-down template-style proposal, which I feel addresses most of the concerns here.
I tried skimming the PEP while I could, but it seems like this might be missing a couple of the features that make JS tagged template literals work so well:
- tags get a strings array that's referentially stable across invocations. This can function as a cache key to cache strings parsing work.
- tags can return any kind of value, not just a string. Often you need to give structured data to another cooperating API.
Deferred evaluation of expressions is very cool, and would be really useful for reactive use-cases, assuming they can be evaluated multiple times.
Thanks to the feedback in the Discourse thread, we will be changing the signature such that a tag function will be passed an object implementing a Template protocol, with attributes source, for the original source string as written; and attribute args for what was *args in the starting version of the PEP.
The string array is not referentially stable across invocations and cannot be because there is a single argument array containing both the "static" bits and the "dynamic" bits. So you can't use it the way that JS' `static_strings` argument can be used as a key in a `WeakMap`.
Tags can return any kind of value, so there's that.
Deferred evaluations can be evaluated multiple times and is in fact one of the biggest foot-guns in this API (in my opinion).
I wonder if each of the string args is stable? Then you could just use the first as the key.
The JS API where the strings are separate might seem awkward at first, but it ends up being a really elegant design.
Deferred evaluation is really powerful and I wish JS had it. It's one of the reasons why Solid has a custom JSX compiler and doesn't use tagged literals... to get the-evaluation you need to user to pass in closures, which is cumbersome.
Most managed languages do string interning. But the object passed as the string value doesn't seem to be just a string. It's wrapped in another object, which could have callsite identity.
For people adding insightful critique on the PEP on HN (I saw some on this thread already), please ensure your opinion is represented in the PEP thread itself too.
Given the names attached to the proposal, is this PEP actually up for debate?
Most of the critiques I am reading (with which I fully agree) is that this is more complexity in the language without sufficient payoff. There are now how many stupid or fancy ways to construct a Python string with a variable?
I think this will turn out well. Julia has had this forever as string macros, and it has worked out rather nicely, features like `r"\d+"` for regex, and `raw"strings"` are just string macros. The set of all useful custom literal strings isn't bounded, so a lightweight mechanism to define them and make use of the results is a good thing.
Another kitchen sink to add to Python's world-class kitchen sink collection.
At least in the spirit of "the language shouldn't be able to define things the user can't" (see: Java string concatenation) this seems like a good change.
Looks good. Would have been nice if they included a way to express type checking of the format_spec. That’s going to be an unnecessary source of runtime errors.
Hi, I'm one of the PEP authors. This is precisely my primary interest. TSX makes for a good HTML templating experience. Python templating approaches usually create a parallel universe of scope, syntax, imports, linting, type checking, IDE assistance, etc. where little of Python tooling exists.
... is this any different than a function like this:
greet("hello {world}")
which walks the call stack to find the variables, and uses them as locals / arguments?
If so: why not just do that? I would expect the performance to be kinda terrible compared to a language-intrinsic, but this hardly seems like a thing worth truly optimizing. And if it's too costly at runtime, someone can implement a parse-time AST rewriter, like many Python things already do. Heck, that example's `assert` is probably using pytest (because everyone uses pytest) and it's doing exactly this already, and it isn't a language feature, it's just a normal library using normal Python features.
It's been nearly 23 years, but Python 2.2 fixed this issue (https://docs.python.org/3/whatsnew/2.2.html#pep-227-nested-s...), and it's also why JavaScript added let (and const). f-string support also uses lexical scope, and it's an important part of its success.
This is actual code; you can run this. It will convert the curl command into a Req request and you will get a response back. This is really great, because we have been able to increase the expressiveness of the language.
"""
And somewhere there is the logic implemented to know what ~CURL is and what needs to happen with the part in square brackets, how it needs to be translated into a request call.
And if that is merely an arguments equals arguments thingy, then it would be kind of useless, since there are usually more things one specifies when making a request from inside Python. Things like headers or whether or not to verify TLS certs, and that is not a 1 to 1 mapping between curl arguments and a requests call.
So I remain doubtful, as long as no way is shown to me, how the user of the language can define this syntactic abstraction themselves, which is unlikely to happen or exist in Python.
The concern isn't having features that will make it easier to write DSLs, my problem is that people will misuse it in regular Python projects.
I know that one of the authors are Guido, but I'm not buying the motivation. Jinja2 and Django template are pretty much just using Python, it's not really much of an issue, and I don't believe that business logic should exist in your templates anyway. As for the SQL argument, it will still be possible for people to mess it up, even with Tag Strings, unless you completely remove all legacy code. The issue here isn't that the existing facilities aren't good enough, it's that many developers aren't aware of the concepts, like prepared statements. If developers aren't reading the docs to learn about prepared statements, why would they do so for some DSL developed using tag strings?
Obviously Guido is a better developer than me any day, so I might be completely wrong, but this doesn't feel right. I've seen tools developed to avoid just doing the proper training, and the result is always worse.