yield was first introduced as a statement similar to return. In Python not every statement is an expression; another example of non-expression statement is assignment, which was banned from expressions so that it couldn't be confused with ==.
Since then someone proposed (PEP 342) that yield could be used not just to yield values from coroutines, but to send one back in. yield becomes an expression, mostly used as the right side of an assignment. The parentheses are there because the grammar seems to handle the yield statement and yield expressions differently (the PEP mentions that "yield 12, 42" would have become confusing).
I might misunderstand the point you are making, but this actually stems from a "one and only one" mentality: there is only a single entry for "yield" in the grammar, and it is defined to take a specific kind of expression as its argument.
(edit:) (Additionally, there is the "one and only one" mentality that if yield is valid in a expression for a normal method, it should be valid in an expression for a lambda as well; however, that doesn't cause this syntax issue: statement vs. expression does.)
This level of expression is allowed to contain a comma. As an example, you can do "yield 1, 2", which will yield a 2-tuple containing 1 and 2 as elements.
However, if you wish to embed it as an argument in a function call, you will end up with an ambiguity, as arguments to functions are comma separated. You therefore must embed this expression in parentheses, lest you get a weird grammar conflict.
To be clear, this is identical to the case with tuples themselves, with the one caveat that they decided to make it actually be a syntax error to use it in a way that causes the ambiguity (I will be clear that I do not know if I agree with their decision on that regard).
"return 1, 2" <- valid
"a, b = 1, 2" <- valid
"a((1, 2))" <- valid
"a(1, 2)" <- not what you meant
You can figure out the difference once you get to the next token: ")" (the former) or ";" (the latter), but once you get that far you've already committed to specific parse tree formulations for the code you've already seen.
Ah, ok; that didn't seem related to this specific discussion of yield (yield is valuable to have as an expression even in a multi-statement function), but I can see how it is reminiscent.
It should be noted, though, that when you use the lambda keyword, the result is a function, and has the same properties and behaviors as a function (including that it can contain yield and be a generator).
The lambda syntax is simply a way to declare a function as an expression. You can't have a multi-statement block embedded in an expression due to fundamental limitations of normative whitespace.
There really, though, is only one kind of "callable": "function". There are two syntaxes for declaring it, however: "def", which takes a block of statements, and "lambda", which takes an expression.
This is (to me) drastically different than the behavior difference that I believe is what you are referring to in Ruby, where Proc and lambda have different semantics for embedded usages of the "return" keyword.
(That said, I want to be clear, to both you and any random audience: I personally agree with Ruby's decision to have these two forms of callable, per real-world conversations I've had with wycats.)
(afterword) Someone else might find this interesting: I was going to also make the point that both lambda and def result in the same class (function), but it turns out that Proc.new and lambda do as well (Proc).
Whatever goes inside a lambda must be an expression. The parentheses around yield make it a yield expression. Otherwise yield would be a statement which is a different thing as far as Python is concerned.
You can also use it inside generators:
>>> def f():
xyz = (yield)
>>> g = f()
# nothing happens as xyz == None
You're right, it isn't. The presence of yield makes the return value of f() a generator object. The function body is not executed initially. When you call .next() on the generator object execution starts and runs until it hits the first yield, raising a NameError because g is not defined.