Here's my understanding of how this works in a Poon-Dryja payment channel[1]. I haven't reviewed the Stellar proposal to see if it's substantially different.
Let's assume an Alice-Bob channel funded with 1 BTC, and look at how we get from payment state 1 (Alice: 0.5 BTC, Bob: 0.5 BTC) to payment state 2 (Alice: 0.4 BTC, Bob: 0.6 BTC), and who could do what during the transition. This is just a sketch, but I think it fills in what you're asking.
Payment state 1:
* Alice holds a transaction C1a, that only she can broadcast. C1a has two outputs: 0.5 BTC -> {Alice in the future; or, Bob if signed by Alice's key Ak1}; 0.5 BTC -> Bob.
* Bob holds C1b, which looks like C1a but with names reversed. [Names reversed except both transactions send the same amounts to each party. In this special example case of a 50/50 distribution, that comes to the same thing.]
Transition to payment state 2:
1. Alice and Bob create new transactions C2a and C2b, half-sign them, exchange them, yada yada. This is complicated and has internal steps where only some of the information has been exchanged, but I don't think it's the part you're asking about.
At this point, Alice could broadcast C1a or C2a without penalty, and Bob could broadcast C1b or C2b without penalty, so the channel hasn't conclusively moved to payment state 2. Any broadcast will either close the channel in payment state 1, which it's coming from, or payment state 2, which it's going to. So as long as everyone knows that payment state 2 isn't confirmed (which they do), this isn't really a threat.
(With physical currency: if I hand you a nickel, there's a period where it's not clear if you're going to take it and it's not clear if I'm going to let it go, but at the end of that period either I paid you — the new state — or I didn't — we're back where we were before I started to hand you the nickel, but no earlier.)
2. Alice sends Bob Ak1.
Now if Alice publishes C1a to the blockchain, he can take its both outputs. So she wouldn't do that. Alice can safely publish C2a, and Bob can publish either C1a or C2a. We're halfway to payment state 2.
3. Bob sends Alice Bk1.
Now Bob can't publish C1b without penalty, either.
If Alice and Bob are rational, nobody will ever broadcast C1a or C1b. (If Alice — the only person who can broadcast C1a — does so, she forfeits the channel funds to Bob; and vice versa.)
The channel is now in payment state 2.
---
[1] J. Poon, T. Dryja. “The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments”, pp. 22–26, Section 3.3.4, “Creating a new Commitment Transaction and Revoking Prior Commitments”. This uses the terminology Commitment Transactions, Revocable Delivery Transaction, and Breach Remedy Transaction to describe the concepts that I've either waved my hands about or completely elided, above. It has lots of pictures. “Payment state” is my non-standard neologism.
---
“I got here the same way the coin did.” – C. McCarthy, No Country for Old Men.
Let's assume an Alice-Bob channel funded with 1 BTC, and look at how we get from payment state 1 (Alice: 0.5 BTC, Bob: 0.5 BTC) to payment state 2 (Alice: 0.4 BTC, Bob: 0.6 BTC), and who could do what during the transition. This is just a sketch, but I think it fills in what you're asking.
Payment state 1:
* Alice holds a transaction C1a, that only she can broadcast. C1a has two outputs: 0.5 BTC -> {Alice in the future; or, Bob if signed by Alice's key Ak1}; 0.5 BTC -> Bob.
* Bob holds C1b, which looks like C1a but with names reversed. [Names reversed except both transactions send the same amounts to each party. In this special example case of a 50/50 distribution, that comes to the same thing.]
Transition to payment state 2:
1. Alice and Bob create new transactions C2a and C2b, half-sign them, exchange them, yada yada. This is complicated and has internal steps where only some of the information has been exchanged, but I don't think it's the part you're asking about.
At this point, Alice could broadcast C1a or C2a without penalty, and Bob could broadcast C1b or C2b without penalty, so the channel hasn't conclusively moved to payment state 2. Any broadcast will either close the channel in payment state 1, which it's coming from, or payment state 2, which it's going to. So as long as everyone knows that payment state 2 isn't confirmed (which they do), this isn't really a threat.
(With physical currency: if I hand you a nickel, there's a period where it's not clear if you're going to take it and it's not clear if I'm going to let it go, but at the end of that period either I paid you — the new state — or I didn't — we're back where we were before I started to hand you the nickel, but no earlier.)
2. Alice sends Bob Ak1.
Now if Alice publishes C1a to the blockchain, he can take its both outputs. So she wouldn't do that. Alice can safely publish C2a, and Bob can publish either C1a or C2a. We're halfway to payment state 2.
3. Bob sends Alice Bk1.
Now Bob can't publish C1b without penalty, either.
If Alice and Bob are rational, nobody will ever broadcast C1a or C1b. (If Alice — the only person who can broadcast C1a — does so, she forfeits the channel funds to Bob; and vice versa.)
The channel is now in payment state 2.
---
[1] J. Poon, T. Dryja. “The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments”, pp. 22–26, Section 3.3.4, “Creating a new Commitment Transaction and Revoking Prior Commitments”. This uses the terminology Commitment Transactions, Revocable Delivery Transaction, and Breach Remedy Transaction to describe the concepts that I've either waved my hands about or completely elided, above. It has lots of pictures. “Payment state” is my non-standard neologism.
---
“I got here the same way the coin did.” – C. McCarthy, No Country for Old Men.