Hello, I was one of the people who contributed to this interpreter change. Thank you Nelson for the excellent write-up and for going to such lengths to get to the bottom of this. That said, I wanted to defend Ken a bit. :-)
Ken Jin is a volunteer who's been tirelessly working to make CPython faster over the past couple of years for not enough credit. IMHO, Ken did nothing wrong here, and there's really nothing to be embarrassed about. The fact that it took a month (more if you consider the folks helping to reproduce the original results) for anyone to notice anything wrong shows how complicated the situation was! Put yourself in Ken's shoes -- having updated to LLVM-19 to enable the preserve_none flag, would you then hypothesize that LLVM-19 may have introduced an unrelated performance regression that no one noticed for five months? Lots of people underestimate how hard benchmarking is IMO.
A 1-5% performance improvement is also pretty valuable, just not quite as spectacular as we thought originally :-)
Ken Jin is a volunteer who's been tirelessly working to make CPython faster over the past couple of years for not enough credit. IMHO, Ken did nothing wrong here, and there's really nothing to be embarrassed about. The fact that it took a month (more if you consider the folks helping to reproduce the original results) for anyone to notice anything wrong shows how complicated the situation was! Put yourself in Ken's shoes -- having updated to LLVM-19 to enable the preserve_none flag, would you then hypothesize that LLVM-19 may have introduced an unrelated performance regression that no one noticed for five months? Lots of people underestimate how hard benchmarking is IMO.
A 1-5% performance improvement is also pretty valuable, just not quite as spectacular as we thought originally :-)
reply