The current test types in our benchmarks (I work at TechEmpower) aim to establish a high-water mark for web frameworks and platforms. If you review the implementation requirements, included is a brief rationale for each test type:
For example, the "plaintext" test focuses on raw request routing and HTTP parsing throughput and is intended to allow ultra-high performance servers to shine. Meanwhile, the "fortunes" test is intended to exercise a more diverse spectrum of framework functionality including database connectivity, the ORM, dynamic-sized collections, in-memory sorting, server-side templates, and XSS countermeasures.
Future tests in our project will likely exercise still more framework and platform capabilities, and with more computationally-demanding requirements. I believe this will go further to demonstrate the appeal of high-performance platforms and frameworks. Presently, it is easy to point out that few applications require the capacity to serve 10,000 or 100,000 requests per second. I routinely suggest that a real-world application is likely to perform at 1% to 10% of the high-water mark we show. Where it gets interesting, then, is determining if 10% of 50,000 rps or 10% of 500 rps is workable for you.
Despite conventional wisdom that suggests that a web service is constrained by external systems such as databases and peer services—and not dismissing that as untrue, but rather exaggerated—we believe that real-world web applications are often constrained by the platform or framework. In fact, it is my opinion that selecting a high-performance framework and platform (that suits your development efficiency requirements!) is precisely how you avoid premature optimization. By making a reasonable decision early on, you defer scale pain that is all too common with low-performance platforms until later in your project's life-cycle. Do you want to bring out the big architecture guns and increased system complexity at 10,000 users or at 100,000 or 1,000,000 users? Indeed, nothing is a drop-in replacement for something you know and love. There will be some learning curve if you're moving from one framework to another, and even greater learning is needed if switching platforms or language. But these inflection points in programming are needed sometimes, and you may find it's good to position them at the start of a project rather than midstream.
It is, sort of, but it only shows part of the picture. I'll explain. Yes, TechEmpower will show you some of the fastest servers / frameworks. Maybe you can get close to a million requests per second in some of the native servers delivering static plain text / json on top of the line hardware. Eventually though you need to build and ship an application and some software that does more than a few queries.
High performing/scalable systems are built from lots of pieces of interconnected software like caching software (Redis, memcached, etc), work/job queues (RabbitMQ, ZeroMQ), asynchronous database queries and probably other things? The way these pieces fit together make much more of a difference than the server / framework you pick. I'd say it doesn't largely matter which Python framework you choose when you still have a blocking SQL query that has to be done on each request. The difference will be microseconds compared to how long that query takes.
It is important to benchmark/profile though. Maybe not at the framework level but as part of the continuous integration process. Determine what parts are slow, what parts could be replaced with asynchronous / event based code, and what parts should be replaced with something written in native code (C, Go, or Rust I guess) especially because Python makes it easy to integrate directly with native code (at least in C, I don't have experience with Python talking to Go or Rust).
Maybe this is more useful?