Sentry really is a life saver. We use it for our HTML5 MMO for both the client-side Javascript and the back-end in Node.js, and it has helped us to dramatically improve our end user experience. Really cool seeing this used out of the browser!
I've read the Wikipedia article on Stackless Python but I was wondering if someone could quickly explain why Stackless is more desirable to CCP than regular Python?
Unlike most MMOs, which have multiple instances or "shards" of each location, EVE Online keeps all players in one shard. That means, for busy hub areas, they have to handle thousands of ships interacting with each other all on one server.
The lightweight concurrency lets them write little (and not-so-little) bits of code that execute in parallel without incurring overhead associated with OS threads.
Edit: sorry if I'm confusing "concurrent" and "parallel", I never really liked those terms :p
As far as I know nothing in EVE executes in parallel in a traditional sense. Each solar system is run on one thread and one python interpreter. While stackless is used for concurrent programming it is not used for parallel programing.
Thousands of lightweight threads with cooperative scheduling. Allows you to have a myriad of little programs running, each perhaps servicing a network request, or running some long running game logic. Impossible with threads. Also, threads are pre-emptive, which means that you need to use locking almost everywhere.
This is a handy reminder. We spun up Sentry a while back (like, maybe 1-1.5 years ago?) and spun it back down because logging to it via TCP was unacceptable for us.
For the folks using it, how do you get data to it? Curious how things have shaped up over time.
We didn't want to write to something synchronously without having a fault-tolerant solution for latency... especially when we could afford the end-user latency of seeing the exceptions later. I don't want my logging aggregator to be a SPOF for the main app. Would much rather suck up the data asynchronously from syslog or files, etc.
Hi, at CCP we have a custom transport that does the logging to sentry on a separate tasklet (think lightweight thread). Basically, the logging is done in the backgrond when there is time, freeing the thread whence the error came up to do other stuff immediately.
1) Sentry is not designed to replace logging therefor things like syslog will never achieve the same result. Sentry is explicitly for catching outliers and TCP is the only method able to store it's data.
2) As with CCP most people no longer use synchronous transports. We even changed the default to be the threaded worker transport. That said, when your user hits a 500 adding 50-200ms is generally acceptable as you've already put the user in a bad situation.
Late getting back to this but wanted to clear up some things:
1) Sorry, I think I was misunderstood. Was saying I'd rather populate Sentry from logs not that I think my logs are an alternative for Sentry.
2) Awesome. I'll look at the threaded worker transport stuff. As for the 50-200MS on 500's, I totally agree. Didn't get around to sorting out making connection timeout something like a couple hundred seconds though (at the time it wasn't clear to me if that was an exposed setting for the Sentry lib), and also I was hoping to use Sentry for situations where I might give our users a 200 but warn us internally about and use Sentry to notice a large uptick in the warnings.
