On a side note, i really don't know if the savings will be worth it once the company starts growing and hiring. They'll spend a lot of time maintaining very custom code, running on a custom implementation of a language, alongside erlang codebase... Unless they open source everything to get a community to grow the user base for their tool.
We are able to do this, because starting our LISP-VM and doing all these processes on every
request, is still many times faster that having a VM (with a garbage collector) online all the time.
Having to maintain a small LISP interpreter in C is definitely an extra thing to keep up with, but you have to balance that with the work that would normally go into avoiding long GC pauses once you start tracking a nontrivial amount of objects/data in a VM with garbage collection. Java, for instance, doesn't support separate heaps, which means that a background piggy processing thread that is haphazardly allocating objects can cause "core" threads doing I/O to pause for several seconds for full GCs. Even with Java's fairly sophisticated heap management schemes it is still very difficult to design a system to completely avoid full GC pauses. Erlang is somewhat better in this respect but introduces its own sets of problems to the mix.
In any case, I agree with your sentiment (probably would not be the angle I would have taken), but I can kind of see why the authors might have decided to go this route.
edit: formatting fixes
PS : i'm speaking objective-c here because it's the only language i know that does it that way, not because of its features as a language.
EDIT: There's a downside to reference counting in that either you or the runtime does the retain/release calls very often, whereas a mark-and-sweep collector would just touch each object once per collection. So reference counting might not be faster overall, but typically avoids long pauses. What you probably really want is something like Azul's JVM, which has a kernel extension so they can collect memory concurrently, resulting in shorter pauses than sweeping and faster overall time than reference counting.
And this is another area where a functional paradigm makes sense. Mutations are "hidden" (and the C layer that's responsible for doing them concurrently lives in another "realm" (monad)). So, memory handling is easy and monitoring has shown us that fragmentation is kept to a minimum.
We tried to find something that was fast, cheap to scale (in terms of servers) and easy to extend but we couldn't find any other solutions. The problem is having everything in memory and doing correlations really fast.
i wonder how much engineering talent is going to get sunk into writing a db and whether the management is going to get fidgety while the company's best talent is writing a db rather than the product? (especially with .. two guys)
How close do you work with your clients and how much is your system changing in response to feedback?
What drawbacks (if any) the current tech stack choices had so far on your?
Our clients were involved since day 0 (which I think is the key of our success btw). The first prototype (built in two days) was in Erlang using Mnesia.
The drawbacks are:
+ not having lots of engineers knowing C and Erlang
+ Outsiders having problem with LISP (that's why we implemented PIG)
Also, the data format for input is not tightly-coupled with the LDB system. One can have different processing scripts (lisp) that may organize the data in a different way. Or you can write a C module that extends the underlying db engine to work with your implementation or even your database infrastructure (whether it includes SQL, NoSQL, etc).
During our tests, we saw that SQL databases weren’t a good fit due to the fact that our data where unstructured and we needed a lot of complex “joins” (and many indexes).
I guess that you say it's unstructured because the most important parts of bug reports are stack traces and debug messages? My naive assumption is that stack traces can be stored in a relational way, and it seems that you do:
For us, relations might seem to resemble these of an RDBMS at first sight, but we work with them in a different way.
From 'unstructured', 'complex "joins"' and 'work in a different way' I infer that you do a lot of string processing on the fly, and the real problem was about precomputing everything vs. staying flexible?
I'm wondering: how does one decide to change the TCP_TIMEWAIT_LEN from 60 seconds to 20 seconds? 20 seems an arbitrary picked value: I understand that by doing so they can drastically cut down on the number of open connections after the last FIN is received, but why 20 instead of, say, 17 or, say, 9?
If you decrease the amount of time a connection sits in a TIME-WAIT state, you will increase the probability that a new connection could receive a packet from a previous connection. Given the latency of mobile networks, I would expect the probability of receiving a packet from a previous connection to be even higher.
If you're interested, the beginning of the following paper provides a great overview of the problem: http://www.isi.edu/touch/pubs/infocomm99/infocomm99-web/
Also: "Do not be confused by the /proc/sys/net/ipv4/tcp_fin_timeout config item. The FIN TIMEOUT is not the same as the TIMEWAIT length." -- http://www.stolk.org/debian/timewait.html
- Panagiotis Papadomitsos, Head of Infrastructure @ BugSense