The current solution in this space, that actually works really well at scale, is ElastAlert. The problem is that ElastAlert is kind of a mess to work with. Lots of documentation, but you need to get into the weeds with it to figure out how it really functions.
Once you get it going it's a great tool. Scaling it out (we run hundreds of rules pretty frequently - upwards of 15 times an hour) is just standing up more instances with their own separate rules.
They have a set of docker containers which I find very handy for spinning up deploy specific logging sinks or full on SIEMs.
ELK wants you to massage the data more first.
For example, if someone hacks your Internet-facing web server, your IDS might detect that. They then brute force the password to your production database server, which Active Directory might see. They then use nmap to trace your internal network, which would show up on your internal firewalls. Then they hop server-to-server until they get to a critical server. They then download a payload, infect that server (which your AV might pick up) and start exfiltrating data (which the firewalls and proxy might pick up).
You have all of these security tools, but without some intelligence linking all the logs together and correlating the data, you're stuck tailing these logs individually, hoping you catch the right log at the right time and can remember everything you've ever seen. And we're talking tens of thousands of events every second. A SIEM takes all of that data, does the searching for you, correlates all of the events across different security technologies and vendors, and alerts you when it detects someone doing something they should not be doing.
Popular SIEMs are tools like Splunk, QRadar, ArcSight, LogRhythm, etc.
The SIM came from the requirement to collect events (operations).
SEM stems from the desire to monitor / detect (security).
When we (the operators & security people) realized we where both collecting sort of the same data, the plan was hedged to combine resources and build something that satisfied both our needs. Marketing realized this as a great way to pitch a story that actually made sense to everyone involved: the SIEM market exploded.
Seems a logical progression from Kibana and Logstash - but sometimes I worry search will suffer for all this other stuff.
If you want search, most of the NLP crowd is using Solr.