My current employer uses sourcegraph [1], which also uses n-gram based indexing like github's blackbird. Would be interesting to see a comparative benchmark of the two.
The linked article contains the news that the feature became generally available. Your link isn't a press release, its an explanation of the tech behind this feature published a few months ago.
Oh, this is why this news felt very old and wrong. the "0.01 QPS" number for ripgrep is just a on-paper math, not an actual performance number from a running system.
> The resulting system can manage about 640 queries per second, compared to about 0.01 queries per second for ripgrep.
They are trying to make a point about indexes, but it is overshadowed by the unit mismatch (distributed system QPS vs what sounds like single-machine ripgrep pace)
Yes. ripgrep is used as a basis of comparison for that sentence. ripgrep wasn't GitHub's previous search implementation. (It never was. It's not the right tool for that scale.)
You don't owe debts to open source. That's the point. Or at least that's the point of the free software philosophy.
This is also why I think it's unbelievably lucky we had luminaries like RMS in the beginning. To think of software freedom, what a concept! Legend!
Meanwhile everyone else is constantly trying to turn everything into a transactional interaction. "what are you going to give me for my code? How are you going to pay your debts?"
I think if they did not, and another git host offered free hosting to open source it would have much of GitHub’s share of the market today. They still competed hard on UX and features but it was necessary to win over open source projects, to become the defacto solution for software.
They are using Github for hosting, visibility and ease of contribution. A lot of projects would never have been picked up if they were hosted on other platforms.
With the prior understanding that the code uploaded there would remain free and visible for all those who wished to contribute towards something greater than themselves. Microsoft sweeping in at the last minute to undermine all that wasn't exactly an act met with rapturous applause, nor was it expected by anyone who hosted on Github.
The users are trapped. They've made a time (and some a financial) investment into Github, and to pack up bow and go elsewhere would kill the communities they grew there.
Github owes FOSS everything for the community and buy-in fostered there, and would be nothing without it.
I think it would be accurate to say that the value GitHub extracts from the OSS community (in the form of its core VCS, network effects, ML models, etc.) substantially outstrips the value returned to the community. Which is not to say that the latter is insignificant!
Is that a qualitative or quantitative argument? Like, would you put numbers to either/both sides of that? I'm not confident Github gets more value than it returns (but you may be right).
I don't think this requires belief in a "better" deal, only a good one (or an acceptable one, given the friction and network effects involved in switching).
GitHub had ~$1b in revenues in 2022, but I have no doubt that it created many multiples of that in value among its 90m users and communities in the same period.
I would wager that the cumulative revenues of all third-party services (CircleCI, Sourcegraph etc.) that are part of the GitHub ecosystem also far exceed GitHub, value and revenues that would unlikely exist without GH (or be at an earlier phase of their evolution).
I'd even go so far as to say the value created by pull requests alone exceeds $1bln annually, for the many commercial and open source teams that use it to get work done.
“Symbiotic” by definition means one depends on the other. I think it’s moot who depends on who more. They both depend on each other in a substantial way. Without the other they would likely adapt, but in a suboptimal way, at least for some time.
But I think folks dramatically undervalue what GitHub(Lab) brings, and how expensive it is to host such a system for everyone.
I would describe the relationship as mutualistic: both parties benefit, but not necessarily proportionally. And they don't need to benefit proportionally, but there's also value in highlighting the disproportionality: it behooves the OSS community to remember that extremely valuable ecosystems like GitHub and GitLab derive a significant amount of their value from the continued (and mostly passive) goodwill of the community.
I don't think the OSS services market (I'm assuming you mean Red Hat, etc.) extracts disproportionate value from GitHub. If anything, I'd expect those companies to pay handsomely for their access -- mine certainly does, and we're primarily open source.
In 2008, Microsoft acquired Fast Search and Transfer, an enterprise search company that has decent search technologies [1]. It ran on common Linux and Unix systems as well as Windows at the point of the acquisition.
Yeah, I thought so (that it was internally developed), because someone commented that it's not open sourced.
I was just posting to say Microsoft have (had?) a perfectly nice search engine many years ago. Either GitHub folks didn't want to use it, it's not good (I think it was good, but maybe it's not suitable, or maybe it isn't maintained anymore shrug) or Microsoft butchered it too much over the years.
I don't know much about the inner workings of Microsoft, but I could also imagine than an acquisition from that long ago just isn't known to those in the Github side of the house. Seems pretty typical for a large company.
I hate the new search, most of the time it misses simple things and it is infuriating, I always end up just cloning the repo and searching for things using my own tools
"about twice as fast", yeah twice as fast but twice less effective.. what's the point
Considering "it's written in Rust" is synonymous with "it has been statically guaranteed to not contain a wide class of common bugs that frequently cause severe vulnerabilities", I think it's relatively reasonable for people to advertise when it's been used for a large project. Especially considering how much people like to gripe about its use.
Doesn't apply to this project, but that can be also read as "it has taken on a class of bugs it wouldn't have been exposed in a GC language, with little benefit."
Or even worse, if the app needs to use unsafe, you'd be surprised how many Rust engineers think unsafe somehow "encapsulates" memory bugs inside the block itself. Which is not true. Using unsafe can cause memory issues that bubble up the call stack.
I don't feel any guarantees when a product is made in Rust. There are buggy Rust projects
> [It's not true] that unsafe somehow "encapsulates" memory bugs.
I believe this real-life, 4-year, 1.5 million-line-of-code experiment disagrees with you directly [1]
> In general, use of unsafe in Android’s Rust appears to be working as intended. It’s used rarely, and when it is used, it’s encapsulating behavior that’s easier to reason about and review for safety.
unsafe fn create_null_pointer() -> *const u32 {
std::ptr::null()
}
fn main() {
let ptr = unsafe { create_null_pointer() };
// Dereferencing the null pointer outside of unsafe code
let value = unsafe { *ptr };
println!("Value: {}", value);
}
This will segfault, and the compiler won't warn you about it, even though an error didn't actually happen until you printed the value in "safe" code
What this article is getting at is that the unsafe keyword highlights pieces of code that Google scrutinizes, not that it has functionality that "magically" encapsulates memory bugs. And yes, it's a lot better than C++. But you still need to be aware this can happen and not assume you're safe
All unsafe does is shut off the compiler for sections of the code. You can write code in an unsafe block that bubbles outside of the scope of "unsafe."*
> "you'd be surprised how many Rust engineers think unsafe somehow 'encapsulates' memory bugs inside the block itself,"
you essentially mean:
> "you'd be surprised how many Rust engineers think that they can use unsafe without regard as to whether they are maintaining Rust's safety invariants by hand in their unsafe blocks."
> Considering "it's written in Rust" is synonymous with "it has been statically guaranteed to not contain a wide class of common bugs that frequently cause severe vulnerabilities"
No it isn't. It's not like we've been writing too much applications C-code recently and it's not like Rust offers that much fundamental security over e.g. Golang
> I suggest reading more of the type system literature
Like what, in general? I've been working with variously typed languages of various maturity with stuff like structural typing, null-safety, sealed/enum types, inlined and dto/data types, functional types, various implementations of traiting/inheritance/prototyping, etc for years. Do you have a specific point or did you assume that being vague makes you look mysteriously smart?
Honestly, you are acting like all we ever had is JS, C and Rust.
Okay, so you've listed a lot of type system features... which is fine (and exciting!), but not relevant?
> Do you have a specific point or did you assume that being vague makes you look mysteriously smart?
My point was what I said: you should read the academic literature in this space to understand why Rust is fundamentally safer than languages like Go. It's not just a made-up marketing ploy. The type system is specifically designed to make it very difficult to incur issues with memory, which was Rust's primary goal as a modern systems language. Yes, other languages have fancy features, and some of those features are not in Rust. But the features you mentioned don't specifically relate to memory safety, which was the topic at hand. Most of the features you mentioned are actually just about expressiveness of the type system, but not fundamentally distinct forms of type-checking.
Rust's type system is based on earlier work from Cyclone. You can find the various Cyclone papers here: [http://cyclone.thelanguage.org/wiki/Papers/]. Specifically, you'll want to look at the papers concerned with memory safety, which is where the region-based type system was first introduced.
> Honestly, you are acting like all we ever had is JS, C and Rust.
No, I'm pretty familiar with the landscape of programming languages — it comes with the job! I think you just didn't understand my point, which is probably my fault for not being clearer originally. It happens when I reply from my phone sometimes. Apologies.
I feel like you're being sassy with me, and I don't much care for it.
That Rust is fundamentally more secure (in certain, specific ways) than other languages is a fact, not an opinion. It is a fact well-known by members of the programming languages research community, users of the language, and other sufficiently interested people. I didn't think to qualify it by citing the relevant papers partly because the fact is well-known, partly because the fact is easily researched, and partly because this is an informal conversation where you don't always need to bring primary evidence simply to talk to people.
Just because you were not aware of this fact didn't grant you license to be so dismissive at the start. Your first interaction with me was a stern, corrective "No it isn't". You were wrong, but worse than that, you were authoritative in your wrongness. If you had just asked me to explain what I meant instead of trying to correct me, this conversation could've gone differently.
Now, maybe I've misread your tone, in which case I genuinely apologize. But the certainty with which you made your incorrect statements at the beginning, followed by your kind-of condescending interactions with me in the comments since then, makes me think I am right in my interpretation, and I think that's really unfortunate.
I am honestly not. Please don't read too much into my tone, English is not my native language.
It actually wasn't clear why you've mentioned types theory initially, the semantics of "expressive types" confused me a bit, but I upvoted your latter post where you elaborated.
If it’s written in Rust, the tooling (Cargo; the Rust language server; etc) make it much easier than most other languages to jump in and fix something or add a new feature. I’ve contributed to projects written in Ruby, Python, Java, C#, C++, Haskell and many others, and every Rust project I’ve ever touched has been much easier/quicker to go from `git clone` to PR submitted.
That may not be appealing to people who never fix or extend the software they use; for me, as someone who finds bugs and deficiencies in almost everything I lay eyes on, knowing that a piece of software is written in a language that is the least likely to frustrate me and suck up my time when I inevitably have to dive in and tweak something is a huge competitive selling point over similar software written in other languages.
1. https://github.com/sourcegraph/sourcegraph/blob/main/doc/dev...