
Sourcegraph Master Plan - snapspans
https://about.sourcegraph.com/plan
======
mattnewport
This all sounds like useful stuff I'd potentially use but I don't see it
solving the core problem that programming is difficult. I've long worked in
environments where "basic code intelligence" works and it is certainly useful
but the problems it solves are not what makes programming hard.

Programming is hard because it's hard to define the problems that need to be
solved, hard to design good solutions and often hard to implement them
efficiently. A smart IDE is a useful tool but it does little to nothing to
address the fundamental difficulty of programming. The real challenge to
making programming more accessible is how to teach/train people to be better
thinkers and problem solvers (and I include myself in needing that).

~~~
bla2
The best way to define the problem is iterating on it. This (and other tools)
allows you to iterate faster. That's why tools like this are useful.

~~~
mattnewport
I'm not saying its not useful. My daily work currently is VR development in
Unity and so I'm using C# in Visual Studio with ReSharper which together give
one of the best and most complete "basic code intelligence" experiences I know
of. It's great and useful but while it makes me more efficient it just doesn't
have much impact on the actual hard parts of programming. It improves the
workflow for the routine parts but they are merely time consuming, not
difficult/hard problems.

------
sqs
Sourcegraph CEO here. Thanks for posting this. Happy to answer any questions!

~~~
dantiberian
What is the relationship between Sourcegraph and the LSP? The Master Plan
reads like you were part of creating it?

~~~
sqs
Sourcegraph CEO here. Microsoft designed and released LSP. We are eternally
grateful for that! We played no part in its creation. The first LSP mention
links to the repo at [https://github.com/Microsoft/language-server-
protocol](https://github.com/Microsoft/language-server-protocol) with more
info.

We have, however, built and/or sponsored many language servers. (Those, in
turn, build on compiler APIs that other people tirelessly created, to which we
also owe thanks.)

~~~
Loic
The LSP is a really wonderful, simple and well executed idea. The simplicity
of the protocol, cross platform (not bound to a COM interface or similar
Windows only technology), ease of creation of an implementation in any
language, all well done.

This is where the deep experience of Microsoft in building developer tools
together with the desire to be cross platform are shining.

As a user of Emacs under Linux for the past 20 years to code in Fortran, they
succeed in me using Visual Studio Code[0] and hack into a Fortran Language
Server!

[0]: [https://www.ceondo.com/ecte/2018/09/vscode-visual-studio-
cod...](https://www.ceondo.com/ecte/2018/09/vscode-visual-studio-code-
fortran/)

~~~
sqs
Yes, and LSP is one of those ideas that I respect even more because it was
laughed at initially! Unlike most other approaches at solving this problem, it
did NOT try to model concepts or structures in programming languages. It is
presentational, not semantic; it is solely concerned with "what does the user
want to do/see". It was radically simpler than other approaches at solving the
same problem, and it has achieved tremendous adoption, so I think it's fair to
say it's a success.

BTW, very cool that you built the Fortran language server!

~~~
Loic
Simplicity is hard to achieve!

I am not the author of the Fortran Language Server, just fixing bugs and
trying to improve it. But as the protocol is simple, this makes the work to
improve it also way simpler. This is the part where I am now sure that the LSP
will take over the world for the benefits of all the code editors, closed or
open source, cross platform or not. And this is good for us as programmers :)

------
Dayshine
So, I've written a language server for a very niche programming language
(Paradox Interactive games' scripting language).

I started it as a VSCode extension (via LSP), and am halfway to getting a
Sublime Text extension to work, and gave up on intellij. Emacs/Vim shouldn't
be hard, but I have no idea how I would release updates. While I agree LSP is
technically making it an "n-plus-m" problem, implementing the plugins that
wire things up to the IDE is _hard_.

My main pain-points have been:

\- Initial integration: Each IDE has its own extension framework, and it takes
a lot of work to get my LSP server to work at all

\- Deploying updates: Once I have it working, I then need to spend days
automating updates _for each IDE_.

\- Distributing LSP updates: For VSCode, I bundle the executable, but that
means extension updates for every LSP update. I could have system level
installation, but I don't want to ask users to take that risk (nor manage
installers myself)

\- Syntax highlighting: There is no good standard for semantic highlighting
(technically textmate, in practice not). LSP doesn't include highlighting.

\- Settings management: Each IDE has a different way of managing settings, and
gluing that to the LSP isn't easy.

Basically, I have to spend a week or two of my free time to support an extra
IDE at all (not well), which is frustrating. Probably a month or two for full
support, and that would be for maybe a dozen users.

~~~
sqs
Sourcegraph CEO here. We have a project underway that lets you write one
editor extension and use it on multiple editors (and other tools with code
views, such as GitHub, GitLab, Sourcegraph, etc.). We intend it to solve your
problem.

[https://github.com/sourcegraph/sourcegraph-extension-
api](https://github.com/sourcegraph/sourcegraph-extension-api)

~~~
sooheon
Awesome mission. But god I wish JS weren't eating the world like this.

------
codetrotter
> At Sourcegraph, we want to fix this and eventually enable everyone to build
> software. For now, we're revealing our master plan for phase 1: how we're
> going to make it easier and faster for today's developers to build software.

These are lofty goals, and sorely needed.

We've come a long way but there is a lot of "suckage" both in developing
software and in using it still.

I wish Sourcegraph all the best :)

~~~
codetrotter
One thing in particular that grinds my gears about a lot of software is how
limited the _actions_ I can perform on _data_ are.

Take Spotify for example, a fantastic product by many measures, but which even
so does suck in some ways. Being able to create playlists and to enqueue songs
is a start, but think about a list as a special case, limited kind of graph,
and think about what kinds of operations we are missing out on by restricting
ourselves to just lists.

And that's not all. Spotify is also annoying in terms of looking at the
currently playing list of songs. If you go someplace else and look at another
list and then go back to the current playlist you'll usually be brought back
to the top and if the list is long then you have to scroll a lot in order to
get back to where you were.

And I would like to make arbitrary queries like "show me all playlists in
which song x occurs".

And I would like to have access to the full history of songs I've listened to,
including information about leaving the song before it was done, and skipping
forward and back. And I would like to have the point in time in a song where I
starred it or added it to a playlist.

Then I could query for example "show me all songs that I've listened to in the
past six months, that I listened to at least 80% of the full length of, but
which I have not listened to within the last two weeks" and play the resulting
list.

Of course, the playlist being a simple concept makes it easy for users to
understand. I guess the amount of people that want what I want is probably not
many...

~~~
vinceguidry
Arbitrary data querying are literally the problem databases exist to solve.
It's a monumentally hard problem.

And there's another issue on top of it, the company 'crown jewels' problem. If
Spotify makes it too easy to query Spotify, then it becomes ridiculously easy
to compete with Spotify. Which isn't a bad thing necessarily, but not
something we should expect Spotify to expend millions of dollars to enable.

~~~
codetrotter
> And there's another issue on top of it, the company 'crown jewels' problem.
> If Spotify makes it too easy to query Spotify, then it becomes ridiculously
> easy to compete with Spotify. Which isn't a bad thing necessarily, but not
> something we should expect Spotify to expend millions of dollars to enable.

Yeah that's true. But mainly the data I want to be able to query is the data
that is directly connected to my own activity. If Spotify made it possible to
get access to metadata about my activity and the songs that were involved in
my activity then it's useful for users but still limits somewhat what a would-
be competitor could do. Combine that with a ToS that forbids redistribution of
the data and I think Spotify could provide this without helping would-be
competitors too much.

Even if a would-be competitor had access to various data from Spotify they
would still need to both negotiate agreements about streaming rights with the
IPR owners of the music itself, and they would need a lot of marketing in
order to get users. In other words they would need a lot of capital. Where
would a would-be competitor even get that kind of money? I don't see why any
VC or startup accelerator would be willing to fund someone to build this given
how big Spotify is and how limited the group of people probably is that would
decide to leave Spotify for this feature.

Whereas on the other hand, this kind of access to data could be a premium
feature, and then it could be another reason for users that cared about these
things to pay for Spotify Premium. Open source third-party clients could
implement query engines, building on exactly what you mentioned in the first
paragraph;

> Arbitrary data querying are literally the problem databases exist to solve.

Open source clients exists today, and using them with Spotify requires
premium. If Spotify provided the data, _someone_ would follow suit in making
use of the data in open source clients. Which _could_ further boost sales of
premium membership, while at the same time like I said would still _not_ make
it _too easy_ for would-be competitors.

Actually, the client wouldn't even need to be open source, it could be a
commercial product, and by integrating with Spotify Premium they would still
not be a competitor, and they would not need to go through licensing, and
marketing might be easier as well. "Enjoy advanced interaction with your
Spotify Premium listening experince by using our client."

------
beaconstudios
Phase 2 sounds particularly interesting to me as it's a problem I also want to
solve. Are you guys willing to talk about the direction you're moving in, or
is this still to come?

I think making development as simple and accessible as possible is essential
for our software-driven economy to scale efficiently.

------
oscargrouch
Isn't Kythe[0] trying to achieve this already, with a graph based, multi-
language framework?

Of course, there are no integration with current dev tools, but the core is
there to make it work, maybe?

[0] - [https://kythe.io](https://kythe.io)

~~~
sqs
Sourcegraph CEO here. Yes, Kythe is a very ambitious project and we have a lot
of respect for the team building it inside Google. source{d}
([https://sourced.tech/](https://sourced.tech/)) is another company doing
awesome things in the code analysis space.

From our POV, the more people trying to accomplish these things, the better,
because:

1\. As developers and humans, we want these things to exist!

2\. Our business and product is a code search and browsing tool, which
benefits from better code intelligence/analysis. We don't sell language
servers directly; we sell a tool that makes use of them.

(For interactive tools (editors, code search, and code browsers), which is
what we're primarily targeting, LSP ([https://microsoft.github.io/language-
server-protocol/](https://microsoft.github.io/language-server-protocol/)) is a
better fit than Kythe.)

~~~
oscargrouch
> From our POV, the more people trying to accomplish these things, the better

Sure, im totally ok with this vision, given this is how people create new
things, even if it looks 'solved' in a given context.

> As developers and humans, we want these things to exist!

The more iteration, the better.. we definitely need more polished tools in
this space.

> LSP ([https://microsoft.github.io/language-server-
> protocol/](https://microsoft.github.io/language-server-protocol/)) is a
> better fit

Oh thanks, thats kind of the context i was looking for.. in what problem
domain you guys were pointing to.

It makes perfect sense giving the current goals of Sourcegraph, to create a
more complete solution to the dev community it targets.

(I didnt wanted to sound the "Dont reinvent the wheel" kind of guy here, just
curious about what you were trying to acomplish, and used Kythe as a reference
to the problem domain)

------
georgewsinger
Someone has read their Thiel. +1

------
simplecomplex
Impressive that they’ve sold a free IDE for 20 million to investors...

~~~
sqs
Sourcegraph CEO here. We sell a code search and browsing product for
developers to companies
([https://about.sourcegraph.com/](https://about.sourcegraph.com/)).

------
dreamcompiler
I don't get what problem this solves that Emacs with appropriate language
modes doesn't already solve.

~~~
sqs
Sourcegraph CEO here. I have been a near-daily Emacs user for 15+ years. The
unsolved problem for you and me as Emacs users is this: I wish I had the same
high-quality extensions that VS Code users get. I use VS Code 1/3 of the time
because the editing experience is so much better in many languages (such as
TypeScript).

Have you tried VS Code? I would highly recommend you try it if you haven't
yet. I think you'll see what you are missing.

(Of course, Emacs has many awesome features/modes that other editors lack,
too. The same principle applies there.)

~~~
bad_login
What the point in using VS Code compared to emacs when we have LSP servers and
an emacs lsp-mode ?

~~~
sqs
VS Code extensions for language support are (usually) a combo of a language
server plus VS Code extension code. Other editors using just the language
server won't get the features implemented in the VS Code extension. So Emacs
lsp-mode and a language server won't get you all the features that VS Code
users enjoy.

For example, the Go language server (if enabled) in vscode-go is only
responsible for hovers, definitions, references, and a few other things. Auto-
imports, code lens for tests, helpful commands, coverage, etc., are all
implemented in vscode-go's TypeScript extension code.

I recommend just trying VS Code for the languages you use in Emacs to see what
you're missing. I bet a blog post about your experience would get to the front
page of HN. :)

(One solution, of course, is to make it possible to write _all_ of those
things in a way that can be consumed by all editors. Check out
[https://github.com/sourcegraph/sourcegraph-extension-
api](https://github.com/sourcegraph/sourcegraph-extension-api) to see our work
to let you write any kind of editor extension in an editor-independent way
(not just the language/code intelligence features). When this gets into beta,
we'll feel comfortable updating this master plan to denote the progress toward
the goals.)

