
Syntax highlighting is a waste of an information channel - adenadel
https://buttondown.email/hillelwayne/archive/syntax-highlighting-is-a-waste-of-an-information/
======
megameter
This kind of nibbles at the edges of the longstanding question: Whether source
code is best represented as a linear document.

Because historically, that was true for the obvious reasons of pragmatism; you
could put it on a punchcard, print it to paper, save it to a file all using
the same character encoding and rendering methods, and when we went to
interactive editing that didn't change: half of the joy of plaintext is that
you can select, copy, paste, search and modify in mostly standard ways across
many tools.

Syntax highlighting is incrementally in line with this paradigm since there's
only one syntax for the language(syntax extension and mixed-language documents
aside) and it too follows a linear order, although it also recognizes that an
AST exists. But as you get into stuff like code folding, Intellisense and
other IDE-like features, deviations emerge. More of the selections deal with
symbolic meanings(declarations, types) and not really the tokens or the
characters they consist of, and only incidentally the delimiters of the AST.

But - when we try to fully re-envision it and express these other concepts
directly we tend to end up with something that is unfamiliar, a bit clunky and
alienating(most graphical languages). So the accepted developments always seem
to come in these little ideas that are mostly-optional add-ons.

Chuck Moore did use semantic color as a language feature in ColorForth, but
his specific goals lean towards a holistic bootstrapping minimalism, so I
haven't seen it taken up elsewhere.

~~~
XorNot
You can extend this further: git is the wrong representation model for code
(text).

It's a useful representation model, but really only because programming
languages are mostly designed to be line-oriented (though not enough: i.e.
Python should disallow multiple imports on one line because it lightly breaks
diff-viewing). Git is the dumbest possible thing that can work, and it does
work, but it's also wrong.

What we really need is a generic way to store graph-structured data (which is
what an AST really is) into something like git. Because then we could finally
get rid of the notion of code-formatting as anything but a user experience
peculiarity (the rise of "push button and blackbox fixes code for commit"
elucidates why this should be).

But more importantly, it would mean we could reasonably start representing
commits with some notion of _what_ they are doing to the control flow and
structure of the code. Think git's rename detection but it notices when a
function is renamed, or can represent refactoring in a diff by showing that
all the control linkages of one block of code have moved to a particular
function (and by extension, diffs would now implicitly show when new links or
dependencies are added).

The trouble of course is doing any of this generically, is an unsolved
problem. I have an idea that you could probably do something interesting like
this with git's existing structure and a language like Golang, where you
decompose the program into a bunch of files representing the hierarchical AST
and commit that rather then actual text maybe.

~~~
Gibbon1
Code as text had long bothered me because of my experience with various
schematic capture and pcb layout tools. With those the design is stored in
form that can be queried directly. And you can change object properties
programmatically.

I feel like the insistence of text based programming languages just leaves the
industry completely mired in the mid 1970's.

~~~
treeman79
I’ve had to deal with several graphical programming languages over the years.

They only work on the most basic of use cases. The moment you want to get real
work done they just get in the way.

~~~
jtwaleson
Very much disagree, and completely dependent on what you define as "real
work". There are many low-cost platforms out there that allow for very rapid
development of big projects.

Disclaimer: I used to work for one of them (Mendix)

~~~
treeman79
Looks like a modern Visual Basic 6. Neat.

VB6 was so horrible and productive at the same time. Single person could build
an app in days that a team of 10 would struggle with today.

And a group of 10 could bring said app to a stretching halt.

So many memories, good and bad.

~~~
jtwaleson
Indeed :) I'd say the biggest problems with platforms like these are 1:
meaningfully diffing (visual diffing is an unsolved problem) and 2: the
enterprise pricing.

~~~
treeman79
Lock in as well.

Was at a company that invented a visual programming tool and forced
development to use it.

Several flat out refused. We all went on with our careers. The people that
used the proprietary system still work there, not then a decade later. They
hated the tool then. But they have no marketable skills set at this point.

------
duskwuff
I don't buy it.

1\. I personally find syntax highlighting incredibly helpful as a way of
setting off syntactic elements, and incidentally as a way of calling attention
to typos (like a missing quote or misspelled keyword). The author seems to
dismiss this without any analysis.

2\. Using color for syntax highlighting doesn't mean you can't _also_ use
color to highlight other useful things (like search results or errors), as the
author suggests. Many editors which support syntax highlighting do exactly
this!

3\. The author's examples aren't particularly compelling. For example, using
colors to denote levels of nesting will make code flash between colors as you
edit it. Highlighting variables by type feels like a form of syntax
highlighting, not something new, and most of the examples of highlighting code
which meets specific conditions (variables which are assigned to multiple
times, long functions, functions without documentation, etc) all feel like
they could be summarized as "linter warnings/failures".

~~~
stevehine
VS Code / Visual Studio both have a 'rainbow indent' extension which colours
the indented space rather than the highlighted code. But it relies on the code
being formatted correctly; rather than anything semantic in the language. It's
still useful !

~~~
SAI_Peregrinus
And the numerous rainbow bracket extensions for various editors, which color
brackets and parentheses differently depending on nesting level.

------
gnicholas
Some people are reacting to the (clickbait-y) title, which is a more extreme
statement than what the author ends up advocating. The ultimate argument is
that it's ok to syntax highlight sometimes, but sometimes it might be helpful
to use color in other ways.

Personally, I find syntax highlighting to be very useful, but I also agree
that using color in other ways can be more useful at times. As it happens,
syntax highlighting was one of the inspirations for the color gradient text
pattern I developed to make reading on screen easier. [1] The existence of
syntax highlighting helped me to think about color in text as an information
channel that can be used to make certain tasks more efficient, so I'm very
grateful for that!

1: [https://chrome.google.com/webstore/detail/beeline-
reader/ifj...](https://chrome.google.com/webstore/detail/beeline-
reader/ifjafammaookpiajfbedmacfldaiamgg?hl=en)

------
mmmkkaaayy
If you haven't before, I'd recommending giving Intellij's semantic highlight a
look: [https://blog.jetbrains.com/pycharm/2017/01/make-sense-of-
you...](https://blog.jetbrains.com/pycharm/2017/01/make-sense-of-your-
variables-at-a-glance-with-semantic-highlighting/)

Every variable name is assigned its own color, so you can track the data
dependencies in your function using color. Similar to what the author suggests
as "Type Highlighting"

Some might consider it distracting/overstimulating, but I find it both useful
and it makes the code look a bit more 'artistic'.

~~~
lstamour
Many IDEs, IntelliJ and Visual Studio Code among them, also have plugins for
"Rainbow brackets" (the ability to use different colours for different pairs
of open and close brackets). WebStorm takes this a step farther and clearly
highlights open and close tags in HTML, etc., if I recall correctly.

------
matharmin
VSCode semantic highlighting is mentioned by the author, and it does support
these type of use cases. It works by first doing normal syntax highlighting,
then asynchronously getting the semantic info for semantic highlighting, since
it's a much slower process.

The current language-specific implementations do use it to just augment the
syntax highlighting, since that is what is what most people would expect. But
out of the box it already supports some use cases the author mentioned, such
as different highlighting for imports or constants. The full AST and type info
can be used for this (depending on the language server), so pretty-much
anything is possible.

What I personally find very useful (and had to hack experimental support for)
is highlighting of TypeScript async/Promise values and functions. This is
something that a language server can identify easier than humans, and can make
a big difference in coding.

------
adrianmonk
Too much color ends up overwhelming. I can find the red circle super quick
because it's the only red thing in a sea of black and white.

But I like the idea of going beyond syntax. Maybe a way to get the best of
both worlds would be to have keyboard shortcuts that toggle cycle through
different views. Press this shortcut to color-code types, press this other one
to color-code error handling, etc.

Also, maybe other visual depictions might be useful. Maybe parentheses could
start off very tall and get slightly shorter as you go a layer inward. Or add
a 3D/depth shading effect so that code each level of parens literally looks
like a level. Or whatever else you can think of, the point being that it
doesn't have to be color.

~~~
avianes
> Too much color ends up overwhelming. I can find the red circle super quick
> because it's the only red thing in a sea of black and white.

> But I like the idea of going beyond syntax. Maybe a way to get the best of
> both worlds would be to have keyboard shortcuts that toggle cycle through
> different views.

I couldn't agree more. When a line has too many different colors I have a
really hard time to intuitively organizing the information. It's kind of a
visual cacophony to me. So I use a minimalist syntax highlight to avoid that.

I believe we could come up with a more clever and interactive highlighting
system.

A simple example could be rainbow parenthesis which is only enabled when many
parentheses are found in a chunk of code. (if it already exists, I'd be glad
to know about it)

------
heapslip
I think too much syntax color is an easy way to shoot yourself in the foot as
a developer. When everything is highlighted, and everything is important,
nothing is important.

We need to choose what is important, I like to highlight language keywords as
grey, function names as blue and numeric values / strings as purple / green.
Everything else is white, and it works great.

tree-sitter[0] should end regex-based syntax highlighting, check this out for
nvim: [https://github.com/nvim-treesitter/nvim-
treesitter](https://github.com/nvim-treesitter/nvim-treesitter)

I really want to see tree-sitter used to power an awesome linter, that works
across languages.

~~~
jayjader
I find it interesting that your comment (to me) seems to caution against "too
much" syntactic highlighting, while the comparison screenshots in the README
for the project you mention seem to indicate an extensive use of colors
(compared to "traditional" highlighting, as per the readme).

On a personal note I _do_ prefer the 2nd screenshot to the first (ie I prefer
tree-sitter's highlighting than whatever they used to generate the
"traditional" image).

In the end I think it comes down to colors being useful to highlight
_meaningful_ differences between code snippets, be it semantic, syntactic, or
whichever other distinction is useful to the person reading/editing the code
_at that time_. As mentioned elsewhere in these comments, in a language that
distinguishes between synchronous and asynchronous functions (à la async/await
in Typescript or Python) it could very well be useful to have a different
color for each.

Then again, if you're "just" opening up your code files to edit some string
"constants" here and there, there's almost no point in having any other
highlighting (at that moment/for that action/activity) than "what is string,
what is not?". So the holy grail is some kind of highlighting that is aware of
your intent (as in, why you are viewing/editing this piece of code), and can
adapt to changes in said intent.

~~~
heapslip
Yes, I agree it's too much, but that comes from the editor, tree-sitter only
provides the metadata (which is way more accurate than the regex-based
approach). I'm on a color diet :)
[https://i.imgur.com/oqCgssd.png](https://i.imgur.com/oqCgssd.png)

------
waf
Visual Studio does semantic highlighting for at least C# (they call it
enhanced highlighting). Static classes, parameters, internal structure of
regex strings, etc. Basically anything returned by the Roslyn Classification
API.

I personally like it, but I think other people feel that syntax highlighting
plus semantic highlighting leads to rainbow soup.

~~~
maxmouchet
Semshi [1] does it for Python and I find it very useful for identifying errors
such as missing imports, misspelled variables, etc.

[1] [https://github.com/numirias/semshi](https://github.com/numirias/semshi)

~~~
deadbunny
Oooh, this is very nice. Thanks for sharing!

------
Animats
Syntax highlighting shows something that's already visible. What could be
highlighted that _isn 't_ already visible?

Undefined variables? Those really should get a squiggly underline, as with
spell checkers. Some IDE probably does that already.

Anything more subtle?

A key point in language design and program tooling is detecting that thing A,
which needs to be consistent with thing B far away in some other module,
isn't. Most hard to find defects involve some distant relationship. What could
be done with color or other automatic annotation to help?

~~~
greggman3
Undefined variables is already a thing in most IDEs? I know VSCode does it. I
can't imagine the others don't.

One interesting thing I've found in JavaScript (and I'm guessing it would be
useful in Python) is the CodeSpell plugin for VSCode

[https://marketplace.visualstudio.com/items?itemName=streetsi...](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-
spell-checker)

It spell checks code by separating camelCaseWords and
identifiers_with_underscores. Sure it's not "smart" but for dynamic languages,
at least for me, it helps prevent bugs. A static language the IDE would
already have the info it needs to highlight a non-existent property but in a
dynamic language generally not so I've found it quite useful.

------
jarofgreen
A really good post, but problem 2 is the biggie here:

> What if something needs to be colored two things for two different reasons?

As another commenter points out, JetBrains already do a lot of the ideas this
person has raised - but they do it in response to actions. Eg. Put the cursor
next to one bracket, the matching bracket is highlighted.

I think the key to breaking this open is an editor that does different
highlights at different times in response to user actions; the key will be
getting the right user actions so this is seamless and easy to use.

------
Symbiote
I don't think this is suggested: highlight all uses of a field in a class, or
a variable in a method.

IntelliJ does this for the field/variable under the cursor, with a background
colour and a margin colour. I'd like to see what it would be like with all
variables in their individual colours.

~~~
recursivecaveat
There are plugins for various editors using that kind of highlighting scheme
if you'd like to try it. Usually it is called "semantic highlighting".

------
smt88
JetBrains IDEs do almost all of these already, mostly without plugins. Other
IDEs probably do, too.

------
mdf
Good points in this post, although I'd add the word "Only" in front of the
headline.

IMO, syntax highlighting definitely has its uses, but why limit ourselves to
only using colors for that? I'd like to see a future, where depending on
whether I'm, for example:

\- sketching the structure for a completely new program,

\- implementing a new feature to an existing program,

\- refactoring an existing program,

\- debugging a problem in my program,

\- writing tests, or

\- trying to find something for reference

I'd be able to seamlessly switch between different highlighting schemes. I
don't (yet) know what I'd prefer for each of these different modes of software
development. The author of TFA lists some ideas for some different ways of
doing semantic highlighting. Maybe over the years I'd grow to like some of
them for some of these tasks.

------
JohnBooty
I'd like to see a _truly_ flexible code display system that treats things as a
matrix.

As the author describes, you have a number of possible concepts we might like
to visually denote:

\- Language constructs (ie, traditional syntax highlighting) \-
Parentheses/bracket pairs \- Context highlighting \- etc etc etc.

The author considers a single "information channel", text color.

There are other possible, largely orthogonal, information channels.

\- Text background color \- Font weight (light/bold/regular) \- Font face \-
Font italic / upright \- Font underlines (single, double, dashed, bold,
squiggle, etc) \- Font underline color? \- Perhaps font effects (shadow, blur,
etc) \- The editor gutter (Sublime and others expose this) \- Hover state (ie,
tooltips)

I'd like a text display system that lets me mix and match inputs and outputs,
matrix style. Perhaps I'd like to use font weights to denote one concept, and
underlines for another.

Perhaps I'd like to use standard text-color choice for language constructs ala
traditional syntax highlighting, and text background color for "context" or
"semantic" highlighting (where each variable gets a unique color within its
scope)

It would most certainly be able to create some real visual nightmares with
this level of control, but applied judiciously, it could be very valuable.

------
heavenlyblue
Written by an author who had never seen IntelliJ IDEAs highlighting framework.

Yes, it does syntax highlighting, but it also manages to mix it up with local
variables, etc.

------
the_other
I've got a muddled response to this.

On one hand, it sounds really helpful and exciting to have tools assist us
with these problems e.g. I've read the "show nesting" idea before and wondered
why we don't have it yet. Yet on the other hand, I think the post misses the
mark in some ways.

Most IDEs and editors working with most typed languages already do more than
colouring in types: they typically hint ahead of entering characters what type
of thing should go in each place where a typed thing can go. I think this is
more useful than colour, and we already have it.

Taking this further, if our editor has toggles for the categories listed, and
thus if it has ASTs (or equivalent things) to work over for semantics, then it
probably could do even more than highlighting. Some editors already help you
refactor: renaming identifiers across a project, pulling functions out etc.
It's not just colouring, it's active involvement with the code, its' types and
behaviours.

Further, the author deployed an overly simplistic view of colour and our
visual perception. In my case, the "red" circle wasn't much easier to find
because it was red, but because it was filled in. If it had the same colour as
the rounded squares, and was filled in, I would have found it just as fast.
The _real trick_ in the grid examples is that some tool (in this case the
author) had identified the significant information and done _something_ about
it, before the reader even got to apprehend the situation. Tooling which does
this would absolutely be more powerful than tooling which doesn't. We already
have tools which do some of these things. The drive shouldn't be "we need more
colour?" the drive should be "what features are most helpful to coders?"

------
Konohamaru
The author raised an interesting point with respect to syntax highlighting for
types. I find the convention that literals be typed (ints, strings, etc...) to
be inappropriate. That is very low-level and these should always be inferred.

Where typing should exist is at the level the author suggests, such as

* Iterators * List-like structures * Functions that return option types

and other more abstract structures.

------
minikomi
I've experimented with using various "semantic depth" color schemes when
writing clojure.

Depending on the code it can be useful.

Some I tried:

\- [https://github.com/istib/rainbow-blocks](https://github.com/istib/rainbow-
blocks) \- rainbow blocks, the "OG" \- [https://github.com/seanirby/rainbow-
blocks-bg](https://github.com/seanirby/rainbow-blocks-bg) \- rainbow blocks
but background style \-
[https://github.com/alphapapa/prism.el](https://github.com/alphapapa/prism.el)
\- Prism, a slightly more "friendly" rainbow blocks implementation

------
perl4ever
I find the unhighlighted circle to be about as quick to identify as the red
one.

Also, I work for someone who is color blind. This whole attitude of "we must
do something really important with color because it's so powerful" goes
against the idea that it has to be optional.

~~~
Finnucane
I am color blind, and some of the examples here just didn’t work for me. Even
normal syntax highlighting is of limited value, and I need to tweak color
schemes. Some thing like this could be useful as long as it didn’t get
overwhelming (selectable context switching) and the user had some control over
the colors used.

------
sradman
The use cases demonstrated are compelling but there seems to be an underlying
assumption that current syntax highlighting techniques are broken. After
reading the article I'd argue that there is a case to be made for enhanced but
temporary syntax highlighting modes where most highlighting classes are muted
except the focus of your attention, such as imported package variables.

Consistency is important, both for style and syntax highlighting. It is hard
to anticipate what a developer is mentally focused on as they scan code.

------
falcolas
What’s amusing to me is how few suggestions around highlighting ignores the
vast amounts of whitespace that surrounds every character. Borders (partial
and full), underlines, strikethrough, background colors, font faces... so many
options, yet are we’re constantly limiting ourselves to the color of a letter
and the occasional red squiggle.

The one valid excuse I can think of is “my editor doesn’t support it”, to
which I say BS. VIM, in a terminal, can do most of those.

------
dragonwriter
> Why aren’t things this way?

They...are. With proper extensions, for example, VSCode does many of those
things, and can switch which of the supported things it is doing, on top of
also doing syntax highlighting, and some of the changes are automatic and
context-driven.

And while VSCode is my current daily driver, that's been true of code editors
for quite some time.

------
raxxorrax
This is somewhat true, although I think keywords should still be highlighted.
I once wrote a plugin that highlighted all labels in different colors. Even
with a good palette it was really bad... The scope or block highlights could
be useful. Difficult for some languages probably, but I would give it a shot.

------
jnxx
He somehow missed rainbow identifiers, which is quite useful to represent the
data flow in a function:

[https://www.emacswiki.org/emacs/ColorIdentifiersMode](https://www.emacswiki.org/emacs/ColorIdentifiersMode)

------
mmgutz
Egads ... that rainbow parentheses suggestion is horrific to my eyes. Most
editors already do a good job of showing matched braces without shouting.

The problems he mentions are not tied to syntax. We need better CONTEXT aware
highlighting is the gist of the article.

------
pragmatick
I use [https://plugins.jetbrains.com/plugin/10080-rainbow-
brackets](https://plugins.jetbrains.com/plugin/10080-rainbow-brackets) for
IntelliJ to color matching brackets.

~~~
LandR
I find this plugin invaluable for Clojure!

------
Already__Taken
I don't know how to make it but I always wanted a vscode extension to text
background colour what scope you were in for teaching. The rainbow vomit some
beautifully indented student python would be...

------
erikbye
This is awful, a mess. I prefer little to no syntax highlighting, just a
grayscale theme with some bold and italics. Actual code clarity (naming, etc.)
and structure (indentation, whitespace, etc.) is what is important.

Recently there was an article here about what makes a good photograph, one
element was few colors, bringing focus to what matters. That is what code
should look like, like a good photograph. Not like a rainbow exploded.

~~~
yesenadam
It seems everyone prefers a different colouring to their syntax highlighting,
which you get used to, and other peoples' looks weird! But code _should_ look
like what you personally want it to, surely. There's no one way. Sure, you
could write a manifesto promoting your opinion to a prescription for everyone,
but why? You don't have to look at theirs, they don't have to look at yours.
Yes, maybe for code sharing, low- or no-colour is more like a common
denominator, and not strange/disorienting for anyone.

I find one of the main benefits of syntax highlighting is it's less tiring on
the eyes. And once you get used to it what your colours mean, much faster to
read. Having it look attractive as well is just a side-benefit, though not a
small one, if you spend a lot of your life looking at code!

------
reportgunner
I don't get it. Author is a programmer, why doesn't he just develop this? I
don't need this and I don't want it bloating my text editor.

------
gsliepen
Syntax highlighting is very useful, if not then it wouldn't be so commonplace.
The other highlighting methods are also useful. But there is not one objective
best highlighting method, it actually changes from moment to moment depending
on what you are doing. And many editors in fact already change how they
highlight things based on the context! For example, when I search for
something, my editor highlights all matches. If my cursor is on an opening
bracket, it will start highlighting the matching closing bracket.

Some of the highlighting methods shown in the article also seem a bit useless
to me. Assigning colors to indentation levels? Surely the indentation itself
is a huge visual clue already that doesn't need colors to be wasted. The
rainbow parentheses example also shows that there are limits to how many
obiously distinct colors you can actually have on screen at the same time, and
the colors don't uniquely show nesting levels.

~~~
rbanffy
> Surely the indentation itself is a huge visual clue

Apple had a security bug that stemmed from indented C code that was indented
but not within brackets. The second line of an if was not under the condition,
but was indented the same as the first line.

I myself prefer to use type to convey structure and color to convey semantics.
Having keywords boldfaced helps structure to stand out while coloring
parameters, locals and globals differently would help meaning to surface more
evidently.

~~~
alblue
For what it’s worth, it was introduced as a failure of an automated merge
process and not direct programmer entry. Had braces been used the error would
likely not have been caused (or would have compilation errors) but the fact
that it appeared indented you the same level is not because a programmer made
a mistake.

It is perhaps more telling that the code was compiled without an error
indicating dead code which was the net effect of this merge, but sadly warning
free code isn’t a goal.

~~~
rbanffy
True, but just by looking at the code it _looked_ right. An autoformat tool
would have made the mistake more visible. I wonder if popular linters would
pick it up (not only the lack of braces as a rule, but the misleading
indentation)

~~~
gsliepen
GCC version 6 and later actually warn about misleading indentation if you use
-Wall (which you should), and you can turn on -Werror to ensure you don't
forget to fix the mistakes.

~~~
rbanffy
One life goal is that all my software can compile without any warnings.

