
Experiments in Constraint-Based Graphic Design - anishathalye
https://www.anishathalye.com/2019/12/12/constraint-based-graphic-design/
======
anishathalye
Hi HN! I've been frustrated with standard GUI-based design tools, because they
don't match the way I think: like a programmer, in terms of relationships and
abstractions. (Does this resonate with the community here?)

So I've been hacking on this new DSL for design that allows the designer to
specify figures in terms of relationships, which are compiled down to
constraints and solved using an SMT solver. This also leads to good support
for abstraction, because constraints compose nicely.

The system is still a WIP, but it's usable enough that I made all the figures
and diagrams in my latest paper and presentation with it, so I thought I'd
write up a blog post about it and share some results.

This post is about how I think about design and how the programming language
implements this philosophy. It also contains a number of case studies of
designing real figures with the tool.

There are also some (probably useless?) but really amusing results, like what
happens if you solve figures via gradient descent:
[https://www.anishathalye.com/#gradient-
descent](https://www.anishathalye.com/#gradient-descent) (the result is oddly
satisfying).

I'd love to hear what you all think about the ideas in the post. Do you share
any of my frustrations with existing design tools? How do you think when doing
design -- does your model match mine, or is it something different? Do you
currently use any design tools that you're really happy with, that I should
know about?

~~~
LeftHandPath
I remember wishing for something like this when I started working in
Illustrator after using things like rainmeter and hand-writing SVG code for
ages. Of course, I don't have the patience or dedication to implement that
sort of thing.

I'll definitely give it a go if I find the time. I love the idea.

If you could make it work with LaTex you could probably find some happy users
in IEEE / ACM type communities that would use it for drawing figures in
research publications. However, it probably won't take on unless it's made
open-source (most researchers are on tight budgets already!)

~~~
anishathalye
Basalt as it exists today already integrates pretty well with LaTeX -- I have
papers written in LaTeX that have Basalt figures. The figures are rendered
into SVGs, converted to EPS using Inkscape's CLI, and placed in the paper with
\includegraphics.

The integration works pretty well, and it's even possible to make the fonts
and sizes match (e.g. see Figure 1 in
[https://pdos.csail.mit.edu/papers/notary:sosp19.pdf](https://pdos.csail.mit.edu/papers/notary:sosp19.pdf)).

------
crazygringo
This is really cool, and looks appropriate for the hacker crowd.

I'll confess that one of my biggest wishes is for a universal standardized
system and visual language+tool for graphic design with constraints -- so that
graphic designers can learn a _single_ system and simply _export_ to HTML+CSS,
to iOS layout, to whatever it is.

Implementing designs in front-end requires so much back-and-forth with
designers -- "what should happen when this text overflows, what should happen
if the viewport width doesn't fit this", etc etc etc. I wish this stuff would
just be _built in_ to visual design tools, zero code required, and _map
perfectly_ to software implementation. It's silly that it's almost 2020 and we
still have to manually map this stuff to code.

~~~
tenaciousDaniel
I'm working on that right now, actually. Though it will be a while before it's
ready to see the light of day. There are a couple of challenges.

The first obvious challenge is consistent rendering and performance. The sheer
number of targets is overwhelming - browser/ios/android/tv/etc.

The second less obvious challenge is to clearly distinguish what designers are
responsible for, vs what developers are responsible for. Some things are easy
- "what should the text color be", for example. Other things, like determining
accessibility roles, are less obvious.

There needs to be a tool that allows designers and developers to collaborate
while maintaining control over their own domain. Not an easy thing to pull
off.

------
anishathalye
Thank you all for the pointers to related projects and research papers!

To address a couple common questions:

Q: Is Basalt open source? A: Not yet, but it will be. The language is still
under heavy development, and it's currently being completely rewritten in
Racket. I'm waiting until it's a bit more stable to release the code.

Q: Are there plans for a GUI tool? A: Yes, I think it would be awesome to have
a GUI tool! It seems like a hard problem to implement this, so it might take
some time. The live preview is the best that I have currently.

~~~
j88439h84
I love this. I really hope to see the Python version.

I'd love to explore what's possible with Basalt in a language I'm comfortable
in, and integrate it with my favorite tooling.

------
rgovostes
I once took a course on AutoDesk Inventor and was blown away by its parametric
sketching tools, which allow you to define constraints on points, lines, and
angles. It was very easy to draw a rough sketch of what I wanted and then add
the necessary constraints to "clean up" the design—I want these objects
horizontally aligned, with identical dimensions, arranged equidistantly from
this point, etc.

I've been longing for a 2D diagramming tool with just this feature.
Unfortunately there are not many options. Solvespace seemed the most promising
last I looked:
[http://solvespace.com/index.pl](http://solvespace.com/index.pl)

I hope in the future there are GUI tools that can help with designing Basalt
documents.

~~~
Animats
_I once took a course on Autodesk Inventor and was blown away by its
parametric sketching tools, which allow you to define constraints on points,
lines, and angles. It was very easy to draw a rough sketch of what I wanted
and then add the necessary constraints to "clean up" the design—I want these
objects horizontally aligned, with identical dimensions, arranged
equidistantly from this point, etc._

Exactly. That's the right way to do layout, with a constraint-based GUI. The
high-end CAD products have had this for well over a decade. The CAD people can
even do constraints on curves, like having a line be tangent to a circle or an
arc and line meet with equal tangents. This is far beyond what the HTML crowd
even aspires to.

The Basalt people, though, have a textural interface for graphic design, which
is pounding a screw. Ask any graphic designer.

------
zwegner
This is very cool! I hope this gets released--even if it's a rough prototype,
I would still use it in its current state.

Like others (apparently), I started writing a system remarkably similar to
this earlier this year. I just wanted to generate some simple diagrams
programmatically, and went down the rabbit hole of creating a Python library
to combine Z3 with SVG generation. For whatever reason, I was getting awful
performance from Z3, so I started writing my own constraint solver. That ended
up being a lot more of a pain than I had the patience to deal with, so I
abandoned it.

------
egypturnash
> What if the figure design were changed slightly, for example the circle was
> to be inscribed in the rectangle? With Illustrator, it requires recomputing
> all the positions by hand; with Basalt, the change is one line of code:

Hi. I've been using Illustrator as my main art tool for most of twenty years.

1\. Draw a circle. Give it a stroke in some color.

2\. Open the Appearance palette.

3\. Using the button at the bottom of the Appearance palette, add a new
stroke. Make it another color.

4\. Select this stroke and add effect>convert to shape>rectangle, relative
sizing, add 0pt in X and Y.

5\. Add effect>distort and transform>transform to the stroke. 70% in X and Y.

6\. Open the Graphic Style palette and make a new style. Give it a name.

7\. Visit the Appearance palette's menu and turn off "new objects have basic
appearance".

8\. Select the Graphic Style you made, draw some circles using it.

9\. Turn off the Transform effect in the Appearance palette; hit 'redefine
Graphic Style' and watch every circle change.

While your later examples like "make a little graph constrained to a shield"
are more complex to make, there's still ways AI could help you do this stuff -
you could maybe make a pattern brush with a circle for the end and corner
pieces, and empty space for the main body, pile that on top of a plain stroke,
and quickly drag some points around until you had a shape you like.

Eventually your examples get out of the territory of things I'd consider sane
to do automatically in the tools AI has. But it can do more than you think.

okay enough procrastinating back to drawing my crazy comic in AI :)

~~~
Atomskun
I don't understand neither the author's nor yours approach to doing this in
Illustrator.

If you want to have a square inset into a circle in Affinity Designer, you:

\- Draw a circle.

\- Create a square from the middle of the circle outwards. The square will
_constraint_ perfectly to the circle.

If you want to do the reverse (circle inside square), it's even easier since
the bounding box of the 2 objects is one and the same ...

I have the feeling that the author is simply unfamiliar with current design
programs. Those haven't stagnated like one might think from the article.

~~~
egypturnash
It's more about drawing _multiple_ squared circles without hassling with cut-
and-paste.

~~~
TeMPOraL
Cut-and-paste isn't a hassle, it's a very ergonomic tool.

I've seen some drawing software once that was particularly nice in that it
pasted under your mouse (instead of next to the copied object, or in the
middle of your viewport, etc.). Unfortunately, I can't remember which software
was that.

------
omeze
I think this is super cool and definitely fits my mental model of how I think
about diagrams and designs! Would love to take a look at the code if its
available. I'm not a front-end engineer but it seems to me like a more
expressive version of CSS parent/child tags and width/height percentages.

------
tobmlt
Eh hem... One of those developers of constrained "CAD programs, not practical
graphic design tools" chiming in: This is cool! Very nice work and beautiful
presentation. It makes me pine to build an actual nice website explaining what
I did for B-splines/could do for subdivision (so I assert). Or you know, a
Rhino plugin, etc. Combining trains of thought, wouldn't constraint
programming for B-spline design be applicable to fonts? I do not know anything
about fonts really; I just had this belief that font design was often done
with splines of some sort.

------
tenaciousDaniel
Wow, this is eerily similar to a thing I've been working on for a few years as
well. Very cool! I love seeing other people who are asking the same questions
as I am, and arriving at similar conclusions.

------
mcphage
> Basalt doesn’t place many restrictions over these equations, which gives
> great flexibility to the user, but it makes the underlying implementation
> more challenging because it has to solve these equations automatically.

Given you seem to focus on diagram layout, is the added complexity in defining
and solving arbitrary constraints worth the power you get out of having them?
Or, can you give an example use of arbitrarily powerful constraints?

------
marc_io
This idea is also closely related to Jen Simmons's “Intrinsic Web Design”
concept. There is already some automation that can be done on web layouts at
the moment.

More about it: [http://www.zeldman.com/2018/05/02/transcript-intrinsic-
web-d...](http://www.zeldman.com/2018/05/02/transcript-intrinsic-web-design-
with-jen-simmons-the-big-web-show/)

------
wcrichton
You may be interested in Penrose ([https://penrose.ink](https://penrose.ink))
which generates graphics of math figures from equation. I don’t know if they
have as much in the way of a constraint language, but it’s certainly in a
similar vein of work, and I know they use some force directed layout.

------
VectorLock
You had my curiosity but seeing it implemented in Python and so Pythonic now
you have my attention.

------
perchard
Figma's recently released Autolayout is somewhat analogous:
[https://www.figma.com/blog/announcing-auto-
layout/](https://www.figma.com/blog/announcing-auto-layout/)

You can deeply nest Autolayout frames.

------
j88439h84
Very cool. How does it differ from Cassowary?

[https://cassowary.readthedocs.io/en/latest/](https://cassowary.readthedocs.io/en/latest/)

~~~
anishathalye
Cassowary is an incremental solver for linear systems of equations and
inequalities, while Basalt's constraints are more expressive (which is why it
needs an SMT solver). One example of something that's not expressible in a
linear system is computing bounds of a group of objects, which has been a
common pattern in my Basalt figure, e.g. to position a group of objects next
to another. The Group element in Basalt computes its bounds by computing
min/max of its elements' bounds, but Cassowary constraints can't contain
min/max.

~~~
Jyaif
I'm pretty sure you can implement min/max with a Cassowary-style constraint
solver using multiple constraints with priorities.

For example to get the max between v0 and v1 you can use those 3 constraints:

[v0 > limit @ priority1, v1 > limit @ priority1, limit == -infinity @
priority0]

|limit| contains the max, which you can use in other constraints.

------
tartoran
Very smart and original approach. I’d love to see a instructional video on how
you implemented it in Racket or the sorce code when it goes open source. Will
be keeping an eye on it for sure

------
NigelTheCreator
Interesting you find these problems a lot on CAD design, in 2D you can look at
tools like AutoCAD that do similar things in a different setting.

------
yters
Could this also result in a better latex? The layout engine leaves something
to be desired.

------
IAmEveryone
Isn't CSS something that most definitely deserves mention as prior art here?

------
pcurve
Stuff like this makes me feel so dumb and grateful that I have a job. lol

