So I've been hacking on this new DSL for design that allows the designer to specify figures in terms of relationships, which are compiled down to constraints and solved using an SMT solver. This also leads to good support for abstraction, because constraints compose nicely.
The system is still a WIP, but it's usable enough that I made all the figures and diagrams in my latest paper and presentation with it, so I thought I'd write up a blog post about it and share some results.
This post is about how I think about design and how the programming language implements this philosophy. It also contains a number of case studies of designing real figures with the tool.
There are also some (probably useless?) but really amusing results, like what happens if you solve figures via gradient descent: https://www.anishathalye.com/#gradient-descent (the result is oddly satisfying).
I'd love to hear what you all think about the ideas in the post. Do you share any of my frustrations with existing design tools? How do you think when doing design -- does your model match mine, or is it something different? Do you currently use any design tools that you're really happy with, that I should know about?
Do you plan to add a GUI editor too? What do you think of Xcode/Figma's constraint system?
You can use something like https://diez.org/ to provide easy cross platform support. (Note it's only partially open source)
The browser and most "responsive UI" platforms like Flutter have a built-in constraint solver to process layouts. However often the language does not provide sufficiently powerful systems to express constraints in relation to each other, so for the backend you can probably take advantage of existing code when Z3 is too heavy a dependency. (I am not sure if this is possible for all layout solvers, you will probably have to prove it mathematically that the CSS (or whichever platform you are targeting) solver is equivalent or of higher abstraction level in relation to your design language so it can generate the appropriate code. Otherwise the purely declarative approach might not work and you will need JS/Dart/the native scripting language.)
I was surprised not to see any reference to at least some of all the previous work, such as the arc from ThingLab to Cassowary that led to commercial systems such as Apple's Autolayout. Or the Garnet/Amulet systems, with lessons learned. Or Feiner's Drawing with Constraints
You also initially write that the alternative to constraints is to position manually/numerically, but that obviously isn't true, as you have link to the code to do it procedurally later. In the same vein, you initially claim that you can't have abstraction without constraints, but that's also obviously not true when you have procedural abstraction.
You do say that the constraint code is "nicer", and I would probably agree with you, it would probably help to focus on that difference.
Metafont was the original system for drawing fonts for TeX and used constraints to describe the character shapes. Metapost extended this to diagrams in general. They might be an interesting point of reference.
I find it interesting that you tend towards SMT solvers instead of gradient descent solvers. Is it because you want to avoid having to specify approximate solution? CAD tools get away with this because the draughtsman is providing the rough starting point.
Also, as a further question? Is it open-source/free software? I couldn't find any links.
I'm currently using SMT for a number of reasons. One issue is that I've found the constraint is often non-convex, and this sometimes causes issues. And yeah, as you guessed, the initialization is also a hard part. With some figures, I've found that gradient descent can find the right solution with a good initialization (or even with just trying a handful of random initializations), but it doesn't work every time, and I wanted to avoid relying on the user to provide a good starting point.
The right thing moving forward might be some hybrid of SMT and gradient descent, especially if an interactive GUI editor comes into the picture. SMT could provide the initial solution, and gradient descent could help when the user manipulates the figure using the mouse.
I'll definitely give it a go if I find the time. I love the idea.
If you could make it work with LaTex you could probably find some happy users in IEEE / ACM type communities that would use it for drawing figures in research publications. However, it probably won't take on unless it's made open-source (most researchers are on tight budgets already!)
The integration works pretty well, and it's even possible to make the fonts and sizes match (e.g. see Figure 1 in https://pdos.csail.mit.edu/papers/notary:sosp19.pdf).
Is there actually no other existing tools that have a similar functionality?
I'll confess that one of my biggest wishes is for a universal standardized system and visual language+tool for graphic design with constraints -- so that graphic designers can learn a single system and simply export to HTML+CSS, to iOS layout, to whatever it is.
Implementing designs in front-end requires so much back-and-forth with designers -- "what should happen when this text overflows, what should happen if the viewport width doesn't fit this", etc etc etc. I wish this stuff would just be built in to visual design tools, zero code required, and map perfectly to software implementation. It's silly that it's almost 2020 and we still have to manually map this stuff to code.
The first obvious challenge is consistent rendering and performance. The sheer number of targets is overwhelming - browser/ios/android/tv/etc.
The second less obvious challenge is to clearly distinguish what designers are responsible for, vs what developers are responsible for. Some things are easy - "what should the text color be", for example. Other things, like determining accessibility roles, are less obvious.
There needs to be a tool that allows designers and developers to collaborate while maintaining control over their own domain. Not an easy thing to pull off.
To address a couple common questions:
Q: Is Basalt open source? A: Not yet, but it will be. The language is still under heavy development, and it's currently being completely rewritten in Racket. I'm waiting until it's a bit more stable to release the code.
Q: Are there plans for a GUI tool? A: Yes, I think it would be awesome to have a GUI tool! It seems like a hard problem to implement this, so it might take some time. The live preview is the best that I have currently.
I'd love to explore what's possible with Basalt in a language I'm comfortable in, and integrate it with my favorite tooling.
I've been longing for a 2D diagramming tool with just this feature. Unfortunately there are not many options. Solvespace seemed the most promising last I looked: http://solvespace.com/index.pl
I hope in the future there are GUI tools that can help with designing Basalt documents.
Exactly. That's the right way to do layout, with a constraint-based GUI. The high-end CAD products have had this for well over a decade. The CAD people can even do constraints on curves, like having a line be tangent to a circle or an arc and line meet with equal tangents. This is far beyond what the HTML crowd even aspires to.
The Basalt people, though, have a textural interface for graphic design, which is pounding a screw. Ask any graphic designer.
Like others (apparently), I started writing a system remarkably similar to this earlier this year. I just wanted to generate some simple diagrams programmatically, and went down the rabbit hole of creating a Python library to combine Z3 with SVG generation. For whatever reason, I was getting awful performance from Z3, so I started writing my own constraint solver. That ended up being a lot more of a pain than I had the patience to deal with, so I abandoned it.
Hi. I've been using Illustrator as my main art tool for most of twenty years.
1. Draw a circle. Give it a stroke in some color.
2. Open the Appearance palette.
3. Using the button at the bottom of the Appearance palette, add a new stroke. Make it another color.
4. Select this stroke and add effect>convert to shape>rectangle, relative sizing, add 0pt in X and Y.
5. Add effect>distort and transform>transform to the stroke. 70% in X and Y.
6. Open the Graphic Style palette and make a new style. Give it a name.
7. Visit the Appearance palette's menu and turn off "new objects have basic appearance".
8. Select the Graphic Style you made, draw some circles using it.
9. Turn off the Transform effect in the Appearance palette; hit 'redefine Graphic Style' and watch every circle change.
While your later examples like "make a little graph constrained to a shield" are more complex to make, there's still ways AI could help you do this stuff - you could maybe make a pattern brush with a circle for the end and corner pieces, and empty space for the main body, pile that on top of a plain stroke, and quickly drag some points around until you had a shape you like.
Eventually your examples get out of the territory of things I'd consider sane to do automatically in the tools AI has. But it can do more than you think.
okay enough procrastinating back to drawing my crazy comic in AI :)
If you want to have a square inset into a circle in Affinity Designer, you:
- Draw a circle.
- Create a square from the middle of the circle outwards. The square will _constraint_ perfectly to the circle.
If you want to do the reverse (circle inside square), it's even easier since the bounding box of the 2 objects is one and the same ...
I have the feeling that the author is simply unfamiliar with current design programs. Those haven't stagnated like one might think from the article.
I've seen some drawing software once that was particularly nice in that it pasted under your mouse (instead of next to the copied object, or in the middle of your viewport, etc.). Unfortunately, I can't remember which software was that.
Given you seem to focus on diagram layout, is the added complexity in defining and solving arbitrary constraints worth the power you get out of having them? Or, can you give an example use of arbitrarily powerful constraints?
More about it: http://www.zeldman.com/2018/05/02/transcript-intrinsic-web-d...
You can deeply nest Autolayout frames.
For example to get the max between v0 and v1 you can use those 3 constraints:
[v0 > limit @ priority1, v1 > limit @ priority1, limit == -infinity @ priority0]
|limit| contains the max, which you can use in other constraints.