
Show HN: Don't code your UI, draw it - karanchahal
https://github.com/karanchahal/DoodleMaster
======
qwerty456127
For my whole life I've been "drawing" real GUIs and writing the logic code,
not GUI code then. It started with VisualBasic 1.0 for DOS, then there were
different versions of VisualBasic for Windows, then Borland C++ Builder and
Delphi (I was using the former), then NetBeans Swing designer and WinForms
designer in VisualStudio. And now building a GUI is such a problem that a huge
number of almost-useless (I really prefer pen and paper over drawing with a
mouse) "mockup tools" has emerged and we have to use artificial intelligence
to produce the actual UI code or just code it manually...

~~~
sien
It really is disappointing how modern web GUI environments are harder to use
than what was around 20+ years ago.

There are still Visual Basic Apps from the 1990s floating around in large
corporations and in government.

Users who were expert in the process but not expert developers could easily
code up tools to their exact specifications.

[https://en.wikipedia.org/wiki/Visual_Basic#1990s](https://en.wikipedia.org/wiki/Visual_Basic#1990s)

Admittedly the deployment of web applications is much better and they can run
on different devices.

Hopefully when web assembly takes off easier to use tools will reappear.

~~~
Animats
_It really is disappointing how modern web GUI environments are harder to use
than what was around 20+ years ago._

Yes. We had drag and drop UIs decades ago. Now you can't even get a decent GUI
program for laying out a web page. CSS/Javascript got so messy that even
Dreamweaver became useless.

Part of the problem is that ad code requires a messy environment, so that ad
blockers and click generators have a hard time. Google, by policy, does not
permit you to put their ads in a iframe, where they belong. You can't even put
Google Hostile Code Loader ("tag manager") in an iframe sandbox.

~~~
madeofpalk
> Yes. We had drag and drop UIs decades ago.

We had _simple_ drag and drop UIs decades ago.

Computers operated within a lot more limited constructs back then. UI windows
didn't resize, or it was reasonable to expect that they're fixed. How did
those drag and drop tools back then handle creating a UI for screens ranging
from 400pt wide to 2560pt wide?

~~~
vthriller
> How did those drag and drop tools back then handle creating a UI for screens
> ranging from 400pt wide to 2560pt wide?

For simpler forms you'd just set anchoring properties of the widgets in
question (akRight/akBottom in Borland's VCL or whatever its counterpart is
called in WinForms or what was its predecessor). Nowadays it's even easier
with things like, say, GTK's HBox/VBox.

> Computers operated within a lot more limited constructs back then.

Resource limits didn't stop web browsers of the era from rendering complicated
tables in a variety of sizes.

~~~
int_19h
In WinForms, it's still called Anchor (it really did take a lot from VCL), but
since enums are scoped in .NET, you have Anchor.Left, Anchor.Top etc.

WinForms also had Dock (Left, Right, Top, Bottom, or Fill) which basically
forced the control against that edge of its parent, taking up full length
and/or height accordingly in one dimension, and using the explicitly specified
size in the other. With multiple controls with Dock, they'd gradually fill the
available space in Z-index order, stacking up against each other; then
whatever control had Dock=Fill, would fill the remaining space.

So yeah, resizable windows were common, and easy to deal with. The real
problem is dynamically sized controls themselves, e.g due to DPI changes, or
because the control contains dynamic content of unknown size. With those,
anchors no longer work well, and you need some kind of box and grid layouts.
Which are available - but it's kinda hard to edit them visually in a way
that's more convenient than just writing the corresponding code or markup.

The closest I've seen to visually editing UI that can accommodate dynamically
sized controls was actually in NetBeans, which had its own layout manager for
Swing. That thing allowed you to anchor controls relative to other controls,
not just to the parent container. Thus, as controls resized according to their
context, other controls would get moved as needed:

[https://netbeans.org/kb/docs/java/quickstart-
gui.html](https://netbeans.org/kb/docs/java/quickstart-gui.html)

[https://youtu.be/_ZW4ktG1DEE?t=129](https://youtu.be/_ZW4ktG1DEE?t=129)

Still, you needed to be very careful in the UI form editor, to make sure that
controls are anchored against other controls in ways that make sense.

Fundamentally, I think the biggest problem with UI designers today is that
they can't really be WYSIWYG, the way they used to be in the RAD era. The DPI
of the user machine might be different, fonts might be different, on some
platforms (Linux) themes can be different in ways that drastically affect
dimensions etc.

~~~
pjmlp
> Fundamentally, I think the biggest problem with UI designers today is that
> they can't really be WYSIWYG, the way they used to be in the RAD era. The
> DPI of the user machine might be different, fonts might be different, on
> some platforms (Linux) themes can be different in ways that drastically
> affect dimensions etc.

WPF and UWP can deal with it perfectly fine.

~~~
int_19h
Last time I checked the WPF designer, it used canvas by default - which allows
you to put elements exactly where you want by simple drag and drop, like the
old times in VB6 or Delphi; but then nothing actually reflows or resizes etc
at runtime.

If you choose to use layouts, it can display them, but editing it with a mouse
is no longer convenient. It's easier to just drop into XAML and hack on it
there. On every project I worked on that used WPF (which is quite a few by
now), nobody on the team actually used the UI designer, and everybody would
switch the default mode for .xaml files in VS to open the markup directly,
rather than the usual split view.

~~~
pjmlp
WPF designer default template is a Grid layout, so it must been a very long
time ago.

As for devs preferring to type by hand, well their loss I guess.

------
GuB-42
Maybe I am not a UI designer but I much prefer rule based approaches than
drawing.

Rule based means I tell use instruction like "this button must be on top on
that button horizontally centered", "this label must fit that text", "this
image must be between this and that", etc... and let the layout engine deal
with it. UIs are usually not paintings, window sizes vary, text length changes
with localization, decorations change depending on the environment, etc...
Approaching a UI like a canvas will certainly yield good results on the
designer machine, but will look out of place everywhere else, if it is usable
at all.

I think it is the basis or what they call "responsive web design".

~~~
intertextuality
This is basically how designing ios/osx apps in xCode works. You set a bunch
of constraints that are relative to one another. You can also specify what
type of screen those constraints will work on (like iphone vs ipad screens)
and portrait / landscape.

I find it quite pleasant once you get used to it.

~~~
daurnimator
It is (or at least was) all based on Cassowary:
[https://en.wikipedia.org/wiki/Cassowary_(software)](https://en.wikipedia.org/wiki/Cassowary_\(software\))

------
akx
This is not the sort of repo I would present to the public.

* There are no instructions on how to actually run the thing.

* There is no requirements.txt or similar, so I have no idea which version of dependencies I'd need.

* The repository is strewn with unnecessary files (.pyc/.ds_store/.so...), random-looking images with names like "plswork.png", a HTML file from some "starter kit"...

* I can't seem to find the React frontend that is mentioned in the readme -- on the other hand, it looks like `server2.py` is looking for them outside the repository (`".././reactExperiments"`).

~~~
Mononokay
Stop. Just because you refuse to do basic troubleshooting doesn't mean this
person shouldn't have showed it off.

~~~
leowoo91
Just fixed it for 15 minutes but it is still far from running, I would rather
wait until project instructions are added.

~~~
karanchahal
Hello, thank you for your interest ! I have fixed the repo structure and added
instructions on how to build and run. Hopefully that is fine, if not please
tell me about it and I shall update it as soon as possible.

~~~
leowoo91
Thank you for fixing, I think it's close. Now, flask says "index.html" is
missing :-) also step 2 should have "pip3 install -r" instead of "python3 -r".

~~~
karanchahal
Ah yes, I'm at work now. I'll fix that as soon as I get back, though that will
probably be after a few hours.

------
nebulous1
Interesting idea, but I'm not sure I see any advantage over dragging
movable/resizable components from a toolbar, but there are several obvious
disadvantages.

~~~
mcbutterbunz
What if you could sketch your UI with a stencil, paper, and pencil. Then take
a picture of it and have it translated into native components?

~~~
taeric
Odds are high, mine would look worse. I'm not exactly an artist.

Unless you mean just sketch using graph paper and translating coordinates. Not
sure why I need a camera for that. :(

~~~
jfolkins
It sounds like you don't. But many in my org come to me with an idea, and
having them think through the logical pieces and build a functional wireframe
would help under resourced people like me so much.

Also, many people just think that business logic for sanitization and
validation "just happens." The barrier to wireframing, for them, is too high
so they don't. But in this idea, I could see someone submitting a wireframe to
me and my response being "well what happens when a phone number is
international?" I'm educating stakeholders on the functional cost of producing
their idea.

This would theoretically create a feedback loop for future ideas and
initiatives as now, they've begun to be educated on the process. They have
direct experience.

Anyway, anything to lower that barrier in order to partner with and teach my
executives and their supporting staff would be a huge win. At least for me.

~~~
nnq
> many people just think that business logic for sanitization and validation
> "just happens." The barrier to wireframing, for them, is too high so they
> don't

Maybe the barrier is where it should be. Or maybe it should be _even higher!_
People who can't understand the logic of an interface have no business
creating or suggesting interfaces. An UI is meant to be _used,_ not _looked
at_ like a pretty picture in a frame. It should _feel good_ and _feel smooth_
and _increase productivity..._ not _look_ good. Some of the best looking UIs
I've ever seem were also the most utterly user-hostile, unituitive and
productivity lowering.

Sure, if you can afford to pay someone 500/hour or smth "outrageous" like that
(hint: you need a word-class artist, with advanced knowledge of user
psychology, that also has the brain of a business logic analist or of
programmer involved in product design) you could get something that both looks
goorgeous and feels smooth and increases user productivity 10x. But usually
you need to make sacrifices, and the ones the user will _hate_ you for are
those that make his life harder despite seeming nice and slick at first.

~~~
kartickv
I take your point about usability as opposed to visual design, and that many
slick-looking UIs actually lower productivity.

But:

In this example, unless not handling international phone numbers leads to
failure of the project, that can be handled later, say once the project is
approved and time estimation is being done. If I'm building a notes app, and
someone is proposing a new sign up form to increase conversions, and it has a
phone number field, handling international numbers is the last thing to worry
about at this stage (unless international numbers are a significant problem
with the old form leading to abandonment).

We shouldn't doom good ideas with irrelevant details, which are absolutely
relevant later, but not now. Product development happens in phases of
increasing fidelity, and issues need to be brought up at the appropriate time,
not too early, not too late.

~~~
nnq
> handling international phone numbers

Imo this is one of those things _best left unhandled,_ eg. "just use a plain
mostly unvalidated textfield, and throw and error only when you want to use
that data via another system like for a text message campaign". Mostly in
real-life if you want to target the entire freaking planet (not just 99% of
phone using people, but 99.999%), you'll get to realize that any validation is
not enough and that some phone numbers _need_ to contain arbitrary letters and
symbols in them (better don't ask... the world is big and weird :P), and that
yeah, those numbers will not be procesable by things like Twillio, but human
users with local knowledge will know how to actually "dial" them...

But it _needs to be a conscious decision,_ to _consciously_ choose to not-
validate and to understand that you give up the ability to 100% target phone
numbers for things like 2-factor-auth later on.

Not "forget that phone numbers need to be validated" and then, go and say,
"oh, let's do phone-based 2-fa mandatory" or whatever user interaction messup
like that.

------
jakelazaroff
Codeless UI is so exciting! This reminds me of Ivan Sutherland's Sketchpad
from the early 60s
[https://en.wikipedia.org/wiki/Sketchpad](https://en.wikipedia.org/wiki/Sketchpad)

~~~
zeroname
Don't get excited.

The thing that this demo shows in minutes could've been typed up in text in
_seconds_. It is not a productive way to do things. There's a reason that
these systems never catched on, they are unnecessary.

You may say, but "non-programmers" will use it! No, they won't. Designers will
use real design tools to create (non-functional) visual designs. Programmers
will bring those visual designs to functionality. That procedure _works_.
It'll keep working. These systems are diversions, not improvements. Worthy of
investigation, but not practical.

~~~
jongold
Yo - I did the Airbnb project linked ^^^ which I believe was the first of this
wave of deep learning-powered sketch->UI projects (though standing on the
shoulders of decades of R&D projects)

Our take was that we really do design on paper or whiteboard first & foremost,
which is why our project emphasized the webcam + sharpie thing rather than
drawing in-browser etc.

Here's a related thing I wrote about the need for design tools to design the
real thing, rather than facsimiles of the thing:
[https://jon.gold/2017/08/dragging-
rectangles/](https://jon.gold/2017/08/dragging-rectangles/) \- so so so much
process waste is because developers have to re-implement static pictures of
designs.

In our case, we didn't get buy-in to keep developing the project, but I'm
kinda jazzed that so many people are running with the idea

~~~
zeroname
> ...so so so much process waste is because developers have to re-implement
> static pictures of designs.

Okay, but did you attack that problem in a way that actually is _more
efficient_ than established UI paradigms?

Let's split the problem in two parts:

\- "Semantic" Design (Checkbox, ImageView, TextInput...)

\- Visual Design (fonts, colors, margins, ratios)

Your solution covers only the "semantic" part. Just look at the data. It's
basically a simple component tree. It _would_ be more efficient to just type
it up. It also would be more efficient to just drag rectangles from a
toolshelf instead of defining the type of the rectangle by drawing extra
hints.

As for the visual design, that's where you use a design tool like Illustrator
or Photoshop. Typing that up (e.g. in CSS) is surely a pain, but _sketching it
all up_ is out of the question. I certainly do see room for improvement in the
workflow here, but a sketchy interface isn't helping.

You have to question a lot of assumptions here, but also consider how
designers are most efficient with the tools they already know and have used
for years. Don't mistake something that you want to create for something that
users will actually want to use.

------
Someone1234
Microsoft also offers this as Sketch2Code:

[https://github.com/Microsoft/ailab/tree/master/Sketch2Code](https://github.com/Microsoft/ailab/tree/master/Sketch2Code)

------
mpolichette
If you already have a consistent design language,and those pre-baked pieces,
it seems like assembling them visually is the easy part. This would still
leave the behavior logic an data flows.

I’d be interested to see new things like this though!

------
atulvi
Why doodle? What difference does it make than to simply type the name of the
widget you want?

~~~
alangpierce
As one example, you might use a tablet and tablet pen to draw up a few
alternative sketches for the same UI, each of which might have tens of UI
elements. With that input mechanism, it seems a bit tedious to have to tell
the computer "this is a text box" for each text box, especially when a human
(or smart algorithm) looking at the drawing can see that pretty clearly.

------
laurentb
this is pretty cool! one thing on the image drawing one is that there are
"industry-wide" common practices when using a placeholder/drawing one to
represent an image as a square/rectangle crossed from each angle (if I explain
myself correctly). this dates back to the print design days, but would be a
shape that's a lot easier to draw than the pseudo picture that's common in OS
UI's. Yotako, which has been linked somewhere in the thread by tyingq (which
reminds me of the Palm Pilot days) uses a similar approach as well. It makes
it easier to draw and faster too

------
steveharman
Neat, but wouldn't it just be easier to have a pallet of controls / components
that user could drag into a view and cut out the need for interpreting a
drawing?

------
johnjohnsmith
Aaaaand we've come full circle back to Microsoft Foundation Class builder that
was all the rage in the early 1990s.

------
mygo
I still create my UI’s with a sketchbook and pen / pencil. For me there’s
nothing that matches it yet. Even though there are a bunch of digital product
design tools nowadays, I still tend to begin with my sketchbook.

It’s fast and expressive, and always there and always on, just a single tool /
interface (pen to paper), which is a huge advantage when just trying to get
concepts down visually. The clincher in this decade though, is the necessity
to think responsively while sketching, understanding that there are all sorts
of device sizes now. Then either mock it up in a design tool later or straight
code it up once I have the general concepts down. I’ve tried all sorts of
things (was really disappointed when SubForm shut down, that one was kind of
interesting). From concept to product, starting off with paper and pen is
still the quickest route for me.

------
wonderofworld
While i am a fan, we took it up a notch by allowing users to upload their UI
and we write the supporting code using A.I.
[https://crane.ai/](https://crane.ai/)

~~~
karanchahal
Do you use some generative AI or is it still quite rule based with AI doing
the classification bit ?

------
GijsjanB
I think this is great. I see a future where designers can draw and create an
interactive prototype. Anything beyond that is, my (educated) guess, long way
off. Anybody hoping this will remove the need for designers or front end devs
will be disappointed.

~~~
DonHopkins
I'm not disappointed because I don't think removing the benefits of thought
and effort by people who have studied and understand user interface design,
interaction, usability, accessibility, Fitts' Law, etc, is such a great idea
or good for users.

Drawing where to put the widgets (and not using constraints or grids or
automatic layout or adaptive rules or responsive design, or user testing and
performance measurement and empirical evaluation) isn't the hard or important
part of user interface design.

Who is supposed to benefit from this? A company who refuses to hire a
competent user interface designer and wants to crank something out really
quick regardless of quality? Users spend much more time using an interface
than you spend designing and implementing it, so optimizing the time and
amount of mental effort you have to put into making a user interface isn't
worth it if it doesn't result in a better, easier to use interface.

~~~
GijsjanB
I imagine a WordPress-like site, where you draw widgets and bring it to life
with a theme. Maybe nothing too rocket sciency, but I reckon there's a market
for it.

------
rcarmo
Only the other day I was looking at
[https://sketch2code.azurewebsites.net](https://sketch2code.azurewebsites.net),
which does something similar atop Azure cognitive APIs (I work on Azure).

------
alangpierce
karanchahal, not sure if this can be renamed retroactively, but this may have
been a good "Show HN" candidate:
[https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

Regardless, it's a cool idea that made me laugh a bit at first (seems almost
absurd at first glance) but then got me thinking about possibilities. Good
job!

It would be great if you could get it in a state where people could really try
it out (even in an unpolished state), either locally or ideally with a web-
based demo.

~~~
karanchahal
Hello, thank you for your comment. I have updated the code to a somewhat
polished state with instructions on how to run and build the project. I hope
this is fine. Please tell me if anything else is required of me in the issues
section of the repo!

Again, thanks :)

------
tyingq
Haven't used it, but Yotako seems similar:
[https://www.yotako.io/](https://www.yotako.io/)

------
platz
Why not create a text file that encodes the structure of the desired output
instead of having a CNN inferior it

------
omvek
Here is another example of this type of approach from a top-tier, peer
reviewed software engineering journal:
[https://arxiv.org/abs/1802.02312](https://arxiv.org/abs/1802.02312)

------
zachguo
Too bad nowadays frontend dev is more about state management than layout and
boxes. Designers have tons of tools turning mockups to interactable prototype.
Non-technical people have drag and drop tools.

------
jack9
Wireframe Sketcher (my fav out of them all) does this in a more efficient
manner.

------
andyidsinga
its a great concept & prototype - really reminds me of the palm pilot graffiti
writing system and alphabet [0].

after just watching the video - seems like adding a plugin system for
targeting various UI library (ex bootstrap) would be really cool (I didn't
read all of the text - maybe he suggested or already has this..)

[0]
[https://en.wikipedia.org/wiki/Graffiti_(Palm_OS)](https://en.wikipedia.org/wiki/Graffiti_\(Palm_OS\))

------
rglover
This is a fun idea! Congrats on getting it working.

It may not seem like much right now, but the fundamental idea behind this sort
of stuff is the future, especially for front-end code.

------
johannesbeil
Congrats! Can you maybe say a bit about how you trained it?

------
punnerud
How was the PyTorch model trained? (Most interested in the training data)

------
zeroname
> "Don't code your UI"

 _Do_ code your UI.

------
LifeQuestioner
woooooa this is so cool!!!

------
treve
7 comments in, and it's all ragging on this. Good job everyone

~~~
scarface74
What’s the purpose of developing in the open if not to get feedback? A pat on
the back and a attaboy doesn’t help anyone improve.

~~~
narenchoudhary
I feel people who are only willing to point out negatives (without any
appreciation for good parts) should refrain from giving feedback. Frankly
speaking, such feedback isn't very motivating.

~~~
wild_preference
Nor is it actionable. Exists solely for the "critic" to signal how unimpressed
they are, as if seeing the defects in something is some challenging feat.

We're in this obnoxious age of confusing useful criticism with any reaction
one can come up with on the fly, no matter how superficial. Like it's their
destiny to weigh in on something as rapidly as they can, and as if they're
doing some critical service for the universe.

The comment above exemplifies this when they say "what, we're supposed to pat
them on the head and say attaboy?" No, the problem is that you think you need
to fire off some undigested response at all. If you have nothing meaningful to
say, then just say nothing. It's okay.

~~~
scarface74
The feedback was actionable

\- add a requirements.txt to list dependencies

\- create a .gitignore file.

------
catacombs
Thanks, but I'll stick with Sketch.

------
KaoruAoiShiho
What are people's thoughts on when this will actually be usable? UI eng is
just another labor intensive automatable step for creatives. Next will be to
automate the design part too. Don't draw your UI, generate it from user
stories.

~~~
DonHopkins
Pff! Yeah. And generating users stories is the PERFECT application of
artificial intelligence. /s

You could automate the entire software design and development process! Could't
be any worse that what we have now, amiright?

