
Training an AI to convert design mockups into HTML and CSS - AshishGupta93
https://medium.freecodecamp.org/how-you-can-train-an-ai-to-convert-your-design-mockups-into-html-and-css-cc7afd82fed4
======
igammarays
The example mockup image shown is really unimpressive. As usual, the
difficulty is in implementing the last 20% of edge cases and design
subtleties, not well-defined black-and-white block layouts like the given
example. It would've been simpler to build a classical program to handle that
example, and the result would be much better and more predictably structured.

It's like programming a self-driving car for a 2D game. Trivial. Now try
taking that to tar roads. I don't know why this toy example is even news, it's
nowhere close to the difficulty of real-world design mockups.

The AirBnB sketch2code example is a lot more impressive, but that's basically
just handwriting recognition with a 1 to 1 mapping of symbols to code pieces.

~~~
jeffmcmahan
Yes, this is typical AI weenie optimism. "We can handle many of the easiest
cases, so in X years we will be able to do all the really hard things that we
haven't even thought of yet!" No.

~~~
beached_whale
You could have made you point in a far better way that encourages and doesn’t
berate. Eg often people are optimistic but are unaware of the hard edge cases,
the edge cases are interesting

------
joe_the_user
Aren't there a bunch of WYSIWYG tools for creating appearance and HTML/CSS
together? Years ago when I looked, these existed (but designers prefered
photoshop just because).

I mean, I'd assume also there's normally a give-and-take between designer,
CSS-artist and client. The question is whether the neural network can also
learn to take calls at 3am from a client wanting a different shade of aqua.

~~~
richjdsmith
I'm pretty sure Adobe is still making Dreamweaver.

I know that's how I built my first website way back in grade school.

------
TheAceOfHearts
Writing HTML and CSS to implement a mockup is trivial for anyone with even a
little bit of web dev experience. If you want high quality code, an AI isn't
going to produce that.

I can't imagine ever using something so complex, if I can implement the same
thing in a few minutes.

There's also further considerations when dealing with the real world. For
example, you need to be aware of how to handle different accessibility
features. Designers rarely seem to care about things like accessibility, but
it needs to be declared somewhere.

~~~
meerita
If we think AI would not do awesome code, then we're ignoring the achievements
of many of the AI bots. I saw an LOL bot outperform lots of world class
videogamers. What makes you think a bot will not outperform any HTML/CSS
frontend dev.

~~~
sametmax
Right now, context. The ai will outperform us.

But it currently lacks a lot of data to do so. A mock up is only a small part
of the data and feedback loop you need to create a ui.

~~~
vanadium
And then layer in techniques for performance and accessibility best practices.
Those are complexity modifiers beyond just slinging code out of an AI.

------
shironineja
This is what we are all working for. If we can automate the stupid stuff we do
then we can work on other interesting stuff and automate it too.

I don't understand why we aren't all working on automating everything that we
do.

Every line of code you pulled out of the code mine today is finite and given a
business logic rule engine it's outcome could have been generated.

~~~
mark_edward
Because we like being employed. I don't get paid by increasing the global
productivity of society. The idiot who automates their own job (and shares it
with their employer) is much more likely to be impoverished than lauded.

~~~
antonvs
I see this attitude all the time while people are being automated out of their
jobs. Those people end up getting laid off anyway, and then wondering why jobs
for their skillset don't pay as much as they used to.

Resisting automation doesn't help, because if you don't do it, someone else
eventually will.

If you're in a job that's at risk of being automated in the near term, you're
much better off learning how to automate it and switching to being one of the
automators, than continuing doing something that's all but proven to be
something machines can do. People capable of automating jobs out of existence
are in high demand, for obvious reasons.

If you're not _capable_ of becoming an automator, that's more of a problem. In
that case you probably need a strategy for moving into a different kind of
job, that's less threatened.

~~~
EGreg
Here is a silly question

Why will everyone need jobs if so much stuff is automated? Why not just give
out free money?

~~~
placebo
I don't think it's a silly question, but I think free money basically means
dismissing the money system altogether. That would work in an ideal world with
ideal people but we are not that far out of the jungle yet (and I'm not
referring to intelligence levels but to qualities that didn't evolve as fast
as our intelligence). So, while you're right in principle, in practice there
will probably still be a lot of injustice and suffering to go around long
after everything is automated. Having said that, I still think it's a step in
the right direction to reduce the ills of society.

------
m3kw9
What’s the point of the code isn’t human readable? You need to train the AI to
write code that is easily modifiable with low dependencies. Otherwise the
conversion isn’t very useful for anything unless your page is completely
static

~~~
sebringj
Right but what if we didn't have to code anymore? Who would be reading it?
Meaning, we don't worry about compiler readability anymore, we used to when
assembly was fine tweaked etc. Just a thought. I work on full stack including
front end animations etc but frankly its tiring and time consuming work to do
that. Imagine if code was just you talking to it or showing it some
whiteboards, then you would just talk to it again or show it other
whiteboards. Code doesn't matter in that case.

~~~
vtange
It's unlikely we will leapfrog from human-written code to relying on black-box
generated code in one step. Everyone's been pointing to AI being used to
augment existing jobs, as opposed to outright replacement.

In order to augment a front-end developer well, we'll need human-readable
code, unless we like reading uglified code/make an AI for that too.

~~~
mattmanser
We did with compilers, isn't this analogous?

------
karmakaze
Honest question: is anyone else sensing some defensiveness here? The timeframe
is open to debate but what's more controversial is whether html/css is near
the frontier of solvable problems. After AlphaGo, my worldview is changed
forever. Glad to be still being valued for something I started out of joy but
in no way feel it's not disruptable. Sometimes I even wonder why it hasn't
advanced further since it's quite well defined, digital data in/out.

~~~
memebox3v
Yup, lots of ppl are very defensive about this! We are the farm workers
looking at a prototype of a combine harvester. We will stop ourselves from
believing it until its taken our job and then we will claim it was obvious all
along. Ppl are emotional.

------
nightski
This is really cool and I believe necessary work. We need to start thinking
about how to automate programming and to free ourselves much further from the
mundane. Trust me, there is still plenty of room for people to analyze
requirements and come up with a high level solution to people's problems that
doesn't require fiddling with float: right and various technical hacks.

That said I wonder if this is the right approach. At some point "AI" as used
in this context is just a function mapping from an input domain to an output
domain. The output domain in this case, "code" was designed for human
readability (whether it succeeded is a whole different approach).

What would a programming language designed for output from an AI system look
like? How could we optimize it to reduce the output domain size of the
function the AI has to train to learn? How could we optimize it to make the
problem more tractable for machines? I feel like there is an entire field of
research here. Maybe it has already been studied and I am just late to the
game.

------
book_mentioned
Turning web design mockups into code with Deep Learning |
[https://news.ycombinator.com/item?id=16115353](https://news.ycombinator.com/item?id=16115353)
(Jan 2018)

>chrisfosterelli: _This is a neural network that takes an image and predicts
very simple blocks (like BODY, TEXT, BTN-GREEN in the bootstrap example) and
then uses a map to convert them to well-formed HTML_

>jamesjyu: _I 've always wanted to do a contest with other frontend coders to
see who could get closest to a complex layout—like the NYTimes—in one go._
>>janneklouman: _these types of contests exist! I went to one of these[1]
maybe two years ago in Stockholm and I had a blast_ [1]
[http://codeinthedark.com/](http://codeinthedark.com/)

Pix2code: Generating Code from a GUI Screenshot |
[https://news.ycombinator.com/item?id=14416530](https://news.ycombinator.com/item?id=14416530)
(May 2017)

~~~
ghostbrainalpha
That second link doesn't have anything to do with A.I. does it?

~~~
book_mentioned
> _a contest with other frontend coders to see who could get closest to a
> complex layout_

I don't know of any record of past entries nor if any participants used A.I.;
it doesn't seem likely.

------
EGreg
Why are design mockups done in photoshop, when HTML and CSS can be faster and
easier once you have the basic components??

~~~
kowdermeister
1) Because that's how most designers create layouts who can't code an not
really interested in doing so.

2) It demonstrates AI capabilities on a new level.

3) If you design in PS or Sketch, then you will create more interesting
layouts, because you won't skip things that are inconvenient or hard to do.

~~~
andanotherthing
Does anyone use SketchApp to design at all? Just curious, I thought Photoshop
was being replaced..

~~~
returnInfinity
Yes, millions of designers use Sketch, Invision and Adobe XD. Photoshop is
getting disrupted at every low end feature-set. Photoshop will remain the
standard for complex image editing despite AI or whatever, because decades of
cutting edge research and patents.

------
efitz
OK this is juvenile but I found it funny that after 250 epochs
([https://emilwallner.github.io/html/250_epochs/](https://emilwallner.github.io/html/250_epochs/))
the AI just came up with busted HTML and: penas penas penas penas penas penas
penas penas penas penas penas penas penas penas penas penas penas penas penas
penas penas penas penas penas penas penas penas penas penas penas penas penas
penas penas penas penas penas penas penas penas penas penas penas penas penas
penas penas penas penas penas penas penas penas penas penas penas penas penas
penas penas penas penas penas penas penas penas penas penas penas penas penas
penas penas penas penas penas penas penas penas penas penas penas penas penas
penas penas orna.

------
wonder_bread
Is this a dupe of FloydHub's blog post from a couple weeks back? Markup looks
very similar

------
sly010
I agree with the need, my issue is with reproduceability and generally things
that are not defined by the sketch (behavior, different screen sizes, etc). If
everything were fully defined, we could just write a compiler from the high
level definition to the low level. The real problem is that static 2D images
are not good representations of dynamically sized interactive documents.

------
adamsea
Kind of related to this, what do people think frontend developers will be
doing in five years, or, where will they migrate to as frontend (either via
tools like this or like AirBnBs SketchToCode) becomes more automated?

------
maffyoo
what about the converse of this? what about producing a basic set of content
and letting AI taxonomise or organise it and then create a design based on
design models. It could work both ways. I guess ultimately the problem will be
solved both ways and AI could create entire experiences based on specific
domains. Also, for those claiming this will never work i would suggest looking
at some of the amazing stuff being done with AI right now. This kind of
problem, including optimising and making best usage of html/css is almost
tailor made for AI.

------
talmand
Interesting outcome.

The final html code from the first example is not that bad. There's the usual
problem of beginner coders of too much div wrapping that probably isn't really
necessary. I'm curious if the system can also create the css. The css also
suggests a beginner coder, such as an overuse of unnecessary clear classes
because of the overuse of floats for layout. Or having extra class names for
things that are easily handled by a parent class reference, such as a "last"
class on the last li in the list. Although, it appears the css is just a
template obtained online. More on that later.

The second example using bootstrap is a soft failure in my eyes. Although the
html does render correctly in the browser, which is because browsers do their
best to render crappy html, the code is rough. The main problem is it decided
to render the head element as a header element. Compared to the first example
I'm shocked that this is the generated output. The usage of bootstrap does
pose an interesting thought in that the content section of the html is more
precise than the first example.

My reaction to this is it's a decent try at generating a website based on very
strict rules assuming that more than half of the website creation process
still requires a human to complete. For example, I could see this working
quite well if one were to design your mockups strictly be bootstrap, or a
template, and provide that css beforehand. If the mockup is custom outside of
the template/bootstrap css then it'll have to generate that css itself. Which
I think I'm more curious if that's possible. Generating html is easy as you
can establish ground rules of "use this series of nested elements for this
situation" and so on. The examples provided could just as easily be created by
a drag-and-drop system that allows a non-coder to build a basic website. For
that matter, use a markdown to bootstrap converter and train your
writers/editors on the bootstrap basics and off you go.

But it sure did look like a fun learning exercise. As a front end dev, I'm not
worried over my future and would be curious to see where it goes.

------
meigwilym
Looking forward to this being a photoshop plugin.

------
toblender
Now we just need it to do the javascript ...

------
sureaboutthis
Up next, an AI for creating paintings and art and Sistine Chapel ceilings.

No. This will never be a thing. It will be copycat pages, never anything
unique.

------
gedy
The really useful AI would be to take business data or workflow and generate
layout and styles to eliminate the hand made mockup stuff.

