
Building Jarvis - tosh
https://www.facebook.com/notes/mark-zuckerberg/building-jarvis/10154361492931634
======
hacker_9
Say what you will about Zuckerberg, but this is a really cool project! Steps
he took:

1\. Connect his home devices (lights, thermostat, doors, music player, TV,
CCTV etc) to his computer, so he can turn them on and off from it and in
general automate his house from his PC.

2\. Add an NLP component to his computer, so he could text instructions to it
and it would apply the correct automation task. This can learn preferences too
and be told about mistakes so it does better in the future.

3\. Add face recognition, so it can recognise people at his door and
automatically open it if they are expected.

4\. Add speech recognition, by creating an app for the phone that is
constantly on and listening for his voice. This is then converted to
automation tasks again.

All in all, an interesting way to spend 100 hours. Calling it AI is a bit of a
stretch though; this is basically linking up a bunch of in-accurate sensory
parsers, coupled with some limited machine learning, connected to a bunch of
if statements. Still a long way off of Jarvis, who is basically a genius by
human standards!

~~~
IanCal
I think what's particularly great about this in a way is that it's not too
remarkable. I'm not knocking it, but it's awesome to be around when creating
an natural language, speech controlled, available anywhere system with face
recognition to automate your own home is something that can be done in 100
hours.

~~~
ocdtrekkie
I think the remarkable bit is that he did this while being the CEO of
Facebook. There's not a huge number of corporate CEOs who still code at all,
and have time to put in a big project like this. And as someone who's worked
with writing some of the same sort of code, it's really enjoyable to see
someone drastically more talented making many of the same observations and
encountering similar challenges.

~~~
skummetmaelk
It goes under CEO time as marketing.

~~~
tomcam
Facile, needlessly cynical, and with zero evidence. I don't like FB or
Zuckerberg, but what I read was a good write up of a fun project done well.

~~~
floatrock
Oh come on, it was clearly both. Try to tell us this excerpt is not an
advertisement for the facebook SDK:

> I'd learn a lot about the state of AI this year, but I didn't realize I
> would also learn so much about what it's like to be an engineer at Facebook.
> And it's impressive.

> My experience of ramping up in the Facebook codebase is probably pretty
> similar to what most new engineers here go through. I was consistently
> impressed by how well organized our code is, and how easy it was to find
> what you're looking for -- whether it's related to face recognition, speech
> recognition, the Messenger Bot Framework [messenger.com/platform] or iOS
> development...

The rest of that paragraph goes through all the other developer tools you can
sign up for in the facebook ecosystem, including convenient links. Then it
continues:

> One of our values is "move fast". That means you should be able to come here
> and build an app faster than you can anywhere else, including on your own.
> You should be able to come here and use our infra and AI tools to build
> things it would take you a long time to build on your own.

It was a neat project, not denying that, but there is no free lunch. The CEO
of a Fortune 500 does not write up a 5-10 min read without plugging his
interests.

You can enjoy what he's doing, but also recognize what it is.

~~~
ocdtrekkie
I think it's worth pointing out that he started his mobile efforts by talking
about how easy it was to make a Facebook Messenger bot, something they're
heavily pushing... and then said shortly thereafter, to do what he wanted, he
had to switch to making his own app, because Messenger couldn't do what he
wanted.

If a Google exec wrote this post, it would be about how amazing a Google
product was as a solution to the problem, and that would be the end of it. But
here's Mark saying "you know what, Messenger is great but I needed more". And
maybe that could turn into some pretty big advice for his Messenger team on
what to add next, but he seemed to be pointing out where the industry
(including his own company) could do better.

------
avitzurel
Here's what I absolutely love about this...

He's still a hacker. CEO of one of the biggest internet companies of our time
and he still hacks.

I explain to people all the time, true hackers are like true writers. I've
heard writers talk about the same thing. It's like an urge of creativity
inside you, you have to release it into something.

You can also see in the comments section, he's replying to people and
providing feedback.

I really dig this.

~~~
timdorr
Funny thing: Based on the screenshot in the post header, he's using
async/await and therefore the Hack language (instead of vanilla PHP):
[https://docs.hhvm.com/hack/async/introduction](https://docs.hhvm.com/hack/async/introduction)

~~~
archon810
I'm not sure why that's funny or remarkable - Facebook invented and is a heavy
user of Hack, it's only natural that he's familiar with it and chose it.

~~~
timdorr
It's funny as an alliteration: He is still a hacker hacking with Hack.

------
smaili
More than anything else, what stood out to me was his discipline for knowing
how to manage his time in a way that enabled him to build something like this
while still juggling the n number of other things on his plate.

~~~
eldavido
No kidding. I find it impressive both that he put aside (not _found_ , this
had to be a very deliberate decision given the demands on his time) 100 hours
to do this, AND that he got all of it done in 100 hours.

Is there anything written about his personal schedule/habits? When does he go
to sleep at night? When does he wake up? What kind of hours does he work?

I mean really, he has a kid, is CEO of a huge company, exercises, and still
has time to do this project. My hands are more than full trying to exercise,
learn Chinese, and run a small company, I have no idea where he finds time to
raise a kid in there. Incredible.

~~~
duderific
When you are a billionaire, you can certainly outsource all that mundane stuff
(like raising a kid) to extremely capable helpers. He only has to do exactly
what he wants to do. Pretty sure Zuck isn't changing any diapers or walking
the dog unless he decides he feels like it right at that moment.

~~~
dbmikus
I'm not sure I'd call raising a kid mundane... hahaha

~~~
duderific
Yeah, I see what you mean... I guess I meant that compared to running a multi-
billion dollar company, changing diapers or reading Cat in the Hat for the
millionth time seems pretty mundane.

~~~
NamTaf
You're missing what raising a child entails if you think that's all it
entails. If you raise a child without the deep emotional investment that comes
from a parent, you're not going to raise a child well at all.

~~~
duderific
I know, I have a two-year-old boy. I'm just saying he can get more of a break
than a regular person that doesn't have nannies and whatnot.

I haven't had two hours to myself since the day my kid was born, so the idea
of having 100 hours to do a fun project like automating my house is completely
out of the realm of possibility for me.

------
brianshaler
Anyone setting out to do something like this will quickly see how much of a
hassle it is to exfiltrate data from relevant services and your own devices.
APIs and SDKs cover a lot of use-cases, and that is all great and commendable,
but we don't truly own our own devices, data, or the algorithms that make them
useful. I think this will continue to hamper our ability to have a truly
useful 3rd party (your home in this case) orchestrate your digital life.

A worry about reading about Zuckerberg digging into this area is that anything
Facebook does will likely be within their walled garden. There is certainly a
glimmer of hope they'll make something on the "open" side, but I wouldn't get
my hopes up.

I wonder to what extent having great advances in this area by proprietary
cloud services will hamper the advancement and adoption of libre alternatives.

~~~
wgyn
> My experience of ramping up in the Facebook codebase is probably pretty
> similar to what most new engineers here go through. I was consistently
> impressed by how well organized our code is, and how easy it was to find
> what you're looking for -- whether it's related to face recognition, speech
> recognition, the Messenger Bot Framework [messenger.com/platform] or iOS
> development. The open source Nuclide [github.com/facebook/nuclide] packages
> we've built to work with GitHub's Atom make development much easier. The
> Buck [buckbuild.com] build system we've developed to build large projects
> quickly also saved me a lot of time. Our open source FastText
> [github.com/facebookresearch/fastTex...] AI text classification tool is also
> a good one to check out, and if you're interested in AI development, the
> whole Facebook Research [github.com/facebookresearch] GitHub repo is worth
> taking a look at.

Face recognition and speech recognition are the most obvious pieces that look
to be proprietary, but there is lots of other useful open stuff that he
explicitly links. FastText looks like it would be really helpful for at least
the natural language piece of Jarvis-like applications.

~~~
brianshaler
I haven't really dug into open source speech recognition, so I can't speak
much to that side of things, except observing that advancements favor deep
learning and cloud-based services over efficient on-device detection (aside
from very specific "hey siri"/"okay google"/"hey alexa" recognizers). When
it's not a library that can be included, but a service that must be
maintained, it'll be that much hard for free and open solutions to get
adoption.

For face detection & recognition, some of the greatest advancements are
getting acqui-hired and locked away (e.g. Facebook acquiring Face.com) there
is still progress being made with tools like OpenCV & Dlib. I just worry
progress in the open-source space will lag behind and continue to be difficult
to work with, as the delta grows between proprietary capabilities and open-
source options.

~~~
pducks32
Never doubt the power and intelligence of a few open source programmers
dedicated to building something.

------
sreyaNotfilc
I hate his ambition...

In reality, I hate that I am not as ambitious as I should be. Say what you
want, Zuck is really killing it and its admirable.

We're the same age, yet I let myself think feel that its someone else's job to
build the "cool" stuff.

I will have this article handy and read it from time to time. Hopefully it
will continue to motivate me.

~~~
user837387
Hey, I also work my ass off to the point of getting sick and I have
accomplished nothing. By that I mean I have not made any money. I've listed
some of my projects here and not even a single up vote. I don't like it but it
is what it is. I'm simply trying to say that I'm no less ambitious, and that
is probably true of a lot of us here. The gods of luck have simply not been on
my side, yet.

~~~
sfeng
It's very true that how hard you work doesn't correlate directly with success.
That said, learning how to work hard is one of the best things you can do to
become successful. Once you master that, you then have to learn from each
experience. If some of the things you're doing haven't worked out, try to
examine them and figure out where you went wrong. Feel free to post some here
for others to chime in.

Also, don't necessarily think not climbing to the top of HN means you're not
successful. It can be a popularity contest (Zuck will always make it to the
top), and it's also somewhat random. Often the content which makes it to the
frontpage was upvoted by friends, or reposted many times, I wouldn't take it
too seriously.

~~~
paradite
I just want to point out that the _upvoted by friends_ part is against Hacker
News policy and not encouraged and I personally consider them as very
unethical:

 _> Can I ask people to upvote my submission?_

 _> No. Users should vote for a story because it's intellectually interesting,
not because someone is promoting it._

 _> When the software detects a voting ring, it penalizes the post. Accounts
that vote like this eventually get their votes ignored._

[https://news.ycombinator.com/newsfaq.html](https://news.ycombinator.com/newsfaq.html)

------
beeker87
Is anyone else seeing the possible security risk shown in the 'Vision and Face
Recognition' section?

The main concern I would have is someone spoofing the visual recognition
system with something as crude as photographs, or something more advanced like
a mask.

He states in that section, "Once it identifies the person, it checks a list to
confirm I'm expecting that person, and if I am then it will let them in and
tell me they're here." My first thought is someone could see if there's a
friend/acquaintance who routinely visits, and then spoof their face with a
mask. Let alone if the system automatically lets Zuckerberg in, then that's
all that would be needed.

~~~
balamaci
Rather what the hell happened to the guy that had tape on his laptop camera
and microphone. Suddenly he's got an always on listening device. How does one
go from one to the other?

~~~
deadbunny
Securing a local network is orders of magnitude easier than a laptop which
accesses the internet constantly.

Stick the cameras on a vlan only accessable to the (local) servers doing the
face recognition, stick the servers on a vlan that has no direct connection to
the internet.

Compare this to a laptop which is connected to the internet with multiple
attack vectors (browser, email client, etc...)

~~~
balamaci
Does he have some custom build network router that he trust? Putting tape over
your camera means you know enough to not trust the laptop hardware neither the
OS or browsers like you said.

But he says: "We use .. Sonos system with Spotify for music, a Samsung TV, a
Nest cam for Max". So all of his appliances do get outside. I did a double-
take on Nest cam. So he's streaming videos from his house on the internet with
a closed source hw and software.

~~~
deadbunny
I didn't read the part about Nest to be fair but with some basic network
design you can easily segregate networks and reduce your attack surface
massively, even if you're using internet connected devices (seperate vlans,
use http proxies with ACLs, no inter device communication where not needed).

The difference between a switch and a laptop is that your switch isn't running
browsers with 0days found regularly, no malicious JS payloads, no phishing
emails.

To exploit a switch you generally need access to the management interface,
something anyone who has any experience with networking does not put on the
same network (virtual or physical) as laptops, iPads, televisions, or internet
connected cameras.

~~~
balamaci
I agree with what you are saying, however tape over the camera means you don't
trust the OS or the hardware manufacturer with a console command like modprobe
to actually do it's thing and disable the web camera.

------
dekhn
I think there's some law or truism that every geek who becomes a billionaire
spends more time automating their house than it would take to just walk over
and turn on the light switch.

~~~
dekhn
(I was referring to:
[http://wiki.c2.com/?LazinessImpatienceHubris](http://wiki.c2.com/?LazinessImpatienceHubris))

------
raz32dust
This is incredibly cool. I can't overstate the effect of having a CEO who is
an active hacker on the tooling and infrastructure in the company.

On top of that, this would give Zuckerberg so much insight into so many things
- the pain faced by developers, the pain faced by the users, how users would
want to use AI products, why developers are not able to build the products
users want. And this plays well into his strengths as a CEO. There aren't many
(any?) big company CEOs who can do this.

------
nodja
If you're interested, you can recreate the work he did in a fraction of the
time.

[https://home-assistant.io/](https://home-assistant.io/)

It's not voice centric like 'jarvis' would be, but it bridges most of the
existing smart devices that exist right now and let's you do a plethora of
many more things than 'jarvis' does. You can for example add gps tracking to
your phone and have it control your thermostat for when you're close to home,
or turn on the the lights at sundown and you're home, or you can program it to
shut off the light when something is playing on the chromecast, etc. It
supports alexa skills as well, so you can have that be a trigger for whatever
you need done.

Hass's oobe is basically just a dashboard for your smart devices, but with
some tinkering you can set up rules and customize it for your personal needs.

~~~
solidr53
OpenHAB is way more robust IMO

~~~
Splendor
Would you mind elaborating? What makes OpenHAB more robust?

------
chime
For me, the primary takeaway was: "This suggests that future AI products
cannot be solely focused on voice and will need a private messaging interface
as well."

> This preference for text communication over voice communication fits a
> pattern we're seeing with Messenger and WhatsApp overall, where the volume
> of text messaging around the world is growing much faster than the volume of
> voice communication. This suggests that future AI products cannot be solely
> focused on voice and will need a private messaging interface as well. Once
> you're enabling private messaging, it's much better to use a platform like
> Messenger than to build a new app from scratch. I have always been
> optimistic about AI bots, but my experience with Jarvis has made me even
> more optimistic that we'll all communicate with bots like Jarvis in the
> future.

I've noticed that myself with Siri. I avoided using it for years but finally
gave in and enabled full Siri access so I can ask simple questions when the
phone is locked. Our nightly lows in Florida have been all over the place
lately and my birds prefer to stay in the porch when it's 65F or higher but if
it's lower, I have to bring them indoors. So my most common question is: "Hey
Siri, what's the low temperature for tonight?" Unlocking the phone, swiping
right, clicking on the weather icon, waiting, and then parsing the screen to
locate the low temperature is a lot more involved.

However, now that I have Siri, I try to use it for more things via voice but
it doesn't always work because (1) Siri is not that good, (2) it is awkward to
try to find out what exact words to use for Siri, in front of other people,
and (3) there is no autosuggest for search when speaking, which helps a lot
when manually typing.

Personally, I don't think voice + text is enough. Facial/body gesture
recognition will definitely have a part to play in this long-term.

~~~
Ajedi32
I personally liked this statement:

> One thing that surprised me about my communication with Jarvis is that when
> I have the choice of either speaking or texting, I text much more than I
> would have expected. This is for a number of reasons, but mostly it feels
> less disturbing to people around me. If I'm doing something that relates to
> them, like playing music for all of us, then speaking feels fine, but most
> of the time text feels more appropriate.

I've found myself doing the same thing. For queries and commands which involve
only myself I prefer to type unless I'm alone, but for questions which involve
the people around me asking verbally seems more useful.

------
superquest
Having worked for a CEO who viewed "code" as a black box, I really appreciate
the shoutout to the infrastructure and tooling teams ...

~~~
mathattack
Indeed - these types of side projects make him a better CEO.

------
jameshart

         zuck-mbpl:~ zuck$ php j play some red hot chili peppers
    

PHP as a shell scripting language. Old habits die hard, huh?

~~~
Klathmon
It's not a bad scripting language in that case.

Quick startup, a good amount of CLI-specific APIs, available on many servers
already, and no compiler or huge amount of tools needed.

It's a bit verbose for little scripts that should be in something like bash,
but it's a great alternative to something like python if you are more familiar
with it.

~~~
mfonda
In addition to this, if a lot of the libraries you're using are written in
PHP, it makes a ton of sense that command line scripts using those libraries
would also be written in PHP.

------
tlrobinson
BTW, I'm definitely not saying this to knock Zuck since this was clearly more
of a fun challenge / learning exercise for him (it's awesome he did this), but
for anyone interested in setting up a similar thing, you can accomplish much
of the voice/music/automation parts without writing much (if any) code using
Alexa, SmartThings, IFTTT, etc.

Alexa can be a little frustrating sometimes but overall once you get her
dialed in it feels like the future.

I even set up a pet feeder like Zuck mentioned ([http://www.super-
feeder.com/](http://www.super-feeder.com/) you can use a normal home
automation 110V relay but I wired up a 12V relay to a C.H.I.P. for more fine-
grained control)

~~~
lyime
Yeah I was thinking the same thing. I use Alexa for controlling my A/V, lights
and music. It works relatively well without too much configuration. Although I
think it could go a step further but building some macros and rules.

------
pinnbert
This, written with PHP and Objective C, is far more inspirational than the
career-infused blog postings typical here.

~~~
bitmapbrother
He also mention Python - which probably does most of the heavy lifting.

~~~
WikipediasBad
I honestly screenshotted the python comment and framed it on my wall. I love
python, my entire startup's backend is coded in python, and getting zuck to
give a direct shoutout to any language other than a PHP family framework made
me elated.

------
satorii
Amazed by how much work he has done in 100 hours while running one of the
world's largest internet company. If anyone interested in trying out some of
the "Jarvis" without hacking, checking out
[https://saymosaic.com](https://saymosaic.com). We have put together a tool to
enable you interact with your smart devices/services with either voice on Echo
or text on FB messenger. Currently, it is in beta with more integrations and
aggregated services to come.

------
jayajay
This is one of the most insightful, down to earth articles I have read about
AI.

I hope Google/Amazon and all of the people in ML can take some time to read
it.

Main Important Points:

0\. Build a simple, easy prototype first. Then add better functionality later
when you have finished the simple things.

1\. Users will be using text over speech more than we think.

2\. Users always have their phones with them. This suggests that assistant
control should be done via the phone, rather than through fixed hubs around
the house.

3\. General AI is more akin to a limit point, rather than some attainable
state.

4\. Users need to be able to interface with training processes of their
assistant to create more personalized interactions. A new user-machine command
protocol may be invented soon to create an interface for this.

5\. New data protocols may be necessary to to jump start AI and IOT together.

6\. Aggressive AI will be needed to make AI more hands-off in the future; more
autonomous.

7\. More advanced mathematics and internal representation will be needed to
allow AI systems to obtain and apply domain knowledge in isomorphic tasks.

------
g12mcgov
I built something to similar this, using a Raspberry Pi, reverse SSH tunnels,
Phillips Hue, Nest, and a range of sensors.

Part of the project was an admin panel, to oversee the status/health of all
the systems, as well as a manual control panel. My biggest concern was
security. Granted, it was an internal network in my house, but if anyone got
access to it, they had camera feeds, and practically full control of my house.

I wonder how someone like Zuck protects this system, given all the nefarious
people who I'm sure would love access to it.

~~~
tomschlick
FB as a company has probably invested in network security for him / other high
level people. He already has physical armed security so it wouldn't be
surprising that they would protect him / his family digitally to prevent
hacking which could grow roots into his privileged access to FB's resources.

------
Apocryphon
Chances that Facebook will strike a deal with Disney to adopt the Jarvis name
and Paul Bettany voice and roll out their own A.I. helper personality in the
future to compete with Siri/Alexa/Cortana/Google Now eventually?

~~~
user5994461
Jarvis is from Iron Man.

~~~
Apocryphon
Which is Marvel, which is owned by Disney.

------
elwell
> is that Beast or a rug in that room?

The true Turing test for any AI: [http://nextshark.com/wp-
content/uploads/2015/10/ZuckDog_26.j...](http://nextshark.com/wp-
content/uploads/2015/10/ZuckDog_26.jpg)

------
roystonvassey
I was frankly surprised at how hands on Zuckerberg is. Brilliant.

Also, the most important takeaway, and one that I experience on a day-to-day
basis, was the idea that most ML systems currently are heavily over-fitted and
scale is going to be a huge challenge.

------
stevehiehn
I think many developers have not yet realized that you dont need a PHD in
mathamatics to creatively and effectively use modern ML frameworks. When
(most) devs need to store data they download there favorite DB, not roll there
own search algorithm.

------
thecity2
Facebook should hire this guy!

------
westoncb
My issue with this is calling it 'Jarvis'. Jarvis increased Stark's ability to
build new, cool things. That's interesting. Something along these lines--also
inspired by Jarvis--is something that interests me greatly. Regardless of
implementation, the end goal of this project is boring in comparison to
Jarvis' actual purpose.

------
bhntr3
> a grey t-shirt cannon would require hardware modifications to work

This almost felt like satire but in the best way possible.

------
letterthief
Personally, I find the idea of artificial intelligence both silly in some
respects, and anti-human in others. There is no benefit to humans in replacing
them. Any benefits are more about specific humans and taking over the world,
as in human's advantage relative to other humans. It is the same as nuclear
weapons. It doesn't help humanity in any way, it just gives certain humans
power over other humans. Also, most of the time, artificial intelligence is no
match for human intelligence. One of his goals was to get the AI to "learn his
tastes", but to me, this just means he will be given poorly selected music, as
what you want to listen to is much more complex then "learning your tastes".

------
Windson
I'm not a fan of facebook or Zuckerberg, but I do appreciate it that he still
keep challenge himself.Maybe some of his projects is not really "AI" stuff.
But it's not easy to keep coding when you are CEO of a big company.

------
mack1001
Mark is at a place that he can really envision and execute a lot of things
relatively easily that is unimaginable for people who are struggling for basic
acceptance and success i.e most of us.

Success usually begets success and more determination.

~~~
pinnbert
Nice honesty. Try having empathy for Mark--maybe that will help?

~~~
mack1001
Trust me I do. I was just trying to highlight just how successful people find
time and resolve to do things which a lot of others cannot even imagine doing
- like learning Mandarin or building Jarvis.

~~~
deadbunny
Or maybe the fact they do find the time for thing like this is why they are
successful?

~~~
mack1001
Cannot argue with that.

------
OJFord
> _If it gets the mood wrong, I can just tell it, for example, "that's not
> light, play something light", and it can both learn the classification for
> that song and adjust immediately._

That sounds awesome.

Although - and I know this isn't a Facebook project - Facebook is very low in
the order of companies I trust with home automation/AI, if he decides to make
Jarvis commercial (probably necessitating a name change!) I think I'm not sure
I'd trust it even if ostensibly "not a Facebook company".

------
automatwon
I'm viewing the article without logging into Facebook. In the comment /
response section where "X, Y, Z, and a billion others liked this", the first
person is 'Priscilla Chan', which I'm guessing is the same person that is
Mark's wife.

I'm curious, is that priority computed based on Mark's circle of friends? Or,
are these values hard-coded / special-cased for the CEO, and potentially other
individuals aswell?

~~~
iso-8859-1
Almost certainly not hard-coded. Anyone of Mark's friends with a lot of
followers and a public profile will probably be famous.

------
snowfly_shubi
Now i don't know how much of this was his "Personal Project" but to me it
seems more or less as facebook's or better yet mark zuckerberg's marketing
strategy to get into the Home Assistance field. I am sure Facebook's Personal
Assistant is not that far.

------
mightybyte
Next up: people start getting into Zuckerberg’s home by holding up a life size
picture of Andrea’s face.

~~~
komali2
I've wanted to test something similar - holding up a giant "Speed Limit:
75MPH" sign when a google car drives by on these little 30mph Mountain View
roads and see what happens.

Probably nothing but I can't shake the comical thought of one of those little
bugs rocketing to turbo mode while the dude inside is frantically slapping the
big red "stop" button.

~~~
tejaswiy
It hopefully uses it's GPS in combination with the Google mapping data for
speed limits along with signage on the road.

Hilarious to think about nonetheless.

~~~
komali2
Probably right, because when I use google maps' navigation mode, it sometimes
has a little speed limit sign in the top right corner.

~~~
user5994461
All GPS have speed limits for all road, and they can beep when you go over the
speed limits.

The speed is part of the map.

------
djhworld
I like that he's made the time for doing stuff like this, I'd imagine his life
is pretty busy.

------
mrcabada
I have this on my home since 2014 and made by me. I offer this solution to
clients in my city with a startup I founded called Hivee
[https://hivee.io](https://hivee.io)

No magic needed.

------
shaan7
"For assistants like Jarvis to be able to control everything in homes for more
people, we need more devices to be connected and the industry needs to develop
common APIs and standards for the devices to talk to each other."

This.

~~~
jodrellblank
Like the industry developed standards and common APIs for people to _send text
to each other_ :)

------
sapeien
It is important to note that Zuckerberg did not claim any original
contributions, nor does he name any of the non-Facebook libraries used,
denying the acknowledgement of others' work which he relied upon. For all we
know, he used some off-the-shelf libraries for face recognition, language
processing, and speech recognition, while having little to no understanding of
how they actually work. This is what a _lamer_ does, not a _hacker_.

There is a huge gap in public perception of the competence of the elite in the
tech industry. People will get the wrong impression that Zuckerberg is an
expert on any of these topics. It reminds me of the antagonist of Ex Machina
who is portrayed as CEO of a giant software company, AI expert, and robotics
engineer, all in one person.

------
lucb1e
> ... reliably delivering push notifications, managing identity and
> permissions for different people, and more. You can learn about the bot
> framework at ...

For a moment I thought I was reading an infomercial.

------
stefs
> I had to reverse engineer APIs for some of these (...)

technically ... is that still legal?

~~~
stefs
to non-answer my own question:
[https://en.wikipedia.org/wiki/Reverse_engineering#United_Sta...](https://en.wikipedia.org/wiki/Reverse_engineering#United_States)

not sure now. maybe, maybe not.

------
nojvek
I did this for my home at a much smaller scale. Using a rasberry pi connected
with a IR emitter I can control light switches in my house. Got the cheap $20
for 5 IR sockets.

With computers getting smaller and faster exponentially, a lot of science
fiction is quite close.

The hard part is how we can handle this new power. Facebook face recognition
can identify me anywhere. Cars can see people/kids in 3d space. Everything I
say can be recorded to speech and be searched .

It doesn't take long for AI to realize humans cause a lot of pain to lesser
species and eat them as food. They also pollute the planet like crazy and a
way to save the planet might be to exterminate some of them.

------
skoocda
I wonder what he used for the speech recognition component- quite certain it's
an API, but is it internal or 3rd party? Maybe something by Wit.ai which is
still unreleased?

------
fergie
"No commercial products I know of do this today, and this seems like a big
opportunity."

 _sound of a million pitches being thrown together_

------
dirkg
"no AI system is good enough to understand conversational speech just yet"

Didn't MS just announce that Cortana has achieved exactly this?

------
perseusprime11
Side question - Are there any good blogs on home automation for folks who want
to start tinkering with this kind of stuff?

~~~
deadbunny
This guy [1] has a fairly decent YouTube channel.

1\.
[http://www.youtube.com/c/bruhautomation1](http://www.youtube.com/c/bruhautomation1)

~~~
perseusprime11
Thanks. He seems interesting.

------
joe563323
Read the post. Want to watch Zuckerberg videos. Forcing to login. Not cool.

------
taneq
Why is no-one talking about the fact that this is completely, 100% NOT cloud
based?

~~~
rthille
It uses FB Messenger for the non-voice API. How is that NOT cloud-based?

~~~
taneq
That's an interface to it, but all of the voice processing, image processing,
logging etc. happens locally and personally sensitive data (like sound
recordings etc.) never leaves the house. At least that's how I read it.

Compare with things like Amazon Echo (and I believe Siri and similar services)
which send audio clips to a cloud service and do all of the processing there.

------
swingbridge
Neat project.

------
trojanh
That's was insane.

------
dimino
If it's literally Zuckerberg building all this, I bet it's full of security
holes (not that he's a bad dev, I wouldn't know, just that one dev can't
reasonably be expected to develop a secure app at a pace even remotely
quickly).

Dunno what that means, just that it would be mildly funny if someone used this
fact to try to get a job at Facebook or something.

~~~
jhartmann
Actually, there are some ways to make this very secure that don't take much
effort that in are common use around facebook. It is quite easy to build
secure thrift services, and to design a thrift api it is just creating a text
file with the necessary idl. You can then access the service from most any
language by generating the right client. I'm sure same thing goes for gRPC and
Stubby on the google side (more familiar with the fbthrift, folly, wangle,
proxygen tech stack myself, but I'm sure its just as easy.)

~~~
dimino
Ah yes, the "security is quite easy" argument.

As long as people like you think security is easy, people like me will
continue to have a job, unfortunately.

~~~
jhartmann
I'm not saying its easy, I'm saying a developer using the tools and processes
available at a large internet corporation building something small need not be
insecure. Basically until we look at the code and the security processes in
place, or do pentesting, we have no clue one way or the other. I'm mostly just
saying we shouldn't discount something like this automatically as insecure. It
might be terrible, it might be very well thought out and have security be one
of the most important considerations that was top of mind when he coded it.

~~~
dimino
I didn't mean to sound like I'm "discounting" this, though I guess I
"discount" everything that hasn't had a cycle of security focused QA, let
alone zero QA at all.

I _bet_ , with no information other than what's supplied, that this system is
insecure. It's not really a knock on Zuckerberg, just a statement based on my
experience. I am totally open to being wrong, I just think we know more than
nothing about hobby projects and their propensity to consider security
secondary, tertiary, or not at all.

Admittedly, this is Mark Zuckerberg, so who knows, it might have been
meticulously coded to be perfect. The guy's a bit of an obsessive, based on
what I've read, so I would also not be surprised to find it's tightly locked
down and very high quality code. I don't the guy, just the general situation.

------
Eduardalalalala
Muito lindaaa

------
soared
-

~~~
switchbak
I remember when he said he'd be embarking on this challenge, and thinking he'd
probably just end up shelving it.

Honestly, even though this stuff has been done before, I'm actually pretty
impressed with how much he's been able to learn and achieve in that time.
Seems like not all of his hacker skills have wasted away, even if some of it
is in PHP :)

I think finding the time to do all this is actually far harder for someone in
his position, making it even more impressive. Not particularly ground
breaking, sure, but not a bad effort.

------
erikb
At first I read "Building Jenkins" and there I was thinking "Well, finally
someone gets how hard that is" (trying it on Linux and not being a Java main)

------
PaulHoule
MOST. AMAZING. SIDE. PROJECT.

------
kelvin0
Zuck, you had the hacker in me at 'hello' :-)

------
cevaris
Saw the title "Build Jarvis" and laughed to myself thinking yeah right.

Then saw it was a Facebook link and was oh...

------
dickbasedregex
Neat but this is underwhelming considering who it is with his resources. I've
seen dozens of setups like this from random nobody developers over the past 10
years or so.

One-off home brew projects like this are neat but until someone creates and
releases a good, extensible foundation to build on top of, meh... And even
then it would probably be terrible because they would try to spin it into some
business effort (which hackers will avoid b/c they don't want to be glued to
someone else's product) instead of an earnest open source effort.

~~~
ocdtrekkie
This was never intended to be a Facebook project, it was a personal project
for him. And arguably, the amount he accomplished is pretty incredible for the
timeframe considering everything else he was doing.

Sure, he has the money to pay people to do it for him, but it was as much a
learning experience as anything else.

I've been working on a system for my home as well, and my goal is to implement
as much code myself as possible, rather than relying on external libraries,
mostly so that I can learn more about how those things work.

------
mgiannopoulos
Honest question on the difficulty of this :

>> In some ways, this challenge was easier than I expected. In fact, my
running challenge (I also set out to run 365 miles in 2016) took more total
time <<

At a casual pace of 10 minutes per mile, that's around 60 hours. Does needing
60 hours to build all that sound plausible to anyone familiar with the task?

Edit: at the end he notes that "I spent about 100 hours building Jarvis", so
Mark is a really slow runner :)

~~~
GuiA
The grumpy snarky HN user in me says that a lot of the _" I deployed a server
to do X"_ and _" I wrote code to do Y"_ in this blog post should actually say
_" A Facebook engineer provided me with direct API hooks to do X and Y"_, but
it's anyone's guess.

~~~
Kudos
API hooks to what though? It doesn't seem like he's integrating with much if
anything from Facebook. TBH, he seems happy enough getting his hands dirty
that I would imagine he purposely did it unaided just for the satisfaction of
it.

~~~
jhartmann
He did reuse things that exist inhouse, like the face recognition piece. To
rebuild a service with Google/Facebook/Microsoft/Amazon level accuracy in that
task, it is definitely more effort than 100 hours by itself.

To use an existing private service, not nearly as much effort.

That being said: Of course he could have learned how to train a model himself,
but even in that case he is using training infrastructure and data that is
available to him that is not public. I highly doubt he had the time to build
those sorts of things himself. But his goal was more talking to his home and
using tools to build an interactive home assistant. Sounds like he was very
successful at doing that. He wasn't out to build the individual AI pieces,
doesn't make it not cool or interesting.

