
Fighting spam with Haskell - vamega
https://code.facebook.com/posts/745068642270222/fighting-spam-with-haskell/
======
LukeHoersten
For those not involved in the Haskell community, Simon Marlow worked full time
on the GHC compiler and specifically run-time system for many years. Along
with Simon Peyton-Jones, he's huge in the Haskell world. Marlow also wrote the
very excellent "Parallel and Concurrent Programming in Haskell" book.

Facebook also employs Bryan O'Sullivan, an epic Haskell library writer (Aeson,
Attoparsec, Text, Vector, and on and on
[http://hackage.haskell.org/user/BryanOSullivan](http://hackage.haskell.org/user/BryanOSullivan)).
Bryan also co-authored the "Real World Haskell" book.

So Facebook has hired two prolific Haskellers and probably others I don't know
about.

~~~
Periodic
It really shows what an investment they are willing to make in the language.
They aren't just training up internal engineers, they're bringing in some of
the best in the community.

I'm hugely excited that Facebook is making this investment and giving some of
the developments back to the community. There are many smaller companies that
would be interested in Haskell but don't have the resources or expertise to
tackle some of these complicated problems.

Facebook is blazing the trail for production Haskell and the rest of us can
follow.

~~~
LukeHoersten
One thing's for sure, Facebook staffing did an amazing job hiring Haskell devs
;-)

~~~
jeremyjh
Bryan was acquhired and I think he hired Simon.

------
dasmoth
Damn you FaceBook.

I dislike the underlying premise, the adverts, and (especially) the "real
names" policy.

But... between great bits of Open Source like React, cool infrastructure
projects like this, and a technical culture which seems a whole lot more open
than many other big companies, it's getting kind-of hard to go on hating. Walk
back a bit from the obsession with open plan offices, and I might just cave...

~~~
Bahamut
Their commitment to open source is truly amazing - I am currently in the
process of interviewing with them, and am increasingly in awe of the company.

Surprisingly, their massive open office is the quietest office I have been in.
It is a really nice workplace.

~~~
xixixao
This. From my experience at Facebook and Google, FB's open space is way more
quiet than cubicles at Google. People in the other cubicle feel like the
others don't hear them (they do).

------
HugoDaniel
This is an amazing effort, implementing ApplicativeDo and using Haxl for
automatic batching and concurrency, doing code hot-swap in a compiled
language, developing per-request automatic memoization, finding a aeson
performance bug, translating C++ from/to haskell to do partial marshalling of
data, implementing allocation limits to ghc threads, creating a shared c++
library to bundle the c++ dependencies in ghci for interactive coding, killing
two ghc bugs, and more... and in the end producing a reliable scalable
solution.

ouch!

~~~
spopejoy
It's definitely not just a "We're really happy with Haskell here at FB" post.
This probably explains the trollish comments elsewhere to the tune of "Haskell
without Simon+bos /= scalable".

It is a little intimidating, though. I think the perception is "Haskell is
hard enough, AND I have to be a C++/GHC internals expert too?" The hard truth
is you have to be this level of expert at _something_ to achieve robustness at
industrial scale.

Simon's comment in this thread hints at some of the power of Haskell alone:

 _The "automatic" bit is that we insert the code that consults the map so the
programmer doesn't have to write it. The map itself is already invisible,
because it's inside the monad. So the overall effect is a form of automatic
memoization._[1]

This pretty much sums up appdev in Haskell. Engineer (or employ) amazing
abstractions with no funny business (hello Spring!) and no loss in safety or
expressiveness.

I'm really happy this team exists, this is the kind of work that will take
Haskell into the mainstream. Kudos!

[1]
[https://news.ycombinator.com/item?id=9787523](https://news.ycombinator.com/item?id=9787523)

------
seddona
Thanks for the overview Simon, great to hear about the use of Haskell at
scale. At CircuitHub we use Haskell to build our entire web app, Haskell is
great for most tasks these days.

~~~
Lewisham
Did you use a particular framework? I've been greatly enjoying my 4th attempt
at learning Haskell (I think only now am I seasoned enough to really _get it_
), and my general proof that I know something and can use it usefully is to
develop a web app (as all apps are now anyways). However, I didn't find
anything which looked like it had majority community buy-in unlike
Rails/Sinatra/Flask et al.

~~~
wz1000
Yesod is the most actively developed and popular web framework for Haskell.

~~~
haxeny1
For my side project, I use yesod/websockets for the server and haxe for the
client. I feel ok with that choice instead using hpaste. Though eventually I
would like to see a complete haskell stack. @Lewissham: about the fourth
attempt, it just so happens that this was my third attempt at getting
something that was right for me as well as maintaining the typesafety. I do
realize that code is still not as seamless as I would like. It still lets me
understand the overall roadmap that the code needs to take to be
usable/maintainable etc.

~~~
psibi
Can you expand upon what you mean by complete Haskell stack ? What do you feel
are currently missing from Yesod ?

~~~
haxeny1
Yesod is awesome I can't see myself going back to less typesafe database
schema tools. The choice I made seemed like a safer path, perhaps not ideal.

The application that I am trying to put together is a realtime websockets
application. Therefore, I am using yesod/websockets for the server. For the
client, I chose haxe simply because of the targets available making it simpler
to write plugins.

On one of my irc chats or somewhere, I did hear good things about haste,
though it is primarily a javascript library, therefore I plan to evaluate it
later.

About the client: it is a single page app to prevent any page refreshes and I
am a bit sold on the frp (using promHx). I guess I should have elaborated a
bit more about this in my earlier comment, because my choices are not
indicative of anything amiss in yesod.

~~~
e12e
Have you looked at Elm[1]? Is that "too much web" and too far from Haskell for
you?

[https://github.com/elm-lang/elm-platform#elm-
platform](https://github.com/elm-lang/elm-platform#elm-platform)

~~~
haxeny1
Elm was my first choice for a different side project. I went away because then
elm did not have, or it was my lack of knowledge,support for crud widgets. I
miss the elm signal approach in haxe for one thing. Here is the link to my
repo so things are a bit concrete [https://github.com/asm-products/ccar-
websockets.git](https://github.com/asm-products/ccar-websockets.git) .
Warning: this is still work in progress and there are way too many issues
before I can even release this. The elm repo is here:
[https://github.com/dservgun/elm_projects](https://github.com/dservgun/elm_projects)
.

Edit: added some repos to set a context. The original comment about 4th
attempt hit home as the latest repo is my 4th(I am going to quickly lose
count) attempt to achieve at a model that might work/scale. There are still
some issues that I would like to solve: whether to use code generators or not.
Code gens will save me some typing, but I found myself tweaking printfs or
equivalents to fit it into a narrow model.

Edit II: Elm, iirc, is being supported by prezi that also happens to be using
haxe quite actively. So I felt a bit more reassured about using haxe.

------
j_m_b
|We implemented automatic memoization of top-level computations using a
source-to-source translator. This is particularly beneficial in our use-case
where multiple policies can refer to the same shared value, and we want to
compute it only once. Note, this is per-request memoization rather than global
memoization, which lazy evaluation already provides.

I would like to know more about this. What is a request exactly? An API call?
If so, when an existing policy is changed, do the memoization tables have to
change as well? How are the memoization tables shared? If this is running on a
cluster, I would imagine that lookups in a memoization table could be a
bottleneck to performance.

~~~
simonmar
For example, let's say that one of the things you want to compute is the
number of friends of the current user. This value is used all over the
codebase, but it only makes sense in the context of the current request
(because every request has a different idea of "the current user"). So this is
a memoized value, even though in the language it looks like a top-level
expression.

Memoization only stores results during a request. It starts empty at the
beginning of the request and is discarded at the end, and it is not shared
with any other requests. It's just a map that's passed around (inside the
monad) during a request.

~~~
j_m_b
Thanks for the response. Just trying to expand my brain here =), so I have a
followup question.

I always thought of memoization as storing the parameters to, and result of, a
function call in a memotable. Doing some quick research, I came across this
definition of memoization from NIST that sounds more general "Save (memoize) a
computed answer for possible later reuse, rather than recomputing the answer."
What I understand from what you said is that when a request is processed, it
produces a map that is passed around for the duration of the request.

Something like:

Request -> (some processes) -> memoized map -> Policy Filters

How is the memoized map reused?

~~~
simonmar
The memo table (map) is a bit of state that is maintained throughout the
request's lifetime. When we compute a memoized value, it is inserted into the
map, and if we need the value again we can just grab it from the map instead
of recomputing it.

The "automatic" bit is that we insert the code that consults the map so the
programmer doesn't have to write it. The map itself is already invisible,
because it's inside the monad. So the overall effect is a form of automatic
memoization.

------
fokz
I am under the impression that a large part of engineering effort at
established companies go into porting existing components to a deemed to be
more appropriate language for that task.

Is it plain impossible to pick the best fit language without implementing a
solution in the first place and fleshing out the requirements and challenges
that specific to the problem space? Or do the problems evolve fast enough that
no matter how well you design the system, it will need to be deprecated once
in a few years?

~~~
liviu-
Could be simply that time uncovers better solutions to the same problems,
which triggers the need of porting.

~~~
taeric
The problem with this is just how often the story seems to involve porting of
solutions. Not necessarily refinement or advancement, but pure porting. With
lots of debate/fighting on the syntactic properties of the solutions.

------
wz1000
How does the hot-swapping work? The only way I had seen of making this happen
is what xmonad does. I'm assuming this is radically different from that.

~~~
agentgt
I was wondering that myself. Maybe they will release code examples on how that
works. I'm only slightly familiar with the Haskell ecosystem but I have also
wondered if it has a plugin like system (e.g. OSGI for Java).

~~~
xnyhps
I've had some mild success with [https://hackage.haskell.org/package/dynamic-
loader](https://hackage.haskell.org/package/dynamic-loader) (which is
basically just a Haskell wrapper around the C-functions in the RTS), but
compared to writing normal Haskell, this is playing with fire as you loose all
type-checking guarantees.

------
VeejayRampay
And two GHC bugs fixed along the way. Well done everyone.

------
reagency
This is impressive, and in line with Haskell's philosophy to "avoid success at
all costs".

As a mere mortal programmer, who knows a little Haskell, my takeaway is that
if you want to run Haskell in at web scale for a a large userbase, you need
the language's primarily compiler author to help build the application and to
modify the Haskell compiler to make it performant. And you also need your team
led by a 20-year-veteran Haskell expert who is one of the language's handful
of luminaries whonwrotr a plurality of its main libraries. What are the rest
of us to do, who aren't at Facebook?

~~~
wz1000
> Haskell's philosophy to "avoid success at all costs".

How? I think you are misunderstanding the philosophy. Like SPJ once explained,
its 'avoid (success at all costs)' and not '(avoid success) at all costs'. It
basically means, 'don't compromise on your principles in order to achieve
success'.

~~~
robertk
avoid $ success at all costs

------
vezzy-fnord
_Loading and unloading code currently uses GHC 's built-in runtime linker,
although in principle, we could use the system dynamic linker._

That they could use the system dynamic linker makes me think they're using
some form of relatively basic dlopen/dlsym/call-method procedure, or something
along those lines. That's fine, though the use of "hotswapping" evokes the
image of some more elaborate DSU mechanism.

------
themeekforgotpw
Does this system also detect propaganda operations?

~~~
jjawssd
No, it generates them /sarcasm

~~~
themeekforgotpw
The question was serious. Malicious actions could certainly include propaganda
operations and indeed there has been a growing investment across social
networks to block content specific to certain ideas considered dangerous,
disinformation and misinformation.

My guess is actually that, yes, this system has rules (and gets live feeds of
them) to block certain political content associated with influence operations.
But I was hoping to get a more serious, direct and charitable answer from
someone who works on the system.

~~~
jjawssd
I'm also waiting for more commentary to this original question. I would wager
that no it is not possible to detect propaganda effectively. Effective
propaganda notably presents only one side to the story and uses various other
tactics to send that message home but detecting one-sided but factual
statements is impossible with today's natural language processing tools, as
they struggle enough with being able to make sense of anything.

On another note, what happens if Facebook is able to detect "propaganda
content" but it only silences propaganda it doesn't like? This is what I
originally meant.

It's similar to how propaganda works in US media. It's not the news stories
themselves, it's all about the news networks which are told from above which
stories to run. I can see Facebook operating in a similar fashion today or in
the future.

~~~
themeekforgotpw
Definitely not all strategic communications could be detected and blocked.
However some research has been tasked on the issue. Take "Antisocial Behavior
in Online Discussion Communities"
([http://arxiv.org/pdf/1504.00680v1.pdf](http://arxiv.org/pdf/1504.00680v1.pdf)).
This paper, though it terms it 'antisocial', studies online influence
operations and is mentioned on NATO's propaganda ("strategic communication")
centre for excellence Facebook page
([https://www.facebook.com/StratComCOE](https://www.facebook.com/StratComCOE)).

The point of having a rule set that can be pushed to like the system described
here is that it can be fed information from a centralized task force. Indeed,
these systems are used across industries with patterns and rules to detect
malware, hacking, and a host of other activity (using formats such as STIX or
TAXI).

It's unlikely that Facebook itself would be trying to determine whether a post
to some content is propaganda - it's more likely that their automated system
will apply the rules it receives from information defense agencies.

As per your other note, the selective censorship of information is absolutely
a form a propaganda of its own. Facebook (and the US government) don't have
the best history here and have 100% certifiably censored information to
prevent effective organization of (for example) May Day protests. I'm not
going to go speak about that at length here - hoping to get some technical
clarity from those involved.

------
edwinnathaniel
I learned a few things from this post outside the usual "technical"
explanations:

1\. They have CORE Haskell contributors on their payroll to deliver this type
of project (what this mean is that no... Haskell isn't any better than other
language, it's just that they have people who know Haskell very very deep to
the compiler level...)

2\. In-house custom language eventually does not scale (the EOL is much much
much shorter than other programming languages), plan for that :)

~~~
Scarbutt
What you should ask yourself then is, why did they hire core Haskell
contributors?

~~~
outworlder
Let's see. because they can afford to? Also perhaps because they are Facebook,
not your average company?

Who should they hire instead? Given the opportunity and the means, wouldn't
you hire someone of that caliber for your preferred language/environment?

------
saosebastiao
Anybody know what sort of policy resolution algorithms are used? Is this based
on Rete, or home grown?

~~~
mark_l_watson
I don't know if Rete is appropriate. Rete optimizes for a large number of
rules and a small amount of data. I used to hack Rete based OPS5 a lot,
including adding support for multiple data worlds. If you like Common Lisp,
then the OPS5 code base is fun to hack.

------
covi
In the throughput graph, why does Haxl perform worse in the 3 most common
request types?

~~~
Periodic
I don't think the graph says which request types are more or less common. The
ordering could be arbitrary.

It would be interesting to know why some are slower. Perhaps they require very
little processing and so the time becomes dominated by FFI transformations? It
would be nice to know!

~~~
JonCoens
One of the cases we found while performance profiling was a tradeoff between
memory usage and computation completion. On certain requests FXL would use too
much memory and be halted by the equivalent to AllocationLimits, while Haxl
would happily plow on using less memory and complete the request. When looking
at many of those requests in aggregate, the end result would have more
successfully completed requests but with longer response times mixed in.
Completing more requests was seen as a win over the apparent decrease in
throughput.

------
nlake44
Facebook censors so much and swaps out links. I can't even post a link to
[https://scientificamerica.com](https://scientificamerica.com). Their SSL cert
is weak and should not be trusted. It is swapped out for phishing scams.

~~~
reagency
That URL is a typosquatting domain-parking site, so....

------
yellowapple
> Fighting spam

> (facebook.com)

Ha.

------
giancarlostoro
It seems like Facebook seems to be the one company developing a lot of the
very interesting and useful tools for developers.

~~~
pekk
That is unnecessarily negative toward the efforts of literally everyone else.

~~~
giancarlostoro
I wrote it poorly my apologies. I guess I could see how my messages is
misinterpreted.

