
Functional core, imperative shell (2012) - fagnerbrack
https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell
======
Twisol
This basic idea has deeply infected how I approach software architecture and
design. When someone asks me why I wrote something a particular way, it's
usually because I'm trying to keep the imperative/environmental concerns
separate from the functional/algorithmic concerns.

I end up recommending this video at least once every year.

(EDIT: Another one from Gary that pairs well is "Boundaries":
[https://www.destroyallsoftware.com/talks/boundaries](https://www.destroyallsoftware.com/talks/boundaries)
. The core idea is that _values_ \-- data, not code -- should form the
boundaries between distinct systems.)

~~~
fpoling
The actor model is prized in the talk. Yet actor model is not functional. The
problem is the message queue of each actor. It is an imperative storage and
one can get all nasty problems of imperative code from apparently functional
code patterns.

~~~
msangi
What would a functional approach be?

~~~
fpoling
It depends on what is the goal. To take advantage of multicore for
computations one can use explicitly parallel functional algorithms. To improve
response under load one can do load balancing within application and schedule
at the very top (shell level) several independent functional pipelines.

------
bcbrown
One of the things I really like about this approach is how well it lends
itself to separating what needs unit testing from what is unsuitable to unit
testing and should instead be validated by higher-level end-to-end/functional
testing.

I think it's also related to the concept of building a DSL in which to
implement business requirements. Once you have the right 'primitives' you can
then combine them in useful ways that are easy to verify (by reading the code)
that the implementation matches the requirements.

~~~
MichaelMoser123
How do you deal with performance problems? I mean to add an element to a list
you need to create a copy of the list and then add the new element, all in
order to remain functional. Isn't that very expensive?

~~~
gary_bernhardt
You only have to copy if your array is implemented as a simple linear sequence
of memory addresses. More advanced implementations don't have to copy
everything. E.g., Clojure's vectors (its array equivalent) are effectively
O(1) for the common array operations that are O(1) on naive arrays, like
indexing and insertion. But Clojure vectors are still purely functional. (The
actual time for some of those ops is O(log32(n)), but log32(1,000,000) = 4, so
it's effectively O(1).)

The term for this is "persistent data structures", usually implemented via
trees, where replacing an object in a vector is implemented by building a new
tree, reusing all of the old nodes except the ones that appear in the path
from the root to the replaced node. That's why Clojure's Vector is log32; it's
a 32 b-tree. (I'm writing this from memory and have little Clojure experience,
but I'm pretty sure I have it right.)

Many languages have implementations now, but most aren't as fast as Clojure's.
E.g., there's immutable.js: [http://facebook.github.io/immutable-
js/](http://facebook.github.io/immutable-js/)

~~~
twtw
For people who are interested in these things, I'd highly recommend Chris
Okasaki's thesis/book on the topic.

I was not familiar at all with this stuff when I read through it the first
time, so it was a tad mind-bending and I probably understood ~10% of it, but
it was certainly educational.

[http://www.cs.cmu.edu/~rwh/theses/okasaki.pdf](http://www.cs.cmu.edu/~rwh/theses/okasaki.pdf)

------
T-R
This concept is a big part of what the "big deal" around monads is - using
monads to model effectful code conveniently puts the information of "this
should be shell code" into the type, in a way that ensures that code that
calls it also gets annotated as "shell" code. Monads are of course also a much
more broadly applicable abstraction, but their application to effectful code,
enforcing this design, is usually the first and most apparent place people run
into them in the ML family of languages.

~~~
pankajdoharey
Monads is taking it too far. Mutation is a reality, the correct approach is
disciplined mutation. Shoving mutation into convenient boxes and convincing
yourself to never look inside it does not mean mutation does not exist. The
best approach is taken by scheme, and more specifically clojure to have a
disciplined and practical approach. Mathematical purity of programs is a myth
propagated by Type theorists dont buy into it.

~~~
adimitrov
I think you misunderstood Monads, or the role type theory has to play in
modern programming.

Large projects inevitably benefit from static guarantees enforced
automatically by your environment. That can be a 3rd party static code
analysis tool or the compiler. Even just a linter will improve code quality
and thus developer happiness and productivity.[ _] Having your compiler_
enforce* the functional core/imperative shell, and exposing your business
logic _only_ through functional components is what makes a strongly typed
language of the ML family stand out over, say Clojure.

Mutating state is no problem in a strongly typed functional language. In
Haskell, just put your computation in an ST Monad. You can even still expose a
_functional_ signature that doesn't leak the ST monad if your algorithm is
faster with mutation.

[*] Overall. Some people will probably be unhappier, because they have to
follow "arbitrary" rules now, but those would usually have been the worst
offenders.

~~~
pankajdoharey
There have been many articles on this topic. There isnt any evidence to
suggest that static guarantees makes your code better. Ofcourse what does make
your code better is immutability. But complete immutability isnt practical and
even Haskell people understand that but they continue to pretend that programs
are about mathematical purity. If that isnt enough claiming static typing
removes the need for testing is complete bunk.

~~~
pwm
Have you written programs in Haskell/F#/Ocaml? Static guarantees, especially
of expressive type systems, absolutely make your code better and their
benefits compound as your system gets bigger. The type checker acts as a
guardian of the soundness of your whole domain. And yes, expressive static
typing removes the need for a whole class of tests, namely the ones that you'd
have to write in other languages if you are disciplined enough to care about
the soundness of your domain model. I personally loath writing these type of
tests but I do when I can't use the power of say Haskell because I care.

Immutability also makes you code better but it's an orthogonal concern and
utilising both is a smart move.

~~~
pankajdoharey
If it were true, we would all be writing in c++ and there would never be a
Stack overflow or Null pointer exception. But thats isnt true.

~~~
pwm
Any language that has null, including C++, does not fall into the category of
languages with expressive type systems. As soon as you have proper sum types
the null issue goes away + the whole big world of working with ADTs open up.

------
abenedic
So, I am a person who wishes there was a proper article I could read for this.
I don't know how much this matters to people, but I can understand written
English far far better than I can understand spoken English. I know auto-
translate works for some. I am trying with applet and get nothing. I like core
idea. I feel often people outside English land get second class experience.

~~~
TeMPOraL
People _inside_ English-land get second class experience too. Audio and video
are inferior tools for this kind of content, in the same way a linked list is
inferior to a vector if you need random access.

In this particular case, I'm annoyed too. I've learned (what I think is
exactly) this concept from other sources, and I've been recently linked to
this video a couple times. What I would love to do is to quickly diff my
existing knowledge with contents of the talk, but I can't do that because it's
in a video format. I've been putting off watching it for couple weeks now.

~~~
mercer
I'd say this one (as well as other classics like 'Simple Made Easy') is worth
it.

That said, I'd also prefer an article that tells me the same thing.

------
lackbeard
Does anyone know of an example of a non-trivial codebase written in this
style?

How do you do this cleanly when, e.g., you need to make a network call and
then based on what the result returned to you is, either do something with a
local database, make a different network call, or return a result to your
user. Also, error handling...

It seems to me like monads must be the logical conclusion to this style of
programming, or else you wind up with a mess (or just abandoning this
technique.)

~~~
justinpombrio
Literally anything written in Haskell. When you write stateful code in Haskell
(such as code that reads from a database, makes a network call, or does IO, to
use your examples), that code will be wrapped in a Monad. Thus the type
signature of the code shows that it's stateful code, and Haskell's type
checker will ensure that any code that calls it also has a stateful type. For
example, if a function takes in an `Int` and returns an `Int`, and along the
way may _directly or indirectly_ perform IO, that function will have type `Int
-> IO Int`.

Now of course, you don't _have_ to cleanly separate a functional core from a
stateful shell. But if you don't, all of your code is going to end up wrapped
in nested Monads declaring all of the ways that it's inadvertently stateful,
and that's a very painful way to program. So Haskell pushes you strongly
towards having a functional core and stateful shell.

For projects written in Haskell, the wiki has a long list:
[https://wiki.haskell.org/Haskell_in_industry](https://wiki.haskell.org/Haskell_in_industry)

~~~
smadge
I agree, but I think it still takes some coding self-discipline to write code
with a functional core and stateful shell in Haskell. There’s little stopping
you from having every return type wrapped in the IO monad. It’s not any more
unnatural to do that than it is to code in any imperative language.

~~~
justinpombrio
> It’s not any more unnatural to do that than it is to code in any imperative
> language.

Yeah, I guess the `do` notation makes it pretty painless. If you start
_mixing_ monads, though, things get hairy quickly.

~~~
waluigi
Monad Transformers aren't _that_ bad, the MTL style of doing things makes it
all pretty painless.

It also provides a huge opportunity for testing. At a very high level, you
describe all of your effects as a series of embedded, compostable DSLs that
you define interpreters for. The awesome part is that you can switch out the
interpreters at will, so you can, for example, replace something that handles
network requests with something that returns dummy data almost effortlessly.

~~~
justinpombrio
I think we have different pain thresholds. I would consider this very painful:

[http://hackage.haskell.org/package/transformers-0.5.5.0/docs...](http://hackage.haskell.org/package/transformers-0.5.5.0/docs/Control-
Monad-Trans-Class.html#g:7)

> It also provides a huge opportunity for testing. [...]

That's a neat point, thanks for bringing it up.

------
dwohnitmok
Although it means something different and is not the the actual semantic
opposite, I've found the syntactically opposite phrase "Functional shell,
imperative core" also useful, which the speaker alludes to when he talks about
local variable rebinding that is invisible to the outside.

That is locally scoped, mutation-heavy, imperative code is fine as long as you
can wrap it in a deterministic, immutable interface for general use. This is
the premise behind things like Haskell's ST type. More generally it's the
usual way FP languages try to recover competitive performance.

~~~
taeric
It is also how your computer does basic addition. It isn't like there is such
a thing as immutable numbers when it gets to the silicon. Look at how a barrel
shifter is implemented someday, or a comparator.

~~~
branislav
It's also how something like React.js works. Wrapping stateful DOM
manipulation in a functional shell, made performant thanks to the Virtual DOM
diffing algorithm.

------
squirrelicus
Often people will ask how to follow these principles in the world of I/O. The
answer is incredibly simple: with values.

Return a DTO that encapsulates all the business information about the result
state of the I/O call. Do not throw exceptions. Log them if you like, but you
must return all the data necessary for the business logic to react to failure
conditions, encoded in your own use case specific structure.

If you don't care about whether you failed due to timeouts or refused
connections or query parse failures what have you, don't return that data,
just return a hasError kind of property on the return structure (bonus points
if your language supports discriminated unions, but this is not necessary). If
the parent logic needs to react to timeouts and failures to connect
differently, then catch those exceptions or state separately and return
didTimeout or cantConnect flags separately.

Values values values

~~~
the_gipsy
I would prefer a ‘Result<ValueType, ErrorType>’ over stuffing weird
‘hasError’, ‘didTimeout’, etc flags everywhere.

It’s much safer to type-check at compile time that you have discriminated
between result and error, than to hope that you are ‘if’-checking some flags
at the right time at runtime.

~~~
squirrelicus
That is another way to accomplish the same goal. This trade-off is more
aesthetic than architectural and is just a matter of language support

------
ryanmarsh
Great to see this. I ended up writing this way after watching Brian Will's
_Why Object Oriented Programming is Bad_ [0]. Near the end he states we should
be writing "procedural code" but that we should favor "pure functions". What I
took away from this was that at a high level I should be able to reason about
a problem procedurally but the solution should be composed of mostly pure
functions.

0:
[https://www.youtube.com/watch?v=QM1iUe6IofM](https://www.youtube.com/watch?v=QM1iUe6IofM)

------
squirrelicus
It's difficult to overstate the importance of the principles taught in this
talk. Almost as difficult as it is to describe why they are so important. The
fundamentals of quality software are found here, not in some Martin Fowler
dissertation on DDD.

Edit: correction from commenter, thanks!

~~~
stevebmark
I think the general concept of using immutability, queues, and isolating side
effects, is lost and muddied by using Ruby, and it's a misuse of the term
"functional."

------
danidiaz
In the Idris language, one best practice is to relegate to the outer shell not
only IO effects, but also "partiality"—in the sense of computations that might
enter an unproductive infinite loop.

(In Haskell, one way of getting into unproductive infinite loops is by
mistakenly asking "parse elements until the first failure" to a parser
combinator that can always succeed without consuming input.)

The ideal is that the core of the application should be both pure and total,
purity and totality being tracked by the type system. In fact, one can often
relegate partiality to a single function of the outer shell.

------
savethefuture
This is one of my favorite videos. I highly recommend DAS videos, definitely
worth the subscription.

~~~
mercer
Seconded! This one and a few others gave me 'aha moments' that I don't get
much without hunting for the right videos (recommendations much appreciated!
In the vein of Simple Made Easy).

~~~
squirrelicus
Simple Made Easy should be mandatory watching for your developer's license.

Seriously though, this talk, Boundaries, and Simple Made Easy are the trifecta
that forms the foundation of the software I design.

------
ceronman
This is the same principle of the Clean Architecture. I like how Brandon
Rhodes explain how to apply it in Python:

[https://www.youtube.com/watch?v=DJtef410XaM](https://www.youtube.com/watch?v=DJtef410XaM)

------
jypepin
And I'd strongly recommend paying for DAS and looking at every.single.video in
there. Gary is brilliant and each video explains stuff in a very simple,
enjoyable way.

Reminds me a bit of Ryan Bate's ruby/rails videos.

------
Flenser
After watching this and his Boundaries talk, and then reading the Out of the
Tar Pit paper, I started watching Pete Hunt's first talks on why they built
React and had an epiphany why React was going to end up being the most popular
front end framework.

------
fpoling
One does not need a pure functional core to have benefits of such design. The
key idea is that the core code should not change or even read the global
state. It should only be allowed to access or change what is explicitly passed
to it.

For example, to avoid excessive memory consumption to sum matrixes one wants
to code it like A += B, not like C = A + B. Yet one still benefits from all
the testing, design etc. benefits of pure functional code. At the end one call
always get a functional version just by turning A += B into C = A; C += B.

------
codetrotter
While the purpose of this screencast is to explain a principle, not to focus
on the particular piece of software that was implemented, I am left wondering
about a couple of things;

The timeline as he explains it, is updated by creating a new timeline instance
which consists of previous timeline + new tweets. It seems to me then, that
his client does not remove from the timeline tweets that were deleted by its
author subsequently to having been downloaded.

To some it might be a feature to capture as many tweets as possible, but at
the same time, if your view of the timeline includes deleted tweets then you
might find yourself trying to reply to a deleted tweet, and then you would get
an error response from Twitter when you try to post your reply. (Though I
don’t know if his client also does posting, or if it’s a read-only client.)

Furthermore, what about tweets that are edited by their author subsequently to
having been downloaded? Seems that you would not see those edits.

~~~
gary_bernhardt
Tweets aren't editable so no need to handle that. Deleted tweets can be
honored by updating the timeline to be `old_timeline - deleted_tweets +
new_tweets`.

------
ridiculous_fish
How do you implement progress reporting and cancellation in this model?

~~~
mvc
Make it an explicit part of the data model.

~~~
ridiculous_fish
A quicksort that supported progress reporting and cancellation might have a
function parameter that it calls periodically to report progress, with the
function returning a bool indicating cancellation. But this is using code, not
data.

How might this be implemented within the data model?

~~~
yen223
How do you measure progress in a quicksort?

~~~
ridiculous_fish
Glad you asked. The progress should be a bound on the worst case time.

If you have N elements, then the initial worst case time is N^2. Say after the
first partition, you are left with pieces N/3 and 2N/3; the worst case time is
now (N/3)^2 + (2N/3)^2. Your progress after the first partition is the
difference between the original worst case and the new worst case.

This can make for uneven progress advancement but it’s monotonic: the progress
bar will never go backwards.

------
mlthoughts2018
I like to imagine that the huge planet-sized villain from the film The Fifth
Element is the physical manifestation of giving one’s self over fully to the
programming style of indisciminantly mixing side-effectful and I/O operations
into arbitrary code whenever it’s locally convenient to do so.

------
blt
I'm struggling to apply this model to code for machine learning experiments.
The functional core takes an extremely long time to compute, e.g. "train the
model". Thus, you want to save lots of intermediate values to disk so you can
reload them over and over while you are experimenting the downstream part of
the program.

I end up with a lot of code writing values to the disk, which is currently
mixed in with the computations. I'm wondering if there's some way to automate
this "save intermediate values to the disk" so I can write the code in a more
functional style without having to constantly go in and out of the imperative
shell portion.

The process of ML experimentation is extremely painful compared to normal
software development where nothing really takes that long to compute.

~~~
waluigi
The way I would approach this is to write what essentially amounts to a
declarative specification of what computation needs to be run, and then define
an interpreter that handles the caching of intermediate values.

The "Embedded DSL + Interpreter" pattern is incredibly powerful, and it's nice
to see it catching on more.

------
jcyw
The idea of functional core is cool. I find domain-driven design a better
guide for implementation. In a nut shell, there are three types of objects:
value, entity, and service. Both value and entity are functional and stateless
objects. A service object maps between stateless objects and may cross the
domain boundary (such as network, IO, etc). When I do this in java, a service
object would have interfaces returning CompletableFuture of Value or Entity
objects.

InfoQ has a very nice summary of DDD book:
[https://www.infoq.com/minibooks/download/domain-driven-
desig...](https://www.infoq.com/minibooks/download/domain-driven-design-
quickly)

------
Apaec
I learned this through Haskell, it enforces the "functional core, imperative
shell" approach through the IO monad. Now I find myself repeating the same
pattern in other languages(js, java).

------
jcyw
This is the github repo featured in the video:
[https://github.com/bf4/toot](https://github.com/bf4/toot)

------
ww520
It's similar to what usually do. Use functional style in inner, smaller,
library level code. At the global level, it's unavoidable to deal with state
when interfacing with outside. State changes just need to be well encapsulated
and managed.

------
GolDDranks
This is the pattern I'm striving to code in, but when developing with a team,
I keep struggling convincing other people (with OOP, mutability-happy
mindsets) to do so...

------
grzm
(2012)

------
stevebmark
"Functional core" littered with classes, attr_readers, data that's not data
(Curosr.new) and += imperative loops. Yikes.

~~~
gary_bernhardt
The classes serve to close some functions over variables for convenience; the
attr_readers serve as destructuring functions for convenience; the += replaces
a recursive function for convenience and familiarity. The function with the +=
doesn't mutate any value and it remains pure from all callers' perspectives.

You're complaining about syntax, but this screencast is about semantics.

~~~
stevebmark
No, something with "functional" in the title that doesn't use functions isn't
related to syntax.

~~~
gary_bernhardt
What is a function?

