
Functional Programming in a Dysfunctional World - GarethX
https://underthehood.myob.com/functional-programming-in-a-dysfunctional-world/
======
agentultra
The idea that "memory doesn't matter" and "we're mostly waiting on the network
anyway" seems like red herring.

It's not a "micro-optimization" to think of your data first. A program is a
sequence of transformations on a stream of data. Figure out what your data is
and how you want to transform it and your program will become evident.

FP is a fine tool for programmers to have but you still have to think about
the data regardless. If you don't realize how the allocator works in your VM
you might end up fragmenting your pools and over-allocating virtual memory for
a simple process. For some applications that might matter.

~~~
rdtsc
Memory didn't matter in the past. At some point caches weren't important, CPU
were not running as fast compared to main memory. Then CPUs were getting much
faster and caches started to become very important.

Then at some point networking was very slow and it didn't matter if you wrote
your code in a scripting language, if all you did was wait for network
packets. That was around Pentium 4 days or so. CPU speed was doubling quickly
and 1Gpbs cards and switches were still kind of fancy and expensive.

But then it all kind of changed. Caches are important now. You thrash your
cache around and you can take a serious performance hit. Even kernel code
can't keep up with network wire speed at 10G range.

Long story short, a lot of performance heuristics and folk knowledge about it
has be re-evaluated periodically.

~~~
taeric
At what point in the past did memory not matter?

~~~
rdtsc
Memory speed didn't matter back during early 486-586 days. You just didn't
think about cache misses as much because the speed disparity wasn't that
great.

~~~
taeric
I question whether memory was literally not a concern, though. Were you as
likely to outrun memory by the CPU, no? Did you still try and minimize the
amount of data that went through memory for overall speed? I would think so.

And this is ignoring the fact that hard drives were still ridiculously slow.
So, really, the concern has always been that there are large chunks of memory
that are not fast. Over time, "not fast" has changed in definition. But
practical considerations have remained that keeping a small data set will be
faster than a larger one.

------
rdtsc
> Did you ever attempt to get to the bottom of an intermittent error that was
> driving you insane, only to discover much later that a private variable
> wasn’t synchronized properly? Have you ever spent hours stepping through the
> lifecycle of an object, going through many different components spread out
> across the codebase and wished that you had spent all that time solving new
> problems instead?

I see 3 issues here.

1) Most people haven't dealt with those issues enough. There are complex
programs but there are also many simple program as well. And unless one deals
with hard to debug concurrency / pointer / mutable state errors they will not
appreciate immutability. They might pay lip service to it, because it is cool,
but they will not appreciate it enough to start rewriting their codebase in
it. That is a bit like fault tolerance, unless they have been woken up at 3am
because their main larger server process segfaulted and had to debug, they
will not think about isolated units of failure, about supervision, durable
peristent snapshots/checkpoints, monitoring etc. They of course will say that
"fault tollerance is good" but it is good in an abstract, general way.

2) Those that dealt with these issue might accept the state (pun indended) of
affairs and never realize there is another way of doing things. I mean they
just accept that you have to have locks and mutable state and mutexes and
dangling pointers and then spend months debugging hard to track concurrency
errors at 3am. That is just how software development works and it is as good
as it gets.

3) People realize there is something better, they know about, read about,
played with it. But they don't have enough energy, political power, or time do
fix it.

~~~
taeric
I see another issue. Have I ever gotten to the bottom of an intermittent error
only to determine it is a synchronization error? Yes.

The problem is I have also gotten down to the bottom of a problem and found it
was a stale data problem. In a system that was performing gymnastics to keep
things immutable. To the point that fixing it was not a simple matter of just
updating the variable.

Worst were the Multiversion concurrency control (MCC) systems I have seen
rolled to "help keep all facts of the system immutable." Suddenly, we had to
have database implementation experts just to run basic software...

~~~
sigmaml
I have similar experiences too.

Trying to keep data immutable introduces scenarios involving stale data, and a
new problem to solve. I have seen interesting problems with cache coherence
that were quite subtle in their manifestations.

Understanding the system-wide implications of immutable-only data is a non-
trivial design task.

~~~
rtpg
the issue I have with this statement is that stale data exists when going
full-imperative as well.

The solutions might be harder to implement in an immutable environment if your
setup isn't right, but not an order of magnitude harder. And you get all the
advantages of immutability.

People complain about stale data issues in immutable programming styles not
because they're more prevalent, but because they can rule out so many other
classes of bugs immediately.

~~~
sigmaml
> People complain about stale data issues in immutable programming styles not
> because they're more prevalent, but because they can rule out so many other
> classes of bugs immediately.

This generalisation is too broad (as with almost any generalisation).

In any case, I want to mention one interesting experience -- a very large
financial portfolio management system that I worked on, over a decade-and-a-
half ago.

The system had multiple in-coming tickers, feeding huge amounts of data. The
central data structures of the computational pipeline system were immutable.
Initially, there was a very haphazard system for determining when a particular
piece of data was to be deemed stale. In time, formal definitions were
established. However, the software was not amenable to those definitions,
since shared data was not `seen' updating instantaneously.

The core was too big to be changed in any non-trivial manner, without risking
the entire business.

We ended up introducing an explicit notion of time into _every_ pipeline
processor to ameliorate (not entirely solve) the situation.

Every major design decision has its own trade-offs that are specific to the
problem on hand. But, those built into the foundations of the system should be
made pragmatically, not by adhering to a philosophy!

~~~
rtpg
I agree that things need to be pragmatic. But there are classes of bugs ruled
out absolutely by things like immutable structures and referential
transparency.

Now, the thing is that there's always a way to model state, but at least I can
rule out things like the term "x" to mean different things depending on how
many times f was called beforehand for the most part.

~~~
taeric
The problem I have with this line, is it ignores the dealing with mutable
state you already do on a daily basis.

Are you confused that the listing on hackernews changes depending on time of
day? Why would you be confused that a call to "incrementCounter" changes the
value of the counter? Or that a call to "printf" adds to the output?

I agree that having a large function where the value of something changes at
the top and at the bottom can be confusing. But so can a function where you
have two variables of type String because you had to sanitize the input and
assign it to a new variable.

Are there ways to avoid that particular error. Certainly. Yet I have
encountered that about as many times lately as I have encountered "mutable"
based errors.

------
kstenerud
The Virtus issue is simply an example of bad design. When you add magic that
can't be guaranteed by the language, you're eventually going to run into
problems. Just look at Hibernate.

As far as performance goes, once you commit to a language with no mutable
types, you have no recourse once you discover a bottleneck due to
immutability. That's a big worry.

And advanced type systems are not unique to functional languages.

I'm also not clear on why the example testing code is better. It doesn't look
much different from what I'd write in any language (excepting boilerplate).

So yeah, correctness and testability. It's available in most languages.

Also, what's up with all the single-letter variable names?

~~~
kazagistar
> Also, what's up with all the single-letter variable names?

Words are not useful if your variables don't really have semantically
meaningful names. Also, if your code is a simple one-liner, it does not really
add too much benefit to name everything, if it is obvious from the (very
local) context.

For example, when I look at something like:

    
    
        ﻿﻿ordered (x:y:xs) = x <= y && ordered (y:xs)
    

I could do something like:

    
    
        ordered (first:second:remainingList) = first <= second && ordered (second:remainingList)
    

... but if anything, that will make it harder to read.

~~~
SparklingCotton
Then why did you call your variables x and y, and not y and x? Surprise:
because there IS a semantic meaning to the order.

The haskell variable naming convention is bad. What is worse is that when
someone points it out, there is ALWAYS this one or two variable example being
put forward, when the problem exists in the 90% of functions that have 5 or
more variables in scope.

I see a lot of haskell code that looks like FORTRAN. Haskell code is yet not
written in large companies where there is a readability requirement. I just
hope more programmers with a better taste for readability joins the Haskell
ranks. The readability story needs to improve (and I see it improving a bit..)

~~~
kazagistar
It is a lot less of a problem then you make out: most of the time, the single
letter naming conventions are actually very useful. Its not like all the
haskellers just forgot the usefulness of names from the other languages they
frequently use; it is an educated and measured choice.

That said, it certainly can be taken too far and turn into a clusterfuck.

    
    
        permutations            :: [a] -> [[a]]
        permutations xs0        =  xs0 : perms xs0 []
          where
            perms []     _  = []
            perms (t:ts) is = foldr interleave (perms ts (t:is)) (permutations is)
              where interleave    xs     r = let (_,zs) = interleave' id xs r in zs
                    interleave' _ []     r = (ts, r)
                    interleave' f (y:ys) r = let (us,zs) = interleave' (f . (y:)) ys r
                                             in  (y:us, f (t:y:us) : zs)

~~~
ridiculous_fish

        addart a = array ((-1,0),(n,m+n)) $ z ++ xsi ++ b ++ art ++ x
          where z = ((-1,0), a!(0,0)) : [ ((-1,j),0) | j <- [1..n] ] ++ [ ((-1,j+n),a!(0,j)) | j <- [1..m] ]
                xsi = ((0,0), -colsum a 0) : [ ((0,j),0) | j <- [1..n] ] ++ [ ((0,j+n), -colsum a j) | j <- [1..m] ]
                b = [ ((i,0), a!(i,0)) | i <- [1..n] ]
                art = [ ((i,j), if i == j then 1 else 0) | i <- [1..n], j <- [1..n] ]
                x = [ ((i,j+n), a!(i,j)) | i <- [1..n], j <- [1..m] ]
                ((_,_),(n,m)) = bounds a
    
    

From Matrix.Simplex
[https://hackage.haskell.org/package/dsp-0.2.1/docs/src/Matri...](https://hackage.haskell.org/package/dsp-0.2.1/docs/src/Matrix-
Simplex.html)

~~~
codygman
To be fair it _looks_ to be a relatively complex operation. Do you know how
any other languages implement this?

~~~
adwn
This is not about the complexity of the code, but about variables with or
without meaningful names.

~~~
codygman
I feel like xsi and art mean something domain specific that might be obvious
to a domain expert.

------
pdpi
"A nice side effect (hmm) of immutability is you end up with pure functions."

No, immutability doesn't magically give you purity. E.g. Rust gives you
immutability by default, yet you don't need anything to be mutable to get IO.

~~~
teacup50
IO is by-definition mutation.

------
vezzy-fnord
Virtually all of the concepts that the author listed are completely orthogonal
to functional programming, and moreover FP is equivocated with statically
typed FP.

~~~
nickpsecurity
What's the best resources you would recommend for someone to really wrap their
head around functional programming, foundations and practical use?

~~~
pdpi
Learn You a Haskell For Great Good is a great introduction, if you're willing
to just pretend you know nothing of programming for a little while.

~~~
nickpsecurity
I was uncertain at first if I was reading a book title lol. Haskell is usually
a tough learning experience. I'll keep the link and check it out. Might need
to learn it anyway given the seL4 and lots of Galois's work (eg Ivory
language) is using Haskell. Figure it's better to start with it than Ocaml
given that Haskell doesn't give an imperative programmer many outs. Gotta
learn the concepts or leave haha.

"if you're willing to just pretend you know nothing of programming for a
little while."

Always good advice for learning a new paradigm. I learned this when smart
people taught me how to learn a foreign language. They said learn the concepts
fresh, do no mental translations, and immerse yourself where you're forced to
solve problems in that language. It's how we learn the native one's so why not
new ones.

~~~
codygman
> Haskell doesn't give an imperative programmer many outs.

What do you mean?

Check these out:

[http://www.haskellforall.com/2012/01/haskell-for-c-
programme...](http://www.haskellforall.com/2012/01/haskell-for-c-programmers-
for-loops.html)

[http://www.haskellforall.com/2013/05/program-imperatively-
us...](http://www.haskellforall.com/2013/05/program-imperatively-using-
haskell.html)

~~~
nickpsecurity
Closer, but not quite. One commenter there pointed out that you still need
foundational stuff to really get it. Otherwise, you're just seeing templates
without really understanding what they mean. That's kind of what I meant.

Nothing wrong with it, necessarily. I'm just saying people learning it say
there's not many shortcuts around learning it.

~~~
pdpi
That's a good thing in my book. It takes you longer to get productive, of
course, but, when you DO get there, you're on much more solid footing

------
z3t4
If your program has state, it doesn't matter if it's mutable or immutable
state, you will still run into the same problems when working with
concurrency. You either have to use locks, make sure functions are executed in
the right order, or the preferred way: designate/central state.

So functions vs object orientated is mostly a matter of preference.

One advantage with functional programming though, is that it will be much
easier so serialize the state.

~~~
codygman
> If your program has state, it doesn't matter if it's mutable or immutable
> state, you will still run into the same problems when working with
> concurrency. You either have to use locks, make sure functions are executed
> in the right order, or the preferred way: designate/central state.

What about using STM:

[https://www.fpcomplete.com/school/advanced-
haskell/beautiful...](https://www.fpcomplete.com/school/advanced-
haskell/beautiful-concurrency/1-introduction)

------
wcummings
> you have to be wary of using Java and .NET types which are still prone to
> NPEs

Java8 optionals are a maybe monad

[http://stackoverflow.com/a/19932439/4658666](http://stackoverflow.com/a/19932439/4658666)

~~~
quchen
I think your post only takes non-null values into account, which is what OP
seems to talk about. What would this code do?

    
    
        Optional<Int> foo = null;
        foo.flatMap(x -> Optional.of(x));
    

Obeying the Monad laws would mean getting null as a result, via `m.flatMap(x
-> Optional.of(x)) = m` for all m (or `m >>= pure = m` if you prefer Haskell
syntax).

