

Things that separate the men from the boys and/or major concepts in software - henning

Here are a handful of things I've had a really tough time with in programming which seems be something you either get or don't, and there's a big difference between those who master it and those who don't. These are in no particular order.<p>* Recursion and thinking in higher-order functions<p>* Continuations<p>* Monads<p>* mod_rewrite<p>* Pointers and the process of building a low-level mental model of machine execution in assembler/C<p>Also there was a huge weedout in my abstract algebra class in college, separating those who could do rigorous proofs from those who couldn't.<p>"Turtles all the way down" in Smalltalk was a significant moment for me as well.<p>What other major things like this have you come across?
======
gregfjohnson
Locality of concerns. I've come to realize that the fundamental issue in
programming is that we all look at the universe through finite brain-sized
portholes. Higher order languages, structuring systems in abstraction
hierarchies, and software architecture research are all attempts to organize
arbitrarily large systems so that they can be understood in brain-sized
chunks. One might say that well-designed large systems have a sort of fractal
nature; it is possible to zoom in and out, and gain useful insight at multiple
different levels.

------
ericb
I have a, perhaps strange, opinion that these topics are not that difficult,
just difficult to explain and usually explained poorly.

Once you've gained knowledge, it takes a certain type of empathy to explain it
effectively. I think this skill occurs less commonly in left dominant
math/engineering oriented brains. This is why some programming books read like
poorly translated stereo instructions and developer positions requiring
customer-service skills tend to pay _very_ well.

This doesn't negate the intelligence or persistence needed to pick these
programming skills up in any way--they remain difficult for these reasons.

~~~
pg
Actually I still find continuations hard to grasp. I still have to stop and
think before writing code involving continuations, the same way I have to stop
and think before driving from A to B in Cambridge.

It doesn't feel like continuations match the way the mind works. Ditto for
nested backquotes, though I've never been able to think of a better way to do
what they do.

~~~
ericb
I was mainly thinking of difficulty understanding as opposed to difficulty
doing.

Our minds aren't impressive short term memory-wise. That might be part of the
nested backquotes problem. Memory prodigies usually don't have greater short
term memories--they're just better at "chunking" which is really a compression
strategy.

Cambridge roads are Escheresque. I've been to Manhattan a few times, but my
mental map of Manhattan is better than of Boston, where I live. You gave a
talk about compression and beauty, if I remember correctly. Maybe it's all
related--beautiful things are more memorable.

------
technoguyrob
How come mod_rewrite is on there? Isn't that basically just regular
expressions?

As for a more general suggestion: databases. Seriously. It is embarrassing how
many aren't, but just about every programmer these days has to be very
comfortable with understanding database querying and optimization, because
that's one of those few lines of code which can make or break an entire
application.

~~~
rshao
I totally agree. Regex gets easier and easier as you do more. It's just hard
to do in the beginning (personally I started with trial and error).

But definitely, robust database design is among the hardest things I've had to
do. I mean, it's one thing to just set up a couple of tables and write queries
to get your desired data. It's a whole different ball game to ensure that
it'll be scalable, and designing with consideration that you'll eventually
have to split your database into shards as it grows.

In a way, it's a subcategory of what I think is the hardest thing, which is
designing for scalability. Just because something works with a handful of
users doesn't mean it'll work when you have millions of users and hundreds of
simultaneous connections, which most of us are striving for.

------
pmorici
What about multi-threaded programming and atomic operations on shared data?
Seems like something many people flub up or aren't even aware of.

------
pius
_"There are only two hard things in Computer Science: cache invalidation and
naming things"_ -Phil Karlton

In all seriousness, I agree with your list with the exception of mod_rewrite.
I'd also add mastery of algorithms and complexity theory to the list.

------
dejb
To me what separates the men from the boys in software development is the
ability to get the job done effectively. In my experience, while the boys are
playing with programming methods for the sake of it, the men are actually
implementing functionality using the techniques that fit the job.

I must admit spending a fair bit of time as a boy though :)

~~~
Hexstream
Well, you first have to be a boy to then grow into a man then (makes sense).
Because to know what techniques fit the job, you have to know the available
techniques, and to know them you have to learn them, and to learn them you
either have to know or trust that they are or might be useful.

You can't really know how useful they are before you know how they work, but
you won't be bothered to learn how they work if you don't know how useful they
are. It's a bootstrapping problem the solution of which is to accept that
there's a percentage of new things you learn that will turn out not to be
useful to you. So, there's a lot of value in using unfamiliar techniques just
for the sake of it.

~~~
dejb
I agree. Aquiring the right tools is important. As long as you remember that
the tools are there to get the job done not the other way round.

Preferably you can do the majority of your 'technique aquisition' on someone
else's time so when you are coding in your own startup (or mine) you can focus
on the problem at hand. I know I've implemented things before in ways that I
now realize were pointlessly clever.

------
mixmax
The notion of pointers is interesting - to me, as a relative novice to
programming, pointers seem as the intuitive way of doing things - and not at
all hard. What this implies is that different people have very different
mental models of the world and how it works. I know people that I would
certainly consider smarter than myself who have trouble understanding the
concept of pointers - so it's not just about intelligence, though a certain
baseline is probably needed.

On the other hand I have serious trouble understanding faith and the thought
of a god - this is one of the things that just doesn't seem to fit into my
mental model of the world. So understanding certain abstract concepts seems to
have some kind of correlation to how you view the world, and what you find to
be easy and what you find to be hard.

I'm not quite sure I brought the point across in a coherent manner - but one
can hope...

~~~
wallflower
This insightful News.YC thread that I stumbled upon a while back _seems_ to
explain C pointers in a different manner better than I have seen elsewhere:

<http://news.ycombinator.com/item?id=87348>

(Other than the stuff about casting between pointers and ints though.. )

"That was the lightening bolt moment. The problem with pointers in C is that
the [star][varname] syntax means two different things, and they are actually
contradictory from a certain perspective. In one context it means "declare a
variable that is a pointer" and in the other it means "deference". Whenever my
languages-brain bit looked at this it couldn't deal with it, recognised it as
a contradictory positions and basically shut me down, but not in a way that
let me work out what was going on. Have a look at it yourself from that
perspective - hopefully you'll see what I always did. Now whenever I look at C
code I instead just think "everything is an int and pointers are all about
casting ints, and a star when you declare something is your way of telling the
compiler that the type of this thing is int-pointer instead of just int". Thus
I now have no difficulties at all."

~~~
mixmax
Very well written, thanks for the link.

------
pg
Macros. Especially macro-defining macros.

------
sbt
1\. Maturity. The ability to not whine about every little problem one runs
into, but Google it down quickly and fix it right away. This is about
persistence. I find it difficult to work with people I have to motivate at
every step of the way through a project.

2\. Holding a program in one's head. I'm not sure really whether to attribute
this to PG or Peter Norvig, or somebody else. The best way to understand this
is really to experience it in practice. Everyone finds it difficult to keep a
mental image of a project once it starts to grow, but it seems like some
programmers have an especially hard time with this.

------
pius
Oh, and I'd add cryptography because it's difficult enough that you probably
shouldn't write your own unless you've been formally trained and have put some
time in as a practitioner.

~~~
marcus
Implementing cryptography isn't that hard, the algorithms are fairly simple.
You can write RSA for example in a couple of hours if you already have BigInt
implemented. The hard part is the math behind the algorithms that makes them
work and be secure, but thats what we have Schneier for...

Although I wouldn't recommend implementing them yourself, unless you have a
very good reason for it, as Linus once said "Given sufficient eyeballs all
bugs are shallow", in Cryptography that is doubly important.

~~~
evgen
Actually, I would say that the math behind the algorithms is not the hard part
(someone else already did the hard work, as you mention); the hard part about
crypto is learning all of the subtle ways that you can screw something up.
This is the reason you should never implement your own crypto code -- by
definition it will suck for a long, long time.

------
giardini
Why are monads on the list?

FWIW I've never read a clear and simple definition of a monad (that more than
one person agreed upon). The word "monad" appears to be very nearly
replaceable by the word "opinion" with all ensuing ramifications. Perhaps
someone can provide a clear definition of "monad" here!-))

But seriously, I can't help but think that monads occupy a position similiar
to Tait's quaternions: useful to some but not popular enough to achieve a
critical level of adoption.

------
davidk01
all that it takes to get a concept is time and effort and nothing more. do
enough examples and exercises related to a concept and you get a pretty good
model in your head of what the concept is actually about.

------
edu
When you say mod_rewrite, you refer to mod_rewrite or to regular expressions?

------
edw519
Benchmarking - optimizing performance (at the right time) is one area when
junior people never even think of the things that senior people routinely do

Elegance - my definition is Elegance = Functionality / Lines of Code.
Sometimes maximizing elegance takes everything you've got. (And dramatically
reduces long term maintenance cost.)

Variable naming - (not a joke, although it seems like one). This shows that
you really understand what you're doing and makes life easier for those who
have to maintain your work. Real programmers never have to protect their jobs.

Structuring of data - both internally and on disk. Streamlining data
structures often reduces the need for complex processing. Eliminating the need
to be elegant is often more elegant than being elegant.

------
serhei
The ability to quickly learn new things like monads and mod_rewrite.

------
thurston
Parsing and translation.

------
sunkencity
really grok source control, merging / it's not a windows share with revisions
on top of it.

~~~
jules
Please explain source control, because I still don't understand it. I don't
understand how a source control system could do merging without knowing about
the format of the files. It's currently based on a string merge, but this
doesn't seem principled and it's not correct in many cases, or is it? On the
other hand it often works in practice.

------
logjam
Category theory in general, because it encompasses areas of computer science,
mathematics, and physics.

~~~
jules
Why should someone who is interested in programming languages learn category
theory and what is the best (preferably online) text?

~~~
mattrepl
Category theory is important to type theory and semantics (Pierce, see below,
also wrote several books on types in programming languages) and offers a
different representation of computational flow (monads, arrows).

Benjamin Pierce's "Basic Category Theory for Computer Scientists" (not online)
is terse and complete [1].

Wikipedia is good for definitions and there's also a Wikibook on category
theory: <http://en.wikibooks.org/wiki/Category_theory>

There are some brief, informative, and fun lectures on YouTube by the
Catsters: <http://www.youtube.com/profile_play_list?user=TheCatsters>

The natural transformations playlist is where to start, just be ready to use
the aforementioned references.

__

[1] Natural transformations are the original point of category theory. A text
that doesn't cover them is incomplete.

~~~
jules
Thank you and thanks neilc. What is the right order to watch the lectures by
the Catsters? Top-to bottom as listed on that page?

Are there any data structures based on category theory, like sets & graphs
which are inspired by mathematical theories?

~~~
mattrepl
Category theory shares a lot with set theory, but they are different - they
make different assumptions and each is better suited at describing certain
types of systems. Graph theory is also its own topic in math, though sets can
be used to talk about them

As for the Catsters' videos, I'm unsure about the best order. The
recommendation to start with the natural transformations playlist came from
someone in #haskell (Cale, maybe?). I'd suggest diving in to whatever sounds
interesting and backing up to review other videos that contain any new terms.

