
Beyond 10,000 Lines: Lessons Learned from a Large Phoenix Project - mikecarlton
https://shift.infinite.red/beyond-10-000-lines-ba1adc17cc7b
======
rdtsc
Also I found Dialyzer (success typing checker) to be helpful in large
projects:

[http://learnyousomeerlang.com/dialyzer](http://learnyousomeerlang.com/dialyzer)

The way success typing works it is a bit like static typing but when it cannot
deduce the types it assumes success. However when it finds a discrepancy it is
always right. So the more type annotations you add, and the more precise they
are the more benefits you get from it.

It has also been there for many years. I think Python only very recently has
started getting the same kind of things via MyPy. Of course, they had to call
it something differently (Optional Static Typing).

Elixir has a wrapper around it, it seems:
[https://github.com/jeremyjh/dialyxir](https://github.com/jeremyjh/dialyxir)
but never used it (I user Erlang mostly).

~~~
crbelaus
Totaly agree. Dialyzer/Dialyxir can provide a high degree of confidence about
types.

The wonderful thing about it is that you don't have to get all type
annotations since the begining, you can add them over the time. This allows to
use "unspecified" types for quick prototyping, and make them more specific
when the project evolves, which is a major complain of traditional type
systems such as Java's.

Another tool that I find useful is Credo[1] which "is a static code analysis
tool for the Elixir language with a focus on teaching and code consistency".

The Ecto[2] project uses a tool called Ebert[3] that automatically runs Credo
for each pull-request and comments with the issues found. Here you can see an
example of Ebert's bot commenting on a PR[4]

[1] [https://github.com/rrrene/credo](https://github.com/rrrene/credo) [2]
[https://github.com/elixir-ecto/ecto](https://github.com/elixir-ecto/ecto) [3]
[https://ebertapp.io/](https://ebertapp.io/) [4] [https://github.com/elixir-
ecto/ecto/pull/1785](https://github.com/elixir-ecto/ecto/pull/1785)

------
yellowapple
I feel like the bit about OTP misses the point of OTP.

Erlang is not just about distributed computing (in fact, it never was; any
affinity for distributed computing was more of a side-effect of Erlang's
design). Rather, it's about fault-tolerance. Supervision trees and "let it
crash" are the cornerstone of Erlang programming, and therefore by extension
the cornerstone of Elixir programming.

Meanwhile, OTP applications build on this in a way that permits composability.
It's kind of like microservices behind the scenes, bit they feel like a
monolith; you build up your system from lots of different OTP applications
that work together to provide a unified whole.

Elixir and Erlang web frameworks (Phoenix, Sugar, Chicago Boss (IIRC), etc.)
already do a lot of this for you by kicking off various OTP dependencies; for
example, your average Phoenix or Sugar application will in turn start Plug,
Ecto, and various other OTP apps, and these will in turn spin up their own
dependencies (like Cowboy and Postgrex, respectively).

Basically, it's not quite right to equate OTP to just distributed computing.
OTP is at the heart and soul of the vast majority of software written for
BEAM.

------
thijsvandien
Could anyone more knowledgable explain whether or not my feeling is correct
that Elixir becoming (relatively) popular is making Erlang a less viable
choice? Whereas the total number of libraries may be increasing, code that
might have been written in Erlang is now written in Elixir and established
Erlang codebases (RabbitMQ for one) have started to migrate (parts) to Elixir.
Calling Erlang from Elixir is easy, but what about the reverse? It reminds of
me of how, at least in the early days, Play Framework (largely Scala) could be
used from Java but not without much friction. Should I prefer Erlang as a
language for its maturity and fewer constructs, it would be a pity to then
miss out on new libraries or frameworks, such as Phoenix, that revolve around
Elixir.

~~~
dsri
Yes, you can call Elixir from Erlang: [http://elixir-lang.org/crash-
course.html#adding-elixir-to-ex...](http://elixir-lang.org/crash-
course.html#adding-elixir-to-existing-erlang-programs)

~~~
thijsvandien
I did expect it to be theoretically possible, but how realistic/sensible is it
really to do an Erlang project with Elixir libraries or frameworks? Elixir
brings a lot of its own tools, conventions and so on. Will it soon take over
my project, making it a better option to just use Elixir for everything, or
would it be very manageable and not interfere much with the rest?

~~~
bombtrack
We use both Erlang and Elixir where I work. We successfully use Erlang libs in
our Elixir projects pretty easily. We recently started trying to do the
reverse and use an Elixir app inside an existing Erlang app. It has been
significantly more work. The main issues have been around dependency
management.

I wasn't personally working on this, so I apologize for being fuzzy on the
details, but I understand that getting rebar3 to fetch all the deps that would
normally be managed by hex was not possible or at least non-trivial. There was
talk of having to manually install each dep that you knew the Elixir lib would
be requiring.

I think it was sorted into something workable, but if anyone has better
understanding I would love to be pointed to some resources!

~~~
bombtrack
Had a quick chat this morning with the people that worked on it. The package
rebar3_elixir_compile[0] makes this pretty easy. However, our target Elixir
lib is not a public hex package, which requires using git submodules.

Not a seamless setup, but it does work. As the sibling comment suggests,
perhaps using mix for everything would help.

[0] [https://github.com/barrel-
db/rebar3_elixir_compile](https://github.com/barrel-db/rebar3_elixir_compile)

------
misterbowfinger
> 5\. Avoid Non-RESTful Routes

Every "large" project gets to a point where some routes aren't totally
RESTful. It happens. But it's not great advice to say that non-RESTful routes
are always a code smell.

~~~
aikah
What a total RESTful route to begin with ? I thought REST had nothing to do
with "routes", url or resources but how the state of the application is
communicated in an API ? It seems to me the Roy wrote something with a certain
perspective, someone came up, read the paper and decided to apply his writings
to something totally unrelated because he didn't like SOAP ... I personally
stopped using that word so I don't have to confront and debate with "REST
purists" and just went back to talking about web service, because that's what
it is.

~~~
Arcaire
The idea is to limit exposed routes to the standard CRUD routes: create
(/new), read (/show), update (/edit), delete (/destroy).

~~~
aikah
I thought the idea was to make an API behave like a webpage by using
hyperlinks that represents the state of the application, so in theory a
"smart" client could navigate an API autonomously like a human interacts with
a web page ? Because otherwise it's no different from RPC.

~~~
anthonybullard
I think you are thinking of HATEOAS. Read about it here:
[https://en.wikipedia.org/wiki/HATEOAS?wprov=sfla1](https://en.wikipedia.org/wiki/HATEOAS?wprov=sfla1)

------
chx
Anyone has a small-ish gig where I could learn Phoenix/Elixir? I am a senior
level programmer but all my experience is with PHP/Drupal. I would very much
like to work on a smaller side (15-20 hrs a week) gig learning Phoenix/Elixir.
A lot of experience is going to transfer, I hope (problem solving is not
really language dependent) and so I think I offer a pretty good deal: you
could get a very experienced coder for a language learner's rates :)

~~~
messel
I've been hoping to pick it up on weekends & time off but startup hours don't
leave much free time. I worked on a few simple heroku / json phoenix apps last
year. Hoping I can put together a useful but small web app next.

~~~
dvcrn
I can recommend to both of you to get the book "Elixir in Action". It's
fantastic! Starts very simple and step by step explores the must-need concepts
of Elixir.

I started phoenix before fully understanding Elixir and had a hard time. Then
I got that book and when I was halfway through I loved Elixir already.

Now I use phoenix for almost everything web related.

~~~
jon-wood
Programming Phoenix is also excellent. I think it helps that it was written by
the core contributors to Phoenix itself (including Jose Valim, who created
Elixir). I really appreciated not only looking at how things work, but also
why they work in the way they do. The authors clearly have a real sense of
excitement about Phoenix and its contagious.

------
akavel
In section titled "3\. Write Fewer, Valuable Tests", I don't understand the
terminology ("controllers and plugs"), which I assume is some stuff in the
Phoenix framework that I have no idea about. Can somebody possibly explain to
me what are the _" controllers and plugs"_ here? I'm very interested in
learning a more general idea (how to effectively reduce lines of tests) from
this nugget, that I could hopefully translate to different languages &
frameworks I use...

Quoting the relevant fragment in full:

 _" We focused our automated tests on our controller actions and plugs rather
than going for 100% test coverage. Since these are the main ways that the
Phoenix application interfaces with the outside world, they’re the critical
points of failure._

 _Controller tests also exercise a lot of the code paths in your application,
making it less necessary to unit test every single module. As a result, you
end up with fewer tests, which makes it easier to do refactoring, provided
your changes preserve the behavior of the controllers and plugs. "_

~~~
gcb0
they did black box unit test instead of proper unit test.

e.g. if a method is private, it's not tested.

that's the usual way when full reliability is not worth the dev time, or when
devs think functional tests only are enough but still want a tap on the back
for having unit tests and coverage numbers.

in their case it's the later as you can see for: "Controller tests also
exercise a lot of the code paths in your application"

classical functional tests being called unit tests excuse. not that functional
is better or worse, but correct names are better no matter what.

~~~
Jtsummers
The thing about unit testing is that you have to decide what your "unit" is.
It could be down to the function/method level, but that's not always
appropriate. Such a strict testing framework can leave you immobilized when
you want or need to refactor.

Their HTTP API _is_ their interface for the application. It makes a great deal
of sense to focus on that level for their testing (note: they didn't say they
didn't test internals, they said they focused on the external interfaces).

------
saalaa
> Lessons Learned from a Large Phoenix Project

More like "large" projects.

------
kriro
Started meddling with Phoenix recently so this is an interesting data point. I
have to say that the very basic architecture diagram from the PragProg book
basically sold me on using it for my next side project(s).

connection |> endpoint |> router |> pipeline |> controller

Not sure what it is but it must be a combination of "|> seems pretty awesome"
and "well yeah it obviously makes sense to get a request as a struct and chain
functions in this way". It just clicked and aligned so perfectly with my
mental model :)

------
jblow
10,000 lines is supposed to be 'big'? wtf?

I write something like 25kLoC/year (of shipping code, generally very complex
stuff) and I don't even program full-time. The two projects I am working on
now are 35kloc (the smaller one) and 250kloc (the medium-sized one).

If someone thinks 10kloc is big, I have a hard time thinking of that person as
a professional programmer.

(Numbers listed here exclude blank lines and comments.)

~~~
xutopia
What does it matter how many you write? How many do you delete?

I have a project that is 5k lines of code and roughly 11k of tests and
specs(cucumber). It is a rewrite of a project that was 50k lines of code with
1.2k lines of tests and had less functionality and features than what it does
now.

Lines of code are meaningless when it comes to how much value they provide. I
personally prefer when a codebase is smaller because it means some thought was
put into it and most likely has less bugs as a result.

~~~
cgag
I believe jblows games (braid, the witness) have less lines of code than one
would typically expect. I think you should assume hes not writing code like a
government contractor.

~~~
mars4rp
upvote for government contractor, I once saw 5k lines of code for simple
change of address in the front end alone !!!

------
crasm
I was expecting the IT and devops book, The Phoenix Project:
[http://itrevolution.com/books/phoenix-project-devops-
book/](http://itrevolution.com/books/phoenix-project-devops-book/)

------
Roboprog
Trivial, but it warms my heart to see short-ish identifiers with gaps in the
names (underlines, but other languages use dashes, dots, etc), rather than
reallyLongStudlyCaps names. I love that Elixir uses the Ruby convention of
short names with underscores in them.

C++ is the tragedy that keeps smiting the industry with its children. (e.g. -
Java, and its silly names)

"But we need long names, and need to compress them since they contain so many
words". No, you need to better partition things so there is enough context
around what the name is attached to, so that the name can be simple.

------
pmontra
Typo in the title : it's Phoenix, not Phoeniz.

~~~
i336_
Not sure if the words "admin", "admins" or "mods" trigger anything? Mentioning
just in case.

~~~
rattray
try also dang

~~~
sctb
No, those don't have any magic, but if something like this persists you can
always email us at hn@ycombinator.com.

~~~
i336_
Ah, okay then. I was curious, thanks for the info.

Idea: make users with high karma capable of using benign, unassuming and non-
common parts of speech to ping the admins?

...I think I've been thinking about security too much

~~~
reitanqild
Just click the post timestamp and click flag first.

~~~
i336_
...I am so dense. That makes perfect sense, thanks.

~~~
reitanqild
> I am so dense.

Maybe just busy with real work?

