
Roll Your Own Frameworks - luord
https://blog.startifact.com/posts/roll-your-own-frameworks/
======
rglullis
I had to RTFA three times to understand what was bothering me about it: it
basically takes sensible principles about software development and turns into
a self-affirmation exercise about the author's work where he applied those
principles and packaged as some grand wisdom about "frameworks".

The first half boils down to (a) research existing solutions, (b) avoid NIH
when you can and (c) refactor your code as you find commonalities and points
of re-use. The second part is mostly about designing a system that could be
used/extended by someone else (separate the generic from the the specific,
treat it as open source).

Those are very valid points. Thing is, in following those very valid points
one could probably write another article with a far greater number of examples
(and a far better lesson) where the best course of action is to _NOT_ roll
your own framework and prefer to adopt some existing code and make the
necessary extensions/modifications.

To illustrate, I checked the morepath "framework". It seems to tick all of the
boxes one would expect from a proper open source "framework". It even has more
documentation than I would expect from such a small open source project. The
one point where it fails is that it never gets to give one simple example of a
pain point that this project solves that all of the other million web
frameworks don't.

In the end, it seems that the author is looking for an ex post facto
rationalization for some of his projects that had a lot of his time invested
but no external interest afterwards. Too bad that it seems that he also took
the wrong lesson out of it.

~~~
skinnyarms
I would read that article, and vote it up, but I don't think we need to be
harsh towards this one. It's a good read, makes some good points and I'm
thankful for it.

~~~
rglullis
Were it something pointing out the trade-offs that he found when rolling his
own frameworks and I wouldn't be so put-off by it. But when an article wants
to argue for something and its justifications could be used to conclude the
exact same opposite? I fail to find those good points, to be honest.

------
ebiester
If you're doing it for yourself and nobody will ever have to maintain your
code? Sure, knock yourself out!

If you have a large organization that can absorb the cost of maintaining the
framework long after you have gone, and can ensure the entire company is
invested in this framework? Sure, someone will hate you in five years because
they will hate your opinions, but you won't sink a company.

But if it's in between, you are doing a disservice to whoever is paying you.
You won't document it like a great framework. You'll be fine with a hack or
need to skimp on testing during a deadline, and three years later people will
be stuck around decisions that seemed right at the time.

I have seen how bad of a drag this can be up close.

~~~
samvher
I feel like mainly this article is about building generic tools as you are
developing your application. I think the word "framework" is used liberally -
base classes and higher order functions are classified as tiny frameworks in
the article.

In some cases I think it's a tricky call. I'm currently working on a
"framework" and am worried about all the issues that you mention. The reason
I'm still building a framework is that as far as I can tell there is really no
existing framework for the task at hand, and we do need the economies of
reusable code (consistency across deployments & avoidance of double work). I
hope the organization is large enough to absorb this and keep it going down
the road, and am aiming to make it open source in the hope that a community
can form around it as well. Fingers crossed, I guess.

~~~
ebiester
There are all sorts of nuance that are not captured in a short post. In 2012,
I built a micro front-end framework for front end because I wanted something
closer to what turbolinks is today, and I am sure people have cursed my name
in years since. (Of course, the alternatives were things like knockout and
backbone, and I had issues with those and my use case.)

So, to be fair, many of us have done it. The best case is that you can open
source it and everyone can benefit from your work.

------
helldritch
I agree with a lot of what this article says.

I've recently started a new application in PHP which has very strict time /
performance requirements. I could build it in another language, but I'm most
familiar with PHP, the thing would take me 4x as long to build in Rust.

The default frameworks (Laravel and Symfony) are very slow out of the box, and
doctrine / propel are pretty slow as well.

I've instead built something entirely brand new using Swoole and an assembled
set of very fast and very light Composer packages (namely Fastroute and PHP-
DI) and everything else I'm building myself (compromising on reusability and
full-featuredness in return for raw performance).

I've split the code in to two: one half is the framework, it's concerned with
routing, dependency injection, models and hydration, event handling and
responses. The other half is focussed on the business concerns: creating
users, creating authentication token, performing searches.

There were a few challenges: moving from single row, single model queries to
hydrating complex joined collections was a major multi-day challenge, but the
end result is that when a request has all the data it needs cached, it
responds in ~3ms, when it needs something from the database it responds in
~10ms.

I was apprehensive when I started, and a half decade ago this codebase would
be an unmaintainable mess, but I'm finding it a joy to work in: there are few
layers of abstraction, not a single line of code is wasted, and when something
goes wrong there's no magic obfuscating the true problem.

~~~
kwhat4
This is one of the primary reasons most people hate PHP. Let me ask a simple
question: What does the documentation for your homegrown framework look like?
Having worked for several organizations that decided to roll their own
framework in PHP, it is an absolute nightmare. No one else knows how it works
or how to use it and it usually comes with no documentation. Furthermore, you
don't get any updates or bug fixes unless you write them, taking time away
from working on the project that is being held up by your homegrown solution.
There are plenty of fast, low bloat, PHP frameworks to choose from: Phalcon,
Slim, Luman and Silex just to name a few. I guarantee they all have better
documentation and support. If you need some custom functionality, extend one
of these existing solutions to suit your needs. Please, please stop
reinventing the wheel for production code.

~~~
helldritch
I agree with everything you've said, with the exception that it doesn't fit
our usecase.

We run and maintain several Symfony applications, we're very familiar with
frameworks, how they work, their strengths and weaknesses.

This was a carefully considered route:

We wanted to be writing PHP code with high performance and strict response
time requirements, as such:

* Symfony was instantly ruled out.

* We knew we would struggle to do this with raw PHP but didn't want to retrain the team to learn a new language (our thinking being we would likely write something inferior and insecure).

* We considered Phalcon and Swoole and ultimately decided on Swoole as we found Phalcon a bit too opinionated. That being said, Swoole's documentation isn't great so we've had to internally document it. (We will probably push a few documentation PRs for swoole-src when we're done).

* We didn't trust a big framework to have zero memory leaks, which is a key issue for long running Swoole processes, since it's not a typical issue for the single-use PHP processes we are used to.

With regards to documentation, it's very well documented (with documentation
explaining the reasoning behind each class, usage, pitfalls and how it fits in
to the larger application, and documentation for the wider ergonomics of the
application, debugging, performance considerations, etc), with a 96% test php-
unit coverage for the Framework.

Your concerns are very well received and are concerns we have had from the
beginning. Constant vigilance and a deep respect for my co-workers is what
keeps this little project on the straight and narrow.

~~~
tiborsaas
What were your constraints in performance? I'm surprised Laravel is considered
slow.

Have you considered sticking to it and applying some caching layer on top of
it?

~~~
helldritch
It's part of a live video platform for a gaming company which needs extra
information to be overlaid on top of their live stream (for the customer in-
video frame performance is incredibly important: the stream run at 30fps (so a
maximum of 33ms per frame) and the overlay information has to be generated in
11ms or less)

We did originally consider writing it in Rust (it would have been my personal
preference) but the consensus was that since the rest of the tooling for this
customer is in PHP we didn't want to add a skillset they didn't have
internally if they were to move away from us in the future (we aren't big on
needless vendor lock-in and prefer to compete on quality and service).

We experimented with PHP7.4 and Symfony / Laravel and found that we didn't
have the performance leeway we needed, even with preloading / opcache.

We opted for Either Swoole or Phalcon, tried them both and found that Swoole
was more to our liking and was somewhat more performant than Phalcon in our
real-world testing.

We are exposing Swoole directly to them within their VPC, so we were able to
do away with NGINX as a reverse proxy as well and have Swoole handle the HTTP
request phase and HTTPS. We were able to make heavy use of Swoole\Table to
save superfluous trips to Redis and run a Redis synchronisation process in a
coroutine thread when the concurrent number of network tasks drops below 16.

Our end service response times (including network overhead) are around here:

\- Mean response time: 4ms

\- Fastest: 2ms

\- 99th percentile: 9ms

\- 999th permille: 10ms

One unforeseen side effect is that it has a crazy amount of scalability built
in by default, the 8core machine the process runs on can easily scale up to
15,000 concurrent requests (as in 15,000 requests per second) without going
over 15ms for response times, and can scale up to 45,000 concurrent requests
before things start to "break".

~~~
helldritch
Also: Our issues with Laravel / Symfony weren't just performance, but also
statefulness and memory leaks. Circular references, etc, which persist
throughout multiple network requests. In a normal PHP flow using php-fpm or
apache-php-cgi this isn't an issue: The PHP process is created, used and
thrown away. For a Swoole setup, these memory leaks quickly get out of control
and cause performance issues, instability and very occasionally superfluous
garbage collection runs.

------
methodin
I would tend to think if you built your own framework the right way you would
end up looking a lot like... an existing framework. If you build it the wrong
way then it would look like an interwoven mashup of code that you would have
no remembrance of if you came back to it after a period of inactivity. If you
are not confident you can do the former _in a production environment that
isn't just a side project_ then I would argue that you shouldn't until you
can.

~~~
BigJono
> I would tend to think if you built your own framework the right way you
> would end up looking a lot like... an existing framework.

Nope, existing frameworks are designed to solve the problems in 100,000
different applications across every domain known to man. Your "framework"
(which if you're doing it right is probably more like a set of patterns and a
carefully selected set of libraries that do one thing well) solves your own
problems, and only your own problems.

If something tailor made to solve your use case specifically looks exactly the
same as something like Django, then you're either Google or you've
overengineered it.

~~~
methodin
I don't necessarily agree with the sentiment as a whole as there's pretty
consistent design patterns for things like routing, http requests and
responses and user roles and permissions across many languages and frameworks
both large and small. Your point holds when you start digging into more niche
things (like form handling) but even if using frameworks you don't have to do
any of that stuff if you choose not to - it's there if and when you need it.
Things normalize because access patterns dictate that. Bucking that trend
because your app doesn't need that _yet_ is a slippery slope that leads to
exponential increase in time as your app gets more complicated since your
framework lacks core features that you have to go back and build and risk
breaking other things you've already built. If you abstracted that away to
avoid that then you are already thinking like a framework, so what's the
point?

------
irrational
I've been struggling with this for awhile. I have been working on the same
(large/complex) website for nearly 2 decades. Of course there was no Vue/React
back when I started building it (this was even in the days before jQuery had
even been invented!) So, I wrote my own framework. Over the decades I have
constantly updated the framework as I've found pain points. At this point it
does everything that I want it to do. But... only myself and one other
developer really understand how it works. Anyone coming in from the Vue/React
world would be lost. Now that we do have these modern frameworks I feel like I
should probably drop my framework and rebuild. The main issue is time. It
would take so much time to do it and I have a list of new features and
enhancements that people are asking for that is as long as my arm. Do I write
up a case for management as to why I need to halt all enhancement work on the
site for a number of months so that I can rebuild it on top of Vue/React when
the current framework is working fine?

~~~
elorant
As long as it works there's no point modernizing it. At some point when you
won't be able to update it anymore you'll pay someone to build you a new one
with all the bells and whistles of contemporary frameworks-which in most cases
are overhyped piece of junks.

~~~
nindalf
Never rewriting is a great way to save costs in the short term and also shoot
yourself in the foot long term. Unemployment systems written in COBOL were
perfectly fine until we found they weren’t in March 2020. Maybe we should have
rewritten those 20 years ago? That would have saved the NJ Governor from
putting out a request for “Cobalt“ programmers in the middle of a pandemic.

Let’s say GP and his coworker switch jobs in the next couple of months. Who is
going to maintain an application built on a framework no one understands? At
that point you’d have no choice but to rewrite it except now you don’t have
the knowledge of how the existing system works.

All of this without even getting into the security implications of using a
framework that no one else is.

~~~
unclebucknasty
> _That would have saved the NJ Governor from putting out a request for
> “Cobalt“ programmers in the middle of a pandemic._

The difference is this framework still runs on a modern platform (assuming it
uses JS and other Web standards). So, it's dated only WRT _other frameworks_ ,
which come and go anyway.

So, unlike with COBOL (and "Cobalt"), it's supported by the platform of the
day with standards many current devs know.

------
amelius
The thing with frameworks is, you can't compose them. And software based on
different frameworks usually doesn't mix well. Therefore, if you want to write
reusable code, I'd say choose libraries instead of a framework.

------
liendolucas
I think the article's title should be: "For educational purposes roll your own
framework". This is something that can greatly add value to a developer
(preferably afterwards she/he has been exposed to many different languages and
frameworks). What's the taste of implementing a minimal crappy but usable ORM?
How do I implement authentication from scratch? How can I implement that cool
feature X from the Y framework? and so on.

At the moment for example I'm doing exactly that in plain Racket (and also
learning Racket along the way) and so far it has been a fun experience because
you realise how much functionality we take for granted everyday when doing
real work in the industry. I wouldn't definitely do any real work from scratch
without a framework because using the least powerful framework out there
surely is going to be way much more productive than starting from plain
scratch.

Being that said there's I think another reason why there could be room for
this: after you have tasted many of frameworks out there and having
substantial programming experience you might want to roll your framework only
if you are able to provide something innovative or extra value that none of
the alternatives have, after all this must have been the way many of
frameworks out there were born and became popular, but even in this case you
might be able to achieve that by either contributing to a well known project
or extend it through a plugin.

------
Gollapalli
>When I build an application, I build frameworks along the way.

I do believe this is also referred to as simply "building an application."

~~~
webmaven
I know the OP, and he really does tend to extract frameworks while he's
working on an application (and for that matter, extract libraries while
working on frameworks) as a matter of course. It is pretty impressive how
effortless he makes doing the right thing for code reuse seem.

~~~
k__
I also know a handful of people who can pull this off while not messing
everything up in the process.

But most people do, I can't count how many bad reinvented wheels I met in my
PHP time. It was rather refreshing to use Node.js with NPM, where I could
simply install all I need.

------
wyattpeak
I think people should be cautious in taking this advice for anything but toy
projects, on a few counts. I spent two years developing on a framework I built
myself, until I finally gave in and spent an inconvenient amount of time
porting to Django.

1) Unless you have a lot of experience, you don't know why various patterns
have been implemented, or various compromises have been made, in existing
frameworks. Frameworks aren't on the whole built by idiots, and if you think a
framework has made an obviously poor choice, count the days until you realise
why they did it. If you're lucky your choice will still make sense for your
application. If you're not, I hope it doesn't take you too long to redo it
all.

2) As the author says, frameworks are by nature declarative. This is great if
you have really good documentation, but if not you're going to scream trying
to work out the control flow of your application. "Why does changing variable
X result in behaviour Y? What is even checking variable X? grep's returning
nothing, the stacktrace is twenty levels deep and I don't recognise any of
these functions. FUCCCCCCKK!"

You're going to forget how your framework works. If you don't forget there was
no point building it at all, because the point of a framework is to abstract
you from having to think about things.

3) Your project's not that special. The framework your considering will
probably work just fine. It's worked for several million other projects, it'll
work for yours. Yes, you might spend a bit of time trying to shoehorn
something in. It'll be an order of magnitude less time than building an
entirely new system.

4) It's much less useful for future work. A high-level understanding of
building frameworks will get you a job at the Django Foundation. An high-level
understanding of Django will get you a job everywhere else. And also at the
foundation, come to that.

It was a fairly informative process. It taught me things I didn't know I
didn't know. I value it in much the same way I value building interpreters.
But much like building interpreters, except in a few niche areas it's probably
not a good use of your time.

------
ibudiallo
I can understand the author and I usually do the same thing. But once you have
done this two or three times, you have your own framework that you don't have
to reinvent anymore.

If you are interested, I did a live recording of building a blog from scratch
[1]. From the content, to a decent framework to run it. This is more of a
purpose oriented framework where you build the features you need.

[1]: [https://idiallo.com/video/website-from-scratch-
part-1-conten...](https://idiallo.com/video/website-from-scratch-
part-1-content)

~~~
BigJono
IMO this whole thread suffers badly from differing definitions of 'framework'.

The 'framework' of your app is a collection of patterns and interfaces that
allow you to implement your actual features as easily as possible. Whether you
get that by importing 2M loc from the same source, piecing together tiny
libraries that do one thing well, or write the entire thing from scratch, it's
the same result.

I'd agree with you if I was using the definition everyone else is, which seems
to only cover the fist case (importing a shitload of code from one place).

Using any pre-made library is a trade-off where you incur some technical debt
in return for up-front speed. This is obvious if you look at the extremes (i.e
if you have infinite time then I'm 100% sure that using Django isn't the
optimal way to build your app, and if you're writing your own framework then
by definition you're spending a bunch of time writing code that doesn't
directly implement features. Everything else is somewhere in-between).

If you've written a framework that you can just sub in across multiple
projects, then it falls victim to the same trade-off you'd have to take with
an existing framework. Unless there's something about your framework that's
straight up better for the general use case than existing frameworks (in which
case well done, but "just build something better than Django/React/Whatever"
isn't really advice that's useful for the average dev).

I think it's far more realistic to have familiarity with a set of patterns and
small libraries for solving specific problems, and when starting a new
project, port over the parts which are relevant to it. I've written tons of
this kind of code in my career, and I don't see any point whatsoever in
releasing my own open source framework. Instead, whenever I implement
something that I think is interesting or might be useful later, I build a
minimal gist that implements the concept in-case I need to refer to it later
and re-use the code.

I think if I had my own framework that was robust enough to just copy
wholesale between projects, and I had to train all of the devs on my team to
use it each time, it'd just be the worst of both worlds. I'd much rather
either use something that's reasonably standardized, or piece together the
right set of tools for the job and educate and familiarise everyone else with
it as I go.

------
d_burfoot
One problem with building good tools for yourself is that if you ever go work
for an established company, you'll be immensely frustrated by how poor the
tooling is.

------
_nalply
I found localization complicated, so I rolled my own simple localization
framework for the web. I mark all texts with one degree character (°) at the
start. I had a script looking for all occurrences and merging them into a CSV
file which I gave to the translator who could edit the translation in Excel.
To help the translator comments for the context were possible, even a limited
option handling because of different grammar of language was there. I just
added what was strictly needed for the project. Then dynamic JavaScript
search-and-replace was applied after loading.

Bonus: it both translated text in HTML elements and JavaScript strings.

NIH at its best? But still today I couldn't find something so simple even for
the non-technical translator. Because of CSV merging we could work with the
translator in several rounds without hassle.

------
wallflower
We once outsourced a native iOS app to a large consultancy. They were able to
develop a working iOS app in a short amount of time. How did they do it? They
threw at least twenty people (at the peak) on the project. And they used their
own framework. It handled everything from network calls to CoreData to action
sheet UI to the “kitchen sink”. Everything in that app extended a base class
in their framework. They did give us a snapshot-in-time of their source behind
their framework, and any iOS dev who had to fix bugs or add new features to
the app they threw together eventually cursed the framework. Because sometimes
adding a new UI animation or different navigation way required mucking through
the framework code, sometimes frustratingly unsuccessfully. The app is no
longer in use, having been rewritten.

------
mnm1
The author just doesn't understand the difference between frameworks and
libraries. A function is not a framework. Neither is a small single purpose
library. None of this applies to actual frameworks unless one's working alone.
Add another developer and most of these points fall flat. The documentation
won't be maintained, bugs won't be resolved, and the other developers will
hate it. Sure, do what you want as a solo developer, but don't subject others
to shitty frameworks they can't learn, modify, or understand without you for
any reason unless rewriting the wheel is your only purpose. Been there way too
many times and not once was it a positive experience.

~~~
commandlinefan
> Add another developer and most of these points fall flat

See, it seems like that _should_ be true, but in my experience it's the other
way around. If I inherit somebody else's code, I can always (or at least, I
have a perfect batting average so far) make sense of how things work and
usually put together why he did things the way he did even if he's not
available for me to talk to. I may not agree with his design choices, and I
may criticize the way he approached things, but I can always work out how/why
things are put together the way they are and how to move them around.
Frameworks like Angular and Spring, on the other hand, are the product of
dozens if not hundreds of people and are so generic as to be like a cloud of
smoke: it takes weeks just to get my head around what (if anything) the
framework is even offering. That would be fine if it wasn't for the fact that
the framework ends up failing in mysterious ways that require days of
debugging to work out.

------
WesolyKubeczek
Good thing people who came up with Django, Symfony, Flask, or Rails themselves
didn’t listen to what most commenters are saying here. I mean, we’d be using
CGI.pm to this day! (Except that someone had to come up with CGI.pm either)

Because it seems like with attitude like this, the art of creating a better
framework will be lost to everyone but maybe some Brahman-like caste in ivory
towers, who also don’t mind others leeching from their effort and telling
everyone how you’re not supposed to roll your own.

------
probablyfiction
Back in 2013 the company I was working for released a touchscreen kiosk. They
rolled their own hardware and wanted a custom software solution. There was
limited space on the device and it would not be internet-enabled. There would
be no physical access to the device beyond the touchscreen, and the target
audience was not known for being technically savvy.

I was tasked with delivering PHP code that would serve pages to the customer.
For whatever reason I wasn't allowed to use CodeIgniter, which was what we
used elsewhere in the company. I wound up creating a mini-framework to serve
pages. It was extremely basic and limited, but in this situation it was what
was needed.

It is the only time I've ever had to create a framework. I doubt I'll ever
have to do so again. The beauty of frameworks is that they free you up to
focus on your application and the logic to support it, rather than figuring
out the best way to handle things like routing or database access.

~~~
commandlinefan
> the best way to handle things like routing or database access

See, I'm always a little concerned handing something like database access over
to an unaccountable framework for fear that they'll do the wrong thing and a)
it'll be my fault and b) I'll have to figure out how to work around it under
time pressure.

------
k__
The author has a very generous definition of framework.

------
laurencerowe
An approach which I've found to work well is extending an existing framework
with domain specific custom directives. This encourages consistency and can
make contributing business logic directly to your code base more approachable
for domain experts. Meanwhile there's always the escape hatch of the
underlying framework available when required.

------
chpmrc
The purpose of a framework is to set constraints so that precious brain cycles
aren't wasted on opinionated, (most of the time) irrelevant technical choices
(and related training). So that, hopefully, the sum of the knowledge each dev
in the team has of the framework covers enough solution space to solve all
problems the team will come across.

> the only way to make a framework do what you want is an ugly way

Most of the time it gets ugly because the developer(s) doesn't know how to do
that thing with that framework, not because the framework doesn't allow it,
and would rather find an escape hatch than improve his knowledge of the
framework and solve it the right way. I've seen this happen many times, both
while studying and in a work environment. And if that wasn't the case it would
be _way_ more beneficial to tweak the existing framework and contribute
upstream instead of creating "yet another framework" (obligatory xkcd:
[https://xkcd.com/927/](https://xkcd.com/927/)), especially if you'd like to
give back to the community.

Some frameworks have been around for decades (Django and Rails are good
examples) and are still in active development, used by people in all sorts of
industries, all over the planet. It's quite presumptuous to think your
_special_ use case hasn't been covered by at least hundreds of other
developers, tens of which came up with what would probably be considered _the
right way_ to do that specific thing.

The only acceptable reasons for rolling out your own framework are:
performance constraints that are imputable to the code architecture itself
(hint: most of the time they are not), licensing issues, the lack of a
framework for the language you're using and visibility (yes, releasing your
own framework is a cool way of advertising your skills).

~~~
commandlinefan
> would rather find an escape hatch than improve his knowledge of the
> framework

Is that because he's being lazy or because he has a 50-deep backlog of JIRA
tickets, a project manager breathing down his neck and an expectation that
every problem can be resolved in about an hour and definitely, positively, no
developer should ever be "wasting time" reading documentation because we pay
you to code, not to read, now get back to work you lazy worthless developer.

~~~
chpmrc
Could be. It's still not the framework's fault though.

------
ex3ndr
React is not a framework. Ant Design is also not a framework, as SQLAlchemy.

~~~
BigJono
True. One of the reasons I prefer React over others is it gives a good
standardised base to work from, but you can still compose the right set of
tools for each project around it.

Of course you can still fuck that all up by starting a project and insisting
on whatever is popular or that you're familiar with. There's plenty of
'framework-y' libraries out there for React that constrain you in all the
wrong ways if that's how you like to roll.

