
Hydra: A framework that simplifies development of complex applications - vquemener
https://engineering.fb.com/open-source/hydra/
======
rad_gruchalski
A framework that simplifies development of complex applications in Python.
There, I fixed it.

~~~
omry
Good point. The concepts of Hydra are not language specific though. At some
point there might be something similar for other languages.

~~~
rad_gruchalski
Indeed, the concepts are not language specific. I was hoping to see something
with a grammar and a reference implementation. Looking briefly at this
project, not all concepts can be abstracted but a specification with a set of
integration tests would make it implementable in other languages.

Edit: it surprises me every time that a project like this always goes for a
single language implementation. There are plenty of good concepts in there. I
know purists will say “it does too much”. But when one can crank out a complex
cli app and spend half time on it thanks to things like this, I’m all up for
it.

~~~
omry
The project grew organically. Evolution was something like:

1\. Let's build a framework for model based RL, we can start by implementing
the PETS paper
([https://arxiv.org/abs/1805.12114](https://arxiv.org/abs/1805.12114))

2\. It sure does have a lot of degrees of flexibility, and the PETS reference
implementation is full of hard coded configuration. It would be nice to be
able to combine configuration files somehow and override them from the command
line.

3\. Looking around, I didn't find anything that was quiet what I was after so
I created OmegaConf
([https://github.com/omry/omegaconf](https://github.com/omry/omegaconf))

4\. Building the framework, I evolved my understanding of what composing
configuration actually means, and added additional features as I needed them.

5\. The MBRL framework became really nice and useful, but was still primarily
an implementation of PETS. I tried to make it a bit more generic to allow
others in my team to use it for other things, and realized the better thing to
do is to go for something totally generic and make it a standalone package,
Thus Hydra was born.

It evolved into what you see today, and it was almost exclusively my work - so
at no point did I have the cycles to really support development in other
languages.

One of the challenges is to convince people that it's useful, It can do many
things as some people point out and people can get uneasy because of that. For
Hydra to succeed, people should be using it, there is no point in spending
years building something no one would use. ML, which is where Hydra grew from
is dominated by Python today, That's why Python is the first implementation.

Building Hydra in another language is possible, especially now that there is a
reference implementation, but it's a lot of work. If you are interested in
trying it out I would be happy to discuss and advise.

~~~
rad_gruchalski
Thank you for a thorough reply. It would be definitely an interesting exercise
to port it to another language. I’ll reach out. Let me do the homework, a deep
dive so to speak, before reaching out.

------
52-6F-62
The name of this library almost feels a little too self-aware...

~~~
chii
heil hydra o/

~~~
drudru11
Given FaceBooks recent troubles, yeah, this seems a little tone deaf

------
whalesalad
This is one giant antipattern. I hope people don’t really use this.

It does too much. Configuration? Running apps remotely? This is a big cluster
of miscellaneous helper tools glued into a ball called a library. I don’t
understand how it’s useful.

~~~
armitron
It can be a good fit for quick throwaway projects, typically found in various
research groups. Python itself is a giant antipattern for anything more
ambitious I would say. Go provides far more rigor and stable foundations. Rust
is also there for systems programming.

~~~
crimsonalucard
Make no mistake. In terms of abstractions.. rust is a higher level language
then golang. You have more safety features of a higher level language in rust
than you have in golang and you get it at zero cost. The exception is
concurrency primitives which is golangs only saving grace.

------
synt4x1k0
[https://cli.dev/docs/intro](https://cli.dev/docs/intro)

~~~
thruflo22
Code examples:
[https://github.com/facebookresearch/hydra/tree/master/exampl...](https://github.com/facebookresearch/hydra/tree/master/examples)

------
w_t_payne
So ... what does it do, exactly?

~~~
barce
It looks like configuration management for devops. If I have an app and want
it run in 3 different environments that each have rapidly changing configs, I
might use this. But I won't b/c Facebook.

~~~
chrisweekly
"But I won't b/c Facebook." Do I understand correctly that you also personally
boycott React?

~~~
Existenceblinks
I do not boycott React because of Facebook. It's because its inferiority.

~~~
aphextron
Inferiority to what? I hate Facebook as much as the next self respecting
Netizen, but I still use React at work because it's just the best tool for the
job. I'd be happy to hear some arguments to the contrary.

~~~
bwblabs
Ever looked at [https://svelte.dev](https://svelte.dev) ?

[https://youtu.be/gJ2P6hGwcgo?t=1055](https://youtu.be/gJ2P6hGwcgo?t=1055)
[https://twitter.com/wolfr_2/status/1164621105476329472](https://twitter.com/wolfr_2/status/1164621105476329472)

------
saadshamim
I initially thought this was post for
[https://github.com/ory/hydra](https://github.com/ory/hydra)

~~~
Semiapies
Or the Emacs library.

------
tyrion
A genuine question. How come command line options are passed like so:

    
    
      python my_app.py db.driver=mysql db.user=omry
    

No intention to criticize. Just wondering if there is any reason you did not
choose the more standard way:

    
    
      python my_app.py --db-driver=mysql
    

Wouldn't it be nice if all CLI applications could agree on a way to pass
parameters? :S

~~~
omry
Something like this could be implemented, but Hydra is using argparse for a
few basic flags (--help, --run, --cfg etc) and argparse would not like us
passing arbitrary undefined flags.

Philosophically speaking, I am not seeing a big advantage for Hydra overrides
syntax to line up with getopt. What it does is different. For example, unlike
with getopt - you MUST specify a value with every override, there is no
support for flags like --verbose. it would have to be verbose=true, the reason
is that those are editing the configuration object, and in 99% of the cases
you want to put in a value and not default to true.

Also, you should be thinking of those command line arguments as editing a
complex object. db.driver=mysql means: in the db map, change value of the
driver key to mysql.

------
sergioisidoro
This is nice, especially the multi run. But it seems to be a boiled down
version of click at first glance...

[https://click.palletsprojects.com/en/7.x/](https://click.palletsprojects.com/en/7.x/)

~~~
omry
Click let's you build really nice command line libraries, but those are rigid
and their structure is hard coded. A good example of what you wouldn't do with
Click is logging configuration. You have a bunch of appenders, formatters and
loggers and this is just not good with something like Click.

Hydra actually comes with built in Python logging configuration, it's just a
part of the configuration object and Hydra is setting it up for the app:
[https://cli.dev/docs/tutorial/logging](https://cli.dev/docs/tutorial/logging)

You can override this from the command line (unlikely but could be useful), or
completely replace it with your own logging config:
[https://cli.dev/docs/configure_hydra/logging](https://cli.dev/docs/configure_hydra/logging)

This kind of flexibility is a core part of how Hydra build the config, it's
not limited to it's internal config (like logging) but extends naturally to
the user config as well.

~~~
choppaface
It might be nice if mlflow adopted something like this as a feature.

~~~
omry
Tell them to take a look at Hydra.

Hydra has some overlap with what MLFlow is already doing, but I am sure MLFlow
can benefit from configuration composition.

Another way to look at it is: What kind of Hydra plugins can be backed by
MLFlow. for example, remote launching to any cloud can be implemented as a
Hydra Launcher plugin. Currently the included launcher plugin is trivial
(local execution), and the more interesting launchers to launch remotely are
not open sourced as hey rely on internal FB infrastructure. Supporting the
major cloud providers in Hydra through plugins is definitely something I would
like to see.

------
m3kw9
Complex stuff to do supposedly complex stuff

------
peteradio
Looks hella useful to me, having developed plenty of python cli before this
organizes in a way I've not really succeeded before.

~~~
omry
Glad you like it!

------
viraptor
This is pretty similar to openstack's oslo.config, but without the schema for
the parameters. If you want validation, and ini rather than yaml, oslo.config
may be your thing.

~~~
omry
The config is the schema to some extent. see the section about strict mode
here: [https://cli.dev/docs/tutorial/config_file#strict-
mode](https://cli.dev/docs/tutorial/config_file#strict-mode)

At the moment it's not as rich as all of the other command line libraries in
term of types, enums etc but it's richer what it you can easily express with
it.

For example, I challenge you do express logging configuration with
oslo.config, or Click, org argparse. You would be writing a whole lot of code
to come up with something very rigid. With Hydra composing hierarchical
configs is trivial so logging configuration is not harder than anything else.

What you would do with those other CLI libraries is just is either either hard
code your logging config (eew) or separate it into a different config file and
add logic to deal with it. In my book this logic is an example of boilerplate
that Hydra can handle cleanly.

------
zeronone
Java Spring!!

------
crimsonalucard
Sounds like inheritance.

~~~
crimsonalucard
I mean how else can you 'compose' data? Either through overriding things
(inheritance) or as a Product (ConfigA, ConfigB) (Also inheritance).

Technically you can do CoProduct (ConfigA | ConfigB) (not inheritance) but the
logic of unraveling such a type is too much for configs so it's very unlikely
Hydra does this.

~~~
omry
It does, it's called configuration groups.
[https://cli.dev/docs/tutorial/config_groups](https://cli.dev/docs/tutorial/config_groups)

(Assuming I understand your exact meaning).

~~~
crimsonalucard
I mean that there's only a couple of things it can be doing and inheritance is
it... what else can it possibly be doing. I'm responding to a vote down.

~~~
omry
I am actually not seeing "Hey, it's like inheritance!" as a vote down,
although I am not normally reading HN so I may be naive.

~~~
crimsonalucard
I’m responding to my own comment because it’s getting voted down, ha.

