
Show HN: Python requirements for Humans - Socketubs
https://github.com/socketubs/requirements
======
JulianWasTaken
This is a nice idea in theory, but setup.py and requirements.txt are not the
same, and you cannot particularly generate one from the other:

[https://caremad.io/2013/07/setup-vs-
requirement/](https://caremad.io/2013/07/setup-vs-requirement/)

Also, unfortunately pip's parser for requirements.txt isn't public, like much
of its internals at the moment, which means AIUI it's likely that the parsing
code here is going to be brittle if it uses it, or miss edge cases if it
doesn't. (I haven't read this carefully yet though).

~~~
thristian
As somebody who deploys a lot of business-critical Python apps at $EMPLOYER, I
don't even _write_ a requirements.txt.

During development and integration testing, I want to get the latest version
of my dependencies that I should be compatible with, so I use SemVer matching
(somepackage~=1.2.3) in setup.py.

For deployment, I want to use the exact dependency packages that the code was
tested against. So after automated testing is complete, it runs "pip wheel" to
build wheels of my package and all its dependencies and put them into a
directory, then _that_ is the artefact I use to recreate the package in
production. Much more reliable than requirement.txt, since it doesn't assume
that PyPI or internet access is available, and it doesn't care if upstream
authors silently update packages without changing the version number.

The last scenario is unit-testing, because sometimes tests have extra
dependencies the rest of the code doesn't share, like a mock library or other
test helper. In theory, these extra things could be put in setup.py's
"tests_require" option, but then they'd only be installed if you run tests
with "python setup.py test", and they'd also be installed with "easy_install",
which is terrible. I'm thinking maybe I should use setuptools' "feature"
syntax, so you only get the test dependencies if you install with the [test]
feature.

~~~
TheCowboy
Another way to do this, that you or someone may be interested in, is by using
pip-tools.

You write a requirements.in text file where you list libraries by just their
name, and then run "pip-compile requirements.in" and it will output a
requirements.txt.

It also has a function called "pip-sync" which will then install/update
everything in the requirements.txt file.

[https://github.com/nvie/pip-tools](https://github.com/nvie/pip-tools)

------
guitarbill
Python dependency management is a hard problem, but better than most languages
[citation needed]. And `pip` and `setup.py` have emerged over several years,
with several influences merged in (remember distutils?).

Honestly, I wish you'd picked a different tag-line though (riffing on
`requests` no doubt). Unlike `requests`, your solution only works for a subset
of deployment situations, because - as already pointed out - `setup.py` and
`requirements.txt` are for different things.

One of the best examples for this I've seen is to use both to deploy to a
server with no internet connectivity. For development the dependencies are
installed from `setup.py`. Then, before deploying, all dependencies are
downloaded via `pip download`. Put the dependencies on the server, finally,
use `requirements.txt` with `--no-index` and `--find-links` to install.
Definitely an _interesting_ setup, but needs must. Unfortunately, your
solution doesn't support `--no-index`, `--find-links` and a few others.

You may want to have a look at tools like `pbr` (Python Build Reasonableness)
[1], which has an interesting way of dealing with some hard problems. It also
shows how to use `setup_requires` so you don't have to have `requirements.py`
hanging around in your repo.

[1]
[http://docs.openstack.org/developer/pbr/](http://docs.openstack.org/developer/pbr/)

~~~
taion
I split my time pretty evenly between the Python and Node ecosystems, and I've
found that Python dependency management is extremely poor compared to Node
dependency management.

For example, Python's lack of support for nested dependencies means that it
can be extremely difficult to use small "utility" libraries like six as
subdependencies without running into potential problems from conflicting
version requirements from other dependencies.

Additionally, Python's management of different types of dependencies is very
weak, specifically with regard to setup.py and requirements.txt. npm has
(among other things) very nice explicit concepts of development dependencies
and regular dependencies, in addition to application shrink-wraps, which gives
library maintainers very easy ways to split out different kinds of
dependencies, or lock down all dependencies and subdependencies, using the
same set of tools.

While I can say that the current version of pip, especially in conjunction
with pip-tools, is significantly better than earlier iterations of Python
packaging, I strongly hold that Python packaging is substantially worse than
at least one other prominent example.

~~~
guitarbill
Interesting, I've never hit this and I though I've deployed some fairly
complex apps, but you're right of course. The site-packages approach isn't
great, and means you pretty much need virtualenv, especially on e.g. Debian
where the system Python is really important (I do love virtualenvs, and for a
workaround they're pretty solid). Nice to see npm has this pretty much baked
in (is my understanding?).

And don't get me started on eggs. I guess the nice thing about Python is that
the packaging genuinely seems to be improving year-to-year, although still
limited by design decisions and backwards compatibility.

Anyway, thanks for this; I've had a pretty informative morning searching for
"npm vs pip". I'm not a Javascript dev, and when I've tried to use it I've
struggled with the sheer bloat, even e.g. npm vs bower. Even though I'm a bit
jealous of npm now, I'll take pip over package management in the usual
enterprise juggernauts (C, C#, Perl or Java) any day!

~~~
taion
npm has a feature that's broadly equivalent to virtualenvs, in that each
directory is implicitly a virtualenv, and you use the packages installed in a
given directory (or its ancestors). This is a bit less powerful than
virtualenvs since it's tied to the directory structure, but in practice it
ends up being really convenient for most use cases. Along those lines, there
are also some cool affordances for managing dependencies – you can do e.g.
"npm install --save <package>" to install a package and update the equivalent
of setup.py with the new package, and it will automatically set that up with a
semver range that matches the current stable release of that package.

That's actually not what I'm talking about, though, since ultimately those are
just DX conveniences. The big difference is that, by default, npm installs
dependencies in a nested rather than a flat manner. This means that you can
install "library A" and "library B", which both depend on "library C", without
worrying about potential incompatibilities between the required versions of
"library C". This isn't possible in Python; one consequence is that you see a
bunch of libraries that vendor their own small subsets of six to avoid having
potential dependency issues – you don't get this in the Node ecosystem,
because you can just pull in that dependency and not worry about potential
version conflicts.

All of this adds up to a much nicer experience when developing libraries.

------
mpdehaan2
Ironically, the package to make your package requirements easier to write has
additional requirements not in the core distribution :)

But more seriously, good idea in thinking that things should already be this
way. The dual maintainance of the dependency list between setup.py and
requirements.txt unfortunately leads to a lot of packages being left out of
one or the other, or a package that works great from source that has a broken
setup.py (which usually applies to everything I do).

A setup.py command for installing deps from setup.py would, to me, seem a bit
more logical - but things already exist the way they do.

In the end though, it's a bit weird to deviate from the standard norms of
package installation, at least in Python circles, which may introduce some
confusion -- contrast this with Javascript circles that replace "the one true
way" about every 3 months :)

~~~
Socketubs
Thank you for your feeback. Same reasons that encourage me to build
`requirements` :)

I know this isn't perfect for now, `requirements` could evolve!

------
reidrac
I see a problem with that "License is MIT". That's not how software is
licensed and if the author wants this to be used, the legal part must be
flawless.

Who's the copyright holder? How can I contact that person? What's the
copyright year? Licensing software with MIT licence is trivial:
[http://choosealicense.com/licenses/mit/](http://choosealicense.com/licenses/mit/)

Depending on the project, if I really care about the legal status of the code
I use, I may contact the author to clarify this kind of thing, but sometimes
it is too much hassle. Just look at it as a critical bug in your software, and
fix it.

On a more personal note I started to dislike the "* for humans", it's been
overused.

~~~
Socketubs
I'll add license file in repository. Thank you.

------
whalesalad
I wish the Python community would adopt a Bundler/Gemfile style. The
project.clj concept is also a fantastic approach. Environments should just be
maps of data that you can modify or compose on the fly.

------
haldean
How would you install a package that depended on this? You can't install it's
dependencies automatically until you've got one of the dependencies installed.

~~~
Socketubs
Like I say in README[0], you have to put `requirements.py` beside your
`setup.py`

[0]:
[https://github.com/socketubs/requirements#usage](https://github.com/socketubs/requirements#usage)

