
Setup.py vs requirements.txt - donaldstufft
https://caremad.io/blog/setup-vs-requirements/
======
lifeisstillgood
Is there really such a thing as an abstract dependency?

There is such a thing as a tested system (ie two or more Independantly built
packages that have been demonstrated to work together). This is the concrete -
I have built these things and tested them - now take them _both_ as they are
and put them live. I now dislikes the idea that I will build packages (with
pip literally from source) on production machines.

No I don't quite get this - dependencies are just that - we depend on the code
they provide to run. A library author cares as much as a sysadmin that they
avoid a version of dependency that breaks the package.

A better spec might be

    
    
        Requests=1.1,1.2,1.4  #oops 1.3 broke our tests
    
    
    

And yes, requirements.txt and setup.py conflict. We don't have a clean
packaging system in Python and we should accept it and fix it.

None of this Precludes the useful ability to specify which location to
retrieve which package from, but these are _build_ instructions, not
_deployment_ instructions.

~~~
donaldstufft
I call them abstract for lack of a better word. Referenced by name?

It's not about providing versions of code that does or doesn't pass tests.
It's about where that code is fetched from. If you reference the dependency by
name then you can easily switch it out for something that looks like the code
the author originally developed against, but actually has some changes
(patches, bugfix, whatever).

Where to retrieve packages from is both a build and a deployment instruction.
After all you need to fetch the built packages from soemwhere yes? In general
"reference by name and version/abstract" allows you to easily swap in a built
package for a source package as well. Otherwise you'd be stuck with whatever
formats the author published because the dependencies would be pointing to a
specific url (likely outside your control).

The very fact you _can_ build a binary package and host it yourself works
because of what I call abstract dependencies (for lack of a better name). If
they wern't abstract then you'd have to fork the dependent package in order to
point it at your built package/repository instead.

------
m0th87
OK, the concept of "abstract dependencies" makes sense, but boy is it a
complicated way to fix an edge case. How many times have you had to swap out a
concrete dependency vs how many times have you bashed your head against the
wall about the proper way to setup a python library? I know my experience
skews heavily toward the latter.

I wish python had an equivalent to npm. npm does a lot of things right: local
packages by default, recursive dependency resolution, a simple declarative
package specification (package.json), and the ability to run your own private
npm registry. And it wraps up the logic of a bunch of tools: virtualenv + pip
+ (distutils / setuptools / distribute / distutils2).

~~~
d23
> how many times have you bashed your head against the wall about the proper
> way to setup a python library

Hmm, almost never actually. The only example I can think of is PIL. I usually
have to manually edit the installer to get libjpeg and zlib and fooimagelib to
install properly. Other than that, pip seems to have all the magic I crave.

~~~
m0th87
What about creating a new library? It's been a giant PITA for me, especially
figuring out which of the 4 major distribution tools to use.

~~~
viraptor
The latest one. There was only the brief period when setuptools/disttools were
confusing as far as I remember. Now, you can just use d2to1 and the new
distribute config format.

~~~
lmm
So what's it called? "distribute"?

~~~
donaldstufft
You should be using setuptools now.

There's distutils which was the original one and is part of the standard
library, setuptools which extends distutils and adds some very useful things
like dependencies, then distribute split off from setuptools because of a
number of reasons. Just recently distribute merged back into mainline
setuptools and the distribute project is now done and it's maintainers have
taken over setuptools.

In the middle there was also distutils2/packaging which were attempts at
replacing the entire thing but have since been abandoned for a number of
reasons such as trying to replace the entire toolchain at once.

------
severb
This is just plain wrong. There is no reason not to use setup.py for your
private "application" as he calls it. Even if it's not distributed trough the
public index you may still benefit from having version numbers and "abstract"
dependencies for your app. That way you may choose where to look for the
dependencies at deploy time and use a private cheeseshop for delivering your
private packages, not to mention that requirements.txt is a pip only thing.

~~~
donaldstufft
Actually you've misread it or I didn't explain it well enough :)

A private library/app can have abstract dependencies. As I said in the post it
deals with the "deploy specific" side of a Python application. So each
specific deployment would get it's own requirements.txt.

And yes requirements.txt is a pip only thing but the concept applies to the
similar file in zc.buildout whose name escapes me at the moment and any other
similar file.

------
dwightgunning
The setup.py approach that loosely couple dependencies is definitely useful as
the author describes.

However I think important advice to library developers that's missing in the
article is to be explicit (somewhere, likely in documentation) about the
version/source of each dependency that the library has been tested against.

Libraries are distributed, applications are deployed.

~~~
donaldstufft
Sure that'd be good too :) This wasn't really aiming to be a comprehensive
best practices guide to distributing things. Mostly trying to explain the
differences between what a dependency in setup.py means vs a requirements.txt
like file.

I have a hard time coming up with wording because sometimes applications are
distributed as well and the "deployment" is just a config file + installing
the dependencies. I hate using the term application because of that confusion
but I haven't thought of a better way to word it yet :(

------
teilo
Yeah, I just can't agree with this.

I maintain a private bitbucket repository for my company. It contains two
things: Django apps, and specific projects which implement those Django apps.

Each Django app is packaged using a standard setuptools/distribute package.
All dependencies of each app is managed in setup.py.

I place the projects under version control, in order to maintain the specific
templates, url.py, fixtures, and configuration for each site. I use a
requirements.txt file to manage the dependencies of the project.

There is nothing abstract about either.

------
tehwalrus
I have a lot of python code from my PhD which will be released when I pass my
viva (with full version history, of course) - I'm not looking forward to
working out how to deal with pypi after reading this!

I have always tried to make self-contained packages with a simple "just run
setup.py" install procedure for my smaller public projects on github, but my
larger codebase uses lots of libraries, and specifying which ones people
should install is going to be a nightmare...

~~~
ubernostrum
I think you're really overcomplicating this if your use case is to make
something installable from PyPI.

Think of it like this:

* "install_requires" in setup.py = "I don't really care how you get this, or possibly even what version of it you get, just get this".

* requirements file = "I have a known-good, tested and functioning setup using these specific versions, from these specific URLs, of these specific packages; I cannot vouch for whether other versions or other distributions of those packages will work".

------
ajanuary
I've been working with Ivy which does dependency management for Java.

It's got a similar split: an ivy.xml file for specifying what it depends on,
and an ivysettings.xml file for specifying what repositories to look in for
those dependencies.

------
kevinastone
Are you suggesting using `pip freeze > requirements.txt` as some kind of
Gemfile.lock compared to setup.py#install_requires as the standard Gemfile?

~~~
donaldstufft
``setup.py#install_requires`` would be the gemspec file.

Ideally We'd have requirements.txt and requirements.txt.lock which would
correspond to Gemfile and Gemfile.lock. However right now that doesn't really
happen. So you're kind of stuck having requirements.txt play a double duty
where for things where you need exact exact dependencies you'd check in a very
exact requirements.txt and things where you have a supported set of deps
(maybe more restricted ranges) and you expect people to Deploy by
cloning/unzipping a tarball etc you'd use a less restrictive requirements.txt.

I would very much love to get a distinction between the two uses of
requirements.txt so that we have a better correlation to Gemfile and
Gemfile.lock

------
philipn
You refer to dependency_links as a "misfeature," but I'm wondering why?

~~~
donaldstufft
Because it makes you depend on urls instead of abstract name/version
requirements. easy_install does provide some tooling to get around it but in
general it shouldn't be used.

~~~
philipn
I guess this is where I'm confused. Isn't this the same as using the '-e'
switch inside of a requirements.txt?

