
My Python Development Environment, 2020 Edition - suraj
https://jacobian.org/2019/nov/11/python-environment-2020/#atom-entries
======
marmada
Does anyone else think this reflects badly on Python? The fact that the author
has to use a bunch of different tools to manage Python versions/projects is
intimidating.

I don't say this out of negativity for the sake of negativity. Earlier today,
I was trying to resurrect an old Python project that was using pipenv.

"pipenv install" gave me an error about accepting 1 argument, but 3 were
provided.

Then I switched to Poetry. Poetry kept on detecting my Python2 installation,
but not my Python3 installation. It seemed like I had to use pyenv, which I
didn't want to use, since that's another tool to use and setup on different
machines.

I gave up and started rewriting the project (web scraper) in Node.js with
Puppeteer.

~~~
lone_haxx0r
Yes. As someone who has never dove deep into python, but has had some contact
with it: the package manager ecosystem is the #1 thing keeping me away from
it.

npm sucks and all, but at least _it just works_ and doesn't get in my way as
much.

~~~
heyoni
What does npm do that python can’t? I’m curious.

~~~
Too
npm is equivalent to combining pip and virtualenv into a single tool. This
gives better ergonomics when switching between projects since you never have
to "activate" your environment, it's always activated when standing in the
project directory.

~~~
Pandabob
Isn't this what Pipenv does? What has been a downer for me is that many of the
cloud providers do not support pipfiles in their serverless app services
(Elastic Beanstalk, App Engine etc.)

~~~
Pandabob
On second thought, at least on GCP I should be able to put the pipfiles into
.gcloudignore and just update the requirements.txt file with each new commit
using git hooks, build scripts or a ci/cd tool.

------
madelyn
I have never understood the need for all the different tools surrounding
Python packaging, development environments, or things like Pipenv. For years,
I have used Virtualenv and a script to create a virtual environment in my
project folder. It's as simple as a node_modules folder, the confusion around
it is puzzling to me.

Nowadays, using setuptools to create packages is really easy too, there's a
great tutorial on the main Python site. It's not as easy as node.js, sure, but
there's tools like Cookiecutter to remove the boilerplate from new packages.

requirements.txt files aren't very elegant, but they work well enough.

And with docker, all this is is even easier. The python + docker story is
really nice.

Honestly I just love these basic tools and how they let me do my job without
worrying about are they the latest and greatest. My python setup has been
stable for years and I am so productive with it.

~~~
Twirrim
I'm firmly set on virtualenv with virtualenvwrapper for some convenience
functions. Need a new space for a project? mkvirtualenv -p /path/to/python
projectname (-p only if I'm not using the default configured in the virtualenv
config file, which is rare)

From there it's just "workon projectname" and just "deactivate" when I'm done
(or "workon otherprojectname")

It has been stable and working for ages now. I just don't see any strong
incentive to change.

~~~
babayega2
I have been doing this starting Ubuntu 14.04. it's been stable even when I
upgraded now to 18.04 which has python 3 as default. The only downside
compared with tool such as pipenv is the automatic update of packages that
pipenv can offer and it's ability to be integrated into CI/CD pipelines.

------
Evidlo
For those new or unfamiliar with python, I think the best solution is the
simplest: pip and virtualenv

~~~
ramraj07
Seriously. The author is bending over backwards to accommodate poetry from
every direction, from the stupidest installation instructions I've heard, to
"can't transfer from requirements.txt" to "it doesn't work well with docker
but doable". Like what exactly does it add that's worth all this complexity?
Make you a maitai every hour?

------
f4stjack
Eeeehhh I think I will be downvoted to hell and back for this but after I read
the article I had the feeling of "why are you making this feel more complex
than it needs to be?"

I mean compared to Java and C# I have a MUCH MORE EASIER time to set up my
development environment. Installing Python, if I am on a Windows box I mean,
is enough to satisfy a lot of the requirements. I then clone the repo of the
project and

source venv/bin/activate

pip install -r requirements.txt

is enough to get me to start coding.

~~~
slig
> "why are you making this feel more complex than it needs to be?"

Because it's more complex if you have projects on multiple Python versions and
if you want to lock your Python packages to specific versions. (Pip can bite
you when different packages have different requirements for the same lib).

------
wp381640
If you're going to be using pyenv + poetry you should be aware of #571 that
causes issues with activating the virtualenv

[https://github.com/sdispater/poetry/issues/571](https://github.com/sdispater/poetry/issues/571)

the OP himself has a fix for this in his own dotfiles repo:

[https://github.com/jacobian/dotfiles/commit/e7889c5954daacfe...](https://github.com/jacobian/dotfiles/commit/e7889c5954daacfe0988fc05ff9e8e87eb1241b7)

~~~
jacobian
Ha, I'd forgotten about that. Thanks for the reminder.

(Though, how'd you find that? Mildly creepy that you know more about my
dotfiles than I do!)

~~~
wp381640
I landed on that issue a while ago and your pull request was linked so I
ripped your solution and added it to my own dotfiles :)

------
nunez
> Although Docker meets all these requirements, I don't really like using it.
> I find it slow, frustrating, and overkill for my purposes.

How so? I've been using Docker for development for years now and haven't
experienced this EXCEPT with some slowness I experienced with Docker Compose
upon upgrading to MacOS Catalina (which turned out to be bug with PyInstaller,
not Docker or Docker Compose). This is on a Mac, btw; I hear that Docker on
Linux is blazing fast.

I personally would absolutely leverage Docker for the setup being described
here: multiple versions with lots of environmental differences between each
other. That's what Docker was made for!

~~~
jsmeaton
The build step for installing or upgrading a package can be a killer with
nontrivial projects.

~~~
maksimum
It seems like long builds are either (a) necessary or (b) user error. (a) If
you have a tree of dependencies and you change the root, you should rebuild
everything that depends on it to make sure it's still compatible. (b) if you
placed your application into one of the initial Dockerfile layers, but then
you're installing dependencies that don't depend on you, it's user error.

What's the situation where your application needs to go first in the
Dockerfile, and then you need to put a bunch of stuff that doesn't depend on
your application?

------
gravypod
The Dockerfile that's provided looks like it would be very slow to build. I
always try to make Dockerfiles that install deps and then install my python
package (usually just copy in the code and set PYTHONPATH) to fully take
advantage of the docker build cache. When you have lots of services it really
reduces the time it takes to iterate with `docker-compose up -d --build`-like
setups.

------
analog31
In addition to the popular conda, it's worth checking out WinPython for
scientific use. Each WinPython installation is an isolated environment that
resides in a folder. To move an installation to another computer, just copy
the folder. To completely remove it from your system, delete the folder.

I find it useful to keep a WinPython installation on a flash drive in my
pocket. I can plug it into somebody's computer and run my own stuff, without
worrying that I'm going to bollix up their system.

------
ninetax
Curious to hear other's experiences with pipenv vs poetry. Has anyone made the
switch?

~~~
kndjckt
I switched from pipenv to poetry over 1 year ago. I love it!

The main reasoning was so that I could easily build and publish packages to a
private repository and then easily import packages from both pypi and the
private repository.

Happy to answer more questions.

~~~
thehesiod
I'd like to use poetry however ran into
[https://github.com/sdispater/poetry/issues/1554](https://github.com/sdispater/poetry/issues/1554)

We have a custom pypi server and need all requests to go through it, however
haven't figured a way to make poetry always use our index server for all
modules instead of pypi.org

~~~
_AzMoo
Add a second source and it will prioritise that over pypi.

    
    
      [[tool.poetry.source]]
      name = "my-repo-name"
      url = "https://myrepo.url/"

------
perlgeek
> On Linux, the system Python is used by the OS itself, so if you hose your
> Python you can hose your system.

I never manged to hose the OS Python on Linux, by sticking to a very simple
rule: DON'T BE ROOT. Don't work as root, don't run `sudo`.

On Linux, I use the system python + virtualenv. Good enough.

When I need a different python version, I use docker (or podman, which is an
awesome docker replacement in context of development environments) +
virtualenv in the container. (Why virtualenv in the container? Because I use
them outside the container as well, and IMHO it can't hurt to be consistent).

------
xchaotic
I love Python syntax, but I still haven't found a sufficiently popular way
that can deploy my code in the same set of setting s as my dev box (other than
literally shipping a VM). So setting up a dev env is one problem, but
deploying it so that the prod env is the same and works the same is another.

------
ausjke

        python -m venv venv
        source venv/bin/activate
        pip install -U pip
        pip install whatever
        # <do you stuff here>
        deactivate
    

no need any third-party tools, venv is built-in the above steps always worked
for me out of the box.

~~~
snypox
But how about replacing all of these commands with two words? poetry install

~~~
ausjke
which is another layer of abstraction and dependency that I do not really
need, e.g poetry no longer maintained, poetry(or whatever) has an urgent
bugfix,etc

------
Lucasoato
This article is great, those are viable solutions for sure. One of the
alternatives is conda: it's common among data scientists, but many of its
features (isolation between environments, you can keep private repository off
the internet) meet enterprise needs.

------
eximius
I would generally reach for conda instead of this, but they seem quite
comparable in aggregate.

And, given that I've been trying NixOS lately and had loads of trouble and
failing to get Conda to work, I will definitely give this setup a try.

(I haven't quite embraced the nix-shell everything solution. It still has
trouble with some things. My current workaround is a Dockerfile and a
requirements.txt file, which does work...)

------
rullopat
I like Python has a language, but when I see how clean are the tools of other
similar languages, for example Ruby, compared to the clusterf __k of the
Python ecosystem, it just make me want to close the terminal. I 'm always
wondering how it became the language #1 on StackOverflow.

------
notus
I recommend asdf for version management if you use more than one programming
language

~~~
pritambaral
[https://common-lisp.net/project/asdf/](https://common-lisp.net/project/asdf/)
?

~~~
rhizome31
[https://asdf-vm.com/](https://asdf-vm.com/)

------
anonu
There are two things that I find a bit elusive with Python:

1\. Highlight to run 2\. Remoting into a kernel

Both features are somewhat related. I want to be able to fire up a Python
Kernel on a remote server. I want to be able to connect to it easily (not
having to ssh tunnel over 6 different ports). I want connect my IDE to it and
easily send commands and view data objects remotely. Spyder does all this but
its not great. You have to run a custom kernel to be able to view variables
locally.

Finally, I want to be able to connect to a Nameko or Flask instance as I would
any remote kernel and hot-swap code out as needed.

------
luord
So far using docker and setup.py files is working for me, I've never felt they
were particularly slow, so I'll keep using them.

I gotta give poetry a try, though.

------
eivarv
Why not just use conda for envs and deps (or env-specific pip), and install
youtube-dl etc. via your platform's package manager?

~~~
cosmic_quanta
In my experience, conda breaks quite often. Most recently, conda has changed
the location where it stores DLLs (e.g. for PyQt), which broke pyinstaller-
based workflows.

In principle, it's a good idea; in practice, I'm not satisfied. On Windows,
it's an easy solution, especially for packages that depend on non-python
dependencies (e.g. hdf5).

------
nsomaru
I’ve been manually deploying my projects for years.

Can anyone comment on the Docker learning & troubleshooting story for python?

~~~
maksimum
Docker + setuptools/pip + python is great for development and production.
Docker is definitely worth learning, and is pretty easy to learn.

------
snorkasaurus
I like pip-tools for venv requirements management, but I don't see it
mentioned much.

~~~
globular-toast
I use pip-tools. It fits in nicely as an additional component to the standard
toolset (pip and virtualenv). But most people probably do not need to freeze
environments so it's great to be able to _not_ use it for most projects.

------
frou_dh
My sole use of Python is writing plugins (mostly single-user: me) for Sublime
Text.

It feels pretty comfy to effectively be on an island and far away from the
hustle and bustle of the industrial Python tooling.

------
dang
Related from 2018:
[https://news.ycombinator.com/item?id=16439270](https://news.ycombinator.com/item?id=16439270)

------
schainks
I've moved to ASDF and haven't really looked back. It's working well with low
fuss, and supporting far more than just python on my machine.

------
kovek
I'll pay anyone who can assist me with my Python setup. Is there a service
like this, where one can find a developer on demand?

~~~
abcininin
I have been using a consistent setup that hasn't yet failed me for the past 2
years.

1\. Install Anaconda to your home user directory .

2\. create environment using (conda create --name myenv python=3.6) .

3\. Switch to the environment using (conda activate myenv) .

4\. Use (conda install mypackage), (pip install mypackage) in that priority
order .

5\. Export environment using (conda env export > conda_env.yaml) .

6\. Environment can be created on an other system using (conda env create -f
conda_env.yaml) .

Anaconda: [https://www.anaconda.com/distribution/#download-
section](https://www.anaconda.com/distribution/#download-section) .

Dockerized Anaconda: [https://docs.anaconda.com/anaconda/user-
guide/tasks/docker/](https://docs.anaconda.com/anaconda/user-
guide/tasks/docker/) .

~~~
bmer
I can vouch for this. Anaconda is especially good for simulation/data stuff
(based on the focus on which packages are included by default).

One pain point though: getting it to work with Sublime Text 3 requires you to
set the `CONDA_DLL_SEARCH_MODIFICATION_ENABLE` environment variable to `1` on
Windows.

Not a flaw of Anaconda: it just pays attention to how to with multiple Python
installations on Windows.

------
mrfusion
Why pipx vs just using pip?

~~~
tedivm
With pipx when you install things they go into isolated environments. With pip
you're just installing things globally.

This difference is important due to dependencies- if you have two different
CLI tools you want to install but they have conflicting dependencies then pip
is going to put at least one of them into an unusable state, while pipx will
allow them to both coexist on the same system.

~~~
AdrienLemaire
I haven't used pipx, but as far as I understand, pipx = pip + venv. If your
pip executable is in a virtualenv, the "globally installed" is locally
installed.

pipx, poetry, pipenv and co are still nice wrappers to have, I suppose. It
just feel less useful now that most of my projects are dockerized.

~~~
yrro
pipx looks nice. Is there any way to persuade it to install 'wheel' before it
installs the desired package?

That way 'pipx install foo' can download and install wheels rather than
downloading source distributions and building/installing them...

------
diminoten
> Governance: the lead of Pipenv was someone with a history of not treating
> his collaborators well. That gave me some serious concerns about the future
> of the project, and of my ability to get bugs fixed.

Doesn't seem fair. You're not abandoning requests, are you?

~~~
oefrha
Just noticed requests moved from kennethreitz/requests to psf/requests.
Interesting.

Edit: From
[https://www.python.org/psf/github/](https://www.python.org/psf/github/),

> ... we have created a GitHub organization, @psf, to support and protect
> projects that have outgrown ownership by their original author.

------
mlthoughts2018
This is so painful to see compared to using conda.

~~~
whalesalad
1\. The author of this post helped to create the Django framework and runs a
successful Python consultancy.

2\. Conda is not used as much as you might think... it's really only used
within the data science community.

~~~
mlthoughts2018
1\. Argument from authority doesn’t mean anything to me. I also don’t believe
creating Django or running a Python consultancy endow someone with especially
useful opinions of Python packaging tooling. (Not that the author isn’t
knowledgeable, just you seem to think there’s an A implies B relationship
between those two items and having good opinions about Python packaging, and
there’s not).

2\. Conda is quite widely used outside of data science. It’s for example part
of Anaconda enterprise offerings used by huge banks, government agencies,
universities, etc., on large projects often with no use cases related to data
science. Conda itself has no logical connection with data science, it’s just a
package & environment manager.

In each of my last 4 jobs, 2 at large Fortune 500 ecommerce companies, conda
has been the environment manager used for all internal Python development.
Still use pip a lot within conda envs, but conda is the one broader constant.

~~~
m000z0rz
> huge banks, government agencies, universities

> large Fortune 500 e-commerce companies

Sorry, but argument from authority doesn't mean anything to me.

In all seriousness though, you literally did not provide any logical reasons
to think conda is better.

~~~
mlthoughts2018
Giving a counterexample is not argument from authority. I did not respond to
the parent comment to discuss any feature of conda, only to dispel the wrong
claim that only mostly data science projects rely on it.

