
How to set up a perfect Python project - pedxing128
https://sourcery.ai/blog/python-best-practices/
======
nabdab
I still struggle to see the point with every project that starts out using
pipenv. Python -m venv & activate & pip install -r requirements.txt works more
often(every time for me) and is faster. I have literally not once seen a
project where the pipenv solution was better, but I’ve seen 12 where it
straight up failed to install correctly and all others it was slower by a
large factor.

For me pipenvs only selling point is that some people don’t know about python
-m venv.

~~~
mixmastamyk
Most projects don't need either.

I only use pipenv on a work project with a hundred dependencies, and venv on
one other large side project. Everything else is installed with --user,
especially dev tools like black and pyflakes. The user folder gets cleaned up
every few years when you upgrade to a new version of python (rm
~/.local/lib/python3.5).

Has worked fine for a decade or two, no conflicts. I think some folks get
spooked at python packaging and overcompensate in response, but it's pretty
easy to troubleshoot when you've learned how it works.

~~~
pbecotte
This is only true in the limited case where you never have to share your code
with anyone. If you want to share it, setting it up so others can get going
without having to figure out which packages are missing is necessary. In that
case, you should probably use that same tool yourself to make sure that your
environment works the same as you expect others to use it.

~~~
mixmastamyk
Nope, share code all the time. That is what requirements.txt (and
alternatives) are for. Where they install them is up to them.

------
notus
I disagree with the usage of githooks for linting and testing. It discourages
frequenting committing which is a tenet of using git. Commit often, perfect
later, publish once
([https://sethrobertson.github.io/GitBestPractices/](https://sethrobertson.github.io/GitBestPractices/)).

Your pipeline should be responsible for ensuring things don't make it into the
master branch that shouldn't. Run tests and linting on a pull request and make
it a requirement for those to pass in order to merge the PR. Otherwise one of
two things is going to happen:

1\. People will not commit frequently and you will have a commit history that
isn't as granular as it should be.

2\. People will just bypass the githook check

TBH I think some of this stems from people's fear of rebasing. I've heard so
many times, "But it rewrite's the history". Yes, but it's the history on YOUR
branch. Nobody should care what you to do your branch and it is yours to
perfect. You should want to rewrite the history of your branch in my opinion.
If you're committing frequently there will inevitably be some commits you want
to squash or fixup.

~~~
philpem
I agree with you. A better solution would be to use CI to check the branches
(so you know what state you're in) and enforce a test and style pass before a
branch can be merged to master.

I don't know why there's such a fear of rebasing, in a lot of situations it's
the right way to go.

I don't like squashing though, but that's a personal choice - sometimes
excessive use of squash can hide the history of why things developed the way
they did, and that can sometimes be very useful information.

------
dfee
I don’t think any of this is wrong or bad, but it feels a bit outdated. Black
and ISort are great, but rather than a setup.cfg, how about using a
pyproject.toml - and maybe swap Pipenv for something like poetry.

Also, huge fan of pytest (as is mentioned) and mypy (not mentioned, but quite
helpful).

[edit] mypy is indeed mentioned! I must’ve missed that on first pass.

~~~
vonseel
I haven't looked into poetry, but your comment makes me shake my head and
wonder... what's wrong with setup.cfg to begin with? Another damn file format?
What?

~~~
dfee
Well, I hope your wandering ends here, at PEP-518 [0], which addresses your
wonder directly.

[0] [https://www.python.org/dev/peps/pep-0518/#sticking-with-
setu...](https://www.python.org/dev/peps/pep-0518/#sticking-with-setup-cfg)

~~~
vonseel
I’ll check it out. I will admit I like being able to combine several configs
into a single setup.cfg or package.json, but I’m usually the guy who has a
separate config file checked in for every tool he’s using.

------
prepend
I wanted to like black but can’t get over the double quote default. I like
that they justify it with the less ambiguity and the ability to not have to
escape single quotes. I even like their proposed fix of just code with singles
and reformat into double.

But I don’t like unnecessary different levels of reality (code one way, commit
another) and hitting shift quote all those times sucks.

It’s funny how an “opinionated formatter” sounds great in theory until you
realize the opinion is “wrong.”

~~~
caymanjim
I'm not sure when single-quoted strings became the default (at least in
Python/Ruby/similar languages), but it's almost entirely about fashion rather
than functionality. Double-quoted strings were the norm for literally decades
until recently. There is no strong argument for favoring one over the other.
So long as everyone uses the same thing (at least within a given project, but
ideally within an entire language's work), everything is fine. It's still hard
for me to break the double-quote habit because single-quoted strings were
unheard of in my entire career until about ten years ago.

~~~
prepend
I used double quotes for years, but appreciate being able to type more quickly
with single quotes. It’s 90% convenience and 10% less clutter since single
quotes are just smaller.

It’s not a big deal to me as I don’t care too much (similar to tabs vs space).
I have a preference but would likely spend zero time caring if my team were
trying to decide and would never in a million years try to change someone
else’s project.

------
physicsguy
Oh look, yet another project I’ve not heard of (pipx) that I’m being told to
use to manage the clusterfuck of Python package management.

------
mmmrk
> Git hooks with pre-commit

In projects where I am involved in, we usually set up hooks that run a few
checks from [https://github.com/pre-commit/pre-commit-
hooks](https://github.com/pre-commit/pre-commit-hooks) plus
[https://github.com/asottile/pyupgrade](https://github.com/asottile/pyupgrade)
plus [https://github.com/psf/black](https://github.com/psf/black) plus maybe
the mypy and flake8 hooks. These hooks are also run in a "lint" stage by by
tox, which is also run on the CI system. This catches some lints that slip
through every now and then, before you go on to commit them. Also, a tox run
will as a side effect also reformat the code before you check it in, which is
a nice laziness feature.

I am currently thinking if I should move to the pytest plugins for mypy and
flake8 because I find they make more sense as tests, especially if I'm
developing inside a venv and repeatedly run pytest instead of tox. That would
also mean that the mypy and flake8 checks are run repeatedly for every Python
interpreter I want to tox, unkess I invest some time into coming up with
better TOXENVs. Hm.

~~~
wisemanwillhear
We use Flake8, PyLint and PyTest on an internal engineering tool / internal
automation libraries with about 200,000 lines of code, including test code. I
would love to use Git hooks, but long before the project grew this big the
time it took to run our code checks was unacceptable to enforce with hooks.

The fastest I've seen them run recently is 3 minutes. With about 800 tests,
PyTest take a minimum of 45 seconds just to do test discovery--the tests
actually run faster. We started with Flake8 and have been slowly adding PyLint
checks, but each check (so far) has added a little more time. We diff the
local "feature" branch against the remote dev branch and only run PyLint
against the differences, but it still takes 1 min on average to run--
experiments to run against a smaller set of files still result in coffee break
worthy execution times most of the time.

Judging from history, if we enforced the use of Git hooks to run these tools
it would result in mutiny with teams "secretly"\--their manager(s) being
onboard--working around and supplanting the tool. Small things like this
result in a lot of frustration and a loss of goodwill.

~~~
mmmrk
Interesting! How exactly do you run the tests and the linters then? Pytest
with flake8+pylint plugins? Separately? Serially? In parallel? Why flake8 and
pylint at the same time, did you find significant differences?

~~~
wisemanwillhear
We ask developers to make it part of their workflow, but the project is shared
across a 50+ engineers and few people do that. We enforce it via Jenkins
pipeline on every commit that get's pushed to our internal BitBucket Git
server. The Jenkins BitBucket plugin is used to enforce a successful build
before pull requests can be merged. It works, but it can be a little
frustrating at times to see an endless steam of alternating failed then passed
builds because few people run the checks before pushing.

We use a combination of Python Invoke and the flake8 & pylint CLI commands. At
some point we plan to reinvestigate our use of Python Invoke as it doesn't run
tasks in parallel--we were new to Python when we started this project two
years ago and didn't understand the limitations of the tools we were using. (I
should say, Python Invoke has worked out well for most of our developer
centric tasks.)

We run both Flake8 and Pylint as we started out with Flake8 and later tried to
use Pylint. The workload to fix the code base was too much for us as most
developers are QA people learning good software design and how to share a
large code base with their "extra", "volunteer" time. We've been slowly
turning on more Pylint checks and cleaning up the code base. When we did try
to move more quickly, we were deluged with complains about the cognitive load
it placed on people and backed off.

We did find some differences, but I don't remember what they were and couldn't
comment on the significance.

~~~
mmmrk
Thanks. It's a tricky topic, as Google has found as well:
[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en/pubs/archive/43322.pdf).

------
caymanjim
While I might choose slightly different tooling and configuration, this
closely matches my practices. In particular, I'm a big fan of the pre-commit
hook for linting/testing/etc. automatically and preventing a commit if they
fail (you can always override it if there's some rare emergency).

Another thing I like to do is create a skeleton project and keep it in Git, so
that this stuff doesn't have to be done manually for each new git project. You
can check out/fork/clone/whatever the skeleton and have most of this stuff
pre-configured, including e.g. a baseline requirements.txt for whatever
frameworks you use.

You can also put the pre-commit stuff into a ~/.git_template so that newly-
created repos already have it configured. It's a bit harder to manage if you
work with multiple unrelated languages/frameworks, but can be a real time-
saver in a uniform environment.

------
namelosw
The most downside of Python is this. Setting up the environment and dependency
management is very confusing and hard to start in the first place. Especially
for Python there are a lot of non-programmer users.

It seems to be easier to do right in the first place than fix everything.
JavaScript yarn is really good but only based on npm already done most things
okayish.

A good news is that modern languages like Go, Rust, Elixir, etc designed more
carefully with this, providing much better experiences.

------
Rainymood
This is part satire part rooted in deep truths and I am conflicted about this.

------
mikedh
I guess for some value of "perfect" and "best practices."

------
vonseel
Pylint is a much better linting tool than flake8.

------
coleifer
What a bunch of shit. This is mere performance art. All you need is a setup.py
and you're ready to package up a basic python project. Unbelievable.

~~~
growse
Article introducing useful tools and how to set them up = "Performance Art"?

Contrarian-for-the-sake-of-it and ultimately contentless comment = "Not
Performance Art"?

Got it.

~~~
coleifer
It seemed easier than vomiting.

~~~
alazel
So very constructive, thanks for the input.

