
Bowler: Safe code refactoring for modern Python - jxub
https://pybowler.io/
======
nuclear_eclipse
Author here: I created Bowler as a "hackamonth" project when I joined
Facebook's internal Python Foundation team. We've already used it for a bunch
of random codemods that touch a large number of source files throughout our
codebase.

Happy to answer any questions you might have!

~~~
loeg
[https://pybowler.io/docs/basics-setup](https://pybowler.io/docs/basics-setup)
mentions a facebook/bowler github repository, but no such repository exists
(or it is private). Where does the source code live?

Edit: Found it — it's currently at
[https://github.com/facebookincubator/bowler](https://github.com/facebookincubator/bowler)
instead.

~~~
nuclear_eclipse
Good catch, I'll get that fixed! [edit: fixed now!]

~~~
loeg
Excellent! Thanks for the quick fix.

------
typon
Could this be used to create a macro system for python?

~~~
hathawsh
I'm sure it could be used that way, but the compile() function and the ast
library (part of the stdlib) are a more direct way to create a macro system:

[https://docs.python.org/3/library/ast.html](https://docs.python.org/3/library/ast.html)

I've built macros using the ast library before and it works, but I also found
I could adjust my DSL's grammar just a bit and express the same thing with
ordinary Python code. That's why Python macro systems rarely gain much
traction.

BTW Bowler looks cool. I expect to try it next time I do a big refactor.

~~~
makmanalp
For bytecode level transformations rather than AST level:
[https://github.com/llllllllll/codetransformer](https://github.com/llllllllll/codetransformer)

------
pypypypypypy
This was first presented at PyconAU 2018 -
[https://www.youtube.com/watch?v=9USGh4Uy-
xQ](https://www.youtube.com/watch?v=9USGh4Uy-xQ)

------
MrSaints
This is akin to the other "codemod" facilities Facebook already uses for
large-scale refactoring in _busy_ codebases (particularly, their JS
codebases), but for Python
([https://github.com/facebook/codemod](https://github.com/facebook/codemod),
[https://github.com/facebook/jscodeshift](https://github.com/facebook/jscodeshift)).

~~~
nuclear_eclipse
Notably, or notoriously, our previous codemod project just used regexes, which
could result in ballooning complexity, especially when needing to modify code
that might include type annotations. Bowler was designed specifically to allow
refactoring against more complicated subjects, such as function signature
changes, where you lose all predictability in formatting at both the
definition and call site.

------
jsmeaton
I could imagine a library of transformers being created using this tool.

For instance we’ve just enabled the pep3101 Flake8 plugin which enforces
newstyle string formatting over % formatting. I’d love to see a transformer
that automates that refactoring.

~~~
nuclear_eclipse
This sounds like exactly the sort of transforms and codemods that Bowler was
designed to handle. Furthermore, we plan to support more linter-style
features, and would like to have it integrated with tools like Flake8 or
Phabricator, so that it can simultaneously find lints, and immediately suggest
modifications to resolve those lints.

------
ausjke
Is this to help migration from python 2 to python 3, or a tool to make python
3 better? it's unclear after I read it except it mentions it's based on 2to3.

~~~
jwilk
Bowler doesn't have much to do with 2to3, except that they have the same
library underneath.

lib2to3, despite the name, is a fairly generic refactoring library.

------
marmaduke
How well does this handle dynamic constructs? Eg if I rename a method and
elsewhere have

    
    
        if hasattr(obj, ‘foo’):
    

does that get caught or not?

~~~
nuclear_eclipse
No, it would not catch that, because it's not tracing elements through the
AST. It would be possible to do some amount of tracing, but it would be
impossible to catch and modify all variants of that, since Python is an
extremely dynamic language. Imagine something like:

    
    
        attr = "foo" if conditionA else "bar"
        obj = A() if conditionB else B()
        if hasattr(obj, attr):
            ...
    

Because of this, Bowler is focused on being more practical, and leans on the
expectation that an engineer will be in the loop and validate the resulting
diff, run unit tests, etc.

~~~
figgis
I apologize but I'm really having trouble understanding what the problem you
are referring too. In python syntax that would be written as -

    
    
        if ConditionA:
            attr = "foo"
        else:
            attr = "bar"
    
        if ConditionB:
            obj = A()
        else:
            obj = B()
    
        if hasattr(attr, obj):
            do_magic()
    

I haven't tested Bowler yet but I would think something that refactors code
would handle that pretty easily?

~~~
ndr
A() and B() could return two different classes that both support foo and bar
methods. If your refactor is to rename one of those methods for one of the
classes this code breaks and it's not trivial to update automatically, is it?

~~~
shoo
in general, given this is python, A and B could do something completely foul
like request data from the internets and `eval(...)` it then return the
result, or redefine `True, False = False, True` or redefine

    
    
      hasattr = lambda *args: False`
    

or something equally insane and difficult to statically analyse

> Eg if I rename a method and elsewhere have `if hasattr(obj, ‘foo’):`

Further, even in the absence of insane dynamic code, it probably isn't
possible to automatically make a correct decision without knowing the _intent_
of the original code.

For example, maybe we're renaming the `foo` method of `Fizz` class to `barr`,
and currently the program is written such that `is hasattr(obj, 'foo'):` only
happens to be called on `Fizz` instances. Is it the case that the programmer
only ever intended that code to be called on `Fizz` instances, or did the
programmer intend it should operate on all types that have a `foo` attribute,
including old-style `Fizz` instances and new types that might be encountered
in future? In the former case, we should rewrite the logic to `hasattr(obj,
'barr')`, in the latter case perhaps we should leave it unchanged, so it will
no longer trigger after the refactor.

i do agree it should be possible to automatically flag this as "here be
something interesting to consider! please intervene and make a manual
decision!"

~~~
marmaduke
I think these sorts of cases of obfuscated code are outside the scope of
refactoring tools. No one who’s looking to use a refactoring tool has to write
or read code that uses eval or redefines True.

------
Twirrim
I haven't had to use it before, but another tool in that space is "undebt"
from Yelp:

[https://github.com/Yelp/undebt](https://github.com/Yelp/undebt)

------
sandGorgon
Is it possible to use this in vscode ?

That would be awesome.

~~~
nuclear_eclipse
The possibility is there, but we haven't pursued any specific integrations
yet. For obvious reasons, we would most likely work on Nuclide integration
first, and something like VSCode support would probably need to come from the
community. As Bowler is primarily providing the API framework, most
integrations will likely just be plugins that make calls to the Bowler API, or
execute the bowler CLI, with appropriate queries.

------
linuxftw
Seems like more work to me. Simple search and replace is usually sufficient
for something like what the intro video shows.

~~~
SketchySeaBeast
I'm finding Visual Studio Code does an excellent job with "rename symbol" or
"change all occurrences".

~~~
drb91
Funny, I've found it seems to struggle with any references that cross language
boundaries.... except with typescript, for some reason.

EDIT: Sorry, I meant 'file' boundary, not 'language' boundary.

~~~
pvg
Why not, you know, edit the actual sentence instead of adding a thing that
says EDIT but leaves the original confusingly unedited?

~~~
yesenadam
Because, I guess, it's courteous to the people who already commented on the
original, so they don't look like they were talking nonsense.

~~~
pvg
You make the edit and then you describe the edit, to avoid that.

~~~
yesenadam
Ah, good point. (I guess I wasn't confused by the edit - though if I'd been
here longer I probably would've been - but I was by your comment!)

~~~
pvg
Wait, what? You can't just politely walk away from a from a pointless argument
over minutia like that. You _are_ new around here. These must extend to a
minimum thread depth of 12. Please review the guidelines.

~~~
yesenadam
Hahaha! Gold. Thanks, you made my day. Sorry you lost mana for it, I'd +100 if
I could. Well, I _was_ reading about
[https://en.wikipedia.org/wiki/Sayre%27s_law](https://en.wikipedia.org/wiki/Sayre%27s_law)
on here recently.

