
Exploit custom codecs to write inline C in Python - ssize_t
https://github.com/georgek42/inlinec
======
pcr910303
Wait... looking at this, doesn't this mean that by deliberately misusing the
`codec` module, meta programming at compile-time (somewhat like Lisp macros)
is possible in Python?

By using a custom parser, people are able to add custom syntax in python
(which is... undoubtedly unpythonic) with just a decorator(or a magic comment)
on front. I can't wait to start abusing this... :-)

Edit: Looks like a pattern matching syntax that hooks in pampy would be my
first hacking project :-)

    
    
        # coding: patmatch
        import patmatch
        
        input = [1, 2, 3]
        patmatch input:
            case [1, 2, x]:
                print(f"{x} is matched!") #=> 3 is matched!
        
        def fibonacci(n):
            patmatch n:
                case 1: return 1
                case 2: return 1
                case _: return fibonacci(n - 1) + fibonacci(n - 2)

~~~
ssize_t
Absolutely :) I "discovered" this when researching implementing a new language
on top of python. The decorator is completely unnecessary, so you can have a
complete language with custom features built entirely on top of the python
interpreter.

~~~
lapnitnelav
I've been toying with the idea of implementing some sort of DSL on top of
python but wouldn't really know where to start.

Would you have any pointers (heh) given the discovery you made?

~~~
ssize_t
I think the `#coding:` approach is promising. You have to fit your
parsing/transpilation into a pretty small time budget (which is a fun
challenge on its own) to keep the startup time hit negligible. The source code
gets reparsed serval times in certain cases, e.g. when printing a stack trace,
so its a good idea to have some sort of caching mechanism.

As far as parsing goes, if you want to stick with python I've had good success
with pyparsing [1], otherwise I have a strong preference to do language-
related things in OCaml with menhir [2]. I've toyed with wrapping the Python
parser in an OCaml library with decent success [3]. But, of course, unless
you're optimizing for fun, it's probably a good idea to stick with Python.

Another, weirder, approach is creating a language that happens to be able to
be parsed by the same grammar as python, then using a decorator (or similar)
to get the source code or the ast to reparse and transpile. As an example you
could have something like `a <\- b` be the syntax for an actor model message
receive dsl (which is valid python).

[1]
[https://github.com/pyparsing/pyparsing](https://github.com/pyparsing/pyparsing)
[2]
[http://gallium.inria.fr/~fpottier/menhir/](http://gallium.inria.fr/~fpottier/menhir/)
[3]
[https://github.com/georgek42/pyparser](https://github.com/georgek42/pyparser)

~~~
lapnitnelav
Thanks a mil for the tips.

------
tjwei
I wrote a similar piece of code called MagicCodec back in 2008, including a
zhpy codec(Chinese Python) and a Lua codec, which translates Lua source code
to Python. (Blog content in Mandarin Chinese, with sample code [http://weijr-
note.blogspot.com/2008/02/python-magiccodec-01....](http://weijr-
note.blogspot.com/2008/02/python-magiccodec-01.html)) There was a time that I
am slightly interested in Chinese Python for education purpose. Another
interesting approach is using ctypes to replace keywords in CPython
implementation, [http://weijr-
note.blogspot.com/2011/06/python-32-keyword.htm...](http://weijr-
note.blogspot.com/2011/06/python-32-keyword.html) (Also a blog post in
Mandarin Chinese with sample code, which is also in Mandarin Chinese)

------
agumonkey
Oh never heard of the # codec declaration

relies on
[https://www.python.org/dev/peps/pep-0263/](https://www.python.org/dev/peps/pep-0263/)
it seems (this is from 2001, maybe there's more recent)

~~~
est
I envy you py3 kids...

encoding is a huge problem in py2

~~~
franga2000
> encoding is a huge problem in py2°

Yes...exactly! Good thing that it's dead now, every meaningful library has
either been updated or replaced and nobody is using it anymore, right? Like
Windows 7! /s

Seriously tho, what's keeping you from updating? Genuinely curious at this
point...

~~~
est
> what's keeping you from updating? Genuinely curious at this point...

My boss doesn't pay the upgrade bills. Only adding new features are billed.
Sad.

And there are thousands of real-time users on the system and their task can
not be interrupted, also if there is any py2 and py3 mismatch, the outcomes
would be quite costy.

------
tyingq
Interesting feature abuse. Perl's had various Inline modules for some time,
but via a deliberate feature.
[https://www.perl.com/pub/2001/02/inline.html/](https://www.perl.com/pub/2001/02/inline.html/)

~~~
baryphonic
Indeed it is. Guido himself maintained a project called Pyxl3 [1] for a while
(forked from one at Dropbox during his tenure there) that abused this codec
feature to implement something similar to JSX within Python source.

[1] [https://github.com/gvanrossum/pyxl3](https://github.com/gvanrossum/pyxl3)

~~~
smabie
Kinda sad when you think that CL could implement this feature in a pretty
simple macro.

------
kilon
Python syntax can be heavily and easily manipulated via the ast module. An
example of this is hylang
[https://github.com/hylang/hy](https://github.com/hylang/hy)

------
smitty1e
This has got to be the least-pythonic thing I've ever seen.

~~~
bigiain
I read it and immediately thought "Hey, I remember doing this in Perl in the
late '90s or early 2000s!" \- very nostalgic...

------
delfick
I have also used this ability to make a dsl with
[https://noseofyeti.readthedocs.io](https://noseofyeti.readthedocs.io) (rspec
style "describe", "it" for writing python tests).

It's a cool technique, but quite a lot of effort and can break a bunch of
tooling (getting my codec to work with black for example was... fun)

------
jokoon
I guess one could also use a lightweight compiler like TCC and write a python
module with it. That's a good idea for a side project.

------
j88439h84
[https://github.com/metatooling/syntactic](https://github.com/metatooling/syntactic)

is a framework for creating custom Python syntax with the codec trick.

------
yellow_lead
Guess I'm adding Python to my resume. Nice work

------
tony1234
nice job

~~~
ssize_t
thanks!

