
Python without an operating system - Macha
http://lwn.net/SubscriberLink/641244/5d1d6d20aeb0a647/
======
amyjess
Next step: Python _as_ an operating system. Like Chrome OS or Firefox OS, just
with Python instead of a web engine. I'm a big enough Python fan that the idea
really intrigues me.

(also, everytime I see an LWN article here, I keep meaning to subscribe, but I
mostly read HN at work, and I usually forget by the time I get home... I feel
really bad about that)

~~~
Macha
That would be interesting. This isn't quite freestanding Python, in that it's
running on Grub so there is a lower layer, but it's still cool despite that.

~~~
amyjess
Well, even Chrome OS and Firefox OS don't have their own kernels; they run on
top of Linux.

If you were to build a Python OS, you'd probably start by writing PID 1 in
Python. It would be a Python init, starting services written in Python,
running gettys written in Python, and you'd log into a shell written in
Python.

Hmm... I can actually see myself writing an init system in Python... (hell, I
did exactly that at my last company.... I'd have to rewrite it from scratch,
but that's no big deal...)

Great, I think I've found a new rabbit hole.

~~~
voltagex_
Message me if you start this!

~~~
pygy_
[https://github.com/justincormack/ljsyscall](https://github.com/justincormack/ljsyscall)
is just that, but for Lua.

FFI to the kernel on Linux, NetBSD Rump kernels and even bare on Xen.

------
rootbear
I seem to recall that the "Pi" in Raspberry Pi was originally "Py" and that
the idea was that the system would boot straight into Python, similarly to how
early personal computers booted into BASIC. This idea was later abandoned in
favor of going with a full operating system.

~~~
duskwuff
There are a few real devices which boot up into interpreters, like the
Espruino, which runs Javascript (!):

[http://www.espruino.com/](http://www.espruino.com/)

~~~
collyw
Ugggh. I love the idea, and its great that JavaScript is everywhere, by why
does it have to be JavaScript?!?!

~~~
justincormack
There are quite a few devices that boot into Lua, see
[http://www.eluaproject.net/](http://www.eluaproject.net/) plus the Tessel
really does and just compiles to js, and there is Lua firmware for a wifi
module [https://github.com/nodemcu/nodemcu-
firmware](https://github.com/nodemcu/nodemcu-firmware)

~~~
semi-extrinsic
The NodeMCU is an amazing piece of kit. I'd say it's the future for DIY
"internet of things". I mean, a microcontroller with GPIO, spi, i2c etc. and
built-in wifi hardware, running Lua, for $9. You can throw together "small
screen showing live information from the internet" or "sensor uploading data
to internet" for less than $20 total cost. The ecosystem is still very new,
but it's growing really fast.

~~~
rasz_pl
> for $9

$3 with free shipping

~~~
semi-extrinsic
Link plz!

~~~
duskwuff
Ebay, search for "ESP-01".

------
e12e
There's also [https://micropython.org](https://micropython.org) \- for even
more embedded/python on microcontrollers. Maybe one can run python on the
network card rom too?

~~~
snops
>Maybe one can run python on the network card rom too? The network card ROM
isn't microcontroller code, its just x86 code that runs on your main
processor, called by the BIOS[1]. You could potentially hence have the OPs
project run from the BootROM, if it would fit. It might be slightly tricky, as
it runs as part of GRUB, which doesn't expect to find itself running from
there.

Alternatively, the article mentions: > Environments like Mirage OS (and other
"just enough operating systems") could also add Python using the BITS code
without too much difficulty, he said.

You could therefore probably add this code to iPXE [2] without too much
trouble to get python on your BootROM, though I'm unsure how much of the C
standard library you would need to borrow or reimplement. By the looks of it,
iPXE doesn't have threads or local storage support, so you would be pretty
limited in usability without a lot of work.

[1]
[http://networkboot.org/fundamentals/](http://networkboot.org/fundamentals/)
[2] [http://ipxe.org](http://ipxe.org)

~~~
Macha
> You could potentially hence have the OPs project run from the BootROM

Just pointing out as the OP that I'm not the author of the project. I just
came across the article on r/python

------
JoshTriplett
Author and presenter here; happy to answer any questions people might have.

~~~
raverbashing
Nice Work!

So maybe you can answer me this, how is memory management done in this
environment? Are you free (heh) to do malloc and free or did you have to
provide your own implementation? Is GRUB2 already in protected mode or there
were some hacks needed on that area?

~~~
JoshTriplett
For memory management, GRUB2 fetches the memory map from the firmware (e820
under BIOS, BootServices::GetMemoryMap under EFI), and hands the usable memory
regions to its memory allocator to allocate from. GRUB2's memory allocator
then provides malloc, memalign, and free. (It also has some special functions
for things like allocating low memory, needed for some system operations such
as the CPU setup mentioned below.)

In the BIOS version, GRUB2 is entered in 16-bit real mode, and it transitions
to 32-bit protected mode; it then has its own memory map. In the 32-bit and
64-bit EFI versions, the firmware transitions to 32-bit or 64-bit protected
mode before calling GRUB2. (In all cases, there's no actual protection and the
segment bases are all 0, though 64-bit mode does actually require a page
table.)

However, GRUB2 normally only runs on the boot CPU ("BSP", bootstrap
processor). We have to initialize the non-boot CPUs ("APs", application
processors) ourselves. To do that, we allocate an aligned region of low
memory, copy some bootstrap code and data into that memory, and send each
processor an INIT (to reset them) and a startup inter-processor interrupt
(SIPI) that forces them to jump to that address in low memory. We then
transition them from 16-bit real mode to either 32-bit or 64-bit protected
mode ourselves. (In the case of 64-bit protected mode, we set up paging by
using the same page-table address that the firmware set up on the BSP, so that
they all have the same memory map.) With that done, the AP can then run C
code. Then we put the AP to sleep using mwait, waiting for work to do. When we
want them to do something, we hand them a function pointer and a pointer-sized
parameter.

------
de_Selby
The presentation itself is on youtube[1], I'm glad I watched it, because I
missed this sentence when reading the article at first:

>The presentation and demos were all done in the BITS environment, so, in
reality, the whole presentation was a demo, he said to a round of applause

Pretty impressive!

1
[https://www.youtube.com/watch?v=bYQ_lq5dcvM](https://www.youtube.com/watch?v=bYQ_lq5dcvM)

~~~
JoshTriplett
Thanks! That was definitely the most fun part of giving the talk; several
people told me afterward that that was the point where the stuff I'd been
talking about suddenly became real (accompanied by various degrees of "wat"),
because it wasn't just talking to some abstract "hardware and firmware", it
was drawing on the screen, and that made it _real_. And then following that up
with Python-generated fractals really got people thinking about the old PEEK
and POKE days.

My first programming environment was C under DOS, where you could create a
pointer to video memory at 0xA0000000 or 0xC0000000 and start scribbling on
the screen. I learned from a book ( [http://smile.amazon.com/Microsoft-C-
Programming-Robert-Lafor...](http://smile.amazon.com/Microsoft-C-Programming-
Robert-Lafore/dp/0672226618) ) which had an appendix showing how to draw the
Mandelbrot set directly to video memory, so this was really nostalgic for me.

------
rdtsc
There is similar "Erlang as an OS" project

[http://nerves-project.org/](http://nerves-project.org/)

It is used for embedded systems but can work on a larger system too I suppose.

Basically it is just a kernel (for drivers) and on top (init) is Erlang VM.

The interesting aspect is an Erlang process is similar to a modern OS process
-- has an isolated heap. So it kind of maps elegantly to an safe embedded
multi-process OS.

Find the Erlang Factory talk by Frank Hunleth for more details.

~~~
tlo
Interesting, reminds me of HaLVM (Haskell on Xen) and Mirage OS (OCaml):

\- [https://github.com/galoisinc/halvm](https://github.com/galoisinc/halvm)

\- [http://www.openmirage.org/](http://www.openmirage.org/)

------
leephillips
Fortran predates the first examples of what we now think of as operating
systems. After turning on the machine you fed in a deck of cards that
comprised the Fortran compiler.

------
voltagex_
[http://biosbits.org/](http://biosbits.org/) and
[http://biosbits.org/downloads/bits-1154.zip](http://biosbits.org/downloads/bits-1154.zip)
\- no Git repo yet.

~~~
JoshTriplett
Working on that.

~~~
voltagex_
Thanks for your hard work on this project - it's really really cool.

------
bane
You know, how awesome would it be if a RPi booted up directly into Python in
the same way old 8-bits booted up into BASIC? This could really help it better
achieve the goal of being an educational instrument for young students.

~~~
bduerst
First thing I thought about - though if you're looking to do this with a
Raspberry Pi as a purely educational tool, it may be easier just to create the
pure-python tool as a layer on top of linux.

~~~
JoshTriplett
Depends on what you're trying to do; both are useful. Booting to Python under
Linux gives you an exploratory environment with access to operating system
services, but not necessarily one that lets you poke directly at hardware,
unless you put Python in the kernel. (And even then, the kernel and drivers
are doing things behind your back.) BITS gives you direct access to hardware
and firmware, and it's intentionally single-tasking, running only what you
tell it to.

So if you want a comfortable Python environment with more services and the
ability to use the full power of Linux, you would want to boot to Linux and
run Python (possibly with some extra modules); if you want raw access to
hardware and firmware, but without any OS (with all the advantages and
disadvantages that implies), you want BITS.

------
chocolatebunny
Holy cow. I've been looking for something like this. I'm currently trying to
do board level testing on an embedded project with Python and the sysfs
infrastructure in Linux. So far it's been decent but there are limitations to
what I can do, such as testing all the ddr address lines.

~~~
JoshTriplett
Raw access to memory, MMIO, IO ports, registers, ACPI, and EFI should be
pretty straightforward. If there's something you're trying to do that requires
access to anything we don't expose in BITS, especially if it would require
writing any new C code, please feel free to send us a patch, or just drop us a
mail and we'll put it on the list to add.

------
dingdingdang
If this was implemented with direct access to GPU as well then a new wave of
scene.org creations may arrive!

~~~
JoshTriplett
Take a look at the video:
[https://www.youtube.com/watch?v=bYQ_lq5dcvM&t=26m15s](https://www.youtube.com/watch?v=bYQ_lq5dcvM&t=26m15s)

We have direct access to video memory in graphics mode. Have fun.

------
phazmatis
This seems to be a trend - python, before it erlang-on-metal. Is there a
stable higher-level language runtime that runs on bare metal (that I can boot
in a VM)? Or, better yet, something that compiles arbitrary LLVM code into
.ISOs for deployment?

------
asgard1024
These modern "GRUB + interpreter" attempts reminds me of CP/CMS. CP was a
hypervisor and CMS was a small single-user operating system that runs on top
of it (scriptable in REXX).

------
cmdrfred
After all these years people keep finding new things to do with Python.
Amazing. Maybe I can take a stab at writing my own python os.

------
voltagex_
Once TCP and hopefully "requests" support is in there I can see this competing
with iPXE - load and boot images from HTTP.

------
vigile_
Does it mean that we gonna have a new smart generation of virus written in
Python now? My my... ;-)

------
nqzero
python's a great operating system, all it lacks is a decent programming
language

------
andrewchambers
pretty sweet, could this make things like bootloader menus easier to
implement?

~~~
JoshTriplett
It did, yeah. We're generating GRUB menus directly from Python, so that we can
dynamically construct menus of tests or lists of firmware tables.

GRUB's own scripting language is _mostly_ up to the task of enumerating OSes
and displaying boot options, though.

