
Lua support in the NetBSD kernel - jboynyc
http://mail-index.netbsd.org/source-changes/2013/10/16/msg048283.html
======
lifeisstillgood
Holy crapola - they are putting a Lua interpreter in the kernel, and it has
access to some limited calls.

This is partly driven by "we need device drivers and no one understands C
anymore", partly because it's cool, but mostly because Moores law is still
alive and well. They are putting a scripting language in a kernel ! Forget
write your app in python / perl / Ruby then optimise in C - the raw power
argument is going to overwhelm us all.

I heard a stat the other day - that a greetings card, the kind that plays a
silly tune, has more processing power than all computers on the planet in
1960. And we should expect a similar growth in the next 50 years. Even if
that's out by two orders of magnitude just let it sink in.

I think we are at a Cambrian explosion period - where the goal is to try out
every possible new body shape, as fast as possible and see which ones get the
Darwin seal of approval.

New organisation forms are possible, some for the first time in human history,
new ways of thinking and communicating - it's stuff like this that makes one
realise the water around the frog is getting hotter.

(Still it's worth remembering that most train companies in 19C England failed
and the average return for stockholders was 10% - just because the world will
change beyond recognition does not mean industry stocks is a great return)

~~~
ketralnis
Kernel programming is an eternal struggle between people trying to pull more
into the kernel for performance, and people trying to pull stuff back out for
safety and modularity. We've all seen the arguments about the in-kernel HTTP
daemons and whatnot. I tend to be on the side that says that more things
should be pulled out.

> This is partly driven by "we need device drivers and no one understands C
> anymore"

I don't buy this argument. Who are the people

* Using NetBSD

* having cutting-edge hardware for which drivers don't exist

* deciding to write device drivers

* that have device specs/docs needed to write them

* that don't know/want to use C

* that would do it if only they could write it in a scripting language?

Do these people exist? I assert that the far more common case is that the
device specs, documentation, BIOS, etc. aren't accessible. If that's true,
they couldn't write the driver regardless of the language, C is hardly the
limiting factor here

~~~
tedunangst
A far better rationale is that it makes fast prototyping easier. You write
three lines of Lua, stuff into the kernel, and run. Did the blinkenlights
blink? Yes, good, now translate that code into the C driver.

~~~
koralatov
My worry in situations like that is that the `fast prototype' ends up also
being the final driver, because it works well enough and nobody feels the
urgent need to rewrite in C.

~~~
drdaeman
If it works and is sufficiently performant - is there anything wrong with the
fact it's written in X, not Y?

And if it doesn't, then there's certainly a reason to optimize.

------
anttiok
It's awesome to see this work finally hit the tree. Now, I cannot say I agree
with all of the motivations. For example, I do not believe that prototyping
should be done in kernel mode at all, nor do I believe that using Lua will
shield from incorrect operation or crashes. However, there is already proof
that device drivers can be written in Lua
([https://github.com/SnabbCo/snabbswitch](https://github.com/SnabbCo/snabbswitch)),
and it's great to see more push in that direction.

After all, why use C if you have a working alternative? We just need real
world experiments to see if Lua is a working alternative for the kernel.

------
simias
Anybody has a link that explains their motivations? There's a lot of
speculation in this thread but surely NetBSD being an open community there
must have been a public discussion before they merged that in?

I write linux drivers for a living and I must say it puzzles be a bit. There
are times C falls a bit short (if only by its syntax and idiosyncracies).
There are times I wish I could use something like a subset of C++ (for
templates and destructors mostly) or maybe Rust these days. But a scripting
language?

You can already write drivers in userland (using /dev/mem for instance). You
can already write device drivers any way you like that way. One of the main
limitations is IRQ handling, but do they propose handling IRQs in a scripting
language? Maybe I'm too close minded (or just brainwashed by years of C) but
it sounds like a recipe for madness.

I actually used lua to write a small userland test driver (it was convenient
because the lua interpreter is very small and easy to compile and cross-
compile).

It was quite painful really. There's no builtin bitwise operator (although
it's now in the standard library), all integers are stored as double floating
point values (floating point in the kernel is hardly ever needed and usually
very poorly supported). Using integers and bitwise operator is like all you do
in a kernel driver (ok, I'm exaggerating, but not by a lot).

I guess they might have changed the syntax a little bit (and use ints instead
of doubles for storing integers) but then it means their lua is not compatible
with the "real" lua so you can't prototype your code in userland with any
stock lua build.

I dug up my userland test driver code, here's what the naughty bits look like:

    
    
         function avi_unset(addr, val)
           local v = assert(devmem.read32(addr))
    
           v = bit32.band(v, bit32.bnot(val))
    
           avi_write(addr, v)
        end
    

For reference here's what it'd look like in C and its bitwise operators:

    
    
        void avi_unset(unsigned addr, unsigned val)
        {
          unsigned v = __raw_readl(addr);
    
          v &= ~val;
    
          __raw_writel(addr, v);
        }

~~~
pygy_
I won't address your general point, because, I never wrote any drivers, but
the following may appease some of your concerns.

Changing the number type in Lua is only a few #DEFINEs away. See
[http://www.lua.org/source/5.2/luaconf.h.html#LUA_NUMBER](http://www.lua.org/source/5.2/luaconf.h.html#LUA_NUMBER)
and the comment above it.

Regarding the lack of infix bitwise operators: usually, the functions are
cached in locals. It is not as readable as C, but less bulky than what you
wrote:

    
    
         local band, bnot = bit32.band, bit32.bnot
    
         function avi_unset(addr, val)
           local v = assert(devmem.read32(addr))
    
           v = band(bnot(v), val)
    
           avi_write(addr, v)
         end
    

If it became apparent that infix operators were needed, adding them to the VM
is trivial (There's a power patch for Lua 5.1, and a patch for Lua 5.2 was
recently posted in the mailing list). If you want to use the stock VM, you can
still use a preprocessor (which makes things a bit more cumbersome).

~~~
simias
I remember looking at this LUA_NUMBER but I wondered if changing it to an
integer type would break something, it's not explicitly mentioned that you can
use an integer type there. I'm sure it can be done but it still means that
your custom lua will behave a bit differently than the "stock" lua (especially
when it comes to things like roundings and overflows). That might be a bit
problematic if you prototype your code on one and then attempt to run on the
other.

As for your improved code I didn't know I could do that, it does look much
better indeed.

Again, maybe I'm just nitpicking but my main point was that lua, scripting
language or not, just didn't seem better suited than C at writing device
drivers.

That being said, having a good in-kernel REPL would revolutionize kernel
hacking, but it seems very hard to do right. Lots of kernel code is very
timing-sensitive and you can't afford to wait for user input when you're in a
critical path. Just look at the difficulty to implement proper kernel
debugging to get a taste of it.

The thing is, the NetBSD people know that and more and yet they decided to go
with it. I would be very interested to know their objective/motivation. I
googled a bit around and found this:

[http://netbsd-soc.sourceforge.net/projects/luakern/](http://netbsd-
soc.sourceforge.net/projects/luakern/)

The page mentions some use-cases (like power management and packet filtering)
but those can already easily be handed to the userland through the sysfs and
tun/tap interfaces for instance. I don't get what problem it's trying to
solve.

~~~
pygy_
LUA_NUMBER can safely be changed to an integer type, Lua is designed for
extreme portablity from the ground up.

If by stock Lua, you mean lua.c, it is partly intended as a minimal
demonstration of how you can embed Lua the library. The authors encourage
users to modify the VM to better fit their use case, and changing the number
type really is trivial.

 _> Again, maybe I'm just nitpicking but my main point was that lua, scripting
language or not, just didn't seem better suited than C at writing device
drivers._

As I said, I'm mostly unqualified to answer that. It's harder to shoot
yourself in the foot when using Lua, which may make things easier at the
prototyping stage.

Maybe having LPeg[0] could be of some use too.

[0] [http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html](http://www.inf.puc-
rio.br/~roberto/lpeg/lpeg.html)

------
almk277
It's good somebody even tries to break the old myth about C as an essential
kernel part. C is _not_ good for drivers: it lacks most of the appropriate
abstractions, just as facilities to create them. Once we switch from asm to C
- why not go further?

Speed matters, of course, but I suppose that even script speed is enough for
many drivers (and, just in case, we have luaJIT).

Don't sure if lua is a right choice, but it's an interesting experiment - and
this is what NetBSD is good for.

~~~
simias
> C is _not_good for drivers

You'll have to back up that assertion with some facts here, there's a whole
industry and millions of lines of code that contradict you.

~~~
almk277
It's just legacy. C as infrastructure is good, C as a language is bad. There's
a whole industry and millions of lines of code in visual basic, but I never
met somebody who says that VB is good.

Is there a way to say "do this under spinlock"?

Is there a way to handle errors not mixing it with main logic? Not wrapping
nearly every line with if/goto?

Is there a way to create a generic collection? No, linux list.h is not that:
it requires manual initialization, it does not work with elements, but with
their special parts, it's macroses requires logically unneeded information
(such as 'member' in list_for_each_entry), and so on. And this is the best you
can do in C.

C is too unobvious, too. Google for "wait4 backdoor".

C works, of course. It just requires too much boilerplate code and too much
programmer resources.

------
chipsy
Some discussion: [http://mail-index.netbsd.org/tech-
kern/2013/10/06/msg015632....](http://mail-index.netbsd.org/tech-
kern/2013/10/06/msg015632.html)

------
jarjoura
Isn't this similar to Windows wanting to rebuild their entire userland on top
of .NET?

~~~
pjmlp
Actually the best example is OS/400 mainframe system from IBM.

The kernel is a mixture of Assembly, C++ and Modula-2, with a JIT.

All user space code compiles to bytecode.

The kernel JIT compiles the applications on first execution. Afterwards the
application is validated against generated native code, and re-JITed again if
the bytecode has changed between executions.

So Microsoft's attempt was actually nothing new.

------
morenoh149
yay. I was just looking at learning lua to do corona development

------
ape4
The note about the copyright on the diffs seems a bit paranoid or something.

