Generally I am quite happy with this approach, but I am not wild about the cost/benefit of writing tools in Smalltalk. I've spent a lot of energy on treading water and trying to keep things working while the environment is being rewritten underneath my feet.
I have also found that Smalltalk works best when data is imported into the Smalltalk heap, and represented by a dedicated model classes, and also represented by dedicated view objects (Bloc.) That's a damned lot of classes to write and the whole image can grind to a halt if I reasonably want to inspect millions of objects. This makes me miss simpler and more scalable data structures like R data frames.
Going forward I plan to take a more perl-inspired "there's more than one way to do it" approach and complement the Smalltalk code with e.g. R, Lua, etc. I expect to spend more time writing code generators (a la SWIG) rather than hand-crafting everything in Pharo.
I also think your focus on making performance more predictable and reducing the mental burden on devolopers who want to write reliably fast code is exactly the right one; even a version of luajit that's half as fast on average but more predictable could be more useful in practice to most people.
Having said that, to keep some already precarious and fringe technology alive by adding as core part of the toolchain an app written in an even more fringe and decidedly more fiddly[*]/unstable language like Pharo, and one which has basically zero community overlap with luajit users seems... bold.
As does moving forward by adding R/codegen targetting pharo to the mix.
I'm viewing the tooling as a separate project. RaptorJIT is one application that it supports and Pharo is one UI.
(You could also think of codegen as a bridging strategy in case the tooling should be ported to some more conventional environment(s) once it is mature e.g. CLI, web, emacs, etc.)
EDIT: Should also note that the Studio tooling can be run via Docker in pure text mode, converting the binary diagnostic data into ascii that's similar to but richer than the classic LuaJIT '-jdump'. So users don't need to use the GUI, or setup any of the dependencies, or even learn different ways of working.
If the quality (and acceptance) of the tooling is pretty important to the success of RaptorJIT and the tooling is unlikely to be used for anything else than RaptorJIT (e.g. pypy or at least another luajit fork), they are not really _that_ separate.
I don't think codegen is in itself bad, but R -> Pharo so you can work on some low level lua JIT stuff – you're mightily reducing the set of people who don't need to learn extra tech to be able to contribute (from an already small pool of people who have the relevant skills and inclination to improve a JIT compiler).
Here we are talking at cross-purposes. I view the tooling as an extensible framework like Emacs or Eclipse. I am talking about my intentions for extending the diagnostic tooling to support other projects beyond RaptorJIT.
Over time I would anticipate reducing the number of moving parts in the RaptorJIT tooling.
The value of a "there's more than one way to do it" approach is to make it easier to support new projects besides RaptorJIT using whatever tools that project's community are comfortable with (which, as you say, probably won't be Pharo but might be R/python/perl/emacs/etc.)
Not going to happen. Too many people have sworn to keep improving it.
For the project to have a healthy future you really need a community of user-developers who are supported by multiple organizations and cooperating and coordinating with each other. LuaJIT doesn't currently have such a community and bootstrapping one is a doosey.