
The microscope makers - lainon
https://www.nature.com/articles/d41586-017-07528-7
======
andbberger
Oh fun, I can speak to this. I was at Janelia for the last year and a half,
and actually just submitted a paper with Keller. I did ML stuff for
reconstruction and otherwise post-magics with the images.

There are a lot more people in that place working on scopes than just Keller
and Betzig! Whole institution is basically vertically integrated fluorescence
microscopy. You got your folks that work on making specific parts of mice or
fly brains glow, your folks that build the special scopes, VR rigs for the
mice to run around in while they're getting imaged...

IMO we are close the limits of what can be done with 'classical' microscopy,
classical meaning basically as practiced now. The real strides are going to be
made in sophisticated reconstruction techniques leveraging the latest and
greatest in ML and high fidelity simulations of the scope itself, which allow
us to get close to reconstructing at the physical limit of information
captured by the scope.

But even with current 'classical' microscopy such as Keller et. al's Simview,
the datasets we are collecting far exceed in complexity the neuro communities
ability to analyze them, in particular at Janelia. Amongst other frustrations
I have with the institution is the complete lack of interest in serious
theory. Instead its all grasping at straws with very expensive and
sophisticated straw grabbing apparati.

Weird place, amazing science nonetheless. But my advice for neuro people
considering a visit is to stay far away.

~~~
peppery
Your work with ML and images sounds incredible!

Do you have a link (e.g. on arxiv or elsewhere) that describes your approach
using ML for image reconstruction in greater detail? How would you recommend
building up one's combined intuition in optical theory, the relevant ML
techniques, and the biological substances themselves, to the level where you
can innovate in this task as you have done?

Also, for those interested in the concept of building better images using
higher fidelity simulations of the microscope itself, presumably Andrew meant
studies along these lines:
[https://arxiv.org/abs/1702.07336](https://arxiv.org/abs/1702.07336)

~~~
andbberger
That paper has the right general idea, you need a generative model of the
scope and if you're clever you can use that to improve your reconstructions.
But it's missing a couple key things that it workable in practice. One missing
ingredient is a probabilistic model of the thing you're imaging. The others
are secret ;)

Secret because I have a dream for a crazy startup based on this idea which I
don't have the means to do now. Although I generally hate being secretive
about, like, knowledge and for sure the value creation happens during
execution - but just humor me this time, ok?

> How would you recommend building up one's combined intuition in optical
> theory, the relevant ML techniques, and the biological substances
> themselves, to the level where you can innovate in this task as you have
> done?

Well, I'm flattered but I haven't done any substantial innovation here... My
recommendation would be to become a theorist, learn math and physics and work
your way up the hierarchy. You need to understand the whole picture, how it
works on each level, and how the levels fit together - then you can run
thought experiments. A good generative model of the world.

~~~
peppery
Thanks for your sharing your thoughts!

> One missing ingredient is a probabilistic model of the thing you're imaging

Ooh, this (making the prior for the true image signal more informative by
incorporating knowledge of the structure of the signal) is clever. Here, when
you say model, you mean a description that is based on the (bio)physics of
your sample? (E.g., knowing that objects being imaged obey diffusion equations
informs your maximum likelihood estimation of the true signal?)

> Secret because I have a dream for a crazy startup based on this idea which I
> don't have the means to do now. Although I generally hate being secretive
> about, like, knowledge and for sure the value creation happens during
> execution - but just humor me this time, ok?

Of course; such is the right (and joy) of an innovator to define how one's own
idea is disseminated/actualized! (~: It is intriguing that you feel your idea
has the character of best being pursued via a startup (addressing some crucial
unmet commercial need), rather than via the academic model (e.g.
transformative _Nature_ publication) more commonly used for improved
microscopy techniques. Best of luck in this pursuit; I look forward to seeing
your startup's innovations someday soon!

> You need to understand the whole picture, how it works on each level, and
> how the levels fit together - then you can run thought experiments. A good
> generative model of the world.

This idea of developing a "good generative model of the world" is a beautiful
aspiration for all of us to have. Thanks for your insights!

------
PoachedSausage
Fascinating and very similar to what we have to go through in the particle
physics community to get experiments to output data.

I find it strange though that a good microscope team doesn't include an
electronic engineer, yet:

>requires a knowledge of optics, mechanics, electronics, computer programming
and biology.

~~~
posterboy
Particle Physics is high power, as far as I know, so the requirements for an
electrical engineer might be quite different to low power drive circuitry. On
the other hand, signal processing is inherently intertwined with electronics,
so I'm sure they have technicians on board to deal with that, too.

