The idea behind a greenscreen is that you can make that green colour transparent in the frames of footage allowing you to blend that with some other background or other layered footage. This has issues like not always having a uniform colour, difficulty with things like hair, and lighting affecting some edges. These have to be manually cleaned up frame-by-frame, which takes a lot of time that is mostly busy work.
An alternative approach (such as that used by the sodium lighting on Mary Poppins) is that you create two images per frame -- the core image and a mask. The mask is a black and white image where the white pixels are the pixels to keep and the black pixels the ones to discard. Shades of gray indicate blended pixels.
For the mask approach you are filming a perfect alpha channel to apply to the footage that doesn't have the issues of greenscreen. The problem is that this requires specialist, licensed equipment and perfect filming conditions.
The new approach is to take advantage of image/video models to train a model that can produce the alpha channel mask for a given frame (and thus an entire recording) when just given greenscreen footage.
The use of CGI in the training data allows the input image and mask to be perfect without having to spend hundreds of hours creating that data. It's also easier to modify and create variations to test different cases such as reflective or soft edges.
Thus, you have the greenscreen input footage, the expected processed output and alpha channel mask. You can then apply traditional neural net training techniques on the data using the expected image/alpha channel as the target. For example, you can compute the difference on each of the alpha channel output neurons from the expected result, then apply backpropagation to compute the differences through the neural network, and then nudge the neuron weights in the computed gradient direction. Repeat that process across a distribution of the test images over multiple passes until the network no longer changes significantly between passes.
AI assisted coding is really good as an enhanced auto-complete, often better as it picks up patterns in the code and will complete whole lines or chunks of code. There, I'll assess the results like any other auto-completed suggestions.
For other things like when asking questions I won't just blindly copy what the LLM is suggesting. I'll often rewrite it in a style that best fits the style of the codebase I'm working on, or to better fit it into what I'm trying to achieve. Also, if I've asked it for how to do a specific one-line query and it has rewritten a whole chunk of code, I'll only make use of that one line, or specific fix/change. -- This also helps me to understand the response from the LLM.
I'll then do testing to make sure that the code is working correctly, with unit tests where relevant.
I don't use AI for the sake of it, I use it where and when it is useful. For example:
1. advanced autocomplete -- if you have or paste the structure of a JSON or other format, or a class fields, it is good at autocompleting things like serialization, case statements, or other repetitive/boilerplate code;
2. questions -- it can often be difficult to find an answer on Google/etc. (esp. if you don't know exactly what you are looking for, or if Google decides to ignore a key term such as the programming language), but can be better via an AI.
Like all tools, you need to read, check, and verify its output.
Personally I find this workflow is jarring. I get into flow typing code and then the AI autocompletes the next four lines on a tab input. Now my flow is screeching to a halt because I have to switch from flow mode to review mode to make sure it actually autocompleted what I wanted
Text editors/IDEs have simple autocomplete and the ability to do some expansion, e.g. a for loop with placeholders to fill in. Those work and are still useful.
JetBrains also has local line-based LLM models for various languages.
With the LLM-based autocomplete it a) generally autocompletes more code at once, and b) will often pick up on patterns in the existing code. E.g. if you have a similar method, list of print/string buffer write statements, or other repetitive code in the file it will often use that as a model for the generated code.
Sitting here on the sidelines having never configured snippets or macros or any of that in any of my editors, which I could have done like 30 years ago but never bothered in all this time, doing quizzical-dog look at all these people thrilled about LLMs.
I guess they might finally get me to use those things since they take the “configuring” and “remembering shortcuts” part out, but so much of this doesn’t look new at all. Super old, actually.
In my objective opinion, almost all AI uses cases (coding or otherwise), are just because of people's extreme laziness in spending a little time setting up some "automated" workflow, be it canned templates or whatever. The non-AI approach has the added benefit of being precise!
Customizable snipping is a feature editors support (which I mentioned as they are related/similar to what the AI is doing), but is different to the AI autocomplete behaviour.
If I have a JSON structure, I can paste that into the file as a comment, e.g.:
# {"foo": 1, "bar": "test", "baz"}
@dataclass
class FooBar:
foo:
and the AI will/can autocomplete/generate that to:
@dataclass
class FooBar:
foo: int
bar: str
baz: int
The JetBrains local autocomplete is hilarious but occasionally useful. I find it really hit and miss in terms of when it will decide to autocomplete and whether it will exhastively complete all elements, miss some out or get itself into a loop over several.
The out-of-the-box stuff is supposed to be kind of stupid. Are you guys really not editing your own snippets and shortcuts? Have people really been typing out "def do_something(foo, bar, baz)\n\t" manually?
XML is really great for text documents with embeds and markup, either semantic (this part of the paper is an abstract) or visual (this part of the document should be 14-point and aligned right). You can do this in JSON, but it's a pain.
JSON is great for representing data. If you have some data structures and two machines trying to exchange them, JSON is great for that.
TOML / yaml / hcl / JSON with comments are great at config. If you have a human writing something that a machine is supposed to understand, you don't want turning completeness and you don't want to deal with the pain of having your own DSL, those are great.
HTML is actually a dialect of SGML. XHTML was an attempt to move to an XML-based foundation, but XML's strictness in parsing worked against it, and eventually folks just standardized how HTML parsers should interpret ill-formed HTML instead.
Ah good to know. It’s interesting (to me) how similar they look to each other but you and other commentators below mention how they’re more like distant cousins
No, HTML was historically supposed to be a subset of SGML; XML is also an application of SGML. XHTML is the XML version of HTML. As of HTML5, HTML is no longer technically SGML or XML.
HTML is far loosier-goosier in its syntax than XML allows. There was an attempt to nail its syntax down in the pre-HTML 5 days; that's XHTML. When HTML 5 pivoted away from that, that spelled the end of these two things ever coming together.
Really, I think you can trace a lot of the "XML is spooky old technology" mindset to the release of HTML 5. That was when XML stopped being directly relevant to the web, though of course it still lives on in many other domains and legacy web apps.
> There was an attempt to nail its syntax down in the pre-HTML 5 days; that's XHTML. When HTML 5 pivoted away from that, that spelled the end of these two things ever coming together.
Exactly the opposite; WHATWG “Living Standard” HTML (different releases of which were used as the basis for W3C HTML5, 5.1, and 5.2 before the W3C stopped doing that) includes an XML serialization as part of the spec, so now the HTML-in-XML is permanently in sync with and feature-matched with plain HTML.
“Warning!
Using the XML syntax is not recommended, for reasons which include the fact that there is no specification which defines the rules for how an XML parser must map a string of bytes or characters into a Document object, as well as the fact that the XML syntax is essentially unmaintained — in that, it’s not expected that any further features will ever be added to the XML syntax (even when such features have been added to the HTML syntax).”
No, HTML was a specific application profile of SGML (modern HTML, I believe, no longer technically is), XML is a newer (than HTML) application profile of SGML inspired by HTML but aiming for greater generality.
XHTML was an attempt to encode HTML semantics (approximately, each version of XHTML also altered some semantics from HTML and previous XHTML versions) in XML, and the XML serialization of modern, WHATWG HTML exactly encodes HTML semantics in XML.
Yes, there's a handful of niches. Still 1/1000th the momentum it had, or adoption it was expected to get, and nobody under 40 even considers it for new stuff.
An alternative approach (such as that used by the sodium lighting on Mary Poppins) is that you create two images per frame -- the core image and a mask. The mask is a black and white image where the white pixels are the pixels to keep and the black pixels the ones to discard. Shades of gray indicate blended pixels.
For the mask approach you are filming a perfect alpha channel to apply to the footage that doesn't have the issues of greenscreen. The problem is that this requires specialist, licensed equipment and perfect filming conditions.
The new approach is to take advantage of image/video models to train a model that can produce the alpha channel mask for a given frame (and thus an entire recording) when just given greenscreen footage.
The use of CGI in the training data allows the input image and mask to be perfect without having to spend hundreds of hours creating that data. It's also easier to modify and create variations to test different cases such as reflective or soft edges.
Thus, you have the greenscreen input footage, the expected processed output and alpha channel mask. You can then apply traditional neural net training techniques on the data using the expected image/alpha channel as the target. For example, you can compute the difference on each of the alpha channel output neurons from the expected result, then apply backpropagation to compute the differences through the neural network, and then nudge the neuron weights in the computed gradient direction. Repeat that process across a distribution of the test images over multiple passes until the network no longer changes significantly between passes.
reply