I'm upvoting this because learning these devices can be wildly useful when prompting LLMs for content generation. More so, if you use these devices in stylometric analysis of your own writing, as it makes "style transfer" a lot more effective.
These words have great lexical density, and can dramatically influence text generation.
I didn't think about that one(!) -- but yes, upon consideration, yes, I could definitely see that working!
So, great idea!
Also, prior to your post, I didn't know the word "stylometric". So let's put a link to the appropriate Wikipedia entry for the "folks tuning in back at home": https://en.wikipedia.org/wiki/Stylometry
Your post also brings up an interesting question -- which relates to lexical density -- or perhaps more specifically the term "lexical density" in the context of LLM's...
That is, something that could be of use with respect to LLM's, AI's, prompting (etc.) -- would be a rigorous mathematical and/or computational/data structure/algorithmic definition for a given word and/or given token's lexical density...
But -- this web page sort of contains the academic Computational Linguistics definition of lexical density.
That's not a bad starting point... but can we do better?
What I'd like to see is a LLM AI -- which would have different types of "debug modes".
One such "debug mode" -- might have the LLM AI showing each word and/or token as a data structure, showing what it relates to as a graph (with percentages) to other tokens/words/symbols in the context of the current conversation.
In other words, let's have a peek behind the curtain as to the actual "low-level" data structure which determines the lexical density of any given token or word or symbol in the context of the current conversation.
Perhaps that would be too much data for most purposes -- but it could be very helpful for other purposes...
Hmm, now that I think about it -- I'm going to guess (but not know!) -- that a future competitor to ChatGPT/Claude/Bard, etc. -- will be completely transparent about democratizing users' access to all internal LLM/AI parameters and data structures... Simply phrased: Full debug modes for everything, with access to as little or as much debug data, as desired...
Anyway, once again, great comment(!) -- and thank you for getting me thinking!
These words have great lexical density, and can dramatically influence text generation.