For me writing is also a much bigger part of the process. Writing out my explanation of the problem (its symptoms and my theories) to a colleague, even if I don't send it off, can spur me to come up with a new idea. Writing blog posts or an outline of such a blog post can help me spot gaps in my understanding and justifications. And writing down the solution, for myself two weeks ago as the intended audience, helps me internalize the important bits while filing away the minutiae for later reference. Publishing it on my blog also ensures that I spend enough time on the writeup that it is actually helpful for myself later, and not just a mess of notes.
I have often thought about writing a blog from my experiences, but after I write the thoughts down seem like common sense in nature or just plain ranting. Writing it down gets it off my chest / mind, but publishing has never happened. I do keep all the notes so I can revisit them. Often do this with emails too but I delete them if I do not send them after 48 hours. It has taught me to never send an angry email in the ‘heat of the moment’ and to reflect on the content as if I were to receive such an email.
The best one I have is letting others fail, they are going to anyway. When put on a new project with a new team who has been working together for a while, they are NOT going to listen to you no matter how much experience you have even if you consider yourself a SME. I often find myself fighting against what I think is the correct solution as the team will want to do the opposite of the ‘new guys’ suggestions…
> I have often thought about writing a blog from my experiences, but after I write the thoughts down seem like common sense in nature or just plain ranting.
Same. Trying to get over that and just write anyway.
What is it with the resistance to new points of view? So often I ask a question about a feature or process to now how it works and they react like I threatened to kill their puppy.
I don't know how many emails I actually never sent, and also never filed into the drafts folder. The mondane version being fending of rage. But other times, you just develop the answer to your question while writing the question down. On a similar thought, this is why coding is an inherent part of my way of dealing with problems. The rigour required to get a piece of code to work helps me understand concepts. So when learning things, I sometimes end up writing a piece of code to model the domain, just to help me understand it.
To be fair, two thirds of the algorithm is writing things down, and in many cases just clearly formulating the question is enough to find enough gaps in your understanding to answer it as well.
> What you think today is the result of what you read yesterday.
I really need to reinforce this. And follow it. I try to read good books and stay away from random tidbits of novelty but sometimes it's really hard and days pass by in which I don't read anything substantial, just blogs and news.
We can't eat just candy and we can't read just fun little pieces of novelty. We also need the protein and fat of dense long form content like books.
"To answer the question, what you do next is to remove yourself from further inputs. As long as you keep feeding your brain with more data, the queries never resolve, and eventually the stuff you fed it first will fall out of the context window and amount to nothing."
Similar/Related: If you are trying to fall asleep, and you mind is filled with thoughts, a solutions is to count (does not need to be sheep just count numbers). Counting distracts and occupies your mind from being distracted with whatever your anxiety (or excitement) thoughts are. (Source: It works for me). Anything can really be substituted for counting as long as it's not exciting or anxiety producing.
For insomnia and sleep problems, a good approach is to try out additional techniques from the lists in CBT-I (Cognitive-Behavioral Therapy-Insomnia) treatment manuals. This is a well-studied, non-medication approach, with large parts that a person can learn independently.
A specific technique that helped me was to challenge the thought that "If I don't sleep for 8 hours, the following day will be ruined," as that was a common thought that kept me up. There are a couple of PDFs of treatment manuals online that list the techniques:
I would also add a step 1.5: write down the wrong solution.
Too often I find myself wondering about what later turns out to be the most trivial part of the problem. The best antidote is to write a naive half-solution to "throw away"; yes, it may be unusably bad, but now I know what sub-problem is worth thinking about when I'm away from the computer.
Agreed. “Do the simplest thing that could possibly work” has sometimes been good advice for me, even if that simple thing is unsuitable at scale, not production-ready, etc. There are problems for which that solution is a stone’s throw from an optimal one. And if not, seeing the downsides of that naive solution can still point the way to what needs to be done in the ‘real’ solution.
In Feynman's 'Surely You're Joking, Mr. Feynman!' there is a chapter where he describes his observations on 'how does it feels to go to sleep' as part of an essay for a class. This and other lectures/writings from Feynman gives cues to his technique and also the almost impish curiosity with which he approaches a problem. IMHO the best way to master or learn from him, is read/watch his works/lectures.
In research, reaching the point of "writing down the problem" is already a huge accomplishment.
What distinguishes the first-rate researchers and the second is their (1) ability to identify which problems are worth their finite time in solving, (2) defining the problem so that it is manageable but impactful, (3) using approaches other people in th field have not thought of to tackle the problem.
If you try to remember what you are trying to remember on the spot often you can't. Then you stop trying and all the sudden (and later) the answer comes to you. (One example: Happens often when trying to remember the name of an actor 'I know who that is' then later 'ok it was...')
I think this is also related to why some people don't test well. A question in a test format or information requested formulated as a question (that need an immediate answer) doesn't work for many people with facts that are memorized. In fact extreme pressure can sometimes prevent recall just because of the anxiety alone that that produces.
"stepping away from the keyboard" takes your Default Mode Network off the project like a burned-out employee that's just running it into the ground
when the DMN is highly active - fixating, ruminating - its focus is narrow and it's less likely to produce as many creative insights or dredge up as many relevant memories
if you make it less active - doing something else, letting your mind wander, shutting it off entirely with sleep or substances - it will keep trying to solve the problem, but also make farther-flung connections that might solve it
these connections can then be picked up and used more effectively by your executive and salience networks
“Write down the problem” is hugely underrated as part of the algorithm. In my experience, at least when doing research, if you can precisely and rigorously define the problem you actually need to solve you’re probably halfway to a solution.
I’ve been working on robot control using physics-based computing devices for about a year. Not yet able to write down the problem clearly enough to attempt a solution.
Not to make everything about AI, but.. he did mention LLMs. What happens if, after clearly defining the problem and listing all potentially relevant clues as to how to solve it, you first give that information to an LLM, before taking a walk or a weekend off?
This is probably another case where the answer depends on the attitude towards LLMs and maybe technology overall.
For me I think that this is largely how I use LLMs for programming. I use the aider program, add the relevant source files, explain what I want to do and the approach I want, and ask it to do it. It does routinely miss obvious things. But then it's also often fairly easy to ask it to correct itself.
It depends on the nature and complexity of the problem though.
But theoretically the LLM would have a couple of useful ideas or feedback if you really give it all of the context.
But maybe defining the problem and the relevant information is the hard part. Perhaps having significantly larger context windows is a bigger deal than some people might realize.
If the LLM or multimodal model has a very large context window and also enough computing resources to constantly or routinely decide what the goal is.. Then the other part would be having a large pool of potentially relevant information to select from in approaching the problem.
But basically we might be able to skip the step where we select the relevant information and let the AI do that, if we have a large enough context window.
Which might lead to the question, why did we even get out of bed. But that's another problem.
I get the impression that diffusion transformers are a big deal. Do they allow for more sophisticated/developed problem solving or "cognition" in some way?
But does the solution manifest (a flower grows in the garden) or does it become apparent (the fog clears to reveal it)?
Because the phenomenon we're discussing could be attributed to either cause. But they are quite different causes. And imply quite different stuff going on there.
More like the flower, I'd say. Your first problem statement plants the seed. The solution (and the shortcomings in that solution and in the original problem statement) are the seed clearing the dirt. As you write, think, and write again, the plant grows and takes form until the time is right for the flower to bloom.
Then the bloom slowly dies, unnoticed by the majority (beautiful in their place, but with no generalizable characteristics), or as the work becomes common knowledge and is integrated into better and more complex solutions to bigger and harder problems. The most beautiful flowers are mass produced for public consumption ($e=mc^2$, and the like), but those are rather few and far between.
You eventually get to the solution if you keep thinking long enough. The trick is to not stop thinking in the background about it. That's the process. I think that hours or days, like the author mentions, even seems very short. Many of the kinds of things that I have experienced, which yielded to this kind of process, took months to years. Sometimes big problems can be broken down into many small ones and only take those minutes to hours to days to complete. But still, the big problem takes a long time to work through.
Well, it is both of course. Why assume a binary process? Several solutions are probably grown, considered, and evaluated and one begins to gain precedence. That one then finally becomes apparent and is revealed.
Most cognition is subconscious. I believe there is an unsung gardener hero working in the background to cultivate the candidate flowers and then put spotlights on a few of the best ones that emerge. You then evaluate them as you walk down your path of consciousness.
A solution may manifest as a series of insights, but the insights themselves are quite binary. You suddenly find yourself having them. For me at least they appear as a thought like any other.
How appropriate, I often have many GitHub issues and corresponding PRs which chip away at problems that are not yet solved. They usually start with some naive approach that doesn't scale. The reasons for its unsuitability to ship are listed then iterated upon. Sometimes work done elsewhere by someone else or myself will find relevance and move the ball closer to the goal. I've always thought of this process as "gardening PRs".
My other process is Emoji-Driven-Development where a whole lot of these PRs can be listed (with high density) and only a single emoji indicating each development status. It's good for refreshing the mental list of unsolved things and where/why they're stuck.
You misinterpreted the article, which is about solutions which just "come" after heavy brain activity. You're talking about iterative, conscious problem solving, which is the complete opposite.
They're not mutually-exclusive. Often I'll have such long-term PRs/branches open until a 'eureka moment' presents itself. The process is merely a way of keeping the back-burners 'warm'.
In the shower is where I have solved my most challenging problems. Taking a shower is just mindless routine, so my brain's "background processing" goes into overdrive.
See also rubber duck debugging, where explaining the problem (verbally or in writing) is essentially the first step of Feynman's algorithm, and allows your brain to figure out the solution in the background.
It's hard to observe your own mind, or indeed others, it's intangible and generally difficult to observe thinking.
Because of this drawback, LLMs are actually a decent model for this sort of process since we can observe how they operate. I'm not claiming they're actually intelligent like we are, but rather that they model the process of drawing connections and making associations close enough to how we think to the point where it's an useful analogy.
I think, if you adopt this mindset, acutely any process that involves precise instruction could be dismissed. Or more generally, your standards for analogy may be so stringent as to render that concept invalid.
There are parallels. You feed an LLM context data and then tell it what to focus on so it can pull relevant data.
Maybe the entire process isn't like feeding an LLM, but that step is. Relevance identification is an interesting part of the process. The LLM can do a decent job of making connections, but it doesn't know what is relevant. In the longer time frame of the thinking process, we constantly throw out data as irrelevant or identify previously unknown relevant data that needs to be added. It's a part of the process completely outside of the LLM.
I'm as AI-wary as they come, but - it's a completely factual, relevant, and helpful statement. To say that any relation or comparison with AI ruins the comparandum is as absurd as saying that anything coloured red is bad because the Nazis used the colour red in their iconography.
I think the most common failure mode is neuroticism; getting stressed and frustrated with the lack of progress, thinking about the progress toward the outcome rather than the problem to be solved. Stress in particular is extremely poisonous to thinking. The expectation should be that it takes as long as it takes, and you need to be calm and well rested.
If you find yourself choosing between working on the problem and taking a nap, always take the nap. (Given the choice between remaining in bed, and taking another peek at the problem, usually take a peek at the problem, after which you should do what feels best.)
Some experimentation should convince you that this is how human minds work (well, most of 'em, and yours is probably one of them), after which you can employ this strategy without guilt during a crisis.
A lack of necessary prior knowledge is often a major reason for struggling with a problem. As a relatable example, suppose you're taking an exam. There's a problem near the end that you don't know how to solve—and the reason is that you haven't studied that topic enough. No matter how smart (as in, fast at learning) you are, you need to practice with similar or related problems to solve that issue.
But suppose you're outside of an exam environment and have time to look up the relevant material. I've known a PhD candidate in a non-mathematics field who had to find a mathematical solution to a certain research problem. That person is smart but still needed a few months to learn the mathematical fundamentals to understand and solve the problem. In contrast, someone with a math background could have solved this far more quickly. But that person would have taken at least some months to get up to speed on the research literature for the non-mathematics part of the research problem, in order to properly understand its constraints and bigger-picture significance.
Lara Alcock's book "How to Study as a Mathematics Major" touches upon this topic more directly. She encourages readers not to be too intimidated if other students in a course seem really smart: much of the time, the reason is not due to an innate difference in smartness, but rather prior exposure by other students to concepts in the course. Students who seem to find the material effortless often have already studied many of the topics in another course or could even be retaking the course after a previous attempt.
When you plant a seed in a garden, there's many factors at play - is the season right? Is the climate right for that seed? Is the soil the right type? Are there pests or varmints roaming that might eat it prematurely? Etc.
The garden of thought has analogous factors: Is this a question your brain actually cares about right now? Do you have the background knowledge necessary to work it out? Is your brain calm enough to process that question? Are there distractions or anxieties that disturb the process?
That said, some people truly are stupid. I recently read John Cleese's autobiography, and he tells a story from when he was a Geography teacher...
There was a lad who he was teaching countries and their capitals. Even when given direct attention, the kid simply wasn't able to name any capitals whatsoever. He would smile and nod, giving no indication of difficulties... But he couldn't recall the info even after being told it 8 seconds previously. This particular type of data slid off his brain.
At the end of the term, the kid got one question right on the final exam, probably by accident. Cleese posted the paper in the teacher's room, attracting the comment from one teacher "The sad thing about true stupidity is that you can do absolutely nothing about it".
Perhaps that kid had a genius for engines or something, but he was never going to be able to understand geopolitics. He lacked even the awareness to know that he was stupid (at least about countries and capitals). He would never have asked if he was stupid, because he was truly stupid.
If your friend is ever curious about their intelligence, they're probably ok and can develop the skill of thinking like this.
This is a difficult fact to accept. We have all been told that people are generally equal, especially in intelligence, if given the same opportunities, but it becomes more clear in time that some problems are intractable to some people and no amount of training or exposure can change that. However, it's a better answer to the problem of why some people like Cleese's example do not absorb information. The alternative is to apply malice and laziness to them when it just isn't so.
We all have these intelligence holes that gives some insight into the mechanism. Eg. I'm bad at remembering names. As in the example, if you tell me someone's name, I'm likely to forget it 5 minutes later. I just spent 3 years reading Douglas Hofstadter's book and had to look up his name to type it here. This seems to happen because I don't see an application to remembering the name. I'm never going to meet Doug and rarely will anyone need to be told about the book, so why remember it? There's definitely a parallel to state capitals in that example.
Well said about not realizing it's important therefore names get down prioritized. Here's a counter point. I'm a movie geek. Yet when I reach for a name of someone from the cast I almost always end up describing, y'know, that guy who played together with that other guy in that movie, y'know, the one with the weird story line? Him! Yes, him. Love him.
While thinking is somewhat of a background process, our brains don't just solve the mysteries of the universe while we eat a ham sandwich. We tell our brains which problems are of high importance and it focuses on them. If you've ever laid down to go to sleep and told yourself to wake up at 6:30 and it worked, you've witnessed an obvious application of this.
The problem comes when we fail to point out the importance of a problem, or when we do so reflexively which means that we tell our brain that everything is important and it simply cannot process all of the requests.
Setting your thinking requests before doing a non-thinking activity is a good way to start. Think about the problem consciously and then specifically ask for an answer to a question. Then go do something physical or mechanical: take a walk, sleep, mow the lawn, watch a non-challenging movie, etc. Be prepared to accept whatever result you get. A common response is: non enough information, but it should point you toward what that additional info looks like.
