I like the memory system, in general. For reference I'm using mostly Opus 4.8 + Max effort. It will often pull things out of memory that are relevant. Like I'll ask it to come up with a few options I should consider for, say, a self-hosted OIDC provider and it'll say things like "Considering the size of your operations team, this might be a better fit because of X and Y".
Now, I'll agree that this is probably the sort of thing I should put in the CLAUDE.md, but in this case it wasn't on my radar to put that in my CLAUDE.md, so it was nice that it surfaced that.
It does sometimes go awry though. Today I was asking about a problem I was having authenticating, and it said "you may be running into this trusted proxy setting because you put your apps behind an haproxy". That is true of 95% of our apps, so it was worth mentioning, but in this case it was not so I had to correct it. But, I'm glad it mentioned it because if we did have it proxied it could have saved me a lot of time.
It seems like a prerequisite is a certain level of world model and associated reasoning ability. Your examples are entirely dependent on the past context being relevant to the current situation. That's particularly tricky if you regularly ask about hypotheticals or problems that you're assisting someone else with. A human would probably ask clarifying questions such as "is this for the operations team at X? are they still size Y?" and "is this app proxied like the others you mentioned in the past?" rather than assuming.
There's also a noticable hierarchy to such context that needs to be correctly modeled - you could for example be involved with multiple teams of different sizes that are subject to different rules which is something a human would understand naturally.
You say that, and that's a possibility, and maybe in your experience that is true. But I'd say I've found actual humans will frequently use existing internal models of something unless they have a reason to believe something has changed to invalidate that. We even have a well worn joke about what happens when you assume.
>you could for example be involved with multiple teams of different sizes that are subject to different rules
Just to clarify, my experience with the memory of these tools doesn't provide any data (for or against) how well/poorly the memory may work with someone who is involved with multiple teams.
> I'd say I've found actual humans will frequently use existing internal models of something
Sure, I didn't mean to claim that humans perform perfectly. Merely to reference a particular approach in order to illustrate my point about modeling concepts.
> doesn't provide any data (for or against)
It seems a foregone conclusion that it would confuse details between the different contexts in a way that a human (ie an entity with a certain level of world model and reasoning ability) would not be expected to.
Now, I'll agree that this is probably the sort of thing I should put in the CLAUDE.md, but in this case it wasn't on my radar to put that in my CLAUDE.md, so it was nice that it surfaced that.
It does sometimes go awry though. Today I was asking about a problem I was having authenticating, and it said "you may be running into this trusted proxy setting because you put your apps behind an haproxy". That is true of 95% of our apps, so it was worth mentioning, but in this case it was not so I had to correct it. But, I'm glad it mentioned it because if we did have it proxied it could have saved me a lot of time.