With how good Claude 3.5 is for coding, I would love if this supported private github repos - just connect my github account and provide a repo link which stays up to date as changes are made instead of having to upload content to a project.
I feel like this will probably be an option down the line - I wonder if it would be possible to replicate this functionality via the API as well, kind of like how there are open source projects attempting to replicate artifacts via the API.
You could absolutely get this working via the API right now. Here's how to use my API CLI tool to do that:
pip install files-to-prompt llm llm-claude-3
llm keys set claude
# paste in your Claude API key
llm models default claude-3.5-sonnet
files-to-prompt my-github-repo/ | llm --system 'say OK'
And now you can start an interactive chat with Claude 3.5 Sonnet pre-seeded with your files like this:
llm chat --continue
The --continue will ensure it is a continuation of the previous chat session where you dumped in all those files.
If you want to run against an updated copy of your files, start a new conversation with the files-to-prompt piping again.
+1 - I've been using LLM CLI to do this, and it's great. The best part is that you can swap models without changing your workflow.
I use Claude 3.5 for my daily coding tasks, but used Gemini Pro 1.5 to analyze a new Github repository the other day because it has a much larger context size.
You simply run `llm -m gemini-1.5-pro-preview-0409 "<your question>"`
With the new Projects feature I was able to hit the message rate limit much faster in Claude, after the API is really the only option until reset time.
I know people have been doing really cool things writing new projects. But everything I try to do with Sonnet 3.5 with an existing codebase, with its own intricacies like library versions is really bad
I've been trying to generate test coverage with 3.5 on a repo that has existed for years. it can take up to 20 attempts of file -> this is the error i get -> error -> error until i get one that runs successfully. It slightly improves when i give it a file, and also give it other files it successfully generated after 20 attempts and ask it to use the same style and learn from the changes it made previously to make it work
But its very hard to get it done with less than 10 attempts.
I bet one problem with supporting repos is that the 200K context can fill up quick. Need to be selective with what files are in the context. If the repo was integrated, and you could select which files are in the current context, that could be good.
I think the first step is to have a .claudeignore file (or maybe we can standardize on .llmignore) to tell it which files you don’t want it to add to the context. You could easily eliminate millions of tokens by ignore node_modules in most JavaScript projects!
You would ignore node modules in any case, but our monorepo has millions of files outside node modules or other generated artifacts, so going per package is still 100k files or so. LLMs don’t like that much yet.
How about a CLI tool that uses Claude's APIs to make changes to the files and do commits? Or, is it a must this is integrated into a web page and/or code editor?
This would also work for local development and then put up a pr, but I feel like that's a temporary thing as well until a model or agent-like workflow is good enough to directly read from a github issue and make the PR directly.
Right now Cursor can already use Claude 3.5 so I feel like that's very close to integrated into a code editor.
Thanks, I was a bit confused about what Project means.
It's funny that they don't organize the document, instead they just put what it is[1] and how to use it[2] on the entire Claude FAQ page[3] which is also not categorized.
It's like they don't need to organize the document cause they can just ask it to LLM.
Same request as for the introduction of GPTs. Make sure you understand which part of your data you are giving the rights away.
"Our use of Materials. We may use Materials to provide, maintain, and improve the Services and to develop other products and services. We will not train our models on any Materials that are not publicly available, except in two circumstances:
If you provide Feedback to us (through the Services or otherwise) regarding any Materials, we may use that Feedback in accordance with Section 5 (Feedback)."
I love the idea but I think they really need integrations to make this useful. They need connectors to cloud storage and other SaaS apps. No one wants to manually monitor a Box/Google Drive/One Drive folder for new or updated documents and manually upload them into a Claude project.
Apparently that’s coming, according to their post about it.
“In the coming months, we’ll continue making Claude easier to use while expanding the types of project knowledge you can bring to Claude via native integrations with popular applications and tools. We’re excited to see how your team works with Claude.”
The obvious comparison here is ChatGPT's GPTs. Claude Projects appear to work by dumping the entire "knowledge base" into the context in one go, hence the 200,000 token limit, which makes it very easy to reason about.
GPTs have a form of RAG built in - you can upload up to 20 documents each of 2 million tokens and those get chunked and vectorized and used to help answer questions - but there's still no internal documentation on how that chunking process works, which makes it hard for me to trust them as I can't make informed decisions about how best to use the knowledge feature.
If it's anything like the "file upload" feature in ChatGPT, it's not very good. CGPT will frequently straight up lie to me about the contents of even short files I uploaded specifically for it to reference. Like, rewriting the code to use the opposite API pattern from what I gave it, or totally ignoring what detail I put in a file specification and making some up - things it basically never does if I paste the code directly in the chat.
I expect it will work better than that, because the Claude project is dumping the whole set of uploaded files directly into the context. My experience with Claude has been that it's really good at working with large chunks of text that you paste straight into the prompt.
It’s nothing like that implementation (I’ve been using it today). It just dumps the entire contents of the files and gives you a progress bar that shows how full the context is.
Here'a feature request in case people from Anthropic are listening:
I'd like to dump our product knowledge base (emails, tickets, support articles, etc..) into a Project and then create an embeddable chat-bot from it that can handle customer support right from our website.
Most support requests we get are similar or equal to previous requests. It could help our support staff save a lot of time to focus on the requests that actually matter.
Unfortunately project files don't seem to have folders or a way organize the hierarchy so I had Claude write a little script to dump all source files into a single markdown file with paths: https://pastebin.com/P6Cc1cmL
It uses fd which respects .gitignore by default. A 10k LoC C++/Rust/QML/JS project came in at about 50% of the context.
With this release they also added chat history search!
Anyone else notice that for cancelled team subscriptions the renew subscription link just jumps them to the help center? It's very weird. Tried with both Chrome and Edge on Windows 11.
Fwiw I've found Sonnet still too censored. Give it a poem to translate or analyzes and it refuse to reproduce it (even in translated form) due to copyright reasons.
"You have control to delete individual conversations, which will be removed immediately from your conversation history and automatically deleted from our back-end within 30 days."
I feel like this will probably be an option down the line - I wonder if it would be possible to replicate this functionality via the API as well, kind of like how there are open source projects attempting to replicate artifacts via the API.
reply