> I could kind of see things either way. Is this like not providing the source code, or is it like not providing the IDE, debugger, compiler, and linter that was used to write the source code?
Do the engineers that made this hand edit this file? Or did they have other source that they used and this is the build product?
> (Also, it feels a bit "looking a gift horse in the mouth" to criticize people who are giving away a cutting-edge model that can be used freely.)
Windows was free for a year. Did that make it open source?
> Do the engineers that made this hand edit this file? Or did they have other source that they used and this is the build product?
Do any open source product provide all the tools used to make software? I haven't seen the linux kernel included in any other open source product and that'd quite frankly be insane. As well as including vim/emacs, gcc, gdb, X11, etc.
But I do agree that training data is more important than those things. But you need to be clear about that because people aren't understanding what you're getting at. Don't get mad, refine your communication.
> Windows was free for a year. Did that make it open source?
Windows didn't attach an Apache-2.0 license to it. This license makes this version of the code perpetually open source. They can change the license later, but it will not back apply to previous versions. Sorry, but this is just a terrible comparison. Free isn't what makes a thing "open source." Which let's be clear, is a fuzzy definition too.
What I'm asking for is pretty clear. The snapshot of code and data the engineers have checked into their repos (including data repositories) that were processed into this binary release.
> This license makes this version of the code perpetually open source.
It doesn't because they didn't release the source.
There's nothing stopping me from attaching an Apache 2 license to a shared library I never give the source out to. That also would not be an open source release. There has to be actual source involved.
> You’re welcome to fuck around and find out. Go release llama2 under Apache 2. You’re saying that’s fine right?
You're missing my point. Obviously you can't release someone else's IP under whatever license you see fit.
You can release your own binary under Apache 2. Doing so without releasing the source doesn't make it open source despite being an open source license.
> The answer to your question is that code stored as a binary is not different from code stored as text. Pickled models are code.
I'm not saying it's not code; I'm saying it's not source.
The data used to derive this model is not different from the brain and worldly observations and learnings of the engineers, which are not part of any open source materials.
What are you talking about "the brain" of the engineers, this is bonkers. Monocasa is being excruciatingly patient with you all but the fact is this was generated with tools and is not a source release, it's a final product or compiled or generated release.
Code generated with tools is still code. This code is the source. The output of the code is the output. Monocasa is failing to understand or perhaps intentionally not understanding the difference. In some contexts a “compiled release” implies an output that is largely immutable for practical purposes. That is not what this is. It’s technically a binary object, but it’s a binary object you can easily unpack to get executable code that you can read and edit. It is a convenient format different from classical text code. The fact that it’s a binary is completely irrelevant. It’s akin to arguing that code that is provided in a zip file cannot be open source. Both because it’s a compressed file and because it doesn’t include the compression algorithm.
With that understood, demanding the “tools” that were used to create the code is like asking for the engineers’ notebooks of design thoughts along the way. It has no bearing on your ability to use or modify it. This is not an open source project to make neural nets. This an open source project of a neural net.
If someone releases math_funcs.py, you don’t need anything about the tools that were used to create math_funcs.py to consider it open source.
So if you use tools that generate some boilerplate code as part of your project you need to include the boilerplate generator otherwise it’s not open source?
Why are you being so obtuse? No, devs don't have to include the source to vim in their repos. They have to include the source files for their product in their repos. I'm confident this just isn't that hard to understand.
These are the source files! I’m going to stop responding to monocasa because I think he is being obtuse and leading me to say things that you are misinterpreting.
There is no expectation to include vim, or any tools required to create a codebase. We agree. And that’s why this repo is sufficient. Asking for the tooling that was used to make this project would be out of scope and unreasonable.
This is a repo that can be used to make predictions. It is not a repo that is used to make models.
The source code of the repo referred to by “open source” is the code of the repo.
You can ask all you want but that is irrelevant as to whether it is open source. If photoshop were open source, the c++ code would need to be available. Not the tooling used to make the c++ code. The c++ code is equivalent to the model. Not the separate pythoj codebase that was involved in making it.
Which is some BSD PyTorch + PyTorch calling code that anyone competent in the field can implement any number of ways and is not special to this output.
> You can ask all you want but that is irrelevant as to whether it is open source.
There's a pretty good definition of open source at OSI [0], point of 2 of which is (emphasis mine):
"The program must include source code, and must allow distribution in source code as well as compiled form. Where some form of a product is not distributed with source code, there must be a well-publicized means of obtaining the source code for no more than a reasonable reproduction cost, preferably downloading via the Internet without charge. The source code must be the preferred form in which a programmer would modify the program. Deliberately obfuscated source code is not allowed. Intermediate forms such as the output of a preprocessor or translator are not allowed."
You can't bump the window on what "program" means. "Program" here doesn't mean "predictions", that's the output of the program. If you had a program that generated images, you wouldn't say that that program was the source code of the images. You would say that that program generates images and has source code.
This just isn't an open source release. It's freely released to the public, but it doesn't contain the source used to create or modify it.
> Which is some BSD PyTorch + PyTorch calling code that anyone competent in the field can implement any number of ways and is not special to this output.
> You can't bump the window on what "program" means. "Program" here doesn't mean "predictions", that's the output of the program. If you had a program that generated images, you wouldn't say that that program was the source code of the images. You would say that that program generates images and has source code
My dude… you have no idea what you’re talking about.
The picked model is the preferred way to interact and modify it. It is the source code. It is not like a compiled program. It is literally code.
I am NOT claiming the predictions are the program. I am saying the pickled model is. You 100% don’t need anything else to do anything more with the model.
I don’t know or care if they released their model generating code but nobody competent who understands what they are talking about cares about this.
It’s pickled because it’s big. Just imagine this as a zip containing algo.py
> Do any open source product provide all the tools used to make software? I haven't seen the linux kernel included in any other open source product and that'd quite frankly be insane. As well as including vim/emacs, gcc, gdb, X11, etc.
BSD traditionally comes as a full set of source for the whole OS, it's hardly insane.
But the point is you don't need those things to work on Linux - you can use your own preferred editor, compiler, debugger, ... - and you can work on things that aren't Linux with those things. Calling something "open source" if you can only work on it with proprietary tools would be very dubious (admittedly some people do), and calling a project open source when the missing piece you need to work on it is not a general-purpose tool at all but a component that's only used for building this project is an outright falsehood.
But what's proprietary here? That's what I'm not getting from the other person. You have the algorithm. Hell, they even provided the model in pytorch/python. They just didn't provide training parameters and data. But that's not necessary to use or modify the software just like it isn't necessary for nearly any other open sourced project. I mean we aren't calling PyTorch "not open source" because they didn't provide source code for vim and VS code. That's what I'm saying. Because at that point I'm not sure what's the difference between saying "It's not open source unless you provide at least one node of H100 machines." That's what you kinda need to train this stuff.
> But what's proprietary here? That's what I'm not getting from the other person. You have the algorithm. Hell, they even provided the model in pytorch/python. They just didn't provide training parameters and data. But that's not necessary to use or modify the software just like it isn't necessary for nearly any other open sourced project.
It's necessary if you want to rebuild the weights/factors/whatever the current terminology is, which are a major part of what they're shipping. If they found a major bug in this release, the fix might involve re-running the training process, and currently that's something that they can do and we users can't.
> I mean we aren't calling PyTorch "not open source" because they didn't provide source code for vim and VS code.
You can build the exact same PyTorch by using emacs, or notepad, or what have you, and those are standard tools that you can find all over the place and use for all sorts of things. If you want to fix a bug in PyTorch, you can edit it with any editor you like, re-run the build process, and be confident that the only thing that changed is the thing you changed.
You can't rebuild this model without their training parameters and data. Like maybe you could run the same process with an off-the-shelf training dataset, but you'd get a very different result from the thing that they've released - the whole point of the thing they've released is that it has the weights that they've "compiled" through this training process. If you've built a system on top of this model, and you want to fix a bug in it, that's not going to be good enough - without having access to the same training dataset, there's no way for you to produce "this model, but with this particular problem fixed".
(And sure, maybe you could try to work around with finetuning, or manually patch the binary weights, but that's similar to how people will patch binaries to fix bugs in proprietary software - yes it's possible, but the point of open source is to make it easier)
Is this code "open source?"[0] It is under a MIT license, has the training scripts, all the data is highly available, etc. But to the best of my knowledge no one has reproduced their results. These people sure couldn't[1] and I'm not aware of any existing work which did. This honestly is quite common in ML and is quite frustrating as a researcher, especially when you get a round of reviewers who think benchmarks are the only thing that matters (I literally got a work rejected twice with a major complaint being about how my implementation didn't beat [0] despite it beating [1]. My paper wasn't even on architecture... so we weren't even trying to improve the SOTA...).
As a researcher I want to know the HPs and datasets used, but they honestly aren't that important for usage. You're right that to "debug" them one method would be to retrain from scratch. But more likely is doing tuning, reinforcement learning, or using a LoRA. Even the company engineers would look at those routes before they looked at retraining from scratch. Most of the NLP research world is using pretrained models these days (I don't like this tbh, but that's a different discussion all together). Only a handful of companies are actually training models. And I mean companies, I don't mean academics. Academics don't have the resources (unless partnering), and without digressing too much, the benchmarkism is severely limiting the ability for academics to be academics. Models are insanely hard to evaluate, especially after RLHF'd to all hell.
> (And sure, maybe you could try to work around with finetuning, or manually patch the binary weights, but that's similar to how people will patch binaries to fix bugs in proprietary software - yes it's possible, but the point of open source is to make it easier)
The truth is that this is how most ML refinement is happening these days. If you want better refinement we have to have that other discussion.
> Is this code "open source?"[0] It is under a MIT license, has the training scripts, all the data is highly available, etc. But to the best of my knowledge no one has reproduced their results. These people sure couldn't[1] and I'm not aware of any existing work which did.
I don't know about ML specifically, but I've seen a number of projects where people publish supposedly "the source" for something and it doesn't actually build. IMO if they're doing it wilfully that makes not open source, whereas if it's just good-faith legitimate incompetence then it can be.
(My litmus test would be: are they giving you all the stuff they'd give to a new hire/assistant working with them? If they've got a "how to build" on their internal wiki with a bunch of steps they're keeping secret, then it's not open-source. But if the process for a new hire is to hand over a code dump and say "huh, it works on my machine, I don't remember what I did to set it up", then at that point I'd consider it open source. I think this aligns with the "preferred form for making modifications" idea in the licenses).
> But more likely is doing tuning, reinforcement learning, or using a LoRA. Even the company engineers would look at those routes before they looked at retraining from scratch.
Sure. But they'd have that capability in their back pocket for if they needed it. It's a similar story for e.g. parts of the Linux kernel code that are generated via a perl script based on the architecture documentation - you only actually re-run that perl script once in a blue moon, but it's important that they publish the perl script and not just the C that was generated by it.
Build scripts are not required for open source. Usually they are provided, because nobody actually wants to maintain them separately, but they are not actually part of the project itself. Often times it's just a few gnu scripts, sometimes there's parts of it missing (because they're reused from another project, or they have secrets in them that the maintainer can't be bothered to remove, or other reasons), and rarely the build script is an entire project itself, and even more rarely there's nothing there at all except a single file of source code that can't be built alone (I've seen this in particular on several old golang projects, and it's incredibly annoying).
I'm not asking for the engineers brains, I'm asking for more or less what's sitting in the IDE as they work on the project.
Robert has provided that there. Mistral has not.
As an aside, I'm more than capable of editing that code; I've professionally worked on FPGA code and have written a PS1 emulator. Taking that (wonderful looking code) and say, fixing a bug, adding a different interface for the cdrom, porting it to a new FPGA are all things I'm more than capable of.
No, but if the Windows binary code was made available with no restrictive licensing, I'd be quite happy, and the WINE devs would be ecstatic. Sure, the source code and build infrastructure would be nicer, but we could still work with that.
'gary_0' being happy with the license terms isn't what defines 'open source'.
I'm fairly happy with the license terms too. They're just not open source. We dilute the term open source for the worst if we allow it to apply to build artifacts for some reason.
We were talking about "looking a gift horse in the mouth", as in it's still a positive thing regardless of the semantic quibbles about open source. Nobody would argue that a hypothetical openly licensed Windows binary-only release is "open source" and I'd appreciate it if you read my comments more charitably in future.
Source code licenses are naturally quite clear about what constitutes "source code", but things are murkier when it comes to ML models, training data, and associated software infrastructure, which brings up some interesting questions.
> We were talking about "looking a gift horse in the mouth", as in it's still a positive thing regardless of the semantic quibbles about open source
Your gift horse in the mouth comment was visibly an aside in the greater discussion being enclosed in parenthesis.
> Nobody would argue that a hypothetical openly licensed Windows binary-only release is "open source" and I'd appreciate it if you read my comments more charitably in future.
That's why I'm using it as an example metaphor in my favor. It's clearly not open source even if they released it under Apache 2. It's not what their engineers edit before building it.
> Source code licenses are naturally quite clear about what constitutes "source code", but things are murkier when it comes to ML models, training data, and associated software infrastructure, which brings up some interesting questions.
I don't think they're all that murky here. The generally accepted definition being
> The “source code” for a work means the preferred form of the work for making modifications to it. “Object code” means any non-source form of a work.
Is this the form of the work that Mistral's engineers work in? Or is there another form of the work that they do their job in and used to build these set of files that they're releasing?
Do the engineers that made this hand edit this file? Or did they have other source that they used and this is the build product?
> (Also, it feels a bit "looking a gift horse in the mouth" to criticize people who are giving away a cutting-edge model that can be used freely.)
Windows was free for a year. Did that make it open source?