This seems less linking and more dependency resolution.
What was wrong with
> ./gradlew dependencies
and then sorting out your duplicate dependencies with improper version issues.
I have spent a lot of time down this road myself and most of it was not fully understanding the framework dependencies, munging of stackoverflow solutions, or apache-commons hell...sometimes a little bit of it all.
Not trying to crap on your hardwork but interested in your motivations if its something bigger which I am not seeing.
Long term I don't think you can properly suggest the appropriate dependency, this is why most build tools provide default selection and let you override with exclude patterns.
Additionally there is a linting tool that will parse through duplicates and offer to clean up your gradle files if that is interesting to you alongside your work.
I dealt with a dependency conflict between two different packages that wanted different versions of Apache Commons. Both of them only claimed that they wanted "greater than or equal to X version." This wasn't actually true, though, because Apache Commons had removed some features and added some features, so the real requirements were
Package A wants Commons >= X and < Y
Package B Commons >= Z
where Z > Y.
Sorting that out meant manually doing exactly what this Depends package does - poking around with different versions and using the missing method exceptions and some other research to figure out the exact nature of the conflict, so that I could pin their versions and set up appropriate package relocation rules.
This tool would not just do that faster, but also more reliably - by doing it manually, I was only able to verify that I had resolved any issues with the code I actually exercised. A gap in test coverage would leave me exposed to run-time errors in production.
To the parent and GP, this is a problem in large code bases. As soon as you bring in a high level dependency like Spring, Spark, or DropWizard the graph goes 20+ levels deep and manual resolution becomes difficult.
Build tools do their best. Gradle has a reasonably sane version based solution. By statically analyzing the method invocations (and eventually properties too), this tool could (one day) just tell you the versions to use that would be most likely to work.
The long term vision would be to actually incorporate this as a dependency resolution strategy in Gradle so you don't have to think at all.
That’s all fine and well if you have one or two broken dependencies from big popular libraries. This technique does not scale if you have a few dozen internal libraries and a few hundred internally deployed artifacts, or more. At that scale you need a more automated solution.
Anecdote: we have a large codebase with probably 100+ 3rd party dependencies, and dozens of internal modules, and that technique works pretty well.
Besides, where's the automated part on the OP solution? I agree the automation, either with this tool, or vanilla project management ones like gradle, could be beneficial.
I can’t speak to gradle, since we don’t use it, but none of the other tools I’ve touched look at ABI compatibility. It’s pretty easy to tell who brought in whatever version after it’s crashed, but catching that in advance is much much harder.
I'm probably too lazy to download and run a JAR but I'd probably run a Maven plugin from the command-line. Try packaging it as a plugin.
I often run into ClassNotFoundException when using an API but not providing an implementation, when using an optional feature that is provided in a separate dependency (Jersey HK2), and when using a feature that has been removed or broken into a separate dependency (JAXB). I believe these scenarios are all fairly common. Will your program help developers resolve those kinds of problems?
Thanks! This is great advice. I will look into it. I would definitely want it to be available for both Maven and Gradle, but I think this should be achievable.
I can't imagine what sort of ill managed corp crapshoot frameworks you're importing in order to necessitate this tool :( but you might also try mvn dependency:analyze to help identify problem areas
- I see this often with Guava. A wants 18 and B wants 21, neither works with the other, but they both with with 20. Because this tool looks at the actual methods being invoked, even if 100% of the methods are present, it's fine as long as 100% of the ones being invoked are.
- Technically, as long as objects aren't exchanged between modules, it would be possible to use ClassLoader isolation to have two libraries each load their own incompatible dependency at run time. It's on my blue-sky TODO list.
> Technically, as long as objects aren't exchanged between modules, it would be possible to use ClassLoader isolation to have two libraries each load their own incompatible dependency at run time. It's on my blue-sky TODO list.
Not only technically, this also works practically. We've done this for years in a POS application in order to be able to load different incompatible versions of vendor-provided hardware drivers at the same time, with the decision on which to use (one, the other, or maybe even both at the same time) being done at runtime. It's based on OSGi for modularization, which also provides other benefits besides classloader isolation, like allowing to bundle multiple native libs for different OSes and bit-nesses within a single jar and automatically choosing the right one to dynamically link into your JVM process that is needed for a particular platform that the application finds itself running on.
Taken to a logical extent, I think this tool could be used to generate conflict-aware ClassLoaders at application start time, and automatically give each module the version it wants.
I guess it would take generating a new jar with a bootloader main() to wrap the existing main()?
What was wrong with
> ./gradlew dependencies
and then sorting out your duplicate dependencies with improper version issues.
I have spent a lot of time down this road myself and most of it was not fully understanding the framework dependencies, munging of stackoverflow solutions, or apache-commons hell...sometimes a little bit of it all.
Not trying to crap on your hardwork but interested in your motivations if its something bigger which I am not seeing.
Long term I don't think you can properly suggest the appropriate dependency, this is why most build tools provide default selection and let you override with exclude patterns.
Additionally there is a linting tool that will parse through duplicates and offer to clean up your gradle files if that is interesting to you alongside your work.