The biggest problem you would have is that they share the function names, and if you have this tree (where a line means "links with dynamic library"):
[my binary]
/ \
[___libA___] [___libB___]
| |
[libreSSL] [OpenSSL]
then they will have some conflicts (some refer to this as "spewing nasal demons"). Suppose that they both have a function called mutex_lock that touches a global, but OpenSSL added an extra field. Now the behaviour is dependent on which one is picked first. Surprise segfaults? crashes only on strict alignment platforms? unlocked behaviour because the global is named something else? All of these can easily happen.
ELF doesn't have any builtin namespacing, and when compiled with C normally, the C compiler doesn't modify the names to create something like namespacing in the way other languages do.
If you make an extra effort to put them in different places (preferably not in the default linker lookup path and using something like a per-executable RUNPATH rather than a global ld.so.conf to find it), you can have conflicting libraries. But you are playing a dangerous game which wastes everyone's time and effort.
the fix is trivial though. If libA resp. libB load openssl resp. libressl with dlopen(..., RTLD_LOCAL) as they should, neither's symbols will conflict with the other.
Thanks for the explanation. So even without the fix described in the sibling comment, I can still have programs that link to OpenSSL and programs that link to LibreSSL, but I can't use both (even if transitively) in the same application.
Why would this be the case? Do they use the same file names?