SQLite also features backticks as literals (though they're not the main option).
And I fail to see how it's much of an oddity, as other RDBs use all kinds of BS for literals. SQL Server for example uses [square brackets].
If anything backticks have some history of being used for string literals in some languages (e.g. Javascript and Go), whereas in many others they are used for command substitution (which is not far removed from what a string literal combined with a processor does in Java).
Overloading that behavior by default would likely blow up a lot of code and certainly violate the principle of least surprise.