If you are using LESS or Sass, it's really not hard at all to only include the parts of Bootstrap that you need. In my experiences, I often need just need maybe half of it and don't bother much with the Javascript stuff. I don't think Bootstrap really has performance or size as a first priority. It's really just a big toolkit, ready to go so you can whip out a functioning app.
This is true of a lot of frameworks. I've honestly not been a huge bootstrap fan, and Foundation 5 and 6 also can be built with SASS and you can easily remove components you don't use.
If you're trying to maximize compatibility while also get designs up quickly, any modern framework is preferable to no framework at all.
I've used semantic UI before and I loved it. But I'd never use it for anything else but a desktop use case. Apart from the css size the render time is very, very slow.
Bootstrap [1]: 118 KB
SemanticUI [2]: 537 KB
Don't forget to add the JavaScript. I know GZip is a thing, but still...
[1] https://github.com/twbs/bootstrap/blob/master/dist/css/boots...
[2] https://github.com/Semantic-Org/Semantic-UI-CSS/blob/master/...