From what I can tell Polymer works closely with the Chrome team and the features that this "framework" offers are really just upcoming tech that is being built into chrome and is being made available by Polymer. Things like web components, html imports, templating etc... are built built into the browser so the longer you use polymer the less of polymer you will use.
When I describe polymer to people I say it's an opinion layer on top of upcoming tech. And in addition to that everything is polyfilled to some older browsers, but right now is aimed towards the evergreen browsers.
When you ask "should I use Polymer", what I hear is, "should I get a jump start on the upcoming future of the web"? Lets be clear here, Mozilla, IE, Chrome etc... are all merging on these technologies. Object observers, templating, web components etc... are all emerging, Polymer is just one way you can use them now and tie them all together in an opinionated way. If you don't like it you don't have to use polymer you could just use plain ol JS and achieve the exact same solution, just slightly more verbose and with less support.
Last thing I have to say about Polymer. It's going to stretch the way you think of the DOM and JS. The way everything is structured and the way you should be thinking about it is that every DOM node should be considered an instance of a class. The DOM node exposes an API and away you go plugging and playing into the rest of your JS. It's really fun once you get used to it. This will really stretch you when they introduce the ajax api as a DOM node. wat. Crazy things are happening, some good, some will need to be changed.
Full disclosure I work at Google as a software engineer that specializes in frontend, so I am exposed to this a lot though I do NOT work on the project. That's about it :)