I go with a " spezzato", which is pants and blazer of different fabric and/or color. Less formal than a suit and allows for more creativity. Neapolitan blazers look especially good on fit people. If tailored or made to measure, don't forget to unbutton the last button on the sleeve, a top player move.
Most, if not all, of my colleagues dress in sweatshirts and ill-fitting jeans. I'm sure some of them see me as "eccentric" or "aloof" or someone who wants to stand out. And they would be right (not aloof though).
Interesting! I still try to keep things fairly casual. If I do some sort of "jacket", it usually a pretty casual sport coat with a crewneck underneath, paired with dark jeans or chinos.
Assuming you're not showing up in something undeniably "dressy," like a dark-colored suit with a white shirt with French cuffs, being "casual" is more about your attitude with your clothes, how you wear them, than the clothes themselves.
Assuming you're wearing the same clothes in both situations, does it look like you're ready for a wedding or that you found the first few clothes in your closet with your eyes half closed and somewhat magically the color and fabric combination is eye-catching?
What's a typical outfit look like for you? Do you have a "uniform"?