Indeed, legally, the first year of any business is considered its "startup" year (and possibly longer, in specific situations), and during this time the "startup" is eligible for tax breaks or exemptions under state and federal tax codes. This use of "startup" precedes Silicon Valley.
Yes, a bakery is a startup if it is just starting up. So is a taco cart, if it the vendor is just starting out. This is the classical definition of startup.