Markdown is very nearly fully sufficient, and for almost any nontechnical work with minimal art or layout, will suffice. It falls flat in some interesting areas:
There's no underbar/underline markup.
There is no native colour markup.
There is no formula support -- not something typically encountered in most texts, but when you need it, you need it.
There is no fine-grained placement control for callouts, boxes, figures, images, etc. They simply appear where they happen to be dropped on a page.
(In several of these cases, you can revert to embedded HTML or styles, which are fine when rendering to HTML, but this won't be picked up by all Pandoc endpoints.)
Mind, if a work consists of nothing more than text, bold, italic, strikethrough, super/sub-script, lists, tables, sections, footnotes/endnotes, and images whose placement is not critical, Markdown is entirely sufficient.
But if you find you need more control, exporting to LaTeX and doing your final editing there will buy you a great deal more control.