No. That's generally because once something gets popular and jumps to the front page, it gets a huge boost in visibility, especially from people who weren't looking at the niche subreddit it comes from.
A lot of those people aren't interested in that content, so it will suddenly get an influx of downvotes.
