One of the most famous songs by the band Styx is Lady, which is by any standards a ballad, a love song. The first half follows the formula to the letter. The song is sweet, despite banal lyrics, and soft. And it absolutely works.
Then, in the middle, the drums come in, playing a military march. If you have heard the song (if you haven't, where have you been the last few decades?) it may never have even occurred to you that that's what he's doing because, in the song, somehow, inexplicably, it feels entirely natural. More than that, it feels inevitable. He plays that because there's nothing else he could be playing. But if you haven't heard the song, or if you step back and think of how incongruous that really is, you may find it astonishing that a military march -- brusque, strident, unforgiving -- should fit into a sweet, tender love ballad at all. Let alone so well that it feels inevitable.
The mystery of why it works is a fascinating one, but to me the even bigger mystery is, how did the band know, as they were writing, performing, and mixing the song, that it would? What ever possessed anyone to even suggest the idea? What made everyone else agree to give it a try? Of all the things that the band could have done at that point, why that?
When you listen critically to pop music and deconstruct it, it's easy to identify things that work and imagine how you could put together a song using those elements. The moment in the beginning of the chorus where the music stops on a beat and is silent for one beat and then starts again -- check. A verse that starts with a stripped-down kick-drum and bass, then builds in more instruments to build to that moment of transition to the chorus -- check. Etc. You can easily take any five pop songs, pull elements from them, mix them together, and make another pop song. (Whether it'll become a top 100 hit is, as a friend has aptly called it, "sorcery", but whether it's immediately recognizably a pop song is almost a certainty. Heck, I could write a program that crafted them.)
But at any given moment when creating such a song, you have a wide variety of things you could put in at that moment. How do you know if this song merits a slow fade at the end, or another repeat of the chorus, or any of the dozens of other things you could put in at each moment? If you look at the history of the decisions made in crafting any big hit pop song, are there endless moments where a small change would have made the song fail, or was the song's success largely assured (because of a catchy melody, for instance) as long as the choices were competently made? Would Lady have been just another forgotten song on a little-known album if not for that incongruous yet seamless military march? How did they know?