I think I was about ten the first time someone told me I should watch Doctor Who. I'd been devouring science fiction from the time I was first able to get to the library (even when I could only go into the kid's side) and was fascinated by any science fiction I could get on TV too. So it only made sense. Trouble is, Doctor Who was only shown on PBS since it was a BBC show, and thus, only at times way past my bedtime. (This is before VCRs, let alone DVRs.) So it was years until I finally got to see some Doctor Who. I think I was 13 or so.
And I really didn't like it. Despite everyone saying I should, and despite that it really seemed like something I should. I tried it again a number of times, but it never really hooked me.
When they revived the show recently, I watched on and off for most of a season and found it okay, but nothing that I really felt a need to keep up with. It felt like mild filler to pass the time, but I didn't mind missing any of it, and sometimes I just zoned out of paying attention. I'm not sure why. There's no single thing about it I can say that I can't also say about other shows I like. It's campy, but sometimes I like campy. It plays cheap with continuity, but that's also true of other shows I like. I am not sure why it almost never gelled for me.
Sometimes I enjoyed the character of the Doctor, particularly his effusively enthusiastic side -- this is the Christopher Eccleston and later David Tennant Doctors, and I particularly enjoyed the latter, though I expect that's as much to the credit of the writers and how they chose to depict that Doctor as to the actor -- but I often found the stories he was in not nearly as interesting.
It's still the case that a lot of people I know with similar tastes really like Doctor Who and I still wonder why I'm the odd one out.