Where I live, the majority of the time spent on an average drive is spent on roads with one lane in each direction and few, if any, chances to pass. In other places, this might be less of an average trip, but I bet it's more of most trips than people really realize, when they consider time rather than distance. This circumstance leads to an inevitable conclusion which nevertheless is not realized, or ignored, or defied, by most drivers:
How you drive has essentially zero impact on when you get there.
That is, every decision you make, how fast to go, whether to let someone out, whether to run the traffic light, etc., makes ultimately no difference in determining when you're going to get where you're going.
Look at that car in front of you. If you drive slow or fast, reckless or cautious, if you stop at yellow lights or race through red lights, odds are you will still end up behind that car, and when you arrive depends entirely on when they arrive, not on what you do.
At most, you might end up one more car-length behind them if you let someone out in front of you. But odds are that they'll turn off eventually and you'll still end up in the same place. And maybe that car in front of you will turn off, too, and then you'll get to race ahead... to the next car, whereupon the same thing happens. None of that actually depended on anything you decide. It all depends on the happenstance of when other cars happen to be getting onto or off the path you're taking.
There are a few moments where a tiny change can have a bigger impact, but they are very few, and very rare, and still likely to cancel each other out, and also rarely depend on your choices. Maybe the act of making or missing a traffic light can cause ten more cars to get in front of you, or avoid that. And maybe if you had sped up just a little bit you could have made that light. But while we imagine this to be an opportunity that happens all the time in driving, in point of fact, when we get caught behind a light, or fail to get caught, the vast majority of the time that depends on the car in front of us, too. Times when our own choices make that much difference are exceedingly rare. And even when they do happen, odds are good most of those ten cars will turn off your path and in the end the effect will be a lot less than you'd expect.
So go as slow as you have to to be safe, because in the end, you will arrive at the same time as if you hurried up to each stop. (Try it sometime. It's hard to make a really definitive test -- one day's traffic is unlike the next. But if you drive recklessly for two weeks and then cautiously for two weeks, the stopwatch will reveal the truth.) Let people out: worst case you'll end up one car farther behind all the way to your destination, and thus arrive ten seconds later, and the more common case is you will end up arriving at precisely the same time you would have. And above all, relax. Listen to the music. Pay attention to the road. Don't worry about the trip. It's going to come out the same either way.