I don't want to sound like a bad person, but there's always a chance that, when you call a guy first, that he'll take it like he can get away with a lot more, because you're so into him. Hopefully this guy has evolved enough to be beyond this primitive way of thinking, but if he hasn't, you could be setting yourself up for dealing with an even bigger jerk than you would have had to deal with before.
If it's any consolation this guy is probably pacing his floor trying to decide how long is fashionably late, so he won't seem like a loser with no life by calling you too early, and ruin his chances with you. Sometimes he'll decide two or three day's is appropiate,but then things may come up, like an unexpected emergency with work, or family where he can't call, and seems like a jerk, when he may have been a really nice guy, who genuinely wanted to call you as soon as possible.
Make him call you. If he seemed like a good guy, not just a good lay, and he has to unfortunately call later than what than what he would have liked, cut him a little slack, it isn't an indication of how interested in you he is. If he takes too long and comes off as cocky when he finally does call, ditch him immediately. Sometimes someone isn't worth wasting your time on, even if they are sexy as hell, and your horny as hell, allowing this to outweigh you're better judgement always ends up being more trouble than it was worth.
I hope you found a nice guy though,and I hope my little rant has helped you're anxiety, at least a little bit.