I guess you can look at it as "here's $20 which is silently implying that I think you're hideous, so excuse me for the insult.". Lol, which is how I always assumed they take it when we do that.
FYI I've never tipped when I said no thanks, but I can understand the possible guilt. There's a few times as I'm walking out I can imagine what's going on in their head as I probably wasn't the first to do it.