It isn't always wrong. In fact, it is wrong if -- and only if your spouse does not explicitly give consent to your having a mistress and you fail to inform the mistress that you're happily married to your spouse (and intend to remain so (if indeed this is the case)) . This is because 1) deceiving your spouse is wrong, 2) your spouse should have the right and opportunity to veto any activity likely to make demands on your time, attention, affection, and money, 3) your mistress should know where she stands and what she's getting into, as well.
I don't give a rat's ass if your wedding ceremony was a religious one. Your marriage is between you and your spouse. Not even your deity of choice has the right to interfere, and certainly no cleric/member of the clergy has the right to intrude.
A couple of responders suggest that to have a mistress is psychologically damaging to children. I say bull. In societies throughout history, children have been raised in homes in which the father has one or more mistresses. They generally turn out just fine. As with all child-rearing issues, the important thing is to teach your child to be kind, open, and honest. Having a mistress doesn't need to adversely affect a child if you can demonstrate that you went about it in an honest manner (as I described above) and treat all parties involved kindly and fairly.