I am the custodial parent. Can I deny visitation?

In theory, of course you could decide to simply not be home at the time your ex comes to pick up the children, or you could decide not to deliver them to your spouse. Although it is possible, you would most likely be in violation of the court order and could be held in contempt of court, depending upon how specific your order is with respect to the visitation. You should have a very compelling reason involving the safety and welfare of the children for withholding visitation, and be prepared to defend your position in court.