I am the non-custodial parent. Can I decide not to accept visitation?

Yes, you can fail to exercise your visitation. Certainly you should let your ex or your children, if they are old enough, know that you cannot pick them up that day, so they are not left waiting for you to appear. If you routinely fail to exercise your visitation, it is possible that the court could order you to pay additional child support to compensate for the extra expenses your ex incurs because of it. This conduct could very adversely affect you if your ex wants to move away from the area at some point in the future.