I was rejected at visitation by my inmate, what should I do?

There must be some hard feelings there. Why did you wait FOUR years? Have you ever attempted to commnicate with him by mail? Letter writing might be the best way for both of you to being an exchange of feelings - about what went wrong between you and how can you fix it.