Should I be concerned if he seems distant in his letters and not answering some of my questions ?

Of course you should be concerned and it is to be expected, from both your perspectives. He is wondering what you are doing, thinks the worst even if you have never given an inclination that you'd stray. It's natural, but it will take a Herculean effort on both of your parts to keep it going. Give him his space for a bit, stop peppering him with questions. He's in prison, he's getting punished - do not pile on with your questions and inflict more pain.  He's hurting and he's depressed. Send him stuff he likes to read, if not reading send magazines or puzzles. If you want it to work, you have to forgive and forget. It's always going to be a fact in his past but to move forward you have to leave your unanswered questions behind. If you cannot forgive the past, it will not work. Big decisions for you. Good luck to you both!