Should I pursue an affair?

I'm in my late thirties with 2 small children, but my relationship with my hub is a disaster. He never touches me and I am always the one to initiate . I cannot begin to explain the pain and desperation this has caused me. I began to blame myself and then I realized that he has a problem. He says he doesn't know why he has trouble with me. I am so sick of wasting these years and I have gotten to the point where I just don't care what he thinks since he doesn't seem to realize or care about the damage he is inflicting. He says I am agressive and angry and he's right. I feel so shitty about myself and yes, I have lost respect for him. A few years ago I started to be attracted to another man and did nothing about it because I felt so guilty. As the years p my attraction has grown stronger and the feelings are definitely mutual. The problem is that I don't know him well and just recently found out that he is married. In all the years I saw him he was NEVER accompanied by his wife. I feel so guilty about wanting him and I can see that he feeling just as guilty, but the feelings are exploding. I cannot erase him from my mind and feel like contacting him. Maybe a fling would satisfy me. I am not looking for a relationship. I know this is not correct, but my situation makes me feel as if I am going to explode. Please help.