I am very confused now.

My marriage went through two crises because my husband loves some other girls.
The 1st time he wanted to break up with me after we married two years,
The 2nd time happened last year when we had two kids; the little one was only 2 years old. He said that he don’t love me anymore because I am not lovable anymore.
I admitted all the mistakes I made (I was spoiled too much before) and begged him come back to me. Now we pretty much recover our relationship, both of us want to make our marriage happier, and forever. But we still fight (oral) with each other sometimes. I could not totally forget that girl he had affair with, and kept asking him if he still keep contact with her and other detail. This always made him mad, and said I do not trust him.
My thought is Frankness is the base we build up our new relationship. If he tell me more about the affair with that girl, it means he don’t care about sharing past with me any more, it means he more care about my feeling now, this will make me feel better.
His thought is we need to forget about past, and start over.
I am so confused now. Maybe I am wrong. I need somebody wise there to help me out with my thought. I am afraid I may make our relationship go crazy again.
I really don’t think it’s all my fault, but he always made me feel it’s my fault, I made him to do such, such thing, to be angry, etc.