Im in love wit a married man who doesnt give me much time but i know he loves me, i want 2hav a child with him?

I was married for 5years and now since divorced its been a year, we have a daughter together with my ex husband. During my divorce i got back to a man i used to date before marriage, He is a Muslim married man who is 15years older to me, he has 4 beautiful children. 11years ago the relationship all started as a sexual relationship but turned into love eventually (during my marriage we had absolutely no relation at all), through this one year of being with him, he has opened up to me a lot of his negative side, but i have accepted and loved him as he is, he always taunts me that he isn't the right guy and i shouldn't love him as do, we seldom meet sometimes after 2 weeks or even after over a month. he keeps his distance so as i should not get used to being with him all the time (he doesn't say it, but i know it), i know he loves and cares for me too, but the loneliness is really unfair. I have been wanting to a baby for so long, and i want to have a baby with this man that i love so much, ive never told him this, but i hinted it and he said it would be great. My daughter has never met him and im not ready for her to meet him because i do not know where this relationship is heading. I sometimes think that I should just let go and move on, but my heart doesn't allow me. Deep down inside i know he isn't the right person, but somewhere in my heart it tells me he is the one. I am totally confused. I need help.