I don't quite understand why love between men and women starts to develop in the first place

#1

I don’t really understand, why did the male and female lords secretly have an affair in the beginning? I think that at the beginning, shouldn’t the little girl’s heart be moved like that towards Young Master Wandering?