Our small 1532 sqft.house is 3 yrs from being paid off, should we keep it or build bigger hse. on our 3 acres?

We bought our 3 acres 2 1/2 yrs ago to build a bigger home on. We have been taking our time designing our new home, however, we keep looking at the #s & we could have our existing home paid off in 3 years or we could go ahead now & pay off & just stay here but it's only 1532 sq.ft.. We have 2 kids & their rooms are small. We were looking at building at least a 2300-2400 sq.ft home. We live just outside a bigger city, 15-20 minutes from shopping & such but our 3 acres is about 15 more minutes further away from our existing one so we would be a little bit further out. The main road going out to our new land has some issues that the county needs to repair before our kids started driving. We have nice neighbors that are surrounding us that have built in the last 3 years. My husband is 43 yrs old & would like to retire Im sure before hes 70. We would like bigger but would it be better to stay & build on a little game room, pay off house, put in a pool or build a bigger home?