Imagine this set up.
It is a small walled town or more accurately a settlement. To the east of the market square is the manor house, to north at ground level are a few shops to cater for trade caravans and above them a hostel or flop house for itinerants. The west has the gate house and the south facing on to the market place is a taverna with seats spilling out into the square.
Our characters should ideally be sat at the taverna, outside. Maybe they are waiting to meet a patron or even waiting to get paid? We will come back to the characters later.