what does the term "boundary" means in the definition of organisation?

It is simply what separates or differentiates the system from its environment. It is the same use as talking about a boundary round a piece of land. What's inside the boundary might be yours and what is outside is someone else's.