Failing to act is still a decision you have made. It is not an absence of one as it is often looked at by many people. Choosing not to make necessary changes and improvements, makes you complicit in maintaining the negative environment you might have inherited.
When you know that changes have to be made, make them!