Is there a way I can not have a member directory?

Or, as an alternative, only include users that have posted a non-private post?

My situation is that all of my paying members will have access to the forum, but if they choose not to participate on the forum, I want to respect their privacy and not have them listed in the member directory (or simply not have a member directory), 

Chris