It doesn't matter the Member has access to forum or not. This page displays all registered users in your WordPress user system. This is user list, this is not filtered by access. Also if a user listed here it doesn't mean he/she should have access to forums. I'm really sorry but there is no option to filter this list.

Just bumping this is up as there doesn't seem to have been a solution for this yet. Hopefully, there is one now-- would even pay for a user-groups-directory-type-add-on if there was one!