Oh one more question.  I run a paid membership site.  Is there a way for me to clear out users who have subscribed to topics if they stop paying and I don't want them to get notifications?  Right now I move a customer who stops paying to "Guest" which gives them no access.....but what happens on the back end?  Are they still getting notifications of new topics?

Please update to 1.4.9 version, navigate to Dashboard > Forums > Dashboard admin page and click on [Update Topics Statistic] button.

Thanks.  Just a report back.... I disabled my smtp plugin and it worked very fast.  However I can't just send email with the default server php so I installed Mailbank (turned off email logging), turned off logging and setup sending through mailgun API and there was a HUGE improvement.  It still takes a few extra seconds when I add a topic into a forum that has a lot of subscribers but it's no longer taking the site down or causing an issue.

I tried that email que plugin but it didn't work...emails wouldn't go out.  Probably cause I have WP Cron off and am running it with a regular cron.