First, some necessary but ultimately misleading issues were found with my database, autoload options and plugins. This comes from years of installing and testing plugins and the fact that most WordPress plugins when they are uninstalled do not automatically clean up their database tables and autoload options. This leads to a bloated database that becomes much more cumbersome than it needs to be. One particular plugin in general Broken Link Checker was one of the most likely culprits for performance issues, this plugin basically scans all outbound links on your site from everywhere (articles, comments….etc) to find out if they are broken, leading to dead pages or bad sights. This is a very useful plugin, but when you have thousands of posts and 30,000+ comments and it has to scan everything this can be a huge drain on performance. It also had many cron entries and autoload options and my performance problems occurred a few weeks after using this plugin so this was the most likely suspect at the time. Database tables and cron entries that start with blc would give you a good indicator it is for Broken Link Checker.
Ultimately though the problems looked like they were caused by the rampant xmlrpc.php attacks going on online, and Iggy first notifed me about an article he read on Ars Technica about how WordPress sites were being compromised to do Denial of Service (DOS) attacks against many other sites. I had support do some digging and sure enough they found many IP’s doing attempts at xmlrpc.php so they put in some deny statements and sent most of the offenders to a 403 error page going forward. I haven’t had any 502 errors since the xmprpc.php was locked down, but this isn’t a permanent solution as more IP’s can crop up, I am thinking of a denyall statement except for only the specific IP’s that need access, and this may be something to implement in the near future.
In the meantime I recommend you read this article http://perishablepress.com/wordpress-xmlrpc-pingback-vulnerability/ about locking down your xmlrpc.php with your .htaccess file.
For general performance troubleshooting, leverage P3 Performance profiler to help identify which plugins may be causing issue, it also shows you how much is loading on every resource to get a good feel of what may be causing performance issues with plugins. For cleaning up autoload options there was a great plugin called http://wordpress.org/plugins/options-optimizer/ which did a great job removing thousands of autoload options from orphaned plugins.