After Optimizing Everything, Still Too Slow

Hi there. I started noticing that my WordPress sites were starting to feel slow. I do web app development for my day job, and do web perf as a hobby, so my themes, browser settings, and whatnot were already optimized. Checking network stats, the largest single cost of page load time is the initial html response, before any other files load.

Digging in, I tried everything that I could find that was suggested:
1. Turned off all plugins
2. Optimized the DB
3. Ran the best caching plugin I could find
4. Reduced the complexity of my theme
5. Updated server to the latest PHP
6. Turned on mod_pagespeed
7. Ran tests inside a browser that had no existing history or caches

None of these steps improved the situation by any noticeable amount.

Narrowing down the problem, I set up the most optimum possible scenario for my shared hosting:
1. Created a brand new, unused WP4 installation with an unshared DB on an unshared sub-domain
2. Removed all plugins
3. Removed all themes
4. Created theme that literally does nothing (the index.php file returns the bare minimum valid HTML and does no PHP processing of any kind)

I also created a PHP file in the same location that has the exact same text as the index.php theme file (that is, no PHP at all). This was to verify that it wasn’t my server’s PHP engine, the apache startup process, or anything else related to my server’s configuration. This allows for a hopefully clean comparison of WordPress to arbitrary PHP code.

The results:
WordPress Page – 450 to 1200 milliseconds load time (impartial WebPageTest result tags it at 687 milliseconds for initial load)
PHP Page – 70 to 200 millisecond load time (impartial WebPageTest result tags it at 196 milliseconds for the initial load)

I’m getting over a half a second of delay on every single page load JUST from the startup process of WordPress.

I went ahead and installed W3 Total Cache again to see if caching would help. No noticeable effect.

Worse, I tested this on my own machine (SSD) using MAMP as a local server, removing all network latency and most disk access time issues.

WordPress Page run locally – 230 to 300 milliseconds
PHP Page run locally – 70 to 150 milliseconds (but that’s because of limitations on how fast the browser can begin rendering, the actual data response completed in 4 to 20 milliseconds)

Given these results, I’m at a loss about what to do. I don’t remember the WordPress processing bottleneck being this large in the past. I’ve already tried to implement a few esoteric solutions like inlining all php imports, selectively turning off parts of WordPress by disabling require statements, and turning on the SHORTINIT value. I was unable to get any of these solutions to work, the server would error out before I was able to achieve any results.

I realize that WordPress does a lot, but I only need the basic stuff: posts, pages, a custom theme I make, tags, categories, feed generation, search, image uploading, and an admin interface capable of modifying these things. How can one could make a sub-single-second website using WordPress when WordPress itself will often take more than a second just to start up? Any suggestions on how to proceed would be most appreciated.