While I am a front end engineer by trade and I do focus on these issues, I'm still just a guy with no budget and limited time when it comes to my own sites. But still, I'm able to build front end optimized sites that generally generate YSlow scores in the 80s and 90s (as long as one's CDNs setting include my *.cloudfront domains.) A lot of the basics are just that basics. We're not talking about some mysterious process. It's just a change in attitude and approach and after that the results just come naturally.

Once you get past the obvious, then sure, squeezing the extra juice out of it may become disproportionate to the resulting benefit, especially for smaller sites. The giants may also have far more complicated setups, with multiple servers in multiple locations with a CDN – with more factors, each may have tweaks applied that, together, give the extra 100ms saving.

Meaning, if, as a front end engineer you say to yourself, "I'm going to minimize the number of files I serve by using fewer CSS and JS files and using CSS sprites" and you stick to that as a philosophical approach, better performance follows for all the sites you touch.

Just using sprites will, without question, improve your performance by a lot more than 100ms. I did some tests for an internal presentation that showed the difference between using a sprite and using 15-20 individual images. I'll see if I can dig it up.