Well, its not hard to draw conclusions this time!
If you can, feel free to enable Joomla cache, especially the plugin-based one. Unfortunately, both caching methods have their drawbacks and can not be used anywhere – for example, if your site heavily rely on logged-in users, both Joomla caches are inapplicable. See the “References” section below for more information.
The silver bullet here is APC: it almost double Joomla throughput and halves latency time. APC is a really good piece of software, so good that it is planned to integrate it into the next PHP version (5.4). Its only drawback is a somewhat higher memory consumption (due to bytecode cache), but the advantages easily overcome it.
Have a nice day!