Filter, sort, search and pagination fetch the new items from the server with AJAX, so the delay is your server’s response time for a page.
A page cache does not speed these requests up, because they are POST requests.
Pro’s AJAX Cache and Preload (Settings → General) answers them with a page that renders only the gallery, and prefetches the next page and hovered links. That removes most of the delay. See General settings.