I just tested Imgix with some of my own images. With your LQIP parameters, I got a response time of 500ms for the first load. That was with an image 1200px wide. I don't know if that would compound significantly if requested 10–15 images at the same time.
However, after the first time you load the image, Imgix caches it. My subsequent requests for the same blurry image loaded in 100ms. Are your images taking 2s to respond every time, or just on the first load?
Also, I'll note that Imgix has just released blurhashes. Which might accomplish a similar result more quickly.
If you'd like, you can share your code with me and I can take a look to see if I can spot any issues. Feel free to send me a DM with your GitHub repo or a ZIP file.
Based on my understanding, the cache never — or rarely — empties. If it ever empties, it is very infrequent, so it should have virtually no effect on your website performance. But, if you're still concerned, let me know and I can try to help with optimizations.