If you going to use 3 columns, then medium large image should do fine. And also due to the infinity scroll, decreasing the post per page won’t make much difference. As it will load more when scrolled down.
The calculation for layout position are done on the fly but it doesn’t make the loading slow. The slow loading is caused due to the wait time to allow to load all the image before placing it in the layout.
Still, i don’t find your site so slow at all. Looks good to me.