Firstly, that's more like a buy and hold... which means, when you get bigger and bigger you will not outperform SP500 (by much)... as you can clearly see in the data... his returns are moving towards SP500.
This is BRK's outperformance to SP500:
View attachment 185053
Since it's more buy hold, so if you trade very actively you can't swing too large amounts. So you have to adapt from a multiple times a day algorithm to a more conservative buy hold... and you would have to put more eggs in the basket, therefore you would be diversifying. Which should normally be a good thing, but that also means you will approach a more market conform return... hence why his returns are sliding towards 0 outperform.
Secondly, he started out with a lot less... so he had no issues with scalability at that time, or any time before he got bigger. Start of BRK was with 2.5mln? Still decent amount... But again, this contributes to him approaching zero in outperformance.