For an LLM pretraining company (or academic organization), the speed and quality of new model development heavily depend on the stability and efficiency of the infrastructure. An unstable infrastructure can cause significant project delays—sometimes by several months. We value a strong engineering culture over a purely academic one, as some researchers tend to overlook infrastructure, resulting in work that lacks continuity and long-term impact.

The entropy of infrastructure determines LLM productivity. That’s a good point of one of my colleages.

Good infrastructure should not overly constrain thinking. With moderate reuse and constraints, it should not significantly slow down the pace of development and validation. A commmon scenario is, to add new feature to an existing system is hard. If develop from scratch, it will take 3 days, however, to continue developing over an existing system requires 7 days of thinking before developing, although the new feature has only 30% correlation with the existing features.

Top-level design may be an underestimated factor in determining success or failure. As a designer, one should constantly observe the operational details of the system and promptly refactor bottlenecks.

Entropy is not necessarily better when lower—it achieves maximum marginal returns when matched with the current goal size, team scale, and personnel capabilities.