This is how the world is. No one is really an expert. No one really has the answer.
But this is really more a product of capitalism than anything else. Under capitalism you just have to keep moving even if you’re just making garbage and debt. There’s no reason to stop and think, because that is seen as a cost (even though it costs more to move without thinking).
Even the best companies that do factor in planning (at least in concept if not actually in practice most of the time) end up with the other problem of “resume driven development” where things that are totally fine and actually working get replaced with things that don’t work because someone needs a new project to get their promotion.
Capitalism produces garbage and puts the people who are least qualified in decision making roles. This still happens in natural systems, but much less. In (healthy) anticapitalist organizing, the people who know the most are generally asked to lead and when they don’t know what to do they stop and figure it out before moving forward.
Aimless wondering can still be a problem, but it’s not forced by the system to continue it’s just people who are learning.
There are people who are knowledgeable and good at their job. Knowledgeable enough to be experts. Those are usually subject matter experts, including developers.
The issue is that no one can guarantee an outcome or that they’ve picked the right approach.
There are answers, they just take a level of experience to reach that most people aren’t cut out for. You gotta be several principal+ IC roles or Dir+ mgr roles in before the patterns congeal into a plan.
Challenge is operating at those levels for extended periods requires a super fucking insane level of competency and dedication. Most people hit that spot and coast till retirement cause you’re at $500k+ at FAANG. Few keep looking for new opportunities unless forced to or they’re those corporate robot sharks with the dead eyes.
and we all look up to the people who look like they know what they’re doing (they don’t but they are very convincing at looking like they do)
In their defense, if they don’t pretend like they know what they’re doing everything would fall apart.