The concept you are describing is called Innovator’s Dilemma and imo the most recent example for it happening is with legacy car manufacturers missing the ev transition, because it would eat into their margins from ICE. But i am not sure if this is a good example for it.
However imo it seems like a great example for what Steve Jobs describes in this video about the failure of Xerox. Namely that in a monopoly position marketing people drive product people out of the decision making forums. Which seems exactly the case here where the concerns of an engineer were overruled by the higher ups, because it didn’t fit within their product segmentation.
I am pretty confident in guessing that they are not doing it for selfless reasons. Imo the reason is that the less information they give the user, the more you are beholden to the algorithm choosing for you.
But depending how they hide it it actually might not just be users, but also companies that e.g. buy ads from them. The less information they get, the more they need to trust whatever metric google offers them