Tom and the financial guys are right.
Disclaimer: I work for Cumulus Networks and have 14 years+ working for hw vendors
Interesting discussion... I wanted to add that both Tom H. and the financial HFT guys seem correct but they are talking about different things. I work directly with many of these customers and bare-metal is very popular in finance, web-scale and enterprise for capex reasons and for opex related to having an open source customizable OS.
HFT/Algo or trading generally needs the intel fulcrum chipset or something like the Cisco 3000 with warp because it has the lowest latency and the features they need like multicast and NAT. Trading in general is very multicast heavy and needs protocols like PIM. A customizable open source OS may be interesting so that they can add certain custom monitoring or analysis tools or provide some special methods of failure detection. This is probably extremely rare though. Almost all of the functions are provided by hw and all that matters from the sw is that it be stable and update the hw as quickly as possible. There may be control plane optimizations here but I doubt it would be much different than on any other switch (why would you improve protocol convergence on one platform and not do it for all?). The only unique sw would be related to the special ASIC features (i.e. Warp which is really a hw feature you turn on) and how they have to manage any sw structures for those features. Any ASIC is a combination of hw, firmware microcode and software working together. In finance today, you generally see bare-metal in modeling and compute farms or application/web services.
The most common usecases for a bare-metal open source OS are to only use the processes you need for a more stable platform. Use proven open source code that you control and can modify if you have the desire/skills. Use your existing automation tools for servers and your server boot environment to provision switches in the same manner as servers (ZTP). A more rare usecase is to customize the OS to perform functions that cannot be done on regular switches easily. Support and function of server automation tools on vendor switches is usually very poor compared to doing it on a Linux OS but there is a lot of effort to improve this. There can be substantial capex and opex savings.
Controversial part: This is all being done by progressive organizations but the tools and skills are filtering down to everyone and it is becoming very common. The majority of data centers will be a highly automated commodity environment within the next 5 years or your CFO will use a highly automated commodity cloud like AWS instead. It doesn't take incredible vision to see this coming. It has already happened in servers and it destroyed some great companies. The days of making the network smart and expensive and the application dumb are over. The network should be cheap and simple (preferably L3 Clos) and the application smart.
Furthermore, my limited understanding of Facebook, Google, etc. is they are doing special customizations of the switch OS related to how they manage their data and workloads, provide security, etc. This requires lots of expertise and only is valuable at large scale and with very smart developers and operators.
AbeG/Pablo: This is SUPER Attractive to large organizations btw.