Disk has one major challenge and one major challenger.
While the capacities of drives have continued to rise, the major challenge for mechanical drives is speed, or lack thereof. Disk spindles are limited today to 15k RPM and have been for a very long time. The first 15k RPM drive shipped almost a decade ago. Processing speeds did not increase over the last decade, or memory speeds, or network bandwidth speeds. Imagine having to use decade-old technology today -- when it comes to mechanical drive speeds, you are.
Tape does not suffer this affliction. Tape drive speeds have continually increased over that time, and while they still don't have the same random read/write speeds as disk, once you get tape going, if you can sustain it with a continuous data feed, it is very fast.
There's been plenty done to get around the per drive speed limitation: large RAID groups, larger and more sophisticated caches, short stroking drives, intelligent data placement on drives, the list goes on. But at the end of the day, storage software will be only able to do so much, especially when there is a more viable and very price competitive alternative available soon...SSD.