Bugs And Unstable Code Threaten Second Life's Future

A group of Second Life's business owners and users are taking Linden Lab to task for ongoing, serious bugs and stability problems in the virtual world which, they say, threatens its survival.

Problems include daily meltdowns in navigation systems, in-world chat, instant messaging, and community groups used to bring together users (known as "residents" in Second Life jargon) with common interests.

The problems are particularly significant as people and real-world companies begin using Second Life as a platform for real business, said Cristiano Diaz, a Web and Second Life developer in Miami who's led a rebellion of residents against Linden Lab.

"Second Life is incredibly unstable and has become more and more unstable as it's grown in the last two years," Diaz said in an interview. "But at the same time, customers are paying hundreds or even thousands of dollars a month for services with Linden Lab."

He added, "It isn't something like World of Warcraft, where you're paying $14.95 a month and if it sucks you can go somewhere else."He said he's concerned that Linden Lab will join the long parade of startups who collapsed and were destroyed under the weight of too much success and rapid growth.

Officials at Linden Lab didn't immediately respond to requests for an interview.

Diaz got so frustrated that he rounded up a petition of frustrated Second Life users and business owners, outlining his complaints. He sent it to Linden Lab on April 30, and posted it to the Web on a new Web site, Project Open Letter, http://www.projectopenletter.com/. By Thursday evening, the petition had 3,800 signatures.

Linden Lab held a Town Meeting Thursday afternoon in a Second Life text chat session to address the concerns. Ironically, the meeting itself was plagued by bugs: The area filled up before the meeting started; that happened two hours before start time, according to reports from residents. Linden Lab broadcast the chat to anyone interested using in-world IM. But the IM portion of the discussion was unreliable, leading to frequent interruptions from users complaining that they weren't receiving text.

At the meeting, Linden Lab CTO Cory Ondrejka -- known as "Cory Linden" in-world -- said improving stability and eliminating bugs are a top priority. "69% of the development staff at LL are currently on scaling and stability and that percentage is rising over the next few weeks," he said. The company is aggressively recruiting developers to add to its staff and help solve problems. "It is the highest priority and the focus of the majority of our design, coding, and QA [quality assurance] work. We are also working on the next-gen architecture to allow far more scaling than the current design." InformationWeek described that next-gen architecture in March.Ironically, Diaz himself was thrown off the session after just 15 minutes. He said, "To have a Town Hall about stability go completely to hell is a testament to exactly what I'm talking about."

Ondrejka posted a response to the open letter Wednesday. The response, which addressed the concerns in some detail, began:

The problem is especially frustrating as individuals and big companies like IBM, Cisco Systems, and the NBA do real business in Second Life, Diaz said.

"Linden Lab is getting so much press in real life, companies are investing millions of dollars in it," Diaz said. "It almost feels like fraud, they can't really deliver on what's being promised."

Linden Lab needs to shift focus from developing new features, like voice, and concentrate on squashing bugs, Diaz said.

Diaz, who goes by "Christiano Midnight" in Second Life, developed the popular Snapzilla service for posting Second Life images to the Web, as well as the SLUniverse community site.

One of the most significant problems is with "inventory" -- the database that keeps track of the virtual goods and services people build and buy in Second Life. The virtual world has its own thriving economy that's a jewel in Linden Lab's marketing of its service. Unlike online games, where players are forbidden from buying and selling goods, Linden Lab encourages the trade.Linden Lab posts the amount of money spent in the previous 24 hours on its home page; that figure was US$1.4 million as of Thursday evening.

All of that virtual merchandise is being tracked using the inventory system, which is buggy and losing items. Second Life blogger "Moo Money" reported that all of her inventory was lost last month. During the Town Hall meeting, resident "Digital Digital" complained of losing a gaming machine valued at 16,000 Linden Dollars or about US$61.

The same inventory is used to store residents' own creations -- residents can have hours, days, or weeks of work wiped out by an inventory glitch.

Second Life also is suffering breakdowns to the most common form of transportation in-world, the instantaneous transfer of avatars from one location to another, known as "teleportation." Teleporting is often blocked when the world has a heavy load, especially on the weekend.

Instant-message communication breaks down, especially the Second Life-to-Internet gateway that sends IMs to e-mail when a user is offline.And residents are complaining of failures in one of the most popular tools for people to get together in-world: Groups. Residents can form groups of common interests -- for example, InformationWeek has a group for its readers and friends in Second Life -- and send out group notices, own server space (known as "land") in common, and have group affiliations displayed on their profiles. Diaz tried to form an Open Letter group in Second Life to discuss the petition, but found he couldn't invite residents to the group, or send out notices to members.

Linden Lab has shut down two-way avenues of communication as the service has grown, Diaz said. They shut down their forums, and transferred discussions to a blog with comments, and frequently delete comments that are critical of Second Life. E-mail to customer support often goes unanswered.

Ondrejka's blog post Wednesday addresses the concerns. He says the inventory problems are related to the distributed MySQL databases in which inventory is stored, and bugs in rebalancing scripts for those databases. Those bugs have been resolved, and other possible causes of inventory problems -- as well as other bugs -- are being investigated.

Linden Lab posted a transcript of the Town Hall meeting Thursday afternoon.

One of the biggest challenges the company faces is "testing without full load" -- testing new software before it's subjected to the traffic demands of Second Life, Ondrejka said in the chat. Linden Lab is working on improving its "synthetic testing" tools and several times in the meeting he urged attendees to go to the beta area in Second Life and report bugs.Linden Lab runs Second Life on data centers in New York and Dallas, and hopes to open data centers overseas to help manage the load. Ondrejka said in the chat: "We have an architectural quirk in how we talk to the [databases] -- a layer of single-threaded data servers -- that would suffer in situations with greater ping latency, so we need to fix that problem first, which is being worked on. Once that is fixed, we will begin international deployments. This is critical since over 65% of our use is from outside the US."

Second Life now supports only a limited range of graphic cards on client PCs; Linden Lab is working on improving that range, Ondrejka said.

The session concluded with a question from resident "Radslns Hutchence," who said: "As residents invested in making SL work, what can we do, in addition to being patient, to help with any of these initiatives?"

Ondrejka responded: