More People…And More Issues

Second Life has now broken the 15,000 peak-concurrent-user mark, as reported by Shockwave and confirmed by Resident statistician* Tateru (in comments). According to things I heard from Sumar Morgan and others at The Shelter in Exile, 16,000 has, in fact, been broken as well. (As I heard that from them, I pulled up the Web site myself, to see just over 15,000 logged in…15,074, as I recall.)

This figure is probably a more important one than the “one million registered users” figure that I’ve written about previously. After all, one million users, not logged in, merely take up space in the databases. Fifteen thousand users, logged in all at once, represent load on the network connections between LL and the rest of the Internet, load on the sim servers that have to pump visual information back to those users while at the same time keeping track of all their actions to update the state of the world (in as close to real time as can be managed), load on the database servers and asset servers that have to keep track of every object they pick up or rez and every Linden Dollar they spend or receive, load on the network interconnecting all those servers as people move about and teleport from place to place…oh, the list goes on.

If you think there’s some correlation between the high user load and the appearance of more problems with lag, dropped packets, database server glitches that result in inventory strangeness or sudden teleport failures, and so forth…it’s probably not just a coincidence.

As I have remarked elsewhere, not just on Evans Avenue Exit but on other sites, as well as in-world (including in earshot of Teeple Linden, when he dropped into The Shelter in Exile momentarily), Second Life is perhaps the single most complex software system I have ever seen or used. Yes, even more complicated than, say, Windows Vista. 🙂 With SL, you’re dealing with distributed processing on a vast scale–something I can speak on with some authority, as I work for a company selling supercomputing cluster systems. Even the largest clusters we’ve ever shipped, though, pale in comparison to the amount of hardware required to keep the Grid functional. (The servers for Anshe Chung’s land holdings alone, for instance, would fill at least three and possibly four standard racks.) Program coding on that scale is decidedly a “nontrivial” problem; even in the more mundane cluster-computing field, whole projects and commercial products have sprung up to deal with the complexity of writing and running massively-parallel programs. SL is so specialized, though, it’s hard to know whether any of that would help it (or, indeed, if LL may already have made use of such tools).

How do you build a world on this scale? Of course, SL didn’t start out at this scale; it started very small, and then, like Topsy, “just growed.” At a certain point, the software underlying SL has to be not so much “designed” as “evolved”…and sometimes, you get so involved in fighting fires, it’s difficult to make forward progress. Yet forward progress is being made; I look at SL as it was when I first started, and, even in six months or so, things have pushed ahead quite a bit.

If we are to automatically assign the blame to LL when things go wrong, they at least deserve the credit for things that go right…and sometimes, when I look at SL, I am reminded of the old saying, “The most amazing thing about a waltzing bear is not how gracefully it waltzes, but that it waltzes at all.” In the meantime, we would do well to learn patience. (Cue Ed Gruberman: “Yeah, yeah, patience, how long will that take?”)

UPDATE: Tateru comes through with some pretty pictures of the concurrent user load. As you can see, we didn’t actually break 16,000, but we came damn close. And pretty much never is the load under 6,000, except when the grid is down for maintenance. I don’t know about you, but even 6,000 simultaneous users is way more than any software I’ve ever written has had…
* – Among her many other roles. Sometimes it’s easy to believe in a small army of Taterus…



Filed under Philosophy, Technical

5 responses to “More People…And More Issues

  1. Everyone’s been debunking the numbers, you’re the first person I’ve read who seems to have found the right number. Congrats. 🙂

    This is a really good entry… saving it for reference in the future. Thanks.

  2. Well, at least I managed to lay eyeballs on a user count over 15,000, even if I was unable to personally verify the 16,000 figure…

    Hopefully Tateru will have a post soon with some statistics so we can see if we really broke 16K.

  3. Some of the residents posts on the Linden blog could be cut and pasted from Ordinals “The Random Linden Lab Flame Generator”

    It seems clear that it’s not widely recognized what is involved in making SL work. All those sims, database servers, all the routing and networking etc. The mind boogles.

    It’s so easy to say, “Oh it just runs on 3000 servers or so”, but when you have been in a datacentre and seen the racks and racks of gear that just seem to go on into the fluoresent forever.

    Then the questions like, who pays for the power, who pays for the switches? It all adds up.

    I might expect LL will be after more money to keep it all going.

  4. I’m afraid peak was the 15,312 figure that I reported on Shockwave’s blog. We’ve yet to exceed that.

    I can provide raw data to back that up (rather too lengthy here). Still hoping to turn it into some graphs.

  5. Well, I’ve got a graph up now, for the last 14 days’ worth 🙂

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s