Tuesday, 23 November, 2004
Don't be a menace to your local subnet while shaping your packets on the hub: Part I

(This started out life as an e-mail to some of my geek friends, but then I hubristically decided it would be interesting enough to post here, and have accordingly added some contextual background.)

I've been following this thread over at DSLReports (or whatever it is they're calling themselves now) with a fair bit of interest since it hyperbolically hit BoingBoing two weeks ago. The gist of it is that BitTorrent users in greater Vancouver are finding that their downloads are crawling to a halt. Some people started screaming "censorship" (demonstrating they know squat all about censorship), Shaw said, "uh, not really," and Cory Doctorow then more or less accused Shaw of lying about what they were really doing to the packets.

To be clear, I don't know what's going on here. I'm not a particularly heavy BT user (I use BT to fill in TV shows I didn't see because I can't figure out how to work my new VCR), and I've noticed that things are slower than they used to be, but at the same time it's not like I think I have some right to the instantaneous arrival of multi-hundred megabyte files on my hard drive from wildly topologically disparate servers, so it's not like I'm going to call up someone and complain when my transfer rate drops below 200 kb/s.

To get the full flavor of this "discussion" (which is really little more than omnidirectional bitching and threats to move to Telus), you can probably sample a few posts out of every page from 19 through to 30-something. Along the way you'll encounter people who are shockingly clueless about the realities of Internet provision and the costs associated with running a big, real network. One of the more interesting types of complaints comes from users who routinely download in excess of 50GB/month -- which is, as it turns out, about where Shaw's transfer cap is, i.e., they get unhappy if you move more than 50GB of data down in a month. "I paid for unlimited high speed Internet access," this complaint goes (I summarize). "So why is Shaw limiting us?" (Note that you could make this argument even if you didn't break your transfer cap.)

Back in the caveman days of the Internet, when we thought 28.8 was blazingly fast (and when I couldn't get my off-board 28.8 modem to connect at 28.8 owing to a crappy on-board UART), Netcom got burned by offering "unlimited dialup connections" to their users, many of whom went out, connected, and never disconnected. At all. Tying up Netcom's modems and phone lines and costing the company way more than they were getting in revenue from these users. Netcom should have known better, but at the same time, so should the users: Someone's offering to give you a 28.8 connection to the network that's always on for $24/month? Get outta here. In comparison, when I was helping to set up an ISP, I seem to remember paying about $225/month for a pair of 28.8 connections to Insync.. a connection which wasn't any more complicated than taking a pair of 28.8 modems, plugging them into two phone lines, and configuring some scripts to redial if the carrier ever got lost. (Why were we running around with 28.8 dialup links? Because the 56k leased line was too expensive for us at the time, of course. We used the 28.8 lines for our mail server, as I recall.)

Someone should have realized something was up. And so when I signed up with Shaw, all those years ago, I knew that while I was buying a connection that was theoretically on all the time, and theoretically would not limit my usage, there would probably be some kind of limit to what I could and couldn't do with it. In the 5+ years I've had cable access, I haven't yet found those limits.

Here's the problem with moving 50GB/month worth of data over a connection that costs $30/month. 50GB/month works out to 1.6GB/day, 71MB/hour, 1.2MB/minute, or about 20kB/second. In bandwidth terms, that comes to around 165kbit/second. Not too shabby, but the problem is that you're not paying for a 165kbit/second connection. Let's do some comparison shopping with real bandwidth costs here.

A T1 is 1.5Mbit/second, or 1,536Kbit/second. For the priviledge of having your very own T1 in Canada, you can expect to pay around $550/month after tax (if you're able to negotiate a killer deal with your NSP, which, if you have no history of doing such things.. well, lotsa luck with that). Don't forget to budget another $1,000 for a router that'll do what you need it to do. (No, the Linksys box you have is not a router in this sense; it doesn't do BGP routing.) At the end of the day you'll have a connection with a provider who won't care if you download 1.5Mbit/second all the time -- that's what you paid for, after all.

A T1 will cost you $0.36 per kbit/second. Cable costs you about $0.18 kbit/second. I can promise you that neither number accurately represents the actual costs of providing IP service over those lines. But let's say for the sake of argument that the T1 cost is close to the right figure. At T1 rates, a 165kbit/second connection should cost about $54 -- roughly double what you're paying for it right now.

And that's all I have time for right now. Up next: Usage patterns, the economics of big networks, and why providing consumer-grade bandwidth services is a lot like running an airline.

Sunday, 28 November, 2004
Why the bandwidth biz is like owning an airline: Part II

Apologies for the time it took me to get back to this topic: I was waiting on actual hard numbers. One of the reasons it's tough to price bandwidth is because no one will actually tell you what they want you to pay; in this it's a lot like buying a car, or getting airline tickets. ("So.. let me get this straight. You're not going to tell me how much you want me to pay for this vehicle until I've given you a deposit and made an opening offer?" was one of the more notable things I had to say while shopping for a new car back in the spring.) For kicks, I called a couple of local bandwidth providers who would probably prefer to remain nameless (we'll call them T and G) and asked about T1 pricing to my doorstep. Vendor T wanted $150 just to come out and look at my doorstep; Vendor G didn't think they wanted to do business with some yutz doing research, and I can't really blame them for taking that view. "Lookit," I said, "I'm just trying to get a handle on how much a T1 line from you would cost."

It turns out that the first rule of bandwidth pricing is that you do not talk about bandwidth pricing. The second rule of bandwidth pricing is not, surprisingly, that you do not talk about bandwidth pricing but rather that everyone pays less than you do. By the time I finished escalating calls and explaining myself over and over again to Vendor T and Vendor G, they agreed to provide me with service quotes so long as I didn't actually try to buy anything from them. "Oh, by the way," they said, "This is for the line only. We're not actually including connectivity here." This was Wednesday. By Thursday I managed to convince them that those numbers weren't useful if they wouldn't provide connectivity. (This is pretty common: You need to arrange the circuit and the IP separately, and if you ever spent any time in ISDN hell you know exactly what I'm talking about.) So my sales reps (I had reps now -- you see what I'm willing to put up with for the sake of some research?) did a little bit of soul searching and decided, again, that if I wanted to have this information I couldn't hold them to it. "We haven't properly assessed your situation," the rep from Vendor T said, "and we haven't evaluated your location so I can't even begin to provide you with any information on installation costs." It seemed they were concerned that I might try to use this information to leverage a cheaper rate later on, which makes me think that the real mechanism for pricing T1 service is to figure out how much you think the client is able to afford, take all of their money, and ask for another $300/month for "service." (In that sense, they're exactly like every other big computer-related company I've ever dealt with.)

I promised them I wouldn't hold them to it, since it was just a ballpark quote and, as I reiterated, it was only for comparison purposes. Both Vendors said, basically, "$900 month," or $0.585/kbit/second. The 165kbit/sec connection we were speculating about is now suddenly $96/month.

(I have no idea whether they were fleecing me or not. You can look at someone else's experiences with T1 ordering and pricing, bearing in mind that was four years ago and things have changed. The installation charges are still the same, incidentally.

Oh, did I mention you need to get something called a CSU/DSU? You do. Pretty much the only brand worth talking about in terms of CSU/DSU equipment these days is Vendor C; you're going to pay about $700 CAD for a used 175x router with a WIC-1DSU-T1 module installed on eBay. (I didn't spend any time on the phone with Vendor C to figure out new prices.) The box works more or less the same way a cable or a DSL modem does, only it's much bigger and it costs a lot more. Did I also mention what a total pain in the ass IOS is? I didn't? Oh, my bad.

The point is, there's a lot more to T1 ownership than just getting the line and saying, "Giddy up." Once again, this story should be eerily familiar to anyone who spent any time trying to get an ISDN connection to work, right down to the bend-over-and-pound-me-in-the-ass charges imposed by the communications companies.)

So. We've established that a connection that costs $30 should, if you price the service based on heavy usage of 50GB/month, cost about three times as much. How, then, does Shaw make money on a $30/month cable connection? Simple -- they don't count on you to use that much bandwidth. If you think about it, it makes sense; most people want Web access, and Web access (even with some P2P usage) is inherently bursty. There are small packets of data moving back and forth; once in a while you'll see a traffic spike, but most of the time the usage curve is pretty flat. Because of the topology of a cable IP network, bandwidth that you're not using is bandwidth that I can be using. It's kind of like a party line.

But because the bandwidth is shared -- all bandwidth, even DSL bandwidth, is shared after some point -- and this is a zero-sum game, the more movies Johnny downloads, the harder it'll be for Aunt Bea to get to her e-mail. And if Johnny's pal Freddy starts hauling down movies, too, Bea's going to have some serious issues. Freddy and Johnny didn't pay for the bandwidth they're consuming, though they do have access to it; Bea's paying more than her fair share. So guess who gets the boot?

Shaw makes money because they charge the majority of their customers -- who don't use a majority of the bandwidth, so far as I can tell -- more than what they actually use. In other words, it costs Shaw the same amount of money to provide the service -- say $0.50/kbit/sec -- but some of us get better value for our money. By my math, the break-even point, assuming Shaw is actually charing what it costs them to provide the service, which they arent', is around 15GB/month. Beyond that, you're using something you're not paying; Shaw's not going to get too pissed, however, because their space is kind of like a commune that we all pay for, and if you use more and I use less, I'm not going to notice.

Unfortunately it isn't that simple. No one in the bandwidth biz -- at least, in the consumer bandwidth biz -- actually charges their customers what it really costs to provide the service. In this it's a lot like owning and running an airline: I bought tickets to Calgary for the end of next month and paid Air Canada $450 for the round trip. I can promise you that it does not cost Air Canada $450 to fly my ass and my luggage between Vancouver and Calgary -- it's going to cost at least three times as much. But AC makes up the difference in a variety of ways, or at least it tries to (by, say, fleecing business class passengers, and those people stupid or desperate enough to buy the ever-elusive "full fare" tickets), and when it doesn't, it ends up in bankruptcy protection (whoops, too late). Airlines like WestJet have figured out how to charge passengers $170 per flight segment and still be profitable because they've figured out how to keep their costs low; companies like United get into fare wars with Southwest and end up losers because there's no way giant airlines can compete with smaller, more nimble organizations -- not in that way, at least.

The analogy breaks down because Shaw doesn't have a lot of competition in the cable Internet business. But a company providing customers a service and charging them less than what it costs to actually provide the service is going to get nailed sooner or later when the customer base shifts, and that's more or less what has happened in the last two or three years.

When cable Internet service -- an dDSL for that matter -- first came out, the big bandwidth-intensive applications were mostly pipe dreams, and almost everyone figured they'd involve video on demand. After all, the cable company was providing the service, and nobody has ever accused the cable company of being visionary. This is the same organization, remember, that won't let you order your cable channels a la carte, and so you end up with a collection of dumb crap -- the golf channel, the motoring channel, the high life insurance premium channel, the old person's channel -- that you don't want and won't watch simply because it came bundled with something you do want. So it's not like they saw P2P coming.

With the advent of P2P, however, and in particular the advent of P2P coupled with big, fast pipes, it became possible to move multiple gigabytes of data across the network with relative ease. And at that moment, it became possible for a minority of users to have a deleterious effect on the majority -- the commons, as it were, became overrun by a small number of citizens, and the place was wrecked for everyone else. DSL Reports readers might have you believe that QoS, quality of service, is three dirty words, but it's really not; it's an attempt to ensure everybody has at least some speedy access. Fair? If you're a heavy P2P user you'll say no. But as someone on the other side of the fence, or who can at least see both sides of the fence, I'm not so sure. Absent increased capacity, it's about all they can do -- and adding capacity isn't as simple as going out to the local Electrode Hut and buying a couple 2,000' runs of Cat 5. Upgrading the infrastructure is a multi-hundreds of millions of dollars project that takes years.

No one ever guaranteed cable users a certain level of service. That Shaw is attempting to at least ensure that its heaviest users don't have a deleterious effect on their neighbors is laudable, and worth noting as such. It certainly isn't censorship and I doubt very much that they're doing stateful packet inspection on everything that passes through their network, but rather throttling based on usage patterns. P2P users see an oppressive regime; I see attentive network managers. Dozens of pages worth of threatened litigation (over what?) and poorly informed speculation later, no one seems to have grasped the real problem -- how do we ensure that a shared resource like metropolitan connectivity is parcelled out in a fair and equitable manner?

For those who remain unclear on the concept, allow me one last bad analogy: Consider the electric system. There's a finite amount of capacity in the system as a whole, and adding more is a slow, expensive process. If you use more electricty than you pay for, eventually you end up in the dark. If you use a lot more electricity than your neighbors, you end up in the dark too, but so do your neighbors, and they're kind of pissed off as a result. The power company will probably cut you off before you can do that kind of damage, though, and if you really want to use that much juice you're going to have to foot the bill for installing the additional capacity to your neighborhood, along with paying for the power itself.

I suspect that, in the not-too-distant future, you're going to see something similar come into play. Again, though for the first time with broadband services. Broadband access will no longer be flat-fee, or at least, it may be flat-fee up to some point, beyond which it becomes a metered service. This is how the majority of commercial data hosting plans work, and I can't really see why it wouldn't work for individual consumers -- other than the fact that certain segments of the population will threaten to riot as a result of it. There are too many bandwidth-intensive services coming up fast, and not enough capacity. So some kind of restrictions are going to be necessary.

Next: The phantom menace.

Epilogue: That was basically it, as far as the rant and explanation went. I got tired of writing this series and gave up. I gave up on a bunch of other things, too. You might have noticed if you were reading the blog at the time.

phloem@fumbling.com