Samsung's warning: Our Smart TVs record your living room chatter - CNET

[Rob Seastrom]

kf4hcw <kf4hcw at lifeatwarp9.com> writes:

> On 2015-02-11 21:40, Rob Seastrom wrote:
>> Me too.  Email in '82, ARPAnet in '84.  You could at least respond to
>> my points as I did for yours rather than calling what I wrote a
>> "myopic rebuttal" and trying an argument from authority.  Most of us
>> here took debate in high school, college, or both and find that tactic
>> unbecoming.
>
> Your opening remark was
>
>> Ah yes, the "only one side pays" canard.  OK, I'll be happy to debunk that.  When people talk about "fast lanes" it's a clear sign that they are unclear on what's going on.
>
> How is that anything other than myopic -- essentially saying (I
> paraphrase): "First, your a liar. (canard), and second you don't know
> what's going on." My "argument from authority" was simply a rebuttal to
> your apparent claim that I didn't know what was going on.

It was directed more at the sources that you cite than at you.

But if your opinion is drawn solely from those sources, you've got a
problem.  The writers are ill-informed and absolutely pandering to the
audience.  Remember what their job is - selling advertising on the
site, just like TV news - and consider what they write in that context.

You know how you cringe whenever you see media coverage of something
you know a lot about vs. something you know a little about?  Yeah,
that's how I feel about this in the popular technical press.

> I know this: Verizon leveraged their monopoly position to extort
> additional profits from Netflix simply because they could.

Absolutely incorrect.  Verizon refused to give away product (on-net
bandwidth) that has been on their rate card for over 15 years, for
free.

> The problem
> could have been solved for the cost of a few patch cables and a few
> network cards in a handful of data centers, and in fact it finally was.

Have you priced 100 gigabit optics and cards lately?  The slot space
in a CRS3?  The fabric upgrades in the CRS necessary to support the
4-up 100g cards?  That's what's being used here.  The price of a
"handful of cards in a handful of data centers" ends up being several
million bucks even after steep discounts for someone like Comcast or
Verizon are taken into account.  That also doesn't take into account
the costs of moving forward by several quarters internal network
enhancements that were on the roadmap anyway.

> But before that could happen there were many many dollars of potential
> profit at stake so Verizon went after it.

I guarantee you that the amount of profit that Verizon is making once
the fully loaded costs of the interconnections to Netflix are taken
into account are minimal.

> It is my opinion that this is not the way to do business and it's not in
> the interest of the greater good. I will never be able to factually
> support that opinion because all such attempts would require conjecture,
> circular arguments over ideology, or references to predictive models and
> historical events that can be shown to be incomplete, unprovable, or
> irrelevant. So, if you want to control the conversation by framing it as
> a debate then you win. I concede. I never intended to enter a debate. I
> answered a simple question: "... show how the new FCC rulings will help
> any of this..." again I paraphrase... but I took that as an honest
> request for supporting evidence of my opinion and offered precisely that.

I'll stand by my statement that the current proposed FCC regulations
don't have any upside.  If it's a prelude to collecting USF money to
be used to subsidize expanding to underserved areas like they do with
the phone company, that's a potential benefit.  Everyone seems to be
leaning away from saying that they'll do that.  A different approach
might provide some upsides, but language like "no paid prioritization"
is addressing a problem that does not exist now, nor has it ever
existed.

> As for the "only one side pays" canard I will say the following:
>
> Of course Cogent and (3) oversubscribed -- that has been a fundamental
> practice of all networking providers since day 1 and is a cornerstone of
> the business. Verizon, Comcast, AT&T, and every other company in the
> industry does the same thing all the time. The bursty nature of Internet
> traffic almost requires the practice in order to make the business
> models work. How much of this practice a particular provider can get
> away with depends on a lot of factors -- greed balanced against support
> costs and market perception being  just a few of the factors.

"Bursty nature of Internet traffic" is something that is the case at
the edge of the network where there are individuals, small collections
of servers, or both.  It is not the case at server farms or CDN nodes
where hundreds of thousands of TCP streams cancel out each other's
stochastic effects.

Stated another way, if you sell a 10 gigabit pipe to a business to
support an office tower full of desktop users, it will be loaded to
somewhere between 50 megabits and 9.9 gigabits at any given instant.
But the peaks don't last long at all.  You can oversubscribe that
because the odds of two customers' peaks coinciding for any meaningful
length of time.

If you sell a circuit to Netflix, who exclusively runs CDN nodes, you
will find that the circuit is loaded to 1/3 capacity on the off hours
and 99% (which you can get away with when you have homogeneous traffic
on a 10g or 100g pipe) at peak.  You can't oversubscribe that.  10g in
implies 10g out; if not today then very soon as their business grows.
As we say in the business, "no stat mux efficiencies for this circuit".

> My position on the matter is that when I paid Verizon for my Internet
> connection my SLA did not say that Verizon could choose which parts of
> the Internet I could reach reliably. It stated that they would do their
> best to provide my stated bandwidth through their network to any point
> on the Internet I chose to reach to the extent that connectivity is
> under their control.

Verizon didn't choose to have this problem; Cogent and Level(3) did.
Someone put 10 pounds of stuff in a 5 pound bag.

> If I want to pull down a stream from Netflix or Youtube or my
> grandmother's basement web server then I've already paid Verizon for
> that connectivity and it is my expectation that they will resolve any
> problems within their control to ensure they meet their SLA terms. If it
> turns out that a few zillion similar customers make the same choice to
> download from the same location on the 'Net then it is Verizon's duty to
> honor that agreement for all of us and make the stated bandwidth through
> their network available. 

At all relevant times, the stated bandwidth through Verizon's network
was there and available, as demonstrated when things suddently got
better with direct interconnects.  The problem was in (3)'s and
particularly Cogent's networks.

> If the problem is outside of their control and
> grandma's server is falling over then that's not Verizon's problem and I
> don't expect them to fix that. But, if the problem that they
> intentionally allowed connections leading to gradma's server to become
> saturated so that they could bully grandma into giving them money then
> that's not ok. It's extortion.

Verizon didn't intentionally allow circuits to become saturated.
Interconnecting networks sold more bandwidth than they could deliver
over existing circuits and asked Verizon to upgrade for free, outside
of the terms of their contract.  Even settlement-free peering has a
carefully negotiated contract around it; it is 2015 not 1997.  Verizon
said "fine, but your growth rate is X, not Y, we need to renegotiate
this", and the other side balked.

Meanwhile, you suffered as a customer and I don't discount that.
Blaming Verizon in this case is a bit like blaming the grocery store
and their distribution network when a hard winter causes the price of
orange juice to go way up.

> Note also that I don't expect the transmitting side not to pay -- They
> should, and in this case they did. When I want to stand up a server farm
> I pay for my upload bandwidth and my download bandwidth -- and so does
> everybody else. Everybody sort of winks and nods about the unspoken
> over-subscription and the bursty nature of the 'Net and unwritten
> expectations about what the SLAs really mean and then they shake hands
> and it's all fine... that is, right up until somebody gets greedy.
>
> Netflix paid to send. Myself and a large chunk of Netflix customers paid
> to receive. Everybody in between played their version of the "how much
> can we oversubscribe this week" game and everything should have been
> fine -- especially given the extraordinarily low cost of the actual
> hardware required to deliver bandwidth these days... especially at
> peering points in large data centers.

I already explained the cost of the equipment involved.  Remember
that's just for the handoff; it doesn't take into account adding
bandwidth through the network; you don't add interfaces in only one
place to do add capacity for something that is 40% of your bandwidth
at peak hour.

> http://en.wikipedia.org/wiki/Peering
>
> Peering agreements between large network providers have been a key
> factor in building a robust Internet. Everyone benefits from this
> practice because diversity and redundancy are fundamental to
> reliability. If my network helps you connect your customers to servers
> on another side of the network then it's good for everybody if we make
> that connection -- pretty simple really. Better still if I make multiple
> such agreements with multiple providers so that everyone involved can
> benefit from a rich diversity of routs and copious bandwidth.

Since you bring up peering I feel compelled to cite my CV before I
refine your points.  I turned up my first BGP sessions in 1993 and had
a personal router with a personal ASN on the FDDI version of MAE-East
from 1996 until it wa decommissioned all the way (June 2nd 2001, just
checked an old screen shot).  I've answered peering email and been the
guy who sets up peering at three different organizations not counting
the aforementioned personal ASN.

Peering works only when there is a balance of value provided to both
sides.  Traditionally, bandwidth symmetry has been the foil that has
been used to determine whether both sides are benefitting from the
agreement equally.

It's a flawed model, but it works after a fashion and has served the
industry OK for 20 years.  When there is an imbalance in the benefit
to both organizations, "paid peering" is offered.  It's basically
on-net bandwidth, for a cost that is typically less than the cost of
bandwidth to the entire Internet.

Of course Netflix traffic is grossly imbalanced.  Bulk video data goes
one way, acks and HTTP requests go the other.  I'm pretty sure "paid
peering" was always on the table as an offer to Netflix, and indeed
that's what they ended up going with.  But they wanted free for a
direct interconnection.  See the problem here?

> In this case, Verizon found themselves in a position to squeeze Netflix
> by refusing to upgrade their interconnects with intermediate providers.
> Comcast showed them they could get away with it. If they didn't have a
> monopoly position on the last mile to a HUGE chunk of Netflix customers
> they would not have had the leverage to pull it off and they would have
> had to play nice --- again, my opinion, as there is no math to do here
> on what that means or whether my take on good business practice is
> relevant in any way.

Every time there has been paid peering, throughout the history of paid
peering, the side that ends up paying complains about it.  Netflix
wasn't the first and won't be the last, but economics is economics and
the infrastructure upgrades aren't free.

If there had been two or three providers serving the last mile...  the
problem would have still existed inside Cogent and (3), it's just that
the ports to everyone would have been congested instead of the ports
to one provider.  The same complaints would be read on Ars Technica,
just not as focused as they are now since there's more than one
company to hate.

The solution would have been the same too.  Properly sized direct
interconnections to the various networks in question.

> I think it is clear that reducing diversity and redundancy by cutting
> out these middle networks is detrimental to the fabric of the Internet.

Having an A->B->C connection in a dozen cities is marginally worse for
overall reliability than having an A->C connection in the same number
of places.  It is radically worse in terms of enforceable SLAs, as
recent experience has shown.  "Detrimental to the fabric of the
Internet" does not have a technical underpinning; it's handwaving.

> It might be great for Verizon's profits, and those of other monopolies,
> but it definitely works against the greater good; and growing
> consolidation like this weakens the Internet. There is good math on
> those engineering principles and settled science so I shouldn't have to
> debate those points.

See my point supra regarding the cost of the devices involved.  I
suspect that the profit, such as it is, that Verizon may be making on
these fully loaded cost of these interconnects is pretty darned low.
Low single digit percent.

As for the "good engineering principles and settled science", well,
let's just say I am in complete agreement and do this sort of
calculation as part of my day job.  So I agree that we shouldn't have
to debate these points.

> No matter what the facts are, the argument between us is really whether
> the policy and business choices that were made were the right ones and
> whether the FCC's take on this will make things any better --- again, an
> opinion. Actually, I never wanted an argument -- I was just sharing my
> thoughts.

Based on the totality of my knowledge I don't see the FCC's current
position making things any better.  

> There is no way to win a debate on an opinion, and I have no intention
> of trying to do so. Based on the adversarial nature of your initial
> remarks I doubt I would enjoy it, and I have every reason to believe the
> exercise would generate more heat than light.

I'm sorry you took offense at "canard".  Perhaps I should have said
"allow me to provide some details and additional facts that will
totally undermine that argument".  Unfortunately, I have to stand by
my characterization of the "fast lane" metaphor.  It's a bit of a
standing joke in certain parts.

-r