Update on Cox blocking of Craigslist

The original claim of a Cox "blacklist" originated from a statement by Tom Foremski at Silicon Valley Watcher. Foremski originally wrote:

Back on February 23rd Authentium acknowledged that their software is blocking Craigslist but it still hasn't fixed the problem, more than three months later. That's a heck of long time to delete some text from their blacklist.

Now, he says (quoted by George Ou at ZDNet):

I assumed there was a blacklist - I have no idea how Craigslist is being blocked

In fact, we know now that it's a combination of a bug in a firewall driver produced by Authentium software and unusual (but not incorrect) behavior by the Craigslist webserver setting the initial TCP window size to 0. The facts of the problem came out (at least between Craigslist, Cox, and Authentium) at the time the problem was first reported, was fixed in a beta release within weeks, and has only affected Cox customers who use Authentium's security suite.

BTW, I disagree with Richard Bennett and George Ou's remarks which attribute the problem entirely or largely to Craigslist--the behavior of the server is not contrary to the RFC. The initial SYN packet from the client to Craigslist is responded to by Craigslist with a SYN-ACK packet with window size of zero, which means don't send me any data, only an ACK. The client then sends an ACK (completing the three-way TCP handshake), at which point Craigslist sends an ACK packet with a larger window size which the pre-fix version of the Authentium software fails to process. The initial response of the Authentium software to slow down is a reasonable and apparently desired response by Craigslist--they want new clients to hold off transmitting data (an HTTP request) until they give the OK. Authentium took full responsibility for the problem, and they were right to do so.

The story from Foremski was uncritically repeated by Matt Stoller at MyDD, Timothy Karr at Save the Internet (and a couple of other blogs), and now in a Wall Street Journal op-ed piece by Sen. Ron Wyden (D-OR), in a lapse from his normally good judgment about Internet-related matters (e.g., the Cox/Wyden Internet Freedom Act of 1995 and the Cox/Wyden Internet Tax Freedom Act of 1998).

Stoller and Karr went on to repeat the "blacklist" claim even after having the full story, and I don't believe either of them has retracted the claim that this issue is relevant to the network neutrality debate.

Craig Newmark complains that he didn't get good responsiveness from Authentium, which Authentium disputes, but he has indicated satisfaction with Cox.

The story has been picked up by George Ou at ZDNet (here and here) and by Glenn Harlan Reynolds at Instapundit (here, here, and here).

This issue was a user software application issue that had no more to do with network neutrality than a browser incompatibility issue, a webserver disk failure, or a fiber cut. Each of these things can prevent a user from reaching some specific content, but none is imposed by the network provider or remedied by act of Congress or the FCC. Those who continue to treat it otherwise even after knowing the details are demonstrating questionable judgment and integrity.

UPDATE: Craig Newmark has now stated that there was no deliberate blocking here and the Authentium explanation is correct. I've exchanged a few emails with him asking whether the behavior of the Craigslist.org webserver is specifically intended to regulate the rate of new HTTP connections (and whether the behavior is coming from something like an application-layer switch negotiating the TCP handshake); he said he's passed that on to his technical team and I'll report here if I get confirmation or refutation on that point.

One puzzling paragraph of his latest blog post is this one:

One good outcome of this is that we flushed out a swiftboater (in the generic sense), and this helps me understand the way disinformation gangs operate. Unfortunately, in some blogs, a good guy has been linked with the swiftboater, which isn't fair, and hopefully, we can do something about that.

I'm not sure who he's calling a swiftboater, who he's calling a good guy, and who he's calling a disinformation gang. So far as I can see, the disinformation gang in this incident has been the "Save the Internet" crowd, who still have yet to admit the clear facts of the matter. I asked for clarification, but Craig declined to identify who he's referring to (except that he's not referring to Matt Stoller or Timothy Karr).

UPDATE: July 12, 2006: The Craigslist.org webserver has changed its behavior and no longer sends a SYN-ACK packet with a window size of 0; it now gives a window size of 4380. This change by Craigslist.org works as a fix to the Authentium issue. I wonder why they only made the change now.

China's mobile death vans

BLDGBLOG has some photos and information about China's mobile execution chambers, used to bring state lethal injection capability to poor localities that can't afford to build their own execution facilities. Amnesty International says they have evidence that Chinese police, courts, and hospitals are engaged in the organ trade, and suggest that the mobile death vans may be involved.

BLDBLOG cites USA Today reporting that there are 68 different crimes punishable by death in China, more than half of which are non-violent offenses such as tax evasion and drug smuggling. All executions are recorded on audio and video, and shown live to the local law enforcement authorities.

The only other country which had mobile death vans that I'm aware of was Germany under Adolf Hitler. The Einsatzgruppen's mobile killing units were known as "death vans," which used carbon monoxide gas for execution.

Ann Coulter on no evidence for evolution, refuted

P.Z. Myers at Pharyngula has put together an excellent starting point for anyone who would like to see the overwhelming evidence that supports evolution, contrary to Ann Coulter's claim in her new book, Godless, that there is no evidence to support it. 20 science journal articles published per day, a new book published every other day, statements from scientific societies, online tutorials, blogs by scientists, and more.

He also requests that if you can find a single paragraph anywhere in chapters 8-11 of her book that is at all competent or accurate in its description of science, to send it to him.

Matt Stoller lies about site blocking

Matt Stoller has a post up at MyDD dated June 14 titled (with ironic accuracy) "Please lie to me about Net Neutrality" in which he gives the following as an example of unwarranted site blocking that shows the need for net neutrality regulations:

There's a pervasive myth that there has been no discrimination on the internet against content companies. That is simply untrue. For one, Craigslist has been blocked for three months from Cox customers because of security software malfunctions.

Back on February 23rd Authentium acknowledged that their software is blocking Craigslist but it still hasn't fixed the problem, more than three months later. That's a heck of long time to delete some text from their blacklist. And this company also supplies security software to other large ISPs.

Without net neutrality protections, cable and telecom companies will have no incentive to fix these kinds of problems. Already, it's quite difficult to even know that this is happening because they are quite easy to disguise.

However, Ray Dickenson, the VP of Product Management at Authentium, the company which makes Cox's software firewall, had already explained this problem in a post on MyDD back on June 9 when Stoller first brought this up, and it has nothing to do with a software "blacklist":

I'm SVP Product Management at Authentium, Inc. We make the branded security suites that many Internet Service Providers, including Cox Communications, offer to their subscribers. I'd like to take this opportunity to set the story straight on the Craigslist issue that some Cox subscribers have experienced.

In February, we started receiving support calls from users of our branded ESP security suite at ISPs like Cox Communications and Patriot Media. These users had problems accessing the Craigslist.org web site.
Our engineers investigated the issue and found a glitch in our firewall driver that made the Craigslist site very slow to load, or not load at all. (Technical details below)

We contacted Craigslist to learn why only the Craigslist web site was affected and also had our engineers fix the firewall driver. The fixed driver is in QA and will be part of a new release this summer. Our support team has been offering the beta firewall driver to customers who call in and are willing to try it. The support team also assists users uninstalling the software if necessary.

Authentium is dedicated to providing the best possible Internet experience for all users of our security suite, which appears under many brand names. We applaud the efforts of ISPs that go the extra mile to provide free security software to their subscribers and will continue our efforts to make the Internet experience safer and easier.

Technical details:
We found that the Craigslist.org web site sends a TCP packet with a zero-length window. A zero-length window indicates the server is experiencing congestion and cannot handle more data. Our firewall driver responds by sending data only one byte at a time, even after the server increases the TCP window size. This is the glitch we have fixed and are QA testing. Any changes to network drivers must be made carefully, tested thoroughly, and certified before general release.

Authentium's initial response to the Craigslist.org webserver is exactly as specified by RFC 793 (which describes TCP) about the proper behavior when a host to which you initiate a TCP connection specifies a window size of 0, as others have pointed out at the Save the Internet blog:

Flow Control: TCP provides a means for the receiver to govern the amount of data sent by the sender. This is achieved by returning a “window” with every ACK indicating a range of acceptable sequence numbers beyond the last segment successfully received. The window indicates an allowed number of octets that the sender may transmit before receiving further permission.

The bug here is that when the Craigslist.org host later attempts to increase the window size, the Authentium software fails to do so.

It's a bug in Authentium, but it's also arguably a bug in Craigslist.org, which also had the capability of offering a fix but has failed to do so. To characterize this as an example of discriminatory website blocking by Cox is dishonest, and to repeat the claim that this was caused by "text" in their "blacklist" after being informed otherwise is a lie.

Coming on the heel's of Stoller's YearlyKos admission of not understanding the issues and calling for personal vilification of his opponents, this makes a solid case that he's in way over his head and should not be relied upon as a source of information in the net neutrality debate.

UPDATE: Timothy Karr of Save the Internet has jumped on this bogus bandwagon on his Media Citizen blog as well as on the Save the Internet blog (already linked above with the "others have pointed out" text) and at the Free Press Action HQ blog. At the last source, Karr was clearly already informed of the cause of the issue, as he links to this fairly clear explanation from Authentium, in which the Authentium CEO, John Sharp, says that they immediately contacted Craigslist.org and made a beta fix available to their customers (including Cox customers) within a couple of weeks. For no reason I can see, Karr describes this by saying that "The CEO at the 'security software' company in question is equally opaque about the Craigslist blocking." What's opaque about the explanation, and why does he put "security software" in quotes--to suggest that this is malicious blocking?

Graph of Phoenix Housing Inventory

I plugged all the previous data into Excel and generated this graph:


I wonder what happened in December and early January. The trend is amazingly linear, otherwise.

When do we start considering Phoenix a buyer's market? Now? When inventory hits 6oK? When the trend shows clear signs it has reversed? As I said in the comments to the previous housing inventory post, I think I want to start making lowball offers when I get back there!

Josh McDowell helps discover Noah's Ark

Yet another rock formation has been misidentified as Noah's Ark by evangelical Christian explorers ("Arkeologists"). They apparently forgot to bring a geologist or archaeologist with them, but they did bring "some of America’s leading businessmen, an attorney who has argued several cases before the U.S. Supreme Court, and two leading apologists" and take some incredibly unimpressive photographs. The expedition was led by former Costa Mesa, CA police officer turned "international explorer and author," Bob Cornuke, who runs something called the BASE (Bible Archaeology Search and Exploration) Institute. I hope his ethics are better than those of former nurse-anesthetist turned international explorer and author Ron Wyatt, who found a profitable career by claiming to find virtually every possible biblical site and artifact. (Wyatt, a Seventh-Day Adventist, was best debunked in a book by his fellow SDA members Russell R. Standish and Colin D. Standish, Holy Relics or Revelation, a book I highly recommend.)

Ed Brayton has done a good job of dissecting the claims in the announcement article. As he notes, this is far from the first such claimed discovery of Noah's Ark. This one is in Iran rather than the usual location of Agri Dagi in Turkey. I actually give them credit for not looking on Agri Dagi (Mt. Ararat), since the Bible only says that the Ark landed in a region called Ararat, not a mountain of that name (2 Kings 19:37, Jeremiah 51:27).

For a review of some previous claimed Noah's Ark sightings, see my 1993-1994 articles from Skeptic magazine, "Sun Goes Down in Flames: The Jammal Ark Hoax" and "Update on the Ark Hoax".

Andrew Kantor changes his mind on net neutrality

USA Today technology columnist Andrew Kantor has changed his mind, and no longer supports net neutrality regulations:

Not too long ago, I was very much on their side. "Imagine you make a phone call to a friend," I wrote then, "but instead of hearing it ring, you get a recording: We're sorry, but the person you are calling has not paid Verizon to carry his or her conversations.

But I was wrong.

I did what's easy to do: I blew things out of proportion and borrowed trouble. As I learn more, I realize that Net neutrality — at least the way it's being touted today — is a bad idea.

It pains me to say it, because many organizations I respect are fighting for a law. But I'm not.

Kantor now says that net neutrality doesn't force Internet traffic into the slow lane, it prevents the building of a fast lane, and that there is little risk of telcos blocking competing services or content because of the principles in the FCC's August 2005 policy statement (the "four freedoms"). He concludes that

The most a Net neutrality law should say is that A) network providers must carry any legal data regardless of the content or who it comes from, and B) network providers must offer the same services at the same prices to any customer — i.e., they couldn't charge YouTube more for a connection than they charge Disney.

Hat tip to Richard Bennett's Original Blog.

Demonization of adversaries is wrong, Matt Stoller

Ed Brayton's Dispatches from the Culture Wars has an excerpt from an article in Christianity Today by Yale Law School Professor Stephen Carter, a well-known black Christian conservative who authored the book Confessions of an Affirmative Action Baby. In the article, Carter is arguing against the common demonization of the ACLU by Christians, pointing out that while he disagrees with the ACLU on the establishment clause, they are also a big defender of the free exercise clause and have consistently supported Christians in free exercise court cases:

More to the point, the ACLU is often right about the First Amendment's free exercise clause, taking on fights that others refuse. It might surprise some critics that the ACLU defends the free speech and free exercise rights of, well, Christians.

The larger point of the article, however, is to condemn the mode of argument that characterizes those who disagree as irrational, dishonest, or evil simply in virtue of that disagreement:

I am more concerned about a habit of mind that seems to be growing among my fellow Christians, both political liberals and conservatives. That is, we seem to mimic the secular world's conflation of disagreement with wickedness, as if not sharing my worldview places my critic outside the realm of rational discourse...

I've seen similar habits expressed by people on both sides of the net neutrality debate. For example, in Matt Stoller's presentation at the YearlyKos convention, he admits that he doesn't understand the relevant technical issues (and proceeds to demonstrate it by suggesting that "non-neutrality" will cause dropped calls, when in fact it's non-neutral QoS that will prevent them). He asserts that it is fun to beat up on "these bad people" and that it is very important that Mike McCurry be personally vilified. That's explicit endorsement of irrationality, of emotional demogoguery over fact and reason, and should be condemned by everyone in this debate.

Ed Brayton concludes:

But rational people, people who care about truth and accuracy, must fight this tendency. We must try and evaluate every claim using the same criteria. Does the evidence support it? Are the conclusions drawn from the evidence logical? Any claim that fails to meet those criteria should be rejected, regardless of whether it supports our agenda or not. Likewise, any claim that withstands that scrutiny should be accepted as valid, regardless of whether it supports our agenda or not. None of us will ever be Mr. Spock, but we should strive to evaluate all arguments as though we have no stake in the outcome. Some, like the STACLU crowd, make no attempt at all to do so; we should not emulate them.

I agree.

Douglas Ross's Network Neutrality Index

For those looking for a series of arguments in favor of network neutrality, blogger Douglas Ross has put together an index like mine of his postings on the subject. I've not read all of them, and have disagreed with most of the ones I have read (e.g., Ross thinks it's OK to ban QoS because it can't possibly work, even though it does work and is in use in major Internet backbones like Global Crossing's; we had an extended exchange in response to my list of Phoenix-area broadband options).

So check out his writings, and think critically. If you think he's got some good arguments for imposing net neutrality regulations, let me know.

Phoenix housing bubble update

It's been a while since I gave an update on the number of homes for sale in Phoenix--the inventory has continued to balloon since the last report on March 10:

3/7/2006 36953
3/8/2006 37487
3/9/2006 37626
3/10/2006 37531
3/11/2006 38011
3/12/2006 38184
3/13/2006 38169
3/14/2006 38003
3/15/2006 38197
3/16/2006 38574
3/17/2006 38602
3/18/2006 39074
3/19/2006 38972
3/20/2006 38822
3/21/2006 39159
3/22/2006 38982
3/23/2006 39043
3/24/2006 39271
3/25/2006 39381
3/26/2006 39504
3/27/2006 39817
3/28/2006 39784
3/29/2006 39765
3/30/2006 39948
3/31/2006 40192
4/1/2006 40177
4/2/2006 40182
4/3/2006 40012
4/4/2006 40050
4/5/2006 40332
4/6/2006 40739
4/7/2006 40612
4/8/2006 41124
4/9/2006 41393
4/10/2006 41018
4/11/2006 42266
4/12/2006 42327
4/13/2006 42257
4/14/2006 42561
4/15/2006 42592
4/16/2006 42775
4/17/2006 42874
4/18/2006 42523
4/19/2006 42840
4/20/2006 43017
4/21/2006 43236
4/22/2006 43385
4/23/2006 43502
4/24/2006 43697
4/25/2006 43344
4/26/2006 43427
4/27/2006 44024
4/28/2006 43886
4/29/2006 44022
4/30/2006 44290
5/1/2006 44229
5/2/2006 43900
5/3/2006 43966
5/4/2006 44162
5/5/2006 44422
5/6/2006 44094
5/7/2006 44575
5/8/2006 44777
5/9/2006 44609
5/10/2006 44898
5/11/2006 45097
5/12/2006 45356
5/13/2006 45502
5/14/2006 45619
5/15/2006 45697
5/16/2006 45705
5/17/2006 45675
5/18/2006 46064
5/19/2006 46189
5/20/2006 46049
5/21/2006 46734
5/22/2006 46753
5/23/2006 46965
5/24/2006 46856
5/25/2006 47133
5/26/2006 47225
5/27/2006 47582
5/28/2006 47591
5/29/2006 47633
5/30/2006 47722
5/31/2006 47542
6/1/2006 47187
6/2/2006 47191
6/3/2006 47848
6/4/2006 47877
6/5/2006 47979
6/6/2006 48218
6/7/2006 48106
6/8/2006 48365
6/9/2006 48579
6/10/2006 48870
6/11/2006 48889
6/12/2006 49040
6/13/2006 49132
6/14/2006 49237
6/15/2006 49052
6/16/2006 49435

My first report, last October, showed an increase in inventory from 10,748 homes on July 20, 2005 to 19,254 on October 2. We're now at a 459% increase in inventory in the just under 11 months.

(But see Einzige's comment on what counts as evidence of a housing bubble...)