Friday, June 08, 2007

Craigslist blocks Listpic due to usage violation

I discovered Listpic a while back, I think linking to it from LifeHacker. In case you never got a chance to see it, Listpic analyzed Craigslist ads and created a gallery page from the ads that contained photos. It was great when we were looking for furniture, and it apparently came in handy for those who wanted to browse personal ads (a use-case that didn't occur to me).

Craigslist is a great site. Power to the people, grass roots movement, down with newspaper conglomerates and so on. But let's face it. The interface has never been great. I understand the effort is in keeping it lightweight, but Listpic performed a valuable function for me. It saved me time.

I'm never going to buy furniture--or a car, or a computer, or a Hummel Figurine--sight-unseen. I feel like anyone who wants to sell something online who gives half a whit about getting their junk out of their garage is going to attach a picture. If a seller can't be bothered to post one, I'm not going to bother looking at their ad. With Listpic, I could browse their Gallery of items listed on Craigslist and only click through to those posts that interested me, rather than looking in the Craigslist listings for the icon indicating that an ad contains pictures, then clicking through to each and every ad individually.

Browsing Craigslist proper, I wasted a lot of time looking at ads that I never would have clicked through to if I'd had a picture preview. Listpic let me take advantage of the the years of evolution responsible for the development of my visual cortex. Most humans with full use of their sight are much better at quickly scanning a group of photos than scanning separate lines of text. I always figured that Listpic was letting me evaluate more products than I ever would with the standard Craigslist interface.

Well today I learned from the Download Squad that Listpic has been blocked from accessing Craigslist due to what are known as Terms of Use violations. This means that Listpic is dark for the time being. It seems that Craigslist's justification for the blackout can be summed up in a few bullets:
  • Listpic uses ads. Craigslist doesn't, and their community is against the use of their content by other sites to generate revenue.
  • By scraping the ads and pictures from Craigslist, Listpic is leeching a significant amount of bandwidth that Craigslist has to pay for.
  • The Craigslist Terms of Use allow them to ban and fine(!) anyone who republishes or aggregates more than 100 ads without permission.
So let's state the obvious. Craigslist is completely within their rights here. It seems pretty clear that Ryan, the developer behind Listpic, probably didn't read the terms of use and probably didn't ask Craigslist if it was okay for him to reprint their ad content. That's just a dumb mistake. CYA, my friend.

But Craiglist is not handling this situation well, either. Ryan is claiming that he wasn't contacted before receiving the cease-and-desist letter from Craigslist. That seems to fly in the face of the informal, "rootsy" image that Craigslist cultivates. The backlash in the Craigslist forums also indicates that Listpic was providing a service a lot of people appreciated, and there's a lot of testimony that it contributed to people using Craigslist more than they had before. But apparently this is one sticking point: how many people use Listpic anyway, and how much of a bandwidth impact are they having?

In one response to the forum users calling for his head, Craigslist founder Craig Neumark states that the main reason for the shut down was bandwidth related: Listpic was using too much. But then, a few hours later, Craig says in a post that they can tell only a very few people use Listpic. Does this make sense? I'm not privy to traffic figures for Craigslist or Listpic, but this just doesn't pass the sniff test. Listpic has been around for at least 9 months, as the date on this post from the Consumerist proves. Has Listpic's bandwidth imposition happened so dramatically, and in such a short period that the Craigslist Server Admins have only just noticed that someone has been scraping their site, a few thousand images at a time? Maybe someone at Craigslist needs to hire new admins to monitor their web traffic (I am available, Craig...email me). Maybe it just took Craig and Co. a while to figure out what to do, but I find it hard to believe that no one at Craigslist knew what was going on. It's clearly not just about reprint rights or bandwidth usage, or even advertising. All of these things have been going on for 9 months, with nary a peep from Craigslist.

Just out of curiosity, I looked up the Craigslist TOS statement on Archive.org, to see if there was anything hinky going on....maybe Ryan was caught unawares with an unannounced update to the TOS. I'm not accusing Craigslist of changing the TOS just because of Listpic, but I find it interesting that the last version of the Craigslist TOS at Archive.org doesn't mention the $3000 per day fines that the cease-and-desist letter does. In fact, that version of the TOS--logged by Archive.org on Dec. 26th, 2005--doesn't even have a section 19(f), which outlines the fines Ryan may be faced with.

That's not to say that Ryan's off the hook. Even in 2005, you couldn't use Craigslist ads without permission. Oops.

Listpic was a great idea. Craig says that a major problem they had with Listpic was the monetization of Craigslist content through web advertising. Maybe that's not okay according to the terms of use, but it's an idea very much in the 'spirit' of web 2.0. Google uses their index of web content to provide search results, and then display ads relating to those results. This is a leveraging of other people's content to gain revenue. Sure, if I run my own web server, I can tell Google that their bots aren't welcome, but Google doesn't ask permission before they index sites. They assume that if you aren't blocking them, you're okay with their use of your site's content. Interestingly enough, though, Google has realized that having people use their tools in innovative ways only adds value to what they do. The give-back from Google for the use of all that web content comes in the form of all the open APIs that Google allows other people to use, for free. Those are the tools that make all those neat Google Maps mashups possible. There would be no Listpic without Craigslist, but Listpic was giving back to the Craigslist community. This sounds like a very symbiotic relationship, and one that made it a lot easier for the community to access its own content.

Here's how to remedy this situation. Ready your pencils, lawyers and executives.

Craigslist should integrate the Listpic functionality. If it's too much of a bandwidth hog, make it only available to paid members, and continue to offer the lightweight version for non-paying users. Ryan has some prior art in the graphical display of Craigslist ads, so I feel its only fair to offer him some sort of deal. If he's a good coder, then offer him a job. If that's not feasible, then maybe Craig can license or outright buy the Listpic code. Oh, and Ryan, if Craigslist fines you (or deducts that amount from any contracts), pay up and don't complain. Next time, you'll read the fine print and ask permission.

Finally, Craigslist should reconsider its policy regarding usage of ad content, especially when it clearly adds value for the communities of past and future Craigslist users. Craig Neumark says that the Craigslist community has decided that their ads shouldn't be used for revenue by outside sites. Well, everybody's gotta pay for their bandwidth. Craigslist is not a charity, and it's not like Listpic was a link farming site, reprinting Craigslist content and attaching ads to it without giving anything back. Listpic added something that the community found valuable, even if it was operating outside the terms of use. I think that it better serves the online community if people like Ryan, those with innovative ideas, are brought gently into the fold rather than threatened with blackouts and thousands of dollars in fines. Otherwise, Craigslist starts to look more like the coercive newspaper syndicates it has replaced than the user-centric community it purports to be.