Go Crawl Yourself

[mbarrick]

First off, happy new year, blah, blah, blah, [insert navel-gazing year-in-review/big-plans-for-2008 post here]. There, that's done.

At the moment I'm working on some post-relaunch tweaks for Gothic BC and was looking at the logs. In between the people who think I get up at 6:00 a.m. after getting in from the club at 5:00 a.m. and am going to magically download and catalogue 400 photos off my camera, weed through them for the 150 or so worth posting, processes them, and have them posted on the site by 6:15 a.m. (reality: I got up about an hour ago and the camera is still in the bag) who are already scouring the site for last night's pictures, there are a gazillion hits from someone crawling the site for video content with a VEOH client.

This is uncool. First off, there is no video content on Gothic BC, nor will there be for some time to come, if ever. Secondly, as I understand it VEOH is effectively a P2P service which would mean I could potentially have dozens of these clients crawling the site effectively amounting to, given my limited bandwidth, a DDoS attack. No thanks.

Apache rewrite module to the rescue:

RewriteCond %{HTTP_USER_AGENT} veoh [NC]
RewriteRule .* http://www.veoh.com/ [F,L]

The first line checks to see if the program making the request has the string "veoh" in its identifying "user agent" string, if it does, the second line is processed.

The second line says "whatever you are looking for, it's really at veoh.com" so the request is redirected to VEOH's own website, chewing up their bandwidth, not mine.

Of course, the "nice" thing to do would be to issue a "503 forbidden" response and just let the request die, but what VEOH is doing isn't nice. They are using a distributed network to download video off other people's sites, thereby decontextualising it from authors, who in the case of video-bloggers may be relying on the page context to serve up their advertising or other content that represents their income stream. VEOH then presents the content on their own page, with their own advertising, and makes their own money that the authors of the video content never see one red cent of. They deserve to be hoisted on their own petard.

Comments

[captain-bonk.livejournal.com]

creating and updating websites is a lot of work. Everything is always changing out there. I like that soccer site you did.

[evilyn13.livejournal.com]

I see that you got a new printer. Let me know if you ever need any paper, staff discounts!!!

[mbarrick.livejournal.com]

The new printer rocks. How much do you sell the ribbon/paper packages for?

[evilyn13.livejournal.com]

We have the packs of 120 sheets with or 3 ink cartridges for $45 or at staff prices of $28, plus tax.

[mediavictim.livejournal.com]

I upload my video content to YOUTUBE and then re-embed the links back into my site all the time

If they didnt want me to do it they wouldnt have given me the "embed" code in wich to do it. I get streaming video on my site, they get potential clickthroughs - everyone wins.

So is that the type of thing you dont want on Gothic BC - the video comes from the YOUTUBE servers - I dont see what effect it would have on your bandwidth.

[mickmercer.livejournal.com]

Hello - just joined gothicbc so I can tell people about the free goth mag I have available online, and would later like to be able to put up a request for info on any new bc goth bands which may applicable for a new print book I am doing. It says I must post here so you know I am not covered in spam.

[mbarrick.livejournal.com]

OK, Mick, you have posting access now.

[morbioid.livejournal.com]

Neatly karmic.