geek

I noticed that google was going crazy indexing trac for doctrine. Today it downloaded over 90000 pages, transfering 3 gig of data! It was causing quite a bit of load on the server (not huge amounts, but enough to show in my graphs!)

Eventaully , I came up with a nice little trick for reducing the number of hits google will make against a trac install. Google have extended robots.txt to allow some slightly improved pattern matching. Here's my snippet, if you don't understand it, please don't use it.

User-Agent: Googlebot
Disallow: /*?rev*

Took me a while to figure this out, so thought I'd put it here for others. This is useful for Xen setups, where you use a file for the disk image. AFAIK, you can only grow an image, not shrink it.

# dd if=/dev/zero bs=1M count=1024 >> disk.img
# e2fsck -f disk.img
# resize2fs disk.img
# e2fsck -f disk.img

This makes the disk image bigger, checks the image, resizes the file system, and then checks it again