HOME

What exactly does the Googlebot do on my website?

grep 'example\.com.*Googlebot' /var/log/httpd/access.log | \
    awk '{print $7}' > /tmp/googlecrawls.txt
for file in /var/log/httpd/access.log.*.bz2; do
    bzcat $file | \
        grep 'example\.com.*Googlebot'| \
        awk '{print $7}' >> /tmp/googlecrawls.txt
done
sort /tmp/googlecrawls.txt | \
    uniq -c | \
    sort -rn > ~/googlecrawls.txt
rm -f /tmp/googlecrawls.txt

Date: 2007-12-27

Copyright © 2007 Hraban Luyat