GoogleBot ate my bandwidth!
Yesterday I got an automatic e-mail from my web host that this site has consumed 80% of the bandwidth.
I was thinking that it was very stange, it is just in the beginning of the month and I don’t get so many visitors. My bandwidth is on 2 GB.
I checked the my statcounter but no increase in visitors. Hmmm. I checked in AW Stats and there I see that 1,6 GB was used in the last days coming from one IP. And it is not a SE spider.
Ok I go and use the IP look-up in statcounter but that IP was not recorded …
I go to DNSstuff.com to check who is this IP and I find some kind of company in the US …
I checked the IPs of some sites and found out that my site AddURL-Free.com has the same IP, so this is the IP of the servers of the hosting company.
After a moment of figuring I put two and two together I came to the conclusion that Google has been spidering the ODP section of addurl-free.com like crazy and I had a RSS feed there from this site that displayed 8 items on each subpage. So the spidering only costed addurl-free.com about 50 MB but the generation of the RSS feed costed this site 1,6 GB. I have now turned that off.
I have sent an e-mail to my hosting company to see how to solve this, perhaps I will have to upgrade again so that I have 5 GB.
So it could happen that this site goes down for a while “bandwidth limit exceeded” but I hope not.
Anyway I should see it from the good side, I leared that RSS feed consumes bandwidth.
Jim Westergren is a company owner from Sweden who lives together with his wife and son. Some of his interests are SEO, web development, writing and
this is scary…
Hey
Just found exactly the same on one of my client’s sites. They were complaining that despite being a small site, they’d apparently used all of their bandwidth within 4 days.
They had one of these PHP calendars on their site, where you click the day and it tells you what’s on. Googlebot had tried to index EVERY SINGLE POSSIBLE DAY. And, in the first four days of September, had used up all this site’s bandwidth, clocking up an impressive 19,000 hits and 800MB of bandwidth.
You can use robots.txt to tell all decent robots to push off. I’ve just done that. Let’s see if it works!