Down-Times

[MH-Razen]

LFE (and HFE) got two downtimes today and two days ago so I thought it`s time to inform you why:

The bot of chinese searching machine Baidu "visited" our website those two times, but unlike other bots he created tons of traffic so the automatic protection system of my webhost locked my account temporary. Normally our daily traffic is between 15 and 40 GB (LFE is 2GB large), but this bot manage it to create a traffic volume of 70 to 80 GB somehow, dont ask me how.

We locked out the bot now and the problem should be solved, the bad thing is that you can read through the internet that this Bot ignore robots.txt file, so he still might visit LFE - we hope not...

[Ariel]

well right now he is here
hope he wont do that again

[MH-Razen]

i guess thats what i read about that baidu ignors robots.txt...

[YinYin]

so this thing reads lfe 40 times a day when its at it??

thats ridiculous

aint it possible to also block the ip from which this bot creates its requests from?
just in case it creates that amount again ignoring bot settings

[MH-Razen]

so this thing reads lfe 40 times a day when its at it??

thats ridiculous

aint it possible to also block the ip from which this bot creates its requests from?
just in case it creates that amount again ignoring bot settings

haha, yeah when I saw this the first time I wondered why he does that, it`s compleatly stupid...

about ip: yes as far as I know it`s possible even if you just ban one ip ofc. The experts from Alfahosting didnt submit ip ban yet, in case the webspace is locked a third time I will suggest this.

[MH-Razen]

Hello again,

I apologize for the long down-time, my guess is some of you already thought it might be the same reason - and thats true. Baidu-Bot created such a high traffic another time so lfe was locked out from the automatic protection system of my webhost. The Problem: Normal bots read a file called robots.txt before they start crawling, and if this file tells them not to they leave. Baidu-Bot simply ignores this file, so we had to find another solution.

Next step would be an IP-Ban via Htaccess, problem here: Baidu uses multiple IPs (which is not surprising), so it'll hard to keep him out via IP for a long time and it would make lfe slower. So the support came with another solution which is kinda funny, if Baidu comes we tell him to search somewhere else now - technically we send him back home via redirect.

Another problem appears for me in person: If LFE is locked out another time my host will request a payment to unlock my account again because it`s work for them, too. Since I tryout their suggestion right now we came to the agreement that I dont have to pay this if this solution fails, but I would please you to help out here:

If you check the "Whos online"-Bar on the bottom of the forum and you see "Baidu" in there, write me an email ASAP!!!
Here the contact form if you dont have my mail: Contact Formular
In consequence I`ll lock the forum temporary and test the IP-Ban as last solution, but the mainsite will stay online and the webspace wont be locked again - I hope you can see a temporary lock of the forum is better then a long downtime of the compleate webspace untill I payed to my host

[Sonic Boom]

ummm.. i also saw that forbidden page.
But why don't u kinda try to say to the company who owns Baidu to not to send their bots over ur website?
Or maybe u can try to make guests or bots not able to come over the website by some Login necessary thingy, with which if someone wants to see the content then he/she needs to log in...
I don't know much about these websites or bots but I just don't want LFE to be down like this by a stupid bot.
Hope LFE Recovers from this bot...
~Sonic Boom

[Silverthorn]

ummm.. i also saw that forbidden page.
But why don't u kinda try to say to the company who owns Baidu to not to send their bots over ur website?

I highly doubt that they'd even read the email. Baidu is practically the Chinese version of Google which crawls the webpages, checks their content and, if it's politically correct, let it be accessible. Shooting them an email to leave us alone is probably as effective as sending a declaration of war to your government. They'll only laugh about it

Thing with the "block bots/guests from the forums" is that's also "hurting" LFE. You see, services such as google crawl the site and let it appear in search results, attracting new visitors, and, via adsense, generate a bit money to pay the host to keep this site running. Locking it out is counterproductive.

[HappyHouR]

That one nasty bot.That was pretty long down time.

[MH-Razen]

ummm.. i also saw that forbidden page.
But why don't u kinda try to say to the company who owns Baidu to not to send their bots over ur website?
Or maybe u can try to make guests or bots not able to come over the website by some Login necessary thingy, with which if someone wants to see the content then he/she needs to log in...
I don't know much about these websites or bots but I just don't want LFE to be down like this by a stupid bot.
Hope LFE Recovers from this bot...
~Sonic Boom

I tryed to do so since they dont make it as hard as google to find a contact form - but the mail didnt work and came back via Mail Delivery Subsystem, rofl

For the "being down" - me neither, this login you suggest wont work with php methods, it would only work if I protect the whole forum with htaccess and only if you know the password you can enter, this could be something easy like user "lfempire" and password "lfempire", but it`s not comfortable for the users, so lets see how it turns out