Looking for Hitbot IP's

[WiredGuy]

Hi,

I'm looking for a way to defend some of my sites from hitbots and I was wondering if anyone knows where hitbot open proxy IP's are typically collected from. That way I can grab these IP's that they use and blacklist them.

WG

[emgee]

hitbot-IP's are open proxys,so if someone would show you how to find more than enough,it would be the best source for the hitbot-peoples...

my guess is,there are around 1 000 000 open proxys around,even most sizes of lists are 1000's of ip's...so forget blacklisting it.

a other way to find out,if it's a proxy,is to use tools like "aa-tools",and with this tool you do an portscan at the ip's for the open ports 80/1080/8080/3128 http://www.glocksoft.com

[Clocka]

I know it's not quite the same thing but with my AutoRank pro sites, I add a javascript image tag to my gateway page.
The tag actually calls a cgi script which enables the surfers session, then returns a 1x1 pixel image.
I use the onload body tag to submit the form so the surfer doesnt get a "click here to enter" button.

Hit bots generally dont load javascript or images. (they get the page but dont execute it)

There are also some other clues in the ENV varibles to look for.

If you are going to ban proxy i.p's, only ban the anon ones. A lot of ISP's use webcaches and they appear to be a proxy.

Look on hackers sites for "anon proxy lists", then use a proxy judge to check their "level". There are far fewer level 1, 2 and 3 proxies than others, these are the ones to ban first.

Personally I ban all level 1-5 proxies as these are the ones people are going to attempt to use.

Hope that helps a little.

If you want a list of anon proxies that is updated every 3 hours, email me.

Phil

[WiredGuy]

clocka, can you elaborate what are the levels of proxies your referring to? Also, can you private msg me the link to that 3 hour refresh list.

Thanx
WG

[hal]

I have a list of anon proxies if you want it...

[emgee]

the biggest lists i saw are with 10k proxys at once,and 80% from them are not usable (down,offline,or whatever)

10k from 1000k are still only 1%,banning this ip's is NOT a solution.

a solution are scripts to detect it immediately,or checkingtools,like i mentioned above.
"sonicpuke" wrote a small script to extract ucj-logfiles,and follow every ip...thats a good start,but more can be done in this aree for experienced scriptwriters...and they could sell this scripts.

banning a few proxy-IP's get you nowhere.

[arthurdent]

WiredGuy:
Level of anon proxies is determined by mode how proxy work with URL's, especially how proxy represent himself (by proxy, by cache, by IP, by IP without proxy...).

If you need list of anon proxies go to deny.de (In Proxy Dump). You will see lists of the best anon proxies.
However I don't advise you to black listing ALL proxies. This is a heavy work, you will never be able to finish this.

"Problem" with using anon proxies is following most of them do not pass cookies (some of them don't works with "obscene" words in URL's).