![]() ![]() In addition to this, I find it helps to have a private list of working proxies. These days I'm pretty sure Google knows where all the Tor exit nodes are so that might not work anymore. My coder even used to run our requests through TOR which would make the request times appear random. Pmfiorini: good call on the random request time. Over a long period of time, your average wait before a request would be 2 (note the formula above generates a random variable (rv) so it is not always going to generate a "2").Īll the above, I have found, makes it harder for Google to find out that your a "bot", but there are no guarantees, which is a pain. Thread.wait(amt_of_tm_wait_before_next_req) What you would get are exponentially distributed wait times with mean avg_req_time.Īmt_of_tm_wait_before_next_req = -log(u) * 2000 Where u is a uniformly distributed rv between 0 and 1, then In your software, you can deterministically pick a time (e.g., 2 secs) before sending another request, but a better way of doing it (I have found) is to randomly generate wait times that are exponentially distributed (there is a reason why I'm using this distribution - the exponential distribution is "memoryless" distribution, so no correlation can ascertained between requests).Īnyway, to make a long story short, if you could include a formula (or an option) in your code for waiting times that looks like:Īmt_of_tm_wait_before_next_req = -log(u) * avg_req_time ![]() What helps me to overcome getting banned for periods of time is to generate a list of available proxy servers in the region that I'm in (e.g., US), and then run my programs using round-robin scheduling to pick a proxy server.Įven though I do this, often times I've found out that Google is smart enough to figure out that I'm a bot so to overcome this problem what I do is to randomly generate a wait time between requests. Here is a suggestion that helps me (and possibly your tool) I run SEO tools all the time and get banned for a periods of time since Google (or Yahoo! or whatever) thinks I'm a bot (well, I am!). ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |