When you use rank monitoring software, it checks your positions in the search engine result pages just the same way you would do it manually: it sends a query to the search engine via HTTP, usually this query is a simple URL containing the keywords you check rankings for. The search engine then returns a result page, just as it would to your browser. The rank monitoring software parses this result page and picks out URLs of results. If it finds your page among them, it reports the position at which your page was found. If it doesn’t, it requests the next result page from the search engine and reads it, and so on – until it either finds your page or reaches some limit established for the ranking check session.Unlike manual ranking check that would take a huge amount of time (imagine querying the search engines for at least three keywords then browsing 10 result pages for each keyword, picking out for pages of your site, then calculating the position for each of them for every keyword and putting the data into table). Rank monitoring software can facilitate and speed up the process by doing it all automatically.
However, the automated ranking check has some hidden drawbacks.
Once it becomes a common optimization practice to query search engines for many keywords per ranking session, it can produce a significant load on search engine traffic. For this reason, all search engines denounce the automated ranking check. Their programmers have developed a number of techniques to protect themselves against overloading their servers from automatic queries. If the software running on the search engine detects you’ve been sending automatic queries of any type to its database, the most probable penalty assessed will be that access from your IP address to this search engine will be blocked. In other words, if a search engine finds out you’re checking rankings automatically, you will not be able to send any more queries, neither automatically nor manually, to that search engine.
As a rule, this penalty is temporary and lasts only for a certain period, e.g. for one day. However, this can be quite frustrating and can stand in the way of your optimization plans. Besides, if you are sharing your IP with other people they would also be banned from using the search engine. This can happen if you’re inside a corporate network that accesses the internet behind a common firewall or proxy server.
The the answer to this problem is to use software that intelligently emulates human behavior, so that it is impossible for the search engine to differentiate between this software and a real human searching for something in a browser. Such software will comply with the following requirements:
- Making pauses between querying for separate result pages: A human Web surfer won’t ask for the second result page a moment after he has received the first one; the search engine assumes the searcher needs some time to look through the results.
- Making pauses between separate requests sent to the search engine: You won’t ask for a keyword the same second you asked for another one, as an automated ranking checker is likely to do.
- Imitating real browser behavior by fetching images, styles, scripts and logos from search engine result pages: Most browsers, such as Internet Explorer, request these objects embedded into the SERP together with the result page itself. An automated ranking checker may not do this. Enabling ranking checker to fetch images and other page-related information decreases the speed of the ranking check, yet it enhances the chance that you won’t be banned.
- Visiting search engine home pages: As a rule, automated ranking checkers will not visit a search engine’s home pages – they will start right away with sending a direct query URL like
“http://www.google.com/search?q=sports+wear&sourceid=navclient&start=0&ie=utf-8oe=utf-8″
However, Web surfers do not type these URLs in the address bars of their browsers to perform the search; instead, they go to the search engine’s home page and type their query in the search box; only after that does the browser generate the query string shown above and sends it to the search engine. Thus, search engines would detect sessions that start in a strange manner and ban the IP addresses that initiate such sessions.
- Following redirects: Some search engines, like Yahoo!, will give special redirect URLs instead of directly linking to the URLs of the result pages found for your query. The purpose of these redirects is to track your clicks, which helps Yahoo! with result ranking. An example of Yahoo’s search result URL looks as follows:
http://rds.yahoo.com/S=2766679/K=search+tool/v=2/SID=e/TID=F497_109/l=WS1/R=3/IPC=ua/SHE=0 /H=3/SIG=116trk9kp/EXP=1114438203/*-http%3A//search.com/
However, most ranking check software would extract direct result page URLs from Yahoo’s redirect URLs and never follow the redirect URLs themselves. This behavior may seem suspicious to Yahoo! and could result in an IP ban.
Ranking Checker software allows you to adjust human emulation by going to File > Settings, selecting the “SE Requests” tab and going to the “human emulation” section. Here you can tune the exact parameters of requests to search engines so that Ranking Checker software is indistinguishable to the search engines from a real human browsing result pages in Internet Explorer (or another browser you set as user-agent in the appropriate section). It may also be useful to give a couple of proxy server URLs to use so that the ranking check requests will always come to the search engine from different IP addresses.
