[GOOGLE SITEMAPPER TECHNICAL QUERY] WORRYING RESULTS - GOOGLE SITEMAPPER

[sajwal]

Hello People,
                   I used google sitemapper to check the links to make a sitemap for my site and got worrying results displayed...when sitemapper spiders my site it shows thousands of results where pages of ADD TO LIGHT BOX are shown several(most of ) times with amp;amp;amp;amp;amp; characters in line added by session id.

I am attaching the part of spidered results . Pls. let me know if it counts as spaming and how can i reduce these pages to get spidered if its a factor of worries

Pls. see the sitemapper.txt and guide me what it means and what is good/bad about it

[V@no]

I guess, there is nothing you can do about it, unless you do something to limit available functionality for search bots on your website...but its way too many changes to do in the code...

[sajwal]

Thankyou vano for taking time to reply,
                                                       Only thing to help me pls just tell me code that I do not allow spidering of action=lightbox as seen in the attachment

[V@no]

let try this. Asuming you already installed [MOD] Treat bots as users with less rights (if you havent, then do at least Step 3 and remove     $site_sess->login($val, "12345"); line from it).

Then in includes/functions.php find:

Code: [Select]

  if ($user_info['user_level'] != GUEST) {
    $lightbox_url = $self_url;
Replace with:

Code: [Select]

  global $user_bot;
  if ($user_info['user_level'] != GUEST && !$user_bot) {
    $lightbox_url = $self_url;
It should not show lightbox link to the bots anymore.

[sajwal]

Thank you very much V@no you made me happy again

[AntiNSA2]

why not ad a robot.txt nofollow to not allow google to access lightbox.php?

[Anarchology]

I have found it easier, and much faster to use a sitemap program rather than to wait for Google to create a sitemap of your site. Pretty much, you just sit back, while the program trolls through your site under your own permissions. Now, the one that I have used (I think it is the one below), allows you to set words/scripts NOT to crawl. Depending on exactly how many images you have and internet speed, it may take up to a couple of hours to properly scroll through your site.

The programs create exportable sitemap files that you are then able to upload to your root directory, and then point google to it. Personally, I know that I had Google index 9,000 pages thanks to one of the sitemap programs.

This is one, but not sure if this is the one I used. I don't think there are any restricted permissions for the trial, since I only used it during the trial...
http://download.cnet.com/A1-Sitemap-Generator/3000-10248_4-10496253.html?tag=mncol

BUT REMEMBER: Search through the features of sitemap programs allowing you to omit certain words like "admin, lightbox, edit, delete, etc"... which ever words or pages you do not want to be indexed. Also, keep in mind that doing so drastically cuts down on the time consumed when it scrolls your site. Finally, the more you restrict from the robot of these programs, the less amount of bandwidth it will obviously use up when scrolling.

EDIT: Forgot to mention the most important part... LOGOUT of your site (especially if under admin) when these programs are running. This will stop the program from scrolling admin or member actions that could be a security risk!