[MOD] Treat bots as users with less rights

[martrix]

90% of the mod: V@no/Jan

A part of this mod was already published in this forum, but unfortunately it got lost after the hack.

Step 1: Create new users called for example "searchbot" and "googlebot" (or any other aditional you want...)  with password "12345"
(you may use any username or password you want, but then you will need to change the code)
The password 12345 is just for "demonstration" - please use another one for your bots and replace 12345 in the code with your new password...
If you decide not to do that, everybody can visit your page as a "bot"

Step 2: Create a new usergroup "searchbots" and set all bot-users as members of this group.

Step 3: in includes/sessions.php find:

Code: [Select]

$site_sess = new Session();
and replace it with:

Code: [Select]

// mod GoogleBot
$site_sess = new Session();
$robots = array(
"FAST-WebCrawler" => "searchbot",
"Gulliver" => "searchbot",
"AltaVista" => "searchbot",
"Alexibot" => "searchbot",
"asterias" => "searchbot",
"BackDoorBot" => "searchbot",
"Black.Hole" => "searchbot",
"BlowFish" => "searchbot",
"BotALot" => "searchbot",
"NPBot" => "searchbot",
"Pompos" => "searchbot",
"Scooter" => "searchbot",
"SiteSnagger" => "searchbot",
"Slurp" => "searchbot",
"WebmasterWorldForumBot" => "searchbot",
"zyborg" => "searchbot",
"Google" => "googlebot",
"msnbot" => "searchbot",
"ZACATEK_CZ_BOT" => "searchbot",
"EARTHCOM.info" => "searchbot",
"sherlock" => "searchbot",
"Holmes" => "searchbot",
"ia_archiver" => "searchbot",
"lwp-trivial" => "searchbot",
"appie" => "searchbot",
"Scooter" => "searchbot",
"Gigabot" => "searchbot",
"Wget" => "searchbot",
"jyxobot" => "searchbot",
"Xenu Link Sleuth" => "searchbot",
"SeznamBot" => "searchbot",
);
foreach($robots as $key => $val) {
  if(!$_SERVER['HTTP_USER_AGENT'] || preg_match("/".$key."/i", $_SERVER['HTTP_USER_AGENT'])) {
    $site_sess->login($val, "12345");
    $user_bot = 1;
    break;
  }
}
You may add or remove other bots's useragent names as you like or try to put MSIE there for a while, so you may see, how the bot will see the pages... don't forget to delete MSIE from the array aftewards, otherwise ALL your visitors will be logged in as bots (well, or you could just login as a searchbot but if you do that and you do hide the userbox for searchbots, you have to delete the cookies, to login back again under your "real" account.)

Step 4.1: If you did not install Conditional User Group template tags yet, follow the next steps
Open includes/page_header.php and search for each

Code: [Select]

if ($user_info['user_level'] >= USER) {add the following code above

Code: [Select]

$usergroups = array();AND add the following code below

Code: [Select]

  $current_time = time();
  $sql = "SELECT group_id 
          FROM ".GROUP_MATCH_TABLE."
          WHERE user_id = ".$user_info['user_id']."
          AND groupmatch_startdate <= $current_time
          AND (groupmatch_enddate > $current_time OR groupmatch_enddate = 0)";
  $result = $site_db->query($sql);

  while ($row = $site_db->fetch_array($result)) {
    $usergroups[$row['group_id']] = true;
  }Step 4.2 search for

Code: [Select]

//-----------------------------------------------------
//--- User Box ----------------------------------------
//-----------------------------------------------------
and add the following code above

Code: [Select]

$site_template->register_vars(array(
"is_bot" => (isset($usergroups[XXX])) ? 1 : 0
));Where XXX has to be replaced with the number of the usergroup. (If you don't know, how to find out the number of the usergroup, look here)
and from now on you may use this tag:

{if is_bot}
 This will appear only for the searchbots...
{endif is_bot}

if you want to hide some parts or disable some functions for the searchbots, install [MOD] Adding {ifno tagname1}{tagname2}{endifno tagname1} v1.7/v1.7.1
and you will be able to disable thes parts by

Code: [Select]

{ifnot is_bot}
This code will not be shown to searchbots
{endifnot is_bot}
On my site bots may browse the pages, but they can't see links to: download, favorities, user-profile, private messages, logout etc.

That should be everything...
Tell me if it works for you - I did not try it out "again" on a fresh install, so I could forget something...

[mawenzi]

hi martrix,

you are the man ... thanks ... !
that's what I'm searching for some time ...   
nice that you re-publish the MOD with your own statements ... 
thanks again ... !

mawenzi

[JensF]

On my site bots may browse the pages, but they can't see links to: download, favorities, user-profile, private messages, logout etc

Can anyone say me what i must "save" to have this?? Where must i put the tags in??

[martrix]

If you finish all the changes, then you may use the tags {if is_bot} blabla {endif is_bot} or {ifnot is_bot} blabla {endifnot is_bot}

For example: you would like to show the text "Dear bot, welcome to my page" to the bots, then just insert

Code: [Select]

{if is_bot} "Dear bot, welcome to my page" {endif is_bot}
anywhere you wish in the templates...

or if you want to hide favourites button from bots, so they won't use it, then just replace

Code: [Select]

{lightbox_button}
in details.html with

Code: [Select]

{ifnot is_bot}{lightbox_button}{endifnot is_bot}
or if you don't want the bot to see the control-panel of the userbot-account, just replace

Code: [Select]

<a href="{url_control_panel}">{lang_control_panel}</a>
in user_logininfo.html with

Code: [Select]

{ifnot is_bot}<a href="{url_control_panel}">{lang_control_panel}</a>{endifnot is_bot}
The rest is up to you, your imagination and your needs...

But beware: if you don't want to give anybody the possibility of seing the image-details, don't install this - anybody may change the name of his useragent and browse through your pages without registering (that's the reason, why I made up a lot of restrictions for bots)

Have fun

[leedzinh]

do you know yahoo bot?

[V@no]

I know two yahoo bots (dont know if they have more).
Its identify itself as:
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
and
ia_archiver

In the code I used this to identify them:

Code: [Select]

      "ia_archiver" => "bot-yahoo",
      "Yahoo! Slurp" => "bot-yahoo",
      "help\.yahoo\.com" => "bot-yahoo",

[leedzinh]

anyone know more bot please post all of them

thanks!

[bibabobu]

Sorry.
Perhaps i am too stupid to understand.
What does this MOD do? What are the advantages?
 Do searchbots of search engines find better keywords on my site for a better ranking or what?

Please could someone explain me?

[V@no]

its usefull if u have categories not acceptible for guests, but u still want them (categories) to be indexed on search engines.

[~Ally~]

Thank you SO much. I am hoping that this mod will help with the HUGE amount of RAM used up by my galley. .htaccess bans do not work anymore for bots so I'm hoping this will. thank you!

[bibabobu]

Please could someone help me because i am little bit confused.

If i want all of my images and pages to be indexed by search engines when my categories are only accessible for members
i will have to install this MOD.

Is this right???

So is there a problem to also install the MOD Registration with security code validation v1.0 :

Do the search engines also indexing my pages if i install the security MOD?

I hope i explained clear enough with my broken english

[V@no]

The bots will not be registering by themselfs, you as admin will register them, meaning you will need create account(s) for bots, and update names in the mod, so when a bot visit your site 4images will automaticaly log them in as a member you specifyed.

[impss]

The bots will not be registering by themselfs, you as admin will register them, meaning you will need create account(s) for bots, and update names in the mod, so when a bot visit your site 4images will automaticaly log them in as a member you specifyed.

So you will need to register each bot that you want, in the list above?

[mawenzi]

So you will need to register each bot that you want, in the list above?

... NO ...
For the list above you only need to register one new member ... "searchbot" !
... but ...
If you want a searchbot as a single member e.g. Google ... than register a new member "googlebot" ... and set in in list

Code: [Select]

"Google" => "googlebot",

... for other single searchbots is to proceed similar ...

mawenzi

[impss]

Thanks mawenzi