SMF2.0:Search engines

From this area you can decide in how much detail you wish to track search engines as they index your forum, as well as review search engine logs. To view this area, you must enable it via the Core Features page. Search Engines administration may be found under the "Members" button of the Administration Center menu (or under the "Members" heading if the Administration Center sidebar is used).

Stats

Here you can view statistics that pertain to search engines indexing your forum.

You will see some basic statistics related to search engines that have indexed your forum. Note, only search engines that are added on the spiders page will have stats tracked for it. The statistics stored for search engines are cut off by one day increments, thus you will get the statistics related to the search engine throughout a day's span.

From the leftmost column to the rightmost column of the table you will see the following details provided for each search engine that has at least one hit on your forum:

• Date - The date the spider indexed your forum, to which the statistics pertain to.
• Spider Name - The name of the spider that indexed your forum, as according to the name given in the spiders section.
• Page Hits - How many unique hits the spider performed on your forum, such as separate session visits.

At the bottom right of the page is a selection box labelled Jump To Month where you can browse the stats for search engines for whichever month you want by simply selecting that month.

Spider Log

In the Spider Log page ("Administration Center > Members > Search engines > Spider Log or Administration Center > Maintenance > Logs > Spider Log), logs of page views made by spiders browsing the forum are available. Depending on the Search Engine Tracking Level setting selected on the Settings page, this page can vary from showing the action of the spider, or not showing the action at all. If the tracking setting is set to Disabled or Standard, spiders will not be logged on this page.

Viewing Search Engine Logs

• Spider - The name of the spider that indexed your forum, as according to the name given in the spiders section.
• Time - The date and time when the spider viewed the page for the current page view it is associated with.
• Viewing - The section the spider was viewing during the current page view. This will show Disabled if the Search Engine Tracking is set to High instead of Very High.

Delete Entries

At the bottom of this page is an input value that allows you to delete all search engine logs older than the specified amount of days. Once you enter a numeric value, select the Delete button to go through with the pruning of the logs older than the specified amount of days.

Spiders

Lists all the spiders added to your forum that you are tracking the indexing of. You can view some details about the spider as well as add new ones and delete existing ones.

The Spider List

From the leftmost column to the rightmost column of the table you will see the following details provided for each search engine that has at least one hit on your forum:

• Spider Name - The name of the spider that indexed your forum, as according to the name given in the spiders section. Note that these are shown as links and clicking the links will lead you to a page that allows you to modify the details for the spider.
• Last Seen - The date and time the spider last indexed your forum.
• User Agent - The User Agent the spider is detected by. For more information on what a user agent is, search for "user agent" in the search engine of your choice. In general terms, the user agent is one way your forum can identify what viewing the forum is actually a spider.
• IP Addresses - Similar to user agents, IP addresses are another way to identify a spider. Listed here is a comma separated list of IP addresses associated with the spider.

Below this table you can see two buttons:

• Add New Spider - This button will lead you to a new page to add a new spider to your forum.
• Remove Selected Spiders - To the far right of each spider listed on this page is a checkbox that you can select. Selecting this button will completely delete all of the spiders that are checked on this page.

Adding/Editing Spiders

When you select a link of a spider listed on this page or select the Add New Spider button you will be led to a page that allows you to define certain settings for the spider. The following settings are:

• Spider Name - The name you want to give the spider. This can be any name that you want, but you should give it a name so you can easily identify the spider when you come across it viewing your logs and statistics.
• User Agent - The user agent of the spider. User agents are one way to identify spiders. To find out the user agent of a spider, try searching for "user agent <search engine name>" in the search engine of your choice, and chances are one of the first results will tell you what the user agent is.
• IP Addresses - The IP addresses that you want to identify for the search engine. This is similar to the user agent and you can find out the IP address in similar ways as to what is mentioned above for the user agent.

Note that only one of the two identifying fields for the spider are required, either the IP address or user agent. You can, however, input values for both if desired.

Settings

You can change settings for spider tracking from this page. Note, if you wish to enable automatic pruning of the hit logs you can set this up from logs#Log_Pruning.

• Search Engine Tracking Level - Determines the level at which spider activity is logged. Be aware that higher tracking level increases server resource requirement.
  • Disabled - Spider activity is not logged.
  • Standard - Minimal spider activity is logged.
  • High - More accurate statistics about spider activity are logged.
  • Very High - The same as high, but logs data for each page visited.
• Apply restrictive permissions from group - Enables you to prevent spiders indexing some pages.
  • Disabled - Spiders do not belong to a restrictive group.
  • List of groups - By selecting a restrictive group, when a guest is detected as a search crawler it will automatically be assigned any deny permissions of this group, in addition to the normal permissions of a guest. You can use this to provide lesser access to a search engine than you would a normal guest. You might for example wish to create a new group called "Spiders" and select that here. You could then deny permission for that group to view profiles to stop spiders indexing your members profiles. Note that spider detection is not perfect and can be simulated by users so this feature is not guaranteed to restrict content only to those search engines you have added.
• Show spiders in the online list - Determines whether spiders are displayed in the online list, and which members can see them.
  • Not at all - Spiders will simply appear as guests to all users.
  • Show spider quantity - The Board Index will display the number of spiders currently visiting the forum.
  • Show spider names - Each spider name will be revealed, so users can see how many of each spider is currently visiting the forum - this takes effect in both the Board Index and Who's Online page.
  • Show spider names - admin only - As above except that only administrators can see spider status. To all other users spiders appear as guests.

---

As an administrator

Main

• Administration Center
• Support and Credits
• News and newsletters
• Package manager

Configuration

• Core Features
• Features and Options
• Security and Moderation
• Languages
• Server settings
• Current Theme
• Themes and Layout
• Modification Settings

Forum

• Boards
• Posts and Topics
• Calendar
• Search
• Smileys and Message Icons
• Attachments and Avatars

Members

• Members
• Membergroups
• Permissions
• Registration
• Ban List
• Paid subscriptions
• Search engines

Maintenance

• Forum Maintenance
• Scheduled Tasks
• Mail
• Reports
• Logs

Miscellaneous

• Security Tips
• Building Your Community
• Performance enhancements
• Modifications - A mod broke my forum what should I do