Work in progress, expect frequent changes.

Please see versiontemplate for whichever version you are using.

In this area you can decide in how much detail you wish to track search engines and spiders as they index your forum, as well as review search engine logs. You can find this page at Admin Center > Forum > Search Engines. It has four tabs or pages: Stats, Spider Log, Spiders, and Settings.

Stats

Here you can view statistics that pertain to search engines indexing your forum. Note, only search engines that are listed on the Spiders page will have statistics tracked for it here. The statistics stored for search engines are cut off by one day increments, thus you will get the statistics related to the search engine throughout a day's span.

There are three columns in this table:

• Date - This is the date the spider indexed your forum.
• Spider Name - This is the name of the spider that indexed your forum. The name comes from the name given to the spider on the Spiders page or tab.
• Page Hits - This is the number of unique hits, or separate session visits, the spider performed on your forum.

At the bottom right of the page is a dropdown menu labelled Jump To Month where you can browse the statistics for whichever month you want by simply selecting that month.

Spider Log

This log tells you which of your forum pages were visited, and by which spider. Depending on the Search Engine Tracking Level selected on the Settings page, this log can vary from showing the action of every spider, to not showing any activity at all. The Tracking Level must be set to Moderate or Aggressive, to track every spider.

There are three columns in this log:

• Spider - This is the name of the spider that indexed your forum, according to the name given in the Spiders page.
• Time - This is the date and time when the spider viewed a page.
• Viewing - This is the page the spider was viewing when it visited. This will show Disabled if the Search Engine Tracking is set to High instead of Very High.

Delete Entries

Since higher tracking levels can result in a massive number of entries, there is a way to delete log entries at the bottom of this page. Once you enter a numeric value, select the Delete button to prune entries older than the specified amount of days.

Spiders

This table lists all the spiders which your forum recognizes, along with a few details. SMF provides approximately 25, and you can add more or delete spiders, if you don't care to track them, for some reason.

These are the five columns in this table:

• Spider Name - This is the name of the spider that indexed your forum, according to the name given on the Spiders page. Note that these are shown as links and clicking the links will lead you to a page that allows you to modify the details for the spider.
• Last Seen - This is the date and time the spider last indexed your forum.
• User Agent - In general terms, the user agent is the code name for the spider, in the special kind of code which makes the internet work. Try WikiPedia or your favorite search engine, to learn more about this.
• IP Addresses - IP addresses identify specific computers or servers. The IP address column shows the address where each spider comes from.
• Checkbox - Putting a check in a box, and clicking the Delete Selected button at the bottom, allows you to remove spiders from the list.

Adding/Editing Spiders

You can add a new spider by clicking the Add New Spider button, which is right beside the Delete Selected button, in the bottom, right corner of this page. To edit an existing spider, click on its name in the table. In either location, you will be able to add or edit the following fields:

• Spider Name - This is the name of the spider.
• User Agent - User agents are one way to identify spiders. To find out the user agent of a spider, try searching for "user agent <search engine name>" in the search engine of your choice. Chances are one of the first results will tell you what the user agent is.
• IP Addresses - You can identify the IP address like described above for the user agent. In your favorite search engine, enter "IP address <search engine name>".

Note that only one of the two identifying fields for the spider are required, either the IP address or user agent. You can, however, input values for both if desired.

Settings

You can change settings for spider tracking from this page. Note, if you wish to enable automatic pruning of the hit logs you can set this up from logs#Log_Pruning.

• Search Engine Tracking Level - Determines the level at which spider activity is logged. Be aware that higher tracking level increases server resource requirement.
  • Disabled - Spider activity is not logged.
  • Standard - Minimal spider activity is logged.
  • High - More accurate statistics about spider activity are logged.
  • Very High - The same as high, but logs data for each page visited.
• Apply restrictive permissions from group - Enables you to prevent spiders indexing some pages.
  • Disabled - Spiders do not belong to a restrictive group.
  • List of groups - By selecting a restrictive group, when a guest is detected as a search crawler it will automatically be assigned any deny permissions of this group, in addition to the normal permissions of a guest. You can use this to provide lesser access to a search engine than you would a normal guest. You might for example wish to create a new group called "Spiders" and select that here. You could then deny permission for that group to view profiles to stop spiders indexing your members profiles. Note that spider detection is not perfect and can be simulated by users so this feature is not guaranteed to restrict content only to those search engines you have added.
• Show spiders in the online list - Determines whether spiders are displayed in the online list, and which members can see them.
  • Not at all - Spiders will simply appear as guests to all users.
  • Show spider quantity - The Board Index will display the number of spiders currently visiting the forum.
  • Show spider names - Each spider name will be revealed, so users can see how many of each spider is currently visiting the forum - this takes effect in both the Board Index and Who's Online page.
  • Show spider names - admin only - As above except that only administrators can see spider status. To all other users spiders appear as guests.

---

As an administrator

Main

• Administration Center
• Support and Credits
• News and newsletters
• Package manager

Configuration

• Features and Options
• Anti-Spam
• Languages
• Current Theme
• Themes and Layout
• Modification Settings

Forum

• Boards and Categories
• Posts and Topics
• Calendar administration
• Search (admin)
• Smileys and Message Icons
• Attachments and Avatars
• Search engines

Members

• Members
• Membergroups
• Permissions
• Registration
• Warnings
• Ban List
• Paid subscriptions

Maintenance

• Server settings
• Forum Maintenance
• Scheduled Tasks
• Mail
• Reports
• Logs

Miscellaneous

• Security Tips
• Building Your Community
• Performance enhancements
• Modifications - A mod broke my forum what should I do