SMF2.1:Search engines - Online Manual

Work in progress, expect frequent changes.

Please see SMF2.0:Search engines or SMF2.1:Search engines depending on the version of SMF you are using.

In this area you can decide in how much detail you wish to track search engines and spiders as they index your forum, as well as review search engine logs. You can find this page at Admin Center > Forum > Search Engines. It has four tabs or pages: Stats, Spider Log, Spiders, and Settings.

Stats

Here you can view statistics that pertain to search engines indexing your forum. Note, only search engines that are listed on the Spiders page will have statistics tracked here. The statistics stored for search engines are cut off by one day increments, thus you will get the statistics related to the search engine throughout a day's span.

There are three columns in this table:

Date - This is the date the spider indexed your forum.
Spider Name - This is the name of the spider that indexed your forum. The name comes from the name given to the spider on the Spiders page or tab.
Page Hits - This is the number of unique hits, or separate session visits, the spider performed on your forum.

At the bottom right of the page is a dropdown menu labelled Jump To Month where you can browse the statistics for whichever month you want by simply selecting that month.

Spider Log

This log tells you which of your forum pages were visited, and by which spider. Depending on the Search Engine Tracking Level selected on the Settings page, this log can vary from showing the action of every spider, to not showing any activity at all. The Tracking Level must be set to Moderate or Aggressive, to track every spider.

There are three columns in this log:

Spider - This is the name of the spider that indexed your forum, according to the name given in the Spiders page or tab.
Time - This is the date and time when the spider viewed a page.
Viewing - This is the page the spider was viewing when it visited. This will show Disabled if the Search Engine Tracking is set to High instead of Very High.

Delete Entries

Since higher tracking levels can result in a massive number of entries, there is a way to delete log entries at the bottom of this page. Once you enter a numeric value, select the Delete button to prune entries older than the specified amount of days.

You can also use Log Pruning on Admin Center > Maintenance > Logs > Settings page, to automatically prune the Spider Log. Scroll down to the Log Pruning section, and check Enable Log Pruning. See Log Pruning in the manual, to learn how to use it.

Spiders

This table lists all the spiders which your forum recognizes, along with a few details. SMF provides approximately 25 spiders, and you can add more, or delete spiders, if you don't care to track them, for some reason.

These are the five columns in this table:

Spider Name - This is the name of the spider that indexed your forum, according to the name given on the Spiders page. Note that these are shown as links and clicking the links will lead you to a page that allows you to modify the details for the spider.
Last Seen - This is the date and time the spider last indexed your forum.
User Agent - In general terms, the user agent is the code name for the spider, in the special kind of code which makes the internet work. Try WikiPedia or your favorite search engine, to learn more about this.
IP Addresses - IP addresses identify specific computers or servers. The IP address column shows the address where each spider comes from.
Checkbox - Putting a check in a box, and clicking the Delete Selected button at the bottom, allows you to remove spiders from the list.

Adding/Editing Spiders

You can add a new spider by clicking the Add New Spider button, which is right beside the Delete Selected button, in the bottom, right corner of this page. To edit an existing spider, click on its name in the table. In either location, you will be able to add or edit the following fields:

Spider Name - This is the name of the spider.
User Agent - User agents are one way to identify spiders. To find out the user agent of a spider, try searching for "user agent <search engine name>" in the search engine of your choice. Chances are one of the first results will tell you what the user agent is.
IP Addresses - You can identify the IP address like described above for the user agent. In your favorite search engine, enter "IP address <search engine name>".

Note that only one of the two identifying fields for the spider are required, either the IP address or user agent. You can, however, input values for both if desired.

Settings

You can change the tracking level and other settings for spider tracking from this page.

Search Engine Tracking Level

This determines the level at which spider activity is logged. Be aware that higher tracking level increases server resource requirement.

Disabled - Spider activity is not logged, at this setting.
Standard - Minimal spider activity is logged, at this setting.
Moderate - More accurate statistics about spider activity are logged, and for every spider, at this setting.
Aggressive - All possible statistics, for every spider visit, are logged at this level.

Apply restrictive permissions from group

This option allows you to prevent spiders from indexing certain pages, such as member profile pages.

Disabled - Spiders do not belong to any restrictive group.
List of groups - By selecting a particular group, when a guest is detected as a spider, it will automatically be assigned any deny permissions which this group possesses, in addition to the normal permissions of a guest. You can use this to provide lesser access to a search engine than you would a normal guest. For example, you might wish to create a new group called "Spiders" and select that here. You could then deny permission for that Spider group to view profiles, to stop spiders indexing your members profiles. Note that spider detection is not perfect and can be simulated by users. So this feature is not guaranteed to restrict content only to those search engines you have added.

Show spiders in the online list

This option determines whether spiders are displayed in the online list, and which members can see them.

Not at all - Spiders will simply appear as guests to all users.
Show spider quantity - The Board Index will display the number of spiders currently visiting the forum.
Show spider names - Each spider name will be revealed, so users can see how many of each spider is currently visiting the forum - this shows up on both the Board Index and Who's Online page.
Show spider names - admin only - As above except that only administrators can see spider status. To all other users spiders appear as guests.

As an administrator

Main

Configuration

Forum

Members

Maintenance

Miscellaneous

Navigation

Search

Tools

SMF2.1:Search engines From Online Manual

Contents