UTF-8 Readme: Difference between revisions

UTF-8 Readme: Difference between revisions From Online Manual

Jump to: navigation, search

Latest revision as of 08:10, 16 September 2016

UTF-8 is an encoding standard that can represent all Unicode characters. This allows it to show almost any writing system in the world.

UTF in SMF 2.0.x

SMF 2.0.x includes the option to run the forum with or without UTF-8. If you install a new SMF 2.0.x forum with UTF-8, or if you upgrade an existing SMF 2.0.x forum to UTF-8, all posts will be stored in the database using the UTF-8 character set, and every web page will inform the browser that it is using the UTF-8 character set. For each language pack you decide to install for your forum, it will be necessary to choose the UTF-8 version of that character set.

If you choose not to use UTF-8 for your SMF 2.0.x forum, you must choose the non-UTF-8 versions of all language packs that you install on your forum.

If you choose the wrong character set for any of your language packs, you will see some "garbage characters" on the screen. The solution is to use only the correct character set version of the language packs you have chosen for your forum.

You can download language packages with a UTF-8 character set here.

Why would I need UTF-8?

There are a few reasons you might need UTF-8:

If you want to support multiple languages that use different character sets on your forum. For instance, if you want to support both Russian and Turkish, you will need a character set that supports both.
If the software integrating with SMF uses UTF-8. In some cases such an integration can require character sets to match.
If you need better search results or improved sorting. In some cases searching and sorting by the database can be improved by choosing UTF-8 as your character set.

Why would I not need UTF-8?

If none of the above reasons apply to your forum, UTF-8 would probably not be very useful.

How to convert to UTF-8

This procedure will work well if you have used the recommended character sets in your language files in the past.

1. Start with a backup of your database. Character set conversions nearly always work correctly, but it is best to be prepared for the unexpected.

2. Check your default language file (Administration Center » Languages » Edit Languages ) and make a note of the character set used.

3. Go to Forum Maintenance > Convert the database and data to UTF-8 (this option will only be available if SMF detects a database version which supports UTF-8).

4. Select the character sets for your data (member posts) and database. By default, SMF will choose the character set of your default language file.

5. Press proceed, and your database will be converted. Depending on the size of your database, the conversion process might stop temporarily from time-to-time to avoid overloading the server. If this was successful, your forum should be converted to UTF-8.

6. For each of language pack currently in use, replace it with the UTF-8 version of that language pack.

7. Once all the UTF-8 language packs have been installed, convert the language settings of each user by running this query:

UPDATE smf_members
SET lngfile = CONCAT(lngfile, '-utf8')
WHERE lngfile != ''

8. In your Administration Center, change the default language, ensuring that you choose the UTF-8 version.

9. Check to see if all your data was properly converted.

10. If any of your posts contain HTML entities, you will want to convert these to UTF-8 as well by running "Convert HTML-entities to UTF-8 characters"

What to do if your site uses a mix of character sets

If your site began as a non-UTF-8 installation, and you chose UTF-8 language files, the procedure above might not work perfectly. Because there are likely to be hundreds of posts on the database in a character set different to the forum's default character set, site search functions might not work well, and the situation might be holding you back from converting to SMF 2.0.

The good news is that you can fix this problem and convert to UTF-8. You must simply plan the process out a little more carefully, and might need to take one or two extra steps.

Before you begin, double check the default language file/character set you used to install SMF, and also the language file/character set you later added (and which was used for storing the important posts on the forum). Make sure to put the forum into maintenance mode.
Set the default language for your forum to be whatever it was when you first installed the forum (English ISO-8859-1, for example)
In order to change your default language (from English to Greek, for example), it is best to get a copy of the upgrade package for the current version of SMF, for the language you want to be your new default. Copy all these files just as you normally would for an upgrade or install.
Run upgrade.php, still using the old default character set, and then delete upgrade.php
If your character set problems have affected your search take the following steps:
- Go to Administration Center » Search » Search Method, delete any text index (if any) and select "No index" as the search method.
- Remove all rows from tables with names like smf_log_search_*. Use phpMyAdmin to do this. Do not drop these tables.
- Now proceed through the instructions above up to step 4. During step 4, you will have to set the Data character set to the character set used in the posts on your forum.
- Follow the remainder of the instructions above to finish up.

To see how two SMF forum administrators applied this approach to sites that had been created with ISO-8859-1 English with UTF-8 Greek character set added later, please read It's all Greek to me .... :)

Retrieved from "https://wiki.simplemachines.org/w/index.php?title=UTF-8_Readme&oldid=30769"

Category:

@@ Line 1: / Line 1: @@
-{{Cleanup html}}
+UTF-8 is an encoding standard that can represent all Unicode characters. This allows it to show almost any writing system in the world.
-'''What is UTF-8?'''
-UTF-8 is an encoding standard that can represent all Unicode characters. This allows to show almost any writing system in the world.
-'''What&#039;s new in SMF 1.1 RC3 with regard to UTF-8?'''
+==UTF in SMF 2.0.x==
-SMF has always supported multiple character sets. Each language package was written in a specific character set. Nothing has changed to the support of those character sets, but additional support for UTF-8 has been added. It is possible to convert your forum to UTF-8 or (in case of a new forum) install with UTF-8 support. If you have your forum in UTF-8 mode, both the database and website will be using UTF-8.
+SMF 2.0.x includes the option to run the forum with or without UTF-8.  If you install a new SMF 2.0.x forum with UTF-8, or if you upgrade an existing SMF 2.0.x forum to UTF-8, all posts will be stored in the database using the UTF-8 character set, and every web page will inform the browser that it is using the UTF-8 character set.  For each language pack you decide to install for your forum, it will be necessary to choose the UTF-8 version of that character set.
-The following character sets are currently used for SMF&#039;s language packages (both 1.0.x and 1.1):
+If you choose not to use UTF-8 for your SMF 2.0.x forum, you must choose the non-UTF-8 versions of all language packs that you install on your forum.
-<table class="bbc_table"><tr><td>''' Character set '''</td><td>'''Language'''</td></tr><tr><td> big5</td><td>Chinese (traditional)</td></tr><tr><td> gbk</td><td>Chinese (simplified)</td></tr><tr><td> ISO-8859-1</td><td>Albanian, Brazilian, Catalan, Danish, Dutch, English, Finnish, French, German, Portuguese, Norwegian, Spanish, Swedish</td></tr><tr><td> ISO-8859-2</td><td>Croatian, Hungarian, Polish, Romanian</td></tr><tr><td> ISO-8859-9</td><td>Turkish</td></tr><tr><td> tis-620</td><td>Thai</td></tr><tr><td> UTF-8</td><td>Chinese (simplified), Chinese (traditional), Japanese, Persian</td></tr><tr><td> windows-1256</td><td>Arabic</td></tr><tr><td> windows-1251</td><td>Bulgarian, Russian</td></tr><tr><td> windows-1253</td><td>Greek</td></tr><tr><td> windows-1255</td><td>Hebrew</td></tr></table>
+If you choose the wrong character set for any of your language packs, you will see some "garbage characters" on the screen.  The solution is to use only the correct character set version of the language packs you have chosen for your forum.
-As of SMF 1.1 RC3 you&#039;ll be able to also download each of those language packages in UTF-8 character set ([http://www.simplemachines.org/download/?languagesLanguage packs]).
+You can download language packages with a UTF-8 character set [http://download.simplemachines.org/?smflanguages here].
-<span style="font-size: 11pt;" class="bbc_size">'''Why would I need UTF-8?'''</span>
+==Why would I need UTF-8?==
 There are a few reasons you might need UTF-8:
-<ul class="bbc_list"><li>If you want to support multiple languages that use different character sets on your forum. For instance if you want to support both Russian and Turkish, you will need a character set that supports both. UTF-8 is then a logical choice.</li><li>If the software integrating with SMF uses UTF-8. In some cases such an integration can require character sets to match.</li><li>If you need better search results or improved sorting. In some cases searching and sorting by the database can be improved by chosing UTF-8 as your character set.</li></ul>
+*If you want to support multiple languages that use different character sets on your forum. For instance, if you want to support both Russian and Turkish, you will need a character set that supports both.
-<span style="font-size: 11pt;" class="bbc_size">'''Why would I NOT need UTF-8?'''</span>
+*If the software integrating with SMF uses UTF-8. In some cases such an integration can require character sets to match.
-For all the other reasons, UTF-8 would probably not be very useful. Besides, it&#039;s a bit slower too.
+*If you need better search results or improved sorting. In some cases searching and sorting by the database can be improved by choosing UTF-8 as your character set.
-Also keep in mind that you need at least MySQL 4.1 and SMF 1.1 RC3 to be able to use UTF-8 as default character set if you are using MySQL as your database scheme.
+==Why would I not need UTF-8?==
+If none of the above reasons apply to your forum, UTF-8 would probably not be very useful.
-<span style="font-size: 11pt;" class="bbc_size">'''How to convert to UTF-8?'''</span>
+==How to convert to UTF-8==
-<ul class="bbc_list"><li>Start with a <span style="color: red;" class="bbc_color">backup</span> of your database(!) Character set conversions are all but guaranteed to go right.</li><li>Go to &#039;Forum Maintenance&#039; -&gt; &#039;Convert the database and data to UTF-8&#039;</li><li>Select the character set your current data is in. The default setting for this is based on the character set of your default langauge file.</li><li>After pressing proceed, your database will be converted. Depending on the size of your database, the conversion process might stop temporarily from time to time to avoid overloading the server. If that was successful, your forum should be converted to UTF-8.</li><li>You&#039;ll be needing a new set of language files. All language files need to be UTF-8 compatible. Luckily all language packs for 1.1 RC3 are available for both the original character set and UTF-8, so simply download them and you should be ready to go.</li><li>Once all the UTF-8 language packs have been installed, convert the language settings of each user by running the following query:</li></ul><div class="codeheader">: <a href="javascript:void(0);" onclick="return smfSelectText(this);" class="codeoperation"></a></div><code>UPDATE smf_members
+This procedure will work well if you have used the recommended character sets in your language files in the past.
-SET lngfile = CONCAT(lngfile, &#039;-utf8&#039;)
-WHERE lngfile != &#039;&#039;</code><ul class="bbc_list"><li>Also, change the default language in your admin center - Admin -&gt; Server Settings</li><li>Check to see if all your data was properly converted</li><li>If any of your posts contain HTML entities, you will want to convert those to UTF-8 as well...</li></ul>
+. Start with a '''backup''' of your database. Character set conversions nearly always work correctly, but it is best to be prepared for the unexpected.
+. Check your default language file ([[Languages|Administration Center » Languages » Edit Languages]] ) and make a note of the character set used.
+. Go to ''Forum Maintenance > Convert the database and data to UTF-8'' (this option will only be available if SMF detects a database version which supports UTF-8).
+. Select the character sets for your data (member posts) and database. By default, SMF will choose the character set of your default language file.
+. Press proceed, and your database will be converted. Depending on the size of your database, the conversion process might stop temporarily from time-to-time to avoid overloading the server. If this was successful, your forum should be converted to UTF-8.
+. For each of language pack currently in use, replace it with the UTF-8 version of that language pack.
+. Once all the UTF-8 language packs have been installed, convert the language settings of each user by running this query:
+{{code
+|1=<nowiki>UPDATE smf_members
+SET lngfile = CONCAT(lngfile, '-utf8')
+WHERE lngfile != ''</nowiki>}}
+. In your Administration Center, change the default language, ensuring that you choose the UTF-8 version.
+. Check to see if all your data was properly converted.
+. If any of your posts contain HTML entities, you will want to convert these to UTF-8 as well by running "Convert HTML-entities to UTF-8 characters"
+==What to do if your site uses a mix of character sets==
+If your site began as a non-UTF-8 installation, and you chose UTF-8 language files, the procedure above might not work perfectly.  Because there are likely to be hundreds of posts on the database in a character set different to the forum's default character set, site search functions might not work well, and the situation might be holding you back from converting to SMF 2.0.
+The good news is that you can fix this problem and convert to UTF-8.  You must simply plan the process out a little more carefully, and might need to take one or two extra steps.
+*Before you begin, double check the default language file/character set you used to install SMF, and also the language file/character set you later added (and which was used for storing the important posts on the forum).  Make sure to put the forum into maintenance mode.
+*Set the default language for your forum to be whatever it was when you first installed the forum (English ISO-8859-1, for example)
+*In order to change your default language (from English to Greek, for example), it is best to get a copy of the upgrade package for the current version of SMF, for the language you want to be your new default.  Copy all these files just as you normally would for an upgrade or install.
+*Run upgrade.php, still using the old default character set, and then delete upgrade.php
+*If your character set problems have affected your search take the following steps:
+**Go to Administration Center » Search » Search Method, delete any text index (if any) and select "No index" as the search method.
+**Remove all rows from tables with names like smf_log_search_*.  Use phpMyAdmin to do this. Do not drop these tables.
+**Now proceed through the instructions above up to step 4.  During step 4, you will have to set the '''Data character set''' to the character set used in the posts on your forum.
+**Follow the remainder of the instructions above to finish up.
+To see how two SMF forum administrators applied this approach to sites that had been created with ISO-8859-1 English with UTF-8 Greek character set added later, please read [http://www.simplemachines.org/community/index.php?msg=3093945 It's all Greek to me .... :) ]
 [[Category:FAQ]]

Navigation

Search

Tools