# Conjunto de Caractere

## Suporte ao conjunto de caractere UTF-8 e byte-simples

`Zend_Search_Lucene` trabalha internamente com o conjunto de caractere UTF-8. Arquivos de índice armazenam dados unicode no formato de codificação "UTF-8 modificado" usado pelo Java. O núcleo do `Zend_Search_Lucene` suporta esta codificação plenamente, com uma exceção.

`Zend_Search_Lucene` suporta somente os caracteres do Plano Multilingual Básico (BMP) (de 0x0000 a 0xFFFF), não suportando os caracteres suplementares (caracteres acima de 0xFFFF) O Java 2 representa estes caracteres como um par de valores do tipo char (16 bits), o primeiro vem da faixa superior (0xD800-0xDBFF), o segundo, da faixa inferior (0xDC00-0xDFFF). Logo eles são codificados como caracteres usuais UTF-8 em seis bytes. A representação padrão UTF-8 utiliza quatro bytes para caracteres suplementares.

A codificação dos dados de entrada pode ser especificada através da API de `Zend_Search_Lucene`. Os dados serão convertidos automaticamente na codificação UTF-8.

## Analisador de texto padrão

De qualquer modo, o analisador de texto padrão (que também é usado no analisador de consulta) utiliza ctype_alpha() para a sinalização de texto e consultas.

ctype_alpha() is not UTF-8 compatible, so the analyzer converts text to 'ASCII//TRANSLIT' encoding before indexing. The same processing is transparently performed during query parsing.

Conversion to 'ASCII//TRANSLIT' may depend on current locale and OS.

###

Default analyzer doesn't treats numbers as parts of terms. Use corresponding 'Num' analyzer if you don't want words to be broken by numbers.

## UTF-8 compatible text analyzers

`Zend_Search_Lucene` also contains a set of UTF-8 compatible analyzers: `Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8`, `Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8Num`, `Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8_CaseInsensitive`, `Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8Num_CaseInsensitive`.

Any of this analyzers can be enabled with the code like this:

    Zend_Search_Lucene_Analysis_Analyzer::setDefault(
        new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8());

###

UTF-8 compatible analyzers were improved in Zend Framework 1.5. Early versions of analyzers assumed all non-ascii characters are letters. New analyzers implementation has more accurate behavior.

This may need you to re-build index to have data and search queries tokenized in the same way, otherwise search engine may return wrong result sets.

All of these analyzers need PCRE (Perl-compatible regular expressions) library to be compiled with UTF-8 support turned on. PCRE UTF-8 support is turned on for the PCRE library sources bundled with PHP source code distribution, but if shared library is used instead of bundled with PHP sources, then UTF-8 support state may depend on you operating system.

Use the following code to check, if PCRE UTF-8 support is enabled:

    if (@preg_match('/\pL/u', 'a') == 1) {
        echo "PCRE unicode support is turned on.\n";
    } else {
        echo "PCRE unicode support is turned off.\n";
    }

Case insensitive versions of UTF-8 compatible analyzers also need mbstring extension to be enabled.

If you don't want mbstring extension to be turned on, but need case insensitive search, you may use the following approach: normalize source data before indexing and query string before searching by converting them to lowercase:

    // Indexing
    setlocale(LC_CTYPE, 'de_DE.iso-8859-1');

    ...

    Zend_Search_Lucene_Analysis_Analyzer::setDefault(
        new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8());

    ...

    $doc = new Zend_Search_Lucene_Document();

    $doc->addField(Zend_Search_Lucene_Field::UnStored('contents',
                                                      strtolower($contents)));

    // Title field for search through (indexed, unstored)
    $doc->addField(Zend_Search_Lucene_Field::UnStored('title',
                                                      strtolower($title)));

    // Title field for retrieving (unindexed, stored)
    $doc->addField(Zend_Search_Lucene_Field::UnIndexed('_title', $title));

    // Searching
    setlocale(LC_CTYPE, 'de_DE.iso-8859-1');

    ...

    Zend_Search_Lucene_Analysis_Analyzer::setDefault(
        new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8());

    ...

    $hits = $index->find(strtolower($query));