pg_jieba User Guide

pg_jieba is a third-party extension module for PostgreSQL based on the Jieba segmentation library. It is specifically designed for Chinese full-text search, providing efficient segmentation of Chinese text to support full-text retrieval. This article provides a detailed guide on how to install and use pg_jieba in ServBay.

Installing pg_jieba

ServBay comes with the pg_jieba extension module. You just need to enable it in the database. Here are the steps to enable pg_jieba:

1. Connect to the PostgreSQL database:

   psql -U your_username -d your_database

2. Create the extension:

   CREATE EXTENSION pg_jieba;

3. Verify the installation:

   \dx

Configuring pg_jieba

After enabling pg_jieba, you need to perform some configurations to ensure it correctly segments Chinese text and supports full-text search.

Configuring Text Search Configuration

1. Create a text search configuration:

   CREATE TEXT SEARCH CONFIGURATION chinese (PARSER = pg_jieba);

2. Add the dictionary:

   ALTER TEXT SEARCH CONFIGURATION chinese ADD MAPPING FOR n,v,a,i,e,l WITH simple;

Using pg_jieba for Full-Text Search

Below is an example showing how to use pg_jieba for full-text search.

Creating Sample Tables and Data

1. Create a table:

   CREATE TABLE documents (
       id SERIAL PRIMARY KEY,
       content TEXT
   );

2. Insert sample data:

   INSERT INTO documents (content) VALUES
   ('I love natural language processing'),
   ('Chinese word segmentation is an important step in text processing'),
   ('pg_jieba is a great Chinese word segmentation tool');

Creating a Full-Text Search Index

1. Create a GIN index:

   CREATE INDEX idx_gin_content ON documents USING gin (to_tsvector('chinese', content));

Executing Full-Text Search

1. Execute a search query:

   SELECT * FROM documents
   WHERE to_tsvector('chinese', content) @@ to_tsquery('chinese', '中文 & 分词');

   This query will return documents containing both "中文" and "分词".

Custom Dictionary

You can customize the dictionary used by pg_jieba to better meet specific application needs.

Adding Custom Words

1. Create a custom dictionary file:

   /Applications/ServBay/etc/scws/custom_dict.txt

2. Add words to the file, one per line:

   Natural language processing
   Chinese word segmentation

3. Configure pg_jieba to use the custom dictionary:

   SET pg_jieba.dict_path = '/Applications/ServBay/etc/scws/custom_dict.txt';

Reloading the Dictionary

1. Reload the dictionary:

   SELECT jieba_reload_dict();

Summary

pg_jieba is a powerful Chinese word segmentation tool. With simple configuration and usage, you can achieve efficient Chinese full-text search in PostgreSQL. ServBay comes with the pg_jieba extension module, and you can start using it by following the steps outlined in this article. By customizing the dictionary, you can further optimize segmentation results to meet specific application needs.