zhparser Usage Guide

zhparser is a third-party extension module for PostgreSQL designed specifically for processing Chinese text. It provides efficient word segmentation and search functionality suitable for various Chinese text processing scenarios. ServBay comes with scws (Simple Chinese Word Segmentation), and zhparser can use scws to create custom dictionaries. This article will detail how to install and use zhparser in ServBay.

Installing zhparser

ServBay already includes the zhparser extension module. You just need to enable it in the database. Here are the steps to enable zhparser:

1. Connect to the PostgreSQL database:

   psql -U your_username -d your_database

2. Create the extension:

   CREATE EXTENSION zhparser;

3. Verify the installation:

   \dx

Configuring zhparser

After enabling zhparser, you need to perform some configurations for it to correctly handle Chinese word segmentation and full-text search.

Configuring Text Search Configuration

1. Create a text search configuration:

   CREATE TEXT SEARCH CONFIGURATION chinese (PARSER = zhparser);

2. Add dictionaries:

   ALTER TEXT SEARCH CONFIGURATION chinese ADD MAPPING FOR n,v,a,i,e,l WITH simple;

Using zhparser for Full-Text Search

Below is an example showing how to use zhparser for full-text search.

Creating Sample Tables and Data

1. Create a table:

   CREATE TABLE documents (
       id SERIAL PRIMARY KEY,
       content TEXT
   );

2. Insert sample data:

   INSERT INTO documents (content) VALUES
   ('I love Natural Language Processing'),
   ('Chinese word segmentation is an important step in text processing'),
   ('zhparser is a great Chinese word segmentation tool');

Creating Full-Text Search Index

1. Create a GIN index:

   CREATE INDEX idx_gin_content ON documents USING gin (to_tsvector('chinese', content));

Executing Full-Text Search

1. Execute a search query:

   SELECT * FROM documents
   WHERE to_tsvector('chinese', content) @@ to_tsquery('chinese', '中文 & 分词');

   This query will return documents containing both "Chinese" and "word segmentation".

Custom Dictionary

ServBay comes with scws, and you can use scws to create custom dictionaries to better meet specific application needs.

Adding Custom Words

1. Create a custom dictionary file:

   /Applications/ServBay/etc/scws/custom_dict.txt

2. Add words to the file, one word per line:

   Natural Language Processing
   Chinese word segmentation

3. Configure zhparser to use the custom dictionary:

   SET zhparser.dict_path = '/Applications/ServBay/etc/scws/custom_dict.txt';

Reloading the Dictionary

1. Reload the dictionary:

   SELECT zhprs_reload_dict();

Adjusting Segmentation Mode

zhparser supports various segmentation modes, and you can adjust them as needed.

Setting Segmentation Mode

1. Set the segmentation mode to the finest granularity segmentation:

   SET zhparser.seg_with_duality = true;

2. Set the segmentation mode to the coarsest granularity segmentation:

   SET zhparser.seg_with_duality = false;

Summary

zhparser is a powerful Chinese word segmentation tool. With simple configuration and usage, you can achieve efficient Chinese full-text search in PostgreSQL. ServBay comes with the zhparser extension module and scws, allowing you to create custom dictionaries. By customizing dictionaries and adjusting segmentation modes, you can further optimize the segmentation effect to meet specific application needs.