Includes in Sphinx Search

Little known feature of Sphinx Search is dynamic config generation. Sphinx Search programs (indexer, searchd, etc.) take --config option. If config has shebang on first line, mentioned programs will execute config as a program, and will treat everything written to stdout as actual config.

Let’s start with simple task: include contents of few files from /usr/local/etc/sphinx/ dir into config:

#!/bin/sh
/bin/cat /usr/local/etc/sphinx/*.conf

cat concatenates files and writes them to stdout (see man cat for short documentation). /usr/local/etc/sphinx/*.conf means “all files from /usr/local/etc/sphinx/ dir that have .conf extension”. Thus, we concatenate and print all .conf files from /usr/local/etc/sphinx/ dir.

Okay, but what if we need something more advanced? We can use Python: just put #!/usr/bin/env python on the first line. We can put index names and SQL into tuple of tuples, and later generate parts of config from it (simplified example):

INDEXES = (
    ('qa_questions', 'SELECT id, title, text FROM qa_question'),
    ('qa_answers', 'SELECT id, text FROM qa_answer'),
)

No one wants to hardcode SQL into config. What about something DRYer? As this is regular Python script, we can use ORM (Django ORM, SQLAlchemy, Storm, etc.) to generate SQL for us, or any other library.