Skip to content

feat: support dynamic configuration updates via environment variables (not only searchd/common sections) #123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sanikolaev wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat: support dynamic configuration updates via environment variables (not only searchd/common sections) #123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,15 +247,35 @@ Take into account that Manticore search inside the container is run under user `
docker exec -it manticore gosu manticore indexer --all --rotate
```

You can also set individual `searchd` and `common` configuration settings using Docker environment variables.
You can also set individual configuration settings using Docker environment variables.

The settings must be prefixed with their section name, example for in case of `mysql_version_string` the variable must be named `searchd_mysql_version_string`:
### Setting searchd and common options

The settings must be prefixed with their section name, example for in case of `mysql_version_string` the variable must be named `searchd_mysql_version_string`:

```bash
docker run --name manticore -p 127.0.0.1:9306:9306 -e searchd_mysql_version_string='5.5.0' -d manticoresearch/manticore
```

### Setting options in other sections (source, index, etc.)

You can also set options in other configuration sections like `source`, `table`, `search`, and `indexer` using environment variables. The format is `{section}_{section_name}_{setting}`.

For example, to set `sql_user` in a `source min` block:

```bash
docker run --name manticore -e source_min_sql_user='abc' -d manticoresearch/manticore
```

This will update or add `sql_user = abc` within the `source min { ... }` section. If the section doesn't exist, it will be created automatically.

Similarly, for an index section:
```bash
docker run --name manticore -e index_test_path='/var/lib/manticore/test' -d manticoresearch/manticore
```

**Note**: Section names with spaces (like `source min`) should use underscores in the environment variable name (e.g., `source_min_sql_user`).

If you intend to enable the own `listen` directive, utilize the `searchd_listen` environment variable.

You can specify multiple interfaces separated by a semicolon (`|`). To exclusively listen on a network address, employ the `$ip` variable (internally retrieved from `hostname -i`) as an address alias.
Expand Down
203 changes: 165 additions & 38 deletions manticore.conf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,99 @@ if [ -z "$searchd_listen" ]; then
fi
fi

# Function to update a setting within a specific section block
# Usage: update_setting_in_section "section_identifier" "setting_key" "setting_value"
# section_identifier can be "searchd", "common", "source min", "table abc", etc.
update_setting_in_section() {
local section_identifier="$1"
local setting_key="$2"
local setting_value="$3"

# Convert underscores in section_identifier to spaces for matching
# This allows "source_min" in env var to match "source min" in config
local section_pattern=$(echo "$section_identifier" | sed 's/_/[[:space:]]\+/g')
local section_start_pattern="^[[:space:]]*${section_pattern}[[:space:]]*\{"

# Check if section exists
if ! echo "$conf" | grep -qE "$section_start_pattern"; then
# Section doesn't exist, create it
# Convert underscores back to spaces for the actual section declaration
local section_name=$(echo "$section_identifier" | sed 's/_/ /g')
conf="${conf}
${section_name} {
${setting_key} = ${setting_value}
}"
return
fi

# Section exists, use awk to process and update within that section only
# This properly handles nested blocks and only modifies settings within the target section
conf=$(echo "$conf" | awk -v section_start_pattern="$section_start_pattern" \
-v setting_key="$setting_key" \
-v setting_value="$setting_value" '
BEGIN {
in_target_section = 0
section_depth = 0
found_setting = 0
}
{
line = $0
original_line = line

# Check if we are entering the target section
if (match(line, section_start_pattern)) {
in_target_section = 1
section_depth = 1
print line
next
}

# If we are in the target section, track depth for nested blocks
if (in_target_section == 1) {
# Count opening and closing braces on this line
open_braces = 0
close_braces = 0
len = length(original_line)
for (i = 1; i <= len; i++) {
char = substr(original_line, i, 1)
if (char == "{") open_braces++
if (char == "}") close_braces++
}
section_depth += (open_braces - close_braces)

# Check if we have left the target section
if (section_depth <= 0) {
# Add setting before closing brace if not found
if (found_setting == 0) {
# Get indentation from closing brace line
match(original_line, /^[[:space:]]*/)
indent = substr(original_line, 1, RLENGTH)
print indent " " setting_key " = " setting_value
}
in_target_section = 0
found_setting = 0
section_depth = 0
print original_line
next
}

# Check if this line contains the setting we want to update (within the section)
if (match(original_line, "^[[:space:]]*" setting_key "[[:space:]]*=")) {
# Replace the setting value, preserve indentation
match(original_line, /^[[:space:]]*/)
indent = substr(original_line, 1, RLENGTH)
print indent setting_key " = " setting_value
found_setting = 1
next
}
}

print original_line
}
')
}

# Check for searchd/common env vars (backward compatibility)
while IFS='=' read -r envVariable value; do
if [[ "${envVariable}" == searchd_* ]]; then
hasSearchdEnv=1
Expand All @@ -27,15 +120,13 @@ while IFS='=' read -r envVariable value; do
fi
done < <(env)



# Create searchd/common sections if they don't exist and env vars are present
if [[ -n $hasCommonEnv && ! $(echo $conf | grep -E "common\s*{") ]]; then
conf="$(echo "${conf}")
common {
}"
fi


if [[ -n $hasSearchdEnv && ! $(echo $conf | grep -E "searchd\s*{") ]]; then
conf="$(echo "${conf}")
searchd {
Expand All @@ -50,46 +141,82 @@ else
hostip="0.0.0.0"
fi

# Process all environment variables
while IFS='=' read -r envVariable value; do

if [[ "${envVariable}" == searchd_* || "${envVariable}" == common_* ]]; then

if [[ "${envVariable}" == searchd_* ]]; then
section="searchd"

# Skip if not a config variable (doesn't contain underscore)
if [[ ! "${envVariable}" =~ _ ]]; then
continue
fi

# Split on the last underscore to get section_identifier and setting_name
# This handles: searchd_setting, common_setting, source_min_sql_user, table_abc_path, etc.
# Extract section identifier (everything before last underscore) and setting name (after last underscore)
# Use parameter expansion: ${var%_*} removes last _ and everything after, ${var##*_} removes everything up to last _
local section_identifier="${envVariable%_*}"
local setting_name="${envVariable##*_}"

# Safety check: if no underscore was found, section_identifier would be the same as envVariable
if [ "$section_identifier" == "$envVariable" ]; then
continue
fi

# Get the actual value and escape it
actual_value=$(echo ${!envVariable} | sed 's/\//\\\//g')

# Special handling for searchd listen directive
# The listen directive is special because:
# 1. It can appear multiple times in the searchd section (each becomes a separate "listen = ..." line)
# 2. Values are pipe-separated (|) and need to be split into multiple lines
# 3. The $ip variable needs to be expanded to the actual host IP
# 4. All existing listen directives in the searchd section should be replaced (not just updated)
if [[ "$section_identifier" == "searchd" && "$setting_name" == 'listen' ]]; then
# Remove all existing listen directives only from the searchd section
conf=$(echo "$conf" | awk '
BEGIN { in_searchd = 0; section_depth = 0 }
/^[[:space:]]*searchd[[:space:]]*\{/ { in_searchd = 1; section_depth = 1; print; next }
in_searchd == 1 {
# Count braces to track section depth
temp = $0
open_braces = gsub(/{/, "&", temp)
temp = $0
close_braces = gsub(/}/, "&", temp)
section_depth += (open_braces - close_braces)

# Skip listen directives within searchd section
if (match($0, /^[[:space:]]*listen[[:space:]]*=/)) {
if (section_depth > 0) {
next # Skip this line
}
}

if (section_depth <= 0) {
in_searchd = 0
}
}
{ print }
')

# Parse pipe-separated values and expand $ip variable
IFS='|' read -ra LISTEN_VALUES <<<"$actual_value"
count=0
actual_value=""

for i in "${LISTEN_VALUES[@]}"; do
i=${i/\$ip/$hostip}
if [[ $count == 0 ]]; then
actual_value=$i
else
section="common"
fi

value=$(echo ${!envVariable} | sed 's/\//\\\//g')
cleaned_key=$(echo $envVariable | sed "s/${section}_//")

if [[ $cleaned_key == 'listen' ]]; then
conf=$(echo "${conf}" | sed -e "s/^\s*listen\s*=.*$//g" | sed -r '/^\s*$/d')
IFS='|' read -ra LISTEN_VALUES <<<"$value"
count=0

for i in "${LISTEN_VALUES[@]}"; do
i=${i/\$ip/$hostip}
if [[ $count == 0 ]]; then
value=$i
else
value="$value\n listen = $i"
fi
count=$((count + 1))
done
fi

pattern="\s*${cleaned_key}\s*="
if [[ ${conf} =~ $pattern ]]; then
conf=$(echo "${conf}" | sed -e "s/^\s*${cleaned_key}\s*=.*$/ ${cleaned_key} = ${value}/g")
else
conf=$(echo "${conf}" | sed -e "/^\s*${section}\s*{/a \ ${cleaned_key} = ${value}")
fi

actual_value="$actual_value\n listen = $i"
fi
count=$((count + 1))
done
fi

# Update the setting in the appropriate section
update_setting_in_section "$section_identifier" "$setting_name" "$actual_value"

done < <(env)


echo "${conf}" > /etc/manticoresearch/manticore.conf.debug
echo "${conf}"
Loading