Usecase where we need to pressist data to ClickHouse the sooner the better...

[meshpaul]

Hi, we are a bit new to grafite... since we are using ClickHouse we also decided to use graphouse for telemetry. So we are using Grafana with ClickHouse datasource, and going directly to graphouse.data to get telemetry and what we are seeing data is actually 1 min to 5 min lag depending on volume and frequency. So I suspect is graphouse catches some data in memory using the options below (in #Mretric search and tree.. section )... if this is true what is the best setting for pushing metrics to ClickHouse no more than 30 seconds old? Is this possible? What are our options? Any advice is appreciated

We are running on 2 CPU / 8 GB RAM server

graphouse.properties

graphouse.allow-cold-run=false

#Clickhouse
graphouse.clickhouse.host=somehost
graphouse.clickhouse.port=8123
graphouse.clickhouse.db=graphite
graphouse.clickhouse.user=default
graphouse.clickhouse.password=ourpass
graphouse.clickhouse.compress=false

graphouse.clickhouse.data-table=data
graphouse.clickhouse.metric-tree-table=metrics

graphouse.clickhouse.socket-timeout-seconds=600
graphouse.clickhouse.query-timeout-seconds=120

graphouse.clickhouse.retention-config=

#metric server and cacher
graphouse.cacher.bind-address=0.0.0.0
graphouse.cacher.port=2003
graphouse.cacher.threads=2800
graphouse.cacher.socket-timeout-millis=42000

graphouse.cacher.queue-size=10000000
graphouse.cacher.read-batch-size=1000
graphouse.cacher.min-batch-size=10000
graphouse.cacher.max-batch-size=1000000
graphouse.cacher.min-batch-time-seconds=1
graphouse.cacher.max-batch-time-seconds=5
graphouse.cacher.max-output-threads=10000

#Http server (metric search, ping, metricData)
graphouse.http.bind-address=
graphouse.http.port=2005
graphouse.http.threads=25
graphouse.http.max-form-context-size-bytes=500000

#Mretric search and tree
graphouse.search.refresh-seconds=1
graphouse.tree.in-memory-levels=3
graphouse.tree.dir-content.cache-time-minutes=10
graphouse.tree.dir-content.cache-concurrency-level=100
graphouse.tree.dir-content.batcher.max-parallel-requests=3
graphouse.tree.dir-content.batcher.max-batch-size=2000
graphouse.tree.dir-content.batcher.aggregation-time-millis=50


#Host metrics redirect
graphouse.host-metric-redirect.enabled=false
graphouse.host-metric-redirect.dir=
graphouse.host-metric-redirect.postfixes=

#Autohide
graphouse.autohide.enabled=false
graphouse.autohide.run-delay-minutes=30
graphouse.autohide.max-values-count=200
graphouse.autohide.missing-days=7
graphouse.autohide.step=10000
graphouse.autohide.retry.count=10
graphouse.autohide.retry.wait_seconds=10
graphouse.autohide.clickhouse.query-timeout-seconds=600

#Metric validation
graphouse.metric-validation.min-length=5
graphouse.metric-validation.max-length=500
graphouse.metric-validation.min-levels=2
graphouse.metric-validation.max-levels=20
#graphouse.metric-validation.regexp=[-_0-9a-zA-Z\\.]*$
graphouse.metric-validation.regexp=.*

[meshpaul]

@AndreevDm there is a bug in algorithm data is not getting pulled from NonBlockingQ and written to ClickHouse for period of 5 min... . We have identified issue will be giving the details.. Working with our legal to so we can provide fix..