Containers running but no data

[laxxie]

I don't know if i'm too dumb or don't see some obvious failures. Unfortunately i'm not really into Kafka and co.

I followed the Getting Started guide to install everything. Every container is running and i don't see any issues why i don't get any data into Kafka/Postgresql.
Two Juniper routers are configured to send BMP messages and i see incoming packets on port 5000.

root@bmp:/opt# docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED              STATUS              PORTS                                                           NAMES
92745a28b18b   confluentinc/cp-kafka:7.1.1       "/etc/confluent/dock…"   About a minute ago   Up About a minute   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                       obmp-kafka
4ffaafe34a88   openbmp/postgres:2.2.1            "/docker-entrypoint.…"   About a minute ago   Up About a minute   8008/tcp, 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp, 8081/tcp   obmp-psql
09841a154b09   openbmp/whois:2.2.0               "/bin/sh -c '/usr/lo…"   About a minute ago   Up About a minute   0.0.0.0:4300->43/tcp, :::4300->43/tcp                           obmp-whois
18e456d22fda   openbmp/psql-app:2.2.2            "/usr/sbin/run"          About a minute ago   Up About a minute   0.0.0.0:9005->9005/tcp, :::9005->9005/tcp                       obmp-psql-app
bfdfe41361cf   confluentinc/cp-zookeeper:7.1.1   "/etc/confluent/dock…"   About a minute ago   Up About a minute   2181/tcp, 2888/tcp, 3888/tcp                                    obmp-zookeeper
9b44ed3edaa7   openbmp/collector:2.2.3           "/usr/sbin/run"          About a minute ago   Up About a minute   0.0.0.0:5000->5000/tcp, :::5000->5000/tcp                       obmp-collector
191faa80707f   grafana/grafana:9.1.7             "/run.sh"                About a minute ago   Up About a minute   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                       obmp-grafana

root@bmp:/opt# docker exec -it obmp-kafka kafka-consumer-groups --bootstrap-server localhost:29092 --describe --group obmp-psql-consumer

GROUP              TOPIC                    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                             HOST            CLIENT-ID
obmp-psql-consumer openbmp.parsed.collector 6          4               4               0               obmp-psql-consumer-d6c0e62b-3c10-4a8b-afc2-4a4b6c5c03ce /192.168.48.3   obmp-psql-consumer
obmp-psql-consumer openbmp.parsed.collector 3          5               5               0               obmp-psql-consumer-a2d0b2ff-b77e-4343-9eef-272bcecab953 /192.168.48.3   obmp-psql-consumer
obmp-psql-consumer openbmp.parsed.collector 2          0               0               0               obmp-psql-consumer-55491c62-5243-4f70-91c2-8fe776184698 /192.168.48.3   obmp-psql-consumer
obmp-psql-consumer openbmp.parsed.collector 1          0               0               0               obmp-psql-consumer-35ef7d2f-85d4-4ffe-bb2f-27bd3335e802 /192.168.48.3   obmp-psql-consumer
obmp-psql-consumer openbmp.parsed.collector 7          0               0               0               obmp-psql-consumer-db71a141-b1a8-4cc1-a620-35bc7febc0d9 /192.168.48.3   obmp-psql-consumer
obmp-psql-consumer openbmp.parsed.collector 4          242             242             0               obmp-psql-consumer-d1042df6-b402-4b7e-928e-399359335d07 /192.168.48.3   obmp-psql-consumer
obmp-psql-consumer openbmp.parsed.collector 5          0               0               0               obmp-psql-consumer-d1a97a80-1f36-4e44-bf8c-6d9377756da6 /192.168.48.3   obmp-psql-consumer
obmp-psql-consumer openbmp.parsed.collector 0          0               0               0               obmp-psql-consumer-16da5d0c-f242-4fa3-9725-059a2a4f2ddc /192.168.48.3   obmp-psql-consumer

So if i'm right in my understanding the collector should send the data to obmp-kafka but the Topics hasn't been created. I tried it with the default openbmpd.conf and also with a slightly modified one (router_group mapping).

Here is my docker-compose.yml:

---
version: '3'
volumes:
  data-volume:
    driver_opts:
      type: none
      device: /var/openbmp/postgres/data
      o: bind
  ts-volume:
    driver_opts:
      type: none
      device: /var/openbmp/postgres/ts
      o: bind

services:

  zookeeper:
    restart: unless-stopped
    container_name: obmp-zookeeper
    image: confluentinc/cp-zookeeper:7.1.1
    volumes:
      - /var/openbmp/zk-data:/var/lib/zookeeper/data
      - /var/openbmp/zk-log:/var/lib/zookeeper/log
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    restart: unless-stopped
    container_name: obmp-kafka
    image: confluentinc/cp-kafka:7.1.1

    # Change the mount point to where you want to store Kafka data.
    #   Normally 80GB or more
    volumes:
      - /var/openbmp/kafka-data:/var/lib/kafka/data
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: obmp-zookeeper:2181

      # Change/add listeners based on your FQDN that the host and other containers can access.  You can use
      #    an IP address as well. By default, only within the compose/containers can Kafka be accesssed
      #    using port 29092.   Outside access can be enabled, but you should use an FQDN listener.
      #KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://<FQDN>:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://obmp-kafka:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_NUM_PARTITIONS: 8
      KAFKA_LOG_RETENTION_MINUTES: 90
      KAFKA_LOG_ROLL_MS: 3600000
      KAFKA_LOG_SEGMENT_BYTES: 1073741824
      KAFKA_MESSAGE_MAX_BYTES: 100000000
      KAFKA_LOG_CLEANER_THREADS: 2
  # TODO: Move comment to documentation
  # You can use SWAG and Let's Encrypt to front the Grafana HTTP port 3000 instance. Below is an example
  # Create the SWAG direcgtory using "sudo mkdir -m 777 /var/openbmp/swag"
#  swag:
#    image: linuxserver/letsencrypt:version-1.11.0
#    container_name: obmp-swag
#    cap_add:
#      - NET_ADMIN
#    environment:
#      - PUID=1000
#      - PGID=1000
#      - TZ=UTC
#      - URL=<domain of grafana host, this should be the domain only, not the hostname>
#      - SUBDOMAINS=<hostname, without the domain>,
#      - VALIDATION=http
#      - EMAIL=<your email>
#      - ONLY_SUBDOMAINS=true
#      - STAGING=false
#    volumes:
#      - /var/openbmp/swag:/config
#    ports:
#      - 443:443
#      - 80:80 #optional
#    restart: unless-stopped

  grafana:
    restart: unless-stopped
    container_name: obmp-grafana
    image: grafana/grafana:9.1.7
    ports:
      - "3000:3000"
    volumes:
      - /var/openbmp/grafana:/var/lib/grafana
      - /var/openbmp/grafana/provisioning:/etc/grafana/provisioning/
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=i4piW5wP2rz9V63Zd&AS
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_USERS_HOME_PAGE=d/obmp-home/obmp-home
      - GF_INSTALL_PLUGINS=agenty-flowcharting-panel,grafana-piechart-panel,grafana-worldmap-panel,grafana-simple-json-datasource,vonage-status-panel

      # TODO: Move comment block to documentation
      # The below is an example of how to enable OAuth
      #- GF_AUTH_GENERIC_OAUTH_NAME=MyCompany SSO
      #- GF_AUTH_GENERIC_OAUTH_ICON=signin
      #- GF_AUTH_GENERIC_OAUTH_ENABLED=true
      #- GF_AUTH_GENERIC_OAUTH_CLIENT_ID=<the client ID>
      #- GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<client secret>
      #- GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
      #- GF_AUTH_GENERIC_OAUTH_AUTH_URL=<auth url, such as https://domain/as/authorization.oauth2>
      #- GF_AUTH_GENERIC_OAUTH_TOKEN_URL=<token url, such as https://domain/as/token.oauth2
      #- GF_AUTH_GENERIC_OAUTH_API_URL=<userinfo url, such as https://domain/idp/userinfo.openid
      #- GF_AUTH_GENERIC_OAUTH_SCOPES=email openid profile
      #- GF_AUTH_GENERIC_OAUTH_SEND_CLIENT_CREDENTIALS_VIA_POST=true
      #- GF_SERVER_ENFORCE_DOMAIN=true
      #- GF_SERVER_PROTOCOL=http
      #- GF_SERVER_HTTP_PORT=3000
      #- GF_SERVER_DOMAIN=<fqdn of grafana server>
      #- GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/
  psql:
    restart: unless-stopped
    container_name: obmp-psql
    image: openbmp/postgres:2.2.1
    privileged: true
    shm_size: 1536m
    sysctls:
      - net.ipv4.tcp_keepalive_intvl=30
      - net.ipv4.tcp_keepalive_probes=5
      - net.ipv4.tcp_keepalive_time=180
    ports:
      - "5432:5432"
    volumes:
      - data-volume:/var/lib/postgresql/data
      - ts-volume:/var/lib/postgresql/ts
    # alter_job max_runtime in _timescaledb_config.bgw_job ( https://docs.timescale.com/latest/api#alter_job )
    command: >
      -c max_wal_size=10GB
    environment:
      - POSTGRES_PASSWORD=openbmp
      - POSTGRES_USER=openbmp
      - POSTGRES_DB=openbmp

  collector:
    restart: unless-stopped
    container_name: obmp-collector
    image: openbmp/collector:2.2.3
    sysctls:
      - net.ipv4.tcp_keepalive_intvl=30
      - net.ipv4.tcp_keepalive_probes=5
      - net.ipv4.tcp_keepalive_time=180
    ports:
      - "5000:5000"
    volumes:
      - /var/openbmp/config:/config
      - /var/openbmp/config/openbmpd.conf:/usr/etc/openbmp/openbmpd.conf
    environment:
      - KAFKA_FQDN=obmp-kafka:29092

  psql-app:
    restart: unless-stopped
    container_name: obmp-psql-app
    image: openbmp/psql-app:2.2.2
    sysctls:
      - net.ipv4.tcp_keepalive_intvl=30
      - net.ipv4.tcp_keepalive_probes=5
      - net.ipv4.tcp_keepalive_time=180
    ports:
      - "9005:9005"

    volumes:
      - /var/openbmp/config:/config
    environment:
      - MEM=16                                           # Set memory to at least 2GB but ideally 4GB
      - KAFKA_FQDN=obmp-kafka:29092
      - RPKI_URL=https://rpki.cloudflare.com/rpki.json  # define the URL to retrieve json endoed RPKI data
      - RPKI_PASS=None
      - RPKI_USER=None
      - ENABLE_RPKI=1                                   # 1 enables, 0 disables RPKI sync
      - ENABLE_IRR=1                                    # 1 enables, 0 disables IRR sync
      - ENABLE_DBIP=1                                   # 1 enables, 0 disables DBIP import
      - POSTGRES_REPORT_WINDOW='8 minute'               # default POSTGRESS window to select when building
                                                        # summary tables. For deployments that absorb large
                                                        # bursts increase the value, ex 60 minute
      - POSTGRES_PASSWORD=openbmp
      - POSTGRES_USER=openbmp
      - POSTGRES_DB=openbmp
      - POSTGRES_HOST=obmp-psql
      - POSTGRES_PORT=5432
      - POSTGRES_DROP_peer_event_log='1 year'
      - POSTGRES_DROP_stat_reports='4 weeks'
      - POSTGRES_DROP_ip_rib_log='4 weeks'
      - POSTGRES_DROP_alerts='4 weeks'
      - POSTGRES_DROP_ls_nodes_log='4 months'
      - POSTGRES_DROP_ls_links_log='4 months'
      - POSTGRES_DROP_ls_prefixes_log='4 months'
      - POSTGRES_DROP_stats_chg_byprefix='4 weeks'
      - POSTGRES_DROP_stats_chg_byasn='4 weeks'
      - POSTGRES_DROP_stats_chg_bypeer='4 weeks'
      - POSTGRES_DROP_stats_ip_origins='4 weeks'
      - POSTGRES_DROP_stats_peer_rib='4 weeks'
      - POSTGRES_DROP_stats_peer_update_counts='4 weeks'

  whois:
    restart: unless-stopped
    container_name: obmp-whois
    image: openbmp/whois:2.2.0
    sysctls:
      - net.ipv4.tcp_keepalive_intvl=30
      - net.ipv4.tcp_keepalive_probes=5
      - net.ipv4.tcp_keepalive_time=180
    ports:
      - "4300:43"
#    volumes:
#      - /var/openbmp/config:/config
    environment:
      - POSTGRES_PASSWORD=openbmp
      - POSTGRES_USER=openbmp
      - POSTGRES_DB=openbmp
      - POSTGRES_HOST=obmp-psql
      - POSTGRES_PORT=5432

root@bmp:/opt# docker logs obmp-collector
/usr/sbin/run: line 31: /etc/default/openbmpd: No such file or directory
sed: cannot rename /usr/etc/openbmp/sedoHFn0y: Device or resource busy
sed: cannot rename /usr/etc/openbmp/sedML0rFB: Device or resource busy
Waiting 30 seconds to allow for Kafka and other containers to startup.
Running openbmpd collector, see /var/log/openbmpd.log
2023-03-10T19:16:40.346813 | INFO     | runServer          | Initializing server
2023-03-10T19:16:45.451190 | INFO     | runServer          | Ready. Waiting for connections

[kbarton1212]

hey, were you ever able to get this working?

[laxxie]

Unfortunately not =/

[rylyk]

@laxxie from the logs of your collector container it looks like a BMP session is never formed. You also mentioned you confirmed incoming packets on port 5000, how was this determined?

[laxxie]

i checked this via tcpdump (on the server, not inside the docker container).

root@bmp:~# tcpdump -n -i any port 5000 and host <JUNIPER>

09:52:13.351548 vethf2d3b8e P   IP 172.19.0.3.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556869063 ecr 2204352251,nop,wscale 7], length 0
09:52:13.351548 br-d101a5451f88 In  IP 172.19.0.3.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556869063 ecr 2204352251,nop,wscale 7], length 0
09:52:13.351602 ens3  Out IP <SERVER>.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556869063 ecr 2204352251,nop,wscale 7], length 0
09:52:17.348185 ens3  In  IP <JUNIPER>.59754 > <SERVER>.5000: Flags [S], seq 534534272, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 2204386452 ecr 0,sackOK,eol], length 0
09:52:17.348236 br-d101a5451f88 Out IP <JUNIPER>.59754 > 172.19.0.3.5000: Flags [S], seq 534534272, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 2204386452 ecr 0,sackOK,eol], length 0
09:52:17.348247 vethf2d3b8e Out IP <JUNIPER>.59754 > 172.19.0.3.5000: Flags [S], seq 534534272, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 2204386452 ecr 0,sackOK,eol], length 0
09:52:17.348281 vethf2d3b8e P   IP 172.19.0.3.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556873060 ecr 2204352251,nop,wscale 7], length 0
09:52:17.348281 br-d101a5451f88 In  IP 172.19.0.3.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556873060 ecr 2204352251,nop,wscale 7], length 0
09:52:17.348303 ens3  Out IP <SERVER>.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556873060 ecr 2204352251,nop,wscale 7], length 0
09:52:33.575581 vethf2d3b8e P   IP 172.19.0.3.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556889287 ecr 2204352251,nop,wscale 7], length 0
09:52:33.575581 br-d101a5451f88 In  IP 172.19.0.3.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556889287 ecr 2204352251,nop,wscale 7], length 0
09:52:33.575629 ens3  Out IP <SERVER>.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556889287 ecr 2204352251,nop,wscale 7], length 0
09:52:41.548234 ens3  In  IP <JUNIPER>.59754 > <SERVER>.5000: Flags [S], seq 534534272, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 2204410652 ecr 0,sackOK,eol], length 0
09:52:41.548296 br-d101a5451f88 Out IP <JUNIPER>.59754 > 172.19.0.3.5000: Flags [S], seq 534534272, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 2204410652 ecr 0,sackOK,eol], length 0
09:52:41.548303 vethf2d3b8e Out IP <JUNIPER>.59754 > 172.19.0.3.5000: Flags [S], seq 534534272, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 2204410652 ecr 0,sackOK,eol], length 0
09:52:41.548338 vethf2d3b8e P   IP 172.19.0.3.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556897260 ecr 2204352251,nop,wscale 7], length 0
09:52:41.548338 br-d101a5451f88 In  IP 172.19.0.3.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556897260 ecr 2204352251,nop,wscale 7], length 0
09:52:41.548360 ens3  Out IP <SERVER>.5000 > <JUNIPER>.59754: Flags [S.], seq 461924550, ack 534534273, win 65160, options [mss 1460,sackOK,TS val 1556897260 ecr 2204352251,nop,wscale 7], length 0

[rylyk]

That is odd, it looks like your packet capture shows your server with an established session but the collector logs don't see the session. Is it possible that you have two collector pods on port 5000 running? Or did you install the collector on your server prior to moving to docker?

If you stop the collector pod, do you still see bi-directional traffic in your packet captures?

[laxxie]

It's a new server with docker, there were no installation prior.

root@bmp:~# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      101        24727      612/systemd-resolve
tcp        0      0 0.0.0.0:9005            0.0.0.0:*               LISTEN      0          3676312    535823/docker-proxy
tcp        0      0 0.0.0.0:29092           0.0.0.0:*               LISTEN      0          3672388    535156/docker-proxy
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      0          3666631    534543/docker-proxy
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          25736      708/sshd: /usr/sbin
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      0          3673425    534716/docker-proxy
tcp        0      0 0.0.0.0:4300            0.0.0.0:*               LISTEN      0          3673318    534449/docker-proxy
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      0          3662647    534423/docker-proxy
tcp6       0      0 :::9005                 :::*                    LISTEN      0          3664731    535829/docker-proxy
tcp6       0      0 :::9100                 :::*                    LISTEN      1000       25752      661/node_exporter
tcp6       0      0 :::29092                :::*                    LISTEN      0          3663834    535163/docker-proxy
tcp6       0      0 :::3000                 :::*                    LISTEN      0          3673330    534551/docker-proxy
tcp6       0      0 :::22                   :::*                    LISTEN      0          25738      708/sshd: /usr/sbin
tcp6       0      0 :::5432                 :::*                    LISTEN      0          3672250    534724/docker-proxy
tcp6       0      0 :::4300                 :::*                    LISTEN      0          3673321    534460/docker-proxy
tcp6       0      0 :::5000                 :::*                    LISTEN      0          3671298    534430/docker-proxy
udp        0      0 127.0.0.53:53           0.0.0.0:*                           101        24726      612/systemd-resolve
udp        0      0 <SERVER>:68       0.0.0.0:*                           100        26425092   610/systemd-network
udp6       0      0 fe80::f816:3eff:fec:546 :::*                                100        30943      610/systemd-network

After stopping the collector there is no bi-di traffic anyymore, just a RESET back to the Juniper

13:30:47.569240 ens3  In  IP <JUNIPER>.62869 > <SERVER>.5000: Flags [S], seq 4174291627, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 2217496676 ecr 0,sackOK,eol], length 0
13:30:47.569340 ens3  Out IP <SERVER>.5000 > <JUNIPER>.62869: Flags [R.], seq 0, ack 4174291628, win 0, length 0 

but as i checked the Juniper i also saw that the state is connecting, so there seems to be sth wrong with the connection process

[rylyk]

Indeed, it seems like it's not a networking issue as a TCP session is established with incremental seq.

Not sure where the issue lies yet but can we double check the juniper config?

This is a sample we use for our Juniper BMP configs

routing-options {
    bmp {
        station BMPServer1 {
            initiation-message "Development/LAB";  #The initiation message is not required
            local-address x.x.x.x;
            connection-mode active;
            monitor enable;
            route-monitoring {
                pre-policy;
            }
            station-address x.x.x.x;
            station-port 5000;
            statistics-timeout 300;
        }
    }

[laxxie]

Sure, this is our config which we used already before with SNAS (old server which broke so we thought let's use OpenBMP)

station BMPtoBGPtools {
    initiation-message "core1 -> OpenBMP";
    local-address <JUNIPER>;
    connection-mode active;
    monitor enable;
    route-monitoring {
        post-policy;
    }
    station-address <SERVER>;
    station-port 5000;
    statistics-timeout 300;
}

[rylyk]

I get the same message and when I specify the path to the config using -c I get a blank result.

I was hoping to test the version you are using in my setup but unfortunately we are stuck running an older version for now. Right now the only other thing I can think of is the /usr/etc/openbmp/openbmpd.conf you are mounting. You mentioned you has SNAS setup before, has this config file been updated for your new openbmp deployment?

[laxxie]

Sry for my late reply, the project has not the top priority =(
The config hasn't changed. Perhaps i should try a former release? Or any other hints for a good open source BMP collector?