RANGER-5520:Audit Server refactoring to segregate audit ingestion and…

[rameeshm]

… dispatching functionality

What changes were proposed in this pull request?

audit-ingestor

• This would have the functionality of audit log inbounding from various plugin and committing into Kafka
• This will be refactored audit-server microservice application in the current audit-server.

audit-dispatcher.

• This should be micro-services for subscribing the audits from kafka ranger audit topic and send it into Solr / hdfs / opensearch and other destination.
• This should be configurable to take the form of Solr Audit dispatcher, HDFS audit dispatcher etc based on the destination type.
• Each dispatcher for destination type should be a separate service dedicated to commit the audits to respective destination

How was this patch tested?

Tested in Docker setup

//  Ranger Docker

docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-usersync.yml -f docker-compose.ranger-tagsync.yml up -d

// Hadoop + Hive + Kafka

docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-hadoop.yml -f docker-compose.ranger-hive.yml -f docker-compose.ranger-kafka.yml up -d

// start ranger audit server - Single one

docker compose  -f docker-compose.ranger.yml -f docker-compose.ranger-hadoop.yml -f docker-compose.ranger-kafka.yml -f docker-compose.ranger-audit-server.yml  up -d

Commands run:

# HDFS load
docker exec -it ranger-hadoop bash -c '
export KRB5CCNAME=/tmp/krb5cc_hdfs_custom
kinit -kt /etc/keytabs/testuser1.keytab testuser1/[ranger-hadoop.rangernw@EXAMPLE.COM](mailto:ranger-hadoop.rangernw@EXAMPLE.COM)
for i in {1..80}; do
  echo "Iteration $i"
  hdfs dfs -ls /user/
  sleep 1
done'

[Copilot]

Pull request overview

This PR refactors the Ranger audit-server into two separable deployables: an audit-ingestor (plugin → REST → Kafka producer) and an audit-dispatcher (Kafka consumer(s) → Solr/HDFS), and updates distro assembly + Docker tooling accordingly.

Changes:

• Replaces the legacy audit-server module packaging with a new ranger-audit-ingestor WAR and introduces a new ranger-audit-dispatcher distribution with per-destination consumer WARs.
• Updates distro assembly descriptors and distro/pom.xml wiring to build/ship the new tarballs and WAR names/paths.
• Updates docker-compose and Dockerfiles/scripts to run the ingestor and unified dispatcher containers.

Reviewed changes

Copilot reviewed 45 out of 88 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
distro/src/main/assembly/audit-server.xml	Points audit-server distro assembly to the refactored audit-ingestor paths and WAR output.
distro/src/main/assembly/audit-dispatcher.xml	New assembly descriptor to package dispatcher scripts/configs and Solr/HDFS consumer WARs.
distro/src/main/assembly/audit-consumer-solr.xml	Updates legacy Solr consumer assembly paths to new audit-dispatcher module layout.
distro/src/main/assembly/audit-consumer-hdfs.xml	Updates legacy HDFS consumer assembly paths to new audit-dispatcher module layout.
distro/pom.xml	Updates distro dependencies and assembly descriptors to produce audit-server + audit-dispatcher artifacts.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-server.sh	Adapts startup script for audit-ingestor naming + backward-compatible env vars.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-consumer-solr.sh	Updates Solr consumer script to new extracted webapp dir naming and webapp-dir system prop.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-consumer-solr-site.xml	Refactors docker Solr consumer config to support unified startup script discovery keys.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-consumer-hdfs.sh	Updates HDFS consumer script to new extracted webapp dir naming and webapp-dir system prop.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-consumer-hdfs-site.xml	Refactors docker HDFS consumer config to support unified startup script discovery keys.
dev-support/ranger-docker/docker-compose.ranger-audit-server.yml	Switches to audit-ingestor + audit-dispatcher containers; updates ports, commands, volumes, healthchecks.
dev-support/ranger-docker/Dockerfile.ranger-audit-ingestor	Builds a container for the audit-ingestor distribution and updated log/spool paths.
dev-support/ranger-docker/Dockerfile.ranger-audit-dispatcher	New unified dispatcher container that runs destination-specific consumer via start-audit-consumer.sh.
dev-support/ranger-docker/Dockerfile.ranger-audit-consumer-solr	Adjusts legacy Solr consumer Dockerfile comments for refactored origin.
dev-support/ranger-docker/Dockerfile.ranger-audit-consumer-hdfs	Adjusts legacy HDFS consumer Dockerfile comments for refactored origin.
dev-support/ranger-docker/.dockerignore	Updates included distro tarball set to include audit-dispatcher tarball.
audit-server/scripts/stop-all-services.sh	Minor output string change.
audit-server/scripts/start-all-services.sh	Minor output string change.
audit-server/pom.xml	Replaces child modules with audit-common, audit-dispatcher, audit-ingestor and adjusts shared deps.
audit-server/consumer-solr/scripts/stop-consumer-solr.sh	Removes legacy standalone Solr consumer stop script (superseded by dispatcher).
audit-server/consumer-solr/scripts/start-consumer-solr.sh	Removes legacy standalone Solr consumer start script (superseded by dispatcher).
audit-server/consumer-hdfs/scripts/stop-consumer-hdfs.sh	Removes legacy standalone HDFS consumer stop script (superseded by dispatcher).
audit-server/consumer-hdfs/scripts/start-consumer-hdfs.sh	Removes legacy standalone HDFS consumer start script (superseded by dispatcher).
audit-server/audit-ingestor/src/main/webapp/WEB-INF/web.xml	Adds ingestor web.xml for REST + Spring Security filter mapping.
audit-server/audit-ingestor/src/main/webapp/WEB-INF/security-applicationContext.xml	Adds Spring Security config for ingestor REST endpoints (JWT + delegation token filters).
audit-server/audit-ingestor/src/main/webapp/WEB-INF/applicationContext.xml	Adds Spring context wiring for ingestor component scan + scopes.
audit-server/audit-ingestor/src/main/resources/conf/ranger-audit-server-site.xml	Updates ingestor webapp dir and renames kerberos properties under ranger.audit.server.*.
audit-server/audit-ingestor/src/main/resources/conf/logback.xml	Adds ingestor logback configuration.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/server/AuditServerConfig.java	Adds ingestor-specific config loader for ranger-audit-server-site.xml.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/server/AuditServerApplication.java	Updates app name to audit-ingestor.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/NullServletContext.java	Introduces a ServletContext stub used by ingestor security filters.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/FilterChainWrapper.java	Adds filter-chain wrapper to populate Spring Security context from auth cookie/remoteUser.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/AuditJwtAuthFilter.java	Adds JWT auth filter integration using Ranger JWT handler configuration.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/AuditDelegationTokenFilter.java	Updates delegation token filter config prefix to ranger.audit.server.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/AuditAuthEntryPoint.java	Adds entrypoint that returns 401 instead of redirecting.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/rest/RangerJsonProvider.java	Adds Jackson provider wiring using Ranger’s shared ObjectMapper.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/AuditRecoveryManager.java	Adds recovery manager to coordinate writer/retry threads for Kafka outage spooling.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/AuditProducer.java	Adds producer wrapper supporting idempotent config + batch send + selective retry.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/AuditPartitioner.java	Adds plugin-aware partitioner for distributing audit events across topic partitions.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/AuditDestinationMgr.java	Adds ingestor component to initialize Kafka destination and log audit batches.
audit-server/audit-ingestor/src/main/java/javax/ws/rs/core/NoContentException.java	Adds a JAX-RS NoContentException shim to avoid jersey/jackson provider classloading failure.
audit-server/audit-ingestor/scripts/stop-audit-server.sh	Minor output string change.
audit-server/audit-ingestor/scripts/start-audit-server.sh	Minor output string change.
audit-server/audit-ingestor/pom.xml	Renames module/artifact to ranger-audit-ingestor, updates deps/plugins and final WAR name.
audit-server/audit-dispatcher/scripts/start-audit-consumer.sh	Adds unified consumer startup script selecting WAR + main class from type-specific config.
audit-server/audit-dispatcher/pom.xml	Adds dispatcher parent POM aggregating consumer-common, consumer-hdfs, consumer-solr.
audit-server/audit-dispatcher/consumer-solr/src/main/webapp/WEB-INF/web.xml	Adds Solr consumer web.xml for health endpoint via Jersey/Spring.
audit-server/audit-dispatcher/consumer-solr/src/main/webapp/WEB-INF/applicationContext.xml	Switches Solr consumer component scan to org.apache.ranger.audit.
audit-server/audit-dispatcher/consumer-solr/src/main/resources/conf/ranger-audit-consumer-solr-site.xml	Adds Solr consumer config including dispatcher startup metadata + Kafka/Solr destination settings.
audit-server/audit-dispatcher/consumer-solr/src/main/resources/conf/logback.xml	Adds Solr consumer logback configuration.
audit-server/audit-dispatcher/consumer-solr/src/main/java/org/apache/ranger/audit/server/SolrConsumerConfig.java	Loads common + Solr-specific config resources.
audit-server/audit-dispatcher/consumer-solr/src/main/java/org/apache/ranger/audit/rest/HealthCheckREST.java	Adds Solr consumer health endpoint implementation.
audit-server/audit-dispatcher/consumer-solr/src/main/java/org/apache/ranger/audit/consumer/SolrConsumerManager.java	Adds Spring-managed lifecycle to create/start/stop Solr consumer threads.
audit-server/audit-dispatcher/consumer-solr/src/main/java/org/apache/ranger/audit/consumer/SolrConsumerApplication.java	Updates app name/config prefix usage for dispatcher-style execution.
audit-server/audit-dispatcher/consumer-solr/pom.xml	Updates parent pathing, Jersey deps/exclusions, internal deps, adds PMD and final WAR name.
audit-server/audit-dispatcher/consumer-hdfs/src/main/webapp/WEB-INF/web.xml	Adds HDFS consumer web.xml for health endpoint via Jersey/Spring.
audit-server/audit-dispatcher/consumer-hdfs/src/main/webapp/WEB-INF/applicationContext.xml	Switches HDFS consumer component scan to org.apache.ranger.audit.
audit-server/audit-dispatcher/consumer-hdfs/src/main/resources/conf/ranger-audit-consumer-hdfs-site.xml	Adds HDFS consumer config including dispatcher startup metadata + Kafka/HDFS destination settings.
audit-server/audit-dispatcher/consumer-hdfs/src/main/resources/conf/logback.xml	Adds HDFS consumer logback configuration.
audit-server/audit-dispatcher/consumer-hdfs/src/main/resources/conf/hdfs-site.xml	Adds HDFS client kerberos-related config for HDFS consumer deployments.
audit-server/audit-dispatcher/consumer-hdfs/src/main/resources/conf/core-site.xml	Adds core Hadoop security/auth_to_local config for HDFS consumer deployments.
audit-server/audit-dispatcher/consumer-hdfs/src/main/java/org/apache/ranger/audit/server/HdfsConsumerConfig.java	Loads common + HDFS-specific + core-site/hdfs-site configuration resources.
audit-server/audit-dispatcher/consumer-hdfs/src/main/java/org/apache/ranger/audit/rest/HealthCheckREST.java	Adds HDFS consumer health endpoint implementation.
audit-server/audit-dispatcher/consumer-hdfs/src/main/java/org/apache/ranger/audit/consumer/HdfsConsumerManager.java	Adds Spring-managed lifecycle to create/start/stop HDFS consumer threads.
audit-server/audit-dispatcher/consumer-hdfs/src/main/java/org/apache/ranger/audit/consumer/HdfsConsumerApplication.java	Updates app name/config prefix usage for dispatcher-style execution.
audit-server/audit-dispatcher/consumer-hdfs/pom.xml	Updates parent pathing, Jersey deps/exclusions, internal deps, adds PMD and final WAR name.
audit-server/audit-dispatcher/consumer-common/src/main/resources/conf/ranger-audit-consumer-site.xml	Adds shared consumer service config (host/port/context + shared kafka/kerberos placeholders).
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumerRegistry.java	Adds registry to manage destination factories and create consumers based on enabled destinations.
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumerRebalanceListener.java	Adds reusable rebalance listener to commit offsets and log assignment/revocation details.
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumerFactory.java	Adds functional interface for creating consumer instances.
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumerBase.java	Adds base class for Kafka consumer configuration and shared client setup.
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumer.java	Adds common consumer interface contract.
audit-server/audit-dispatcher/consumer-common/pom.xml	Renames and defines the shared consumer-common artifact and dependencies/finalName.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/utils/AuditServerUtils.java	Introduces shared audit-server utility logic for destination config manipulation and topic readiness checks.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/utils/AuditServerLogFormatter.java	Adds structured log helper/builder for consistent startup and status logging.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/server/AuditServerConstants.java	Adds shared constants for server/consumer/producer configuration keys and defaults.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/server/AuditConfig.java	Adds shared configuration base class extending RangerConfiguration with resource loading helpers.
audit-server/audit-common/pom.xml	Adds ranger-audit-server-common shared jar module.

Comments suppressed due to low confidence (3)

dev-support/ranger-docker/scripts/audit-server/ranger-audit-server.sh:68

• The ingestor startup script still sets -Daudit.server.log.file=ranger-audit-server.log. Since this container now runs the audit-ingestor, consider updating the default log filename to something like ranger-audit-ingestor.log to avoid confusing log locations when both ingestor and legacy server naming exist.
  audit-server/audit-dispatcher/consumer-solr/pom.xml:260
• ranger-audit-consumer-common is declared a second time here (it already appears a few lines above with exclusions). This is redundant and can make dependency exclusions unclear; keep a single dependency entry and apply exclusions there if needed.
  audit-server/audit-dispatcher/consumer-hdfs/pom.xml:273
• ranger-audit-consumer-common is declared a second time here (it already appears a few lines above with exclusions). This is redundant and can make dependency exclusions unclear; keep a single dependency entry and apply exclusions there if needed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 88 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 97 out of 116 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mneethiraj]

Consider removing the dependency on setting system property (line 46).

[mneethiraj]

For other values of dispatcherType, how about reading the implementation class name from configuration and initialize - similar to AuditProviderFactory.getProviderFromConfig()?

String className = config.get(CONFIG_PREFIX + dispatcherType + ".classname");

if (StringUtils.isNotBlank(className)) {
    initSuccess = initializeDispatcherManager(className);
} else {
    LOG.error("Unknown dispatcher type: {}. Cannot initialize dispatcher manager.", dispatcherType);
}

[mneethiraj]

It looks like ranger-plugins-common is dragged in only for RangerConfiguration. Getting rid of this dependency would make audit audit-common a lot lighter. Consider AuditConfig not extend RangerConfiguration; instead just make this class self-sufficient.

[mneethiraj]

Following methods don't seem to be used anymore. Please review and remove:

• isSolrConsumerEnabled()
• isHDFSConsumerEnabled()
• isTopicReady()

[mneethiraj]

There will be only once instance of AuditProviderFactory in a JVM - due to use of static AuditProviderFactory sFactory in AuditProviderFactory class. Hence the following in AuditServerUtils, which assumes multiple AuditProviderFactory instances (one per appId per host) is incorrect. Please review and update.

    private final ConcurrentHashMap<String, Map<String, AuditProviderFactory>> auditProviderMap = new ConcurrentHashMap<>();