Assemblyline should be able to be configured to support prefixing Elasticsearch Indexs #391
Description
Is your feature request related to a problem? Please describe.
The problem we have is that we would like to point Assemblyline at a shared Elasticsearch cluster where we pay for Elasticsearch Enterprise licensing to enable Enterprise features and support. This cluster has many nodes and is capable of a lot of underlying storage and compute.
I've outlined a separate issue surrounding Assemblyline's ES permissions here (#390), but
once a user can be provisioned to not require superuser permissions, it would be ideal to limit the scope of the assemblyline user such that it can have full access, but limited to specific indexes that could be wildcarded to the configurable prefix (e.g. assemblyline_* or al_*)
Describe the solution you'd like
Ideally would like to see a way to configure Assemblyline to provide something like a ELASTICSEARCH_INDEX_PREFIX variable which could be set to some string (e.g. assemblyline_* or al_*) and then all Elasticsearch indexes that Assemblyline would create would then contain this prefix.
Maybe something similar can be done for the Index Lifecycle Policies and other things being created in ES.
Additional context
Trying to track down how difficult it would be to implement but I'm not sure I'm following everything, so I'll try to summarize what I see so far:
In the following code, the AssemblylineDatastore appears to register various models.
When looking at a specific model, such as the following User model as an example
- https://github.com/CybercentreCanada/assemblyline-base/blob/5fef232212821f9cbd6605734c102e4ce4dd76bb/assemblyline/odm/models/user.py#L243
It appears that the decorator then defines aindex=Trueandstore=True, which appears to be how Assemblyline will know how to store the data into Elasticsearch as its own index or not
I believe this may all get wrapped up behind how things call the ESCollection class:
- https://github.com/CybercentreCanada/assemblyline-base/blob/5fef232212821f9cbd6605734c102e4ce4dd76bb/assemblyline/datastore/collection.py#L203
so maybe would be something easy as prepending a prefix to the index names here hopefully.