nahcrofDB is an open source, key-value database, designed to be simple, fast, and scalable, even in large datasets.
In order to install nahcrofDB, just run
git clone https://github.com/nahcrof-code/nahcrofDB.git
After installing, make sure that you have the required packages installed, the command for this is as follows.
pip install flask requests
Make sure to look through the config.txt file and update everything. "admin_password" is the password to acces the UI. "password_value" is the password used when accessing the database api.
in order to interact with the database, use client.py on the server that will be accessing the database server. If you are trying to directly interact with the database or create a new database you will use the cli on the server. To create a new database, you wil run...
python3 tools.py create_database DATABASE_NAME_HERE
for more commands use the "help" command, here is the output for the help command
Welcome to NahcrofDB
HINT: location is in reference to the folder the database is in.
reset <location> - resets database for specified location (ONLY USE IF COMPLETELY NECESSARY)
check - check the health of all databases and what keynames they have
structure <location> - view structure file of given location
file1 <location> - view first db file of given location
logs <location> - view log file of given location
fix_structure <location> - attempts to repair a corrupted structure file
view <location> - view database data
queue - view how many write requests are in the queue
folder <location> - view database folder for individual database
backup <location> - create a backup of an existing database
check_backup <location> - compare a database to it's corresponding backup
set_to_backup <location> - set the database to a pre-existing backup
create_database <folder_name> - creates empty database within specified folder
delete <location> - deletes database
st_size <location> - view the size of the structure file
partitions <location> - number of paritions
convert_structure <location> - converts old structure format
rebuild_all_structures - converts every structure file to new format
kill_db - safely shuts down the database program, flushing queue
In order to run nahcrofDB you will run the following command.
python3 main.py
Please note that as of right now, nahcrofDB will only work in linux. Running "main.py" will start both the HTTP handler and "ferris." In short, ferris handles queued write requests.
client.py is the python api wrapper for nahcrofDB. To start, you will use the client.init function.
import client
client.init(folder="my_db_name", url="https://url.com/", password="api_password_here")To make one key, you can do this.
client.makeKey("my_key", "my value")To make multiple keys at once (less database strain) you can use the following function.
client.makeKeys({
"key": "value",
"key2": "value",
"testkey": "testvalue"
})To get one key, you can use getKey
client.getKey("my_key")OUTPUT:
my value
NOTE: if you attempt to use getKey and the key does not exist, the response will be as follows
{'error': True, 'message': 'Key (MISSING KEY) does not exist.', 'status': 404}
To get multiple keys, you can use getKeys
keys = client.getKeys("key", "key2")OUTPUT:
{'key': 'value', 'key2': 'value'}
You can also use getKeysList which allows the user to get multiple keys using a list of keys instead of extra parameters.
keys = client.getKeysList(["key", "key2"])OUTPUT:
{'key': 'value', 'key2': 'value'}
getKeysIncrements functions like getKeysList with the ability to get a list of keys, but instead of sending the entire request at once, it requests chunks, like requesting 100 keys out of the entire list at a time. This helps deal with the max GET request size you'll usually run into.
Here are some examples
keys = client.getKeysIncrements(["key", "key2", ...])(this will get all the keys in increments of 100 and leave no logs)
keys = client.getKeysIncrements(["key", "key2", ...], log=True)OUTPUT:
found values for 1/200 keys!
found values for 2/200 keys!
...
found values for 200/200 keys!
Finally, you can set increment to a value in order to search for a number other than 100 at a time. It looks like this...
client.getKeysIncrements(["key", "key2", ...], increment=50)If this is too slow and you need to get a large number of values in one request, you can use postGetKeys like so:
client.postGetKeys(["key", "key2", ...])(Note: in order to send this request, the database has to load the entirety of this data in memory, as will the client. Make sure the systems you are using are capable of sending and receiving these requests)
To find keys containing data within keynames, you will use the .searchNames function
client.searchNames("key")OUTPUT:
['my_key', 'key', 'key2', testkey']
If you're looking for more precision in your search, change the "where" argument. Here's an example.
client.searchNames("key", where="start")OUTPUT:
['key', 'key2']
Valid values for "where" include the following
"start", "end", Nonethese allow you to search the beginning of the key (where="start"), meaning that the key will only be included in the returned data if it's name starts with the query, the end of key (where="end"), anywhere within the key (None). If you do not specify where, it will be assumed that you are searching for all keys that contain the data somewhere within the name.
In nahcrofDB, there is an incrementKey function. This allows you to increment a key, or a specific value within a key whilst only making one request to the database. Here is a simple example. (IMPORTANT: currently incrementKey does not support enterprise support)
client.makeKey("num", 5)
client.incrementKey(10, "num")
print(client.getKey("num"))OUTPUT:
15
In this basic example, the key "num" was created and then incremented by 10. Here is a more complex example.
client.makeKey("nums", {
"inner_values": [10, 23, 42, 8],
"type": "int",
})
client.incrementKey(2, "nums", "inner_values", 3)
print(client.getKey("nums"))OUTPUT
{
"inner_values": [10, 23, 42, 10],
"type": "int",
}
This change is because nahcrofDB was told to increment a value by 2, the value was in the path nums["inner_values"][3] which was the 4th value in "inner_values", or 8.
HEADERS:
X-API-Key: api-token-here
RESPONSE 200 OK
{
"error": false,
"message": null,
"status": 200,
"value": "value here"
}QUERY PARAMS
?key[]=key1&key[]=key2&key[]=key3
HEADERS:
X-API-Key: api-token-here
RESPONSE 200 OK
{
"key1": "val1"
"key2": "val2"
"key3": "val3"
...
}BODY (application/json)
{
"key": "val1"
"key2": "val2"
...
}RESPONSE 204 NO CONTENT
QUERY PARAMS
?query=Hello%20World
HEADERS:
X-API-Key: api-token-here
RESPONSE 200 OK
[
"key1",
"key2",
...
]QUERY PARAMS
?query=1?where=end
HEADERS:
X-API-Key: api-token-here
RESPONSE 200 OK
[
"key1",
"test1",
...
]BODY (application/json)
{
"amount": 1
}RESPONSE 204 NO CONTENT
example request url: /v2/increment/example_db/nums_list/num1
This is built using the client.py script and is not actually built-in on the database side. If requested, I'd be willing to give a more thorough breakdown on how redundancy can be well handled purely on the client.
The config file to setup your redundancy wherever you happen to be using nahcrofDB is titled nahcrofDB_client_config.py. The absence of this file is the cause for the start message "Failure setting up enterprise usage" followed by an import error. Despite the comments within the file, do not include more than one redundant database. The functionality is not currently there.
nahcrofDB_client_config.py
# you do not need to use this file if you are not using enterprise mode,
# this would be included with your client file if you are using enterprise mode.
# expand this list as much as you need, each database will be written to and utilized in the event of another database failure,
# the one used in the init function will be the default
databases = [
{"url": "http://0.0.0.0:8080", "password": "super_duper_password"},
]as for using it within your code, everything stays the same except for the .init function which will look something like this now:
client.init(folder="my_db_name", url="https://url.com/", password="api_password_here", enterprise=True)as long as enterprise is either not explicitly set or is intentionally set to False, redundant databases will be ignored and requests will ONLY go to the database configured in the init function.
For this I'm just going to link to an external repo, there's already tons of files in this one and it kinda bothers me nahcrofDB-backup
