Issue with larger ini file and duplicate keys?

[andyb2000]

Hi,
Working on a fairly simple ini file but size or the keys may be the issue?

The ini file contents:
[main] scheduler = 0 auto = 0 temperature_set = 30 override = 0 [0] 0 = 1 1 = 1 2 = 1 3 = 1 4 = 1 5 = 1 6 = 1 7 = 1 8 = 1 9 = 1 10 = 1 11 = 1 12 = 1 13 = 1 14 = 1 15 = 1 16 = 1 17 = 1 18 = 1 19 = 1 20 = 1 21 = 1 22 = 1 23 = 1 [1] 0 = 1 1 = 1 2 = 1 3 = 1 4 = 1 5 = 1 6 = 1 7 = 1 8 = 1 9 = 1 10 = 1 11 = 1 12 = 1 13 = 1 14 = 1 15 = 1 16 = 1 17 = 1 18 = 1 19 = 1 20 = 1 21 = 1 22 = 1 23 = 1 [2] 0 = 1 1 = 1 2 = 1 3 = 1 4 = 1 5 = 1 6 = 1 7 = 1 8 = 1 9 = 1 10 = 1 11 = 1 12 = 1 13 = 1 14 = 1 15 = 1 16 = 1 17 = 1 18 = 1 19 = 1 20 = 1 21 = 1 22 = 1 23 = 1 [3] 0 = 1 1 = 1 2 = 1 3 = 1 4 = 1 5 = 1 6 = 1 7 = 1 8 = 1 9 = 1 10 = 1 11 = 1 12 = 1 13 = 1 14 = 1 15 = 1 16 = 1 17 = 1 18 = 1 19 = 1 20 = 1 21 = 1 22 = 1 23 = 1 [4] 0 = 1 1 = 1 2 = 1 3 = 1 4 = 1 5 = 1 6 = 1 7 = 1 8 = 1 9 = 1 10 = 1 11 = 1 12 = 1 13 = 1 14 = 1 15 = 1 16 = 1 17 = 1 18 = 1 19 = 1 20 = 1 21 = 1 22 = 1 23 = 1 [5] 0 = 1 1 = 1 2 = 1 3 = 1 4 = 1 5 = 1 6 = 1 7 = 1 8 = 1 9 = 1 10 = 1 11 = 1 12 = 1 13 = 1 14 = 1 15 = 1 16 = 1 17 = 1 18 = 1 19 = 1 20 = 1 21 = 1 22 = 1 23 = 1 [6] 0 = 1 1 = 1 2 = 1 3 = 1 4 = 1 5 = 1 6 = 1 7 = 1 8 = 1 9 = 1 10 = 1 11 = 1 12 = 1 13 = 1 14 = 1 15 = 1 16 = 1 17 = 1 18 = 1 19 = 1 20 = 1 21 = 1 22 = 1 23 = 1

And the code I'm using to retrieve it:
(DEBUG_PRINT is an alias to Serial.println)

`
const size_t ini_bufferLen = 1000;
char ini_buffer[ini_bufferLen];
const char *ini_filename = "/test.ini";
IniFile ini(ini_filename);

if (!ini.open()) {
DEBUG_PRINT("PANIC - we are in read_inifiledata() and expected the file to exist BUT IT DOES NOT");
} else {
// read ini file and check for valid
if (!ini.validate(ini_buffer, ini_bufferLen)) {
DEBUG_PRINT("PANIC - the ini file did not validate, so we are rejecting it");
DEBUG_PRINT(String(ini.getError()));
} else {
// valid ini file so grab values
int tmp_day=0;
int tmp_hour=0;
while(tmp_day < 7) {
tmp_hour=0;
while (tmp_hour< 24) {
char daybuf[4],hourbuf[4];
itoa(tmp_day, daybuf, 6);
itoa(tmp_hour, hourbuf, 6);
if (ini.getValue(daybuf, hourbuf, ini_buffer, ini_bufferLen)) {
DEBUG_PRINT("Loaded day " + String(tmp_day) + " hour " + String(tmp_hour) + " Value: " + String(ini_buffer));
} else {
DEBUG_PRINT("Ini section NOT found - day " + String(tmp_day) + " and hour " + String(tmp_hour));
};
DEBUG_PRINT("ini error: " + String(ini.getError()));
tmp_hour++;
};
tmp_day++;
};
};
`

It works for the first few iterations, but by the time it gets to day 0, hour 16 it fails and will fail the rest of that day:
16:57:07.867 -> Ini section NOT found - day 0 and hour 16
16:57:07.867 -> ini error: 6
16:57:07.867 -> Ini section NOT found - day 0 and hour 17
16:57:07.867 -> ini error: 6
(and so on)
It then moves to the next day and works:
6:57:08.103 -> Loaded day 1 hour 0 Value: 1
16:57:08.103 -> ini error: 0
16:57:08.103 -> Loaded day 1 hour 1 Value: 1
16:57:08.103 -> ini error: 0
16:57:08.103 -> Loaded day 1 hour 2 Value: 1
(etc, until it gets to hour 16 and fails again):
16:57:08.337 -> Loaded day 1 hour 15 Value: 1
16:57:08.337 -> ini error: 0
16:57:08.337 -> Ini section NOT found - day 1 and hour 16
16:57:08.337 -> ini error: 6
16:57:08.337 -> Ini section NOT found - day 1 and hour 17
16:57:08.337 -> ini error: 6

It continues like that until it gets to day 15, again 16th value fails, it then fails the next day section, and fails all future sections.

16:57:11.624 -> Loaded day 5 hour 15 Value: 1
16:57:11.624 -> ini error: 0
16:57:11.857 -> Ini section NOT found - day 5 and hour 16
16:57:11.857 -> ini error: 6
16:57:11.857 -> Ini section NOT found - day 5 and hour 17
16:57:11.857 -> ini error: 6
16:57:11.857 -> Ini section NOT found - day 5 and hour 18
16:57:11.857 -> ini error: 6
16:57:11.857 -> Ini section NOT found - day 5 and hour 19
16:57:11.857 -> ini error: 6
16:57:11.857 -> Ini section NOT found - day 5 and hour 20
16:57:11.857 -> ini error: 6
16:57:11.857 -> Ini section NOT found - day 5 and hour 21
16:57:11.857 -> ini error: 6
16:57:12.093 -> Ini section NOT found - day 5 and hour 22
16:57:12.093 -> ini error: 6
16:57:12.093 -> Ini section NOT found - day 5 and hour 23
16:57:12.093 -> ini error: 6
16:57:12.093 -> Ini section NOT found - day 6 and hour 0
16:57:12.093 -> ini error: 5
16:57:12.093 -> Ini section NOT found - day 6 and hour 1
16:57:12.093 -> ini error: 5
16:57:12.093 -> Ini section NOT found - day 6 and hour 2
16:57:12.093 -> ini error: 5
16:57:12.093 -> Ini section NOT found - day 6 and hour 3
16:57:12.093 -> ini error: 5

I've checked the ini file is valid, i'm not getting any memory issues or any other error messages, it just appears the iniFile library stops at each of these. It feels like a buffer issue, but I've given it huge buffers to see if that helps and no luck.

[stevemarple]

Does the file contain newlines? Your formatting suggests it does not.

[andyb2000]

Sorry, yes formatting has failed but it does have newlines!

On April 23, 2022 5:31:36 PM GMT+01:00, Steve Marple ***@***.***> wrote: Does the file contain newlines? Your formatting suggests it does not. -- Reply to this email directly or view it on GitHub: #32 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[stevemarple]

Without seeing the input file as it was originally formatted it is hard to follow. IniFile is line-oriented, the buffer only needs to be long enough to handle the longest line in the file. The ATmega328P has only 2kB RAM so dedicating 1000 bytes to the buffer is overkill. What processor are you using?

[andyb2000]

Sorry, will try pasting again and hopefully formatting will work!
EDIT - nope still won't paste, so I've attached
Rename test.txt to test.ini

test.txt

[stevemarple]

I don't have an Arduino with microSD card convenient to use. However, I have replicated your code using the test environment in the extras directory so I can test on my desktop machine. I think it works as you would expect.

You say you don't think it is memory issues but the ATmega328P has a tiny 2kB of RAM. IniFile was carefully written to avoid needing dynamic memory (no malloc()/realloc() or operator new), with the user providing a suitably-sized buffer. Now I don't know what processor you are using but 1000 bytes is much larger than needed, and represents about half of the available RAM on a standard Arduino. For printing you are creating String objects, which create objects using realloc(). This drags in all of the malloc code, which uses a sizeable amount of RAM. You'll see in my test I used the iostream library from C++; on the Arduino you can get equivalent behaviour with the Streaming library (highly recommended, and it does not use malloc() or operator new).

Q1 What processor is it you are using?
Q2 How much memory does the Arduino IDE say your program is using? After compiling your sketch it should print something similar to

Sketch uses 4764 bytes (3%) of program storage space. Maximum is 126976 bytes.
Global variables use 345 bytes of dynamic memory.

(Values taken from compiling a different sketch).

usertest.zip