New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
diskq: disk-buf-size()
related fixes: dqtool cat
, example-diskq-source()
, changing disk-buf-size()
#4308
Conversation
No news file has been detected. Please write one, if applicable. |
@HofiOne If I remember correctly, there is a disk-buffer file upgrade E2E test in testdb, which stores what syslog-ng versions bumped the disk-buffer header version. I think we could/should run that test on this PR and extend the list when a new PE release comes out. Could you kindly help with this? |
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
54097de
to
913a9f8
Compare
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
913a9f8
to
27a0b5a
Compare
disk-buf-size()
related fixes: dqtool cat
, changing disk-buf-size()
disk-buf-size()
related fixes: dqtool cat
, example-diskq-source()
, changing disk-buf-size()
Build FAILURE |
Kira fails with 2 disk-buffer related tests. They seem to be relevant to this change. I am checking those. |
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
27a0b5a
to
07d7455
Compare
So good news, the disk-buffer E2E test case is running in Kira, and it caught an error, which I managed to fix, now it is green. Still, when we have time, we should add some new variants to it as a new disk-buffer header version was introduced. |
I like the patches, those look good. I did some testing, "dqtool info' displays:
The queue file was created with the current syslog-ng version, so it should have the size in the header. Cat seems to work fine. |
I am planning to move around the initialization of disk_buf_size. This change makes it easier, as preallocate() no longer depends on get_maximum_size(). Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
This will be useful for a dqtool fix (see the next couple of commits) and later on for increasing and decreasing the disk-queue file size. Note that with this commit we only store the initially set disk-buf-size and take care of old disk-queue files, which we created when we did not store it yet. Currently we do not base any logic on it. That will come in the next couple of commits. Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
…g it There is a bug in dqtool which was caused by 2 things: 1. We changed the condition of the cursor wrapping from file size to disk-buf-size(). 2. We do not store the disk-buf-size in the disk-queue header. There is no way to tell where we should wrap without knowing the disk-buf-size before loading the disk-queue. The previous commit prepared a fix by storing the initial disk-buf-size in the header. Now we use that value for the wrap condition instead the user given one. We can now detect whether the user tries to change the disk-buf-size of an existing disk-queue file. For now we will reject that change and go with the original disk-buf-size, which is a bugfix itself, as before this commit we did not realize that with a config change and might have tried to overread/overwrite the disk-queue file. We can implement the disk-buf-size change in the future in a safe way, however it is not the scope of this patch set. Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
07d7455
to
98929aa
Compare
@bazsi Nice catch! Fixed. |
Sometimes (e.g. in dqtool) we do not know what disk-buf-size was set for the disk-queue file and we do not care about it. As we store that value in the disk-queue file itself, we can use the value set there without any problems. Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
In theory this should only be done if disk-buf-size() is not set when loading the disk-queue file == dqtool cat or diskq-source(). If the cursors are not wrapped, we can work with any disk-buf-size(), which is larger, than the write head. Additionally, we can assume, that the disk-buf-size() was set to at least the size of the file, which is only not true, if the last message written was started just before the original disk-buf-size() and ended after it. This means that in a non wrapped case, if we use the current file size as the disk-buf-size(), we either use a smaller than originally set, or in lucky cases exactly equal or similarly (un)lucky cases nearly one message larger. I think this is an acceptable compromise, but either way, we are not going to write more messages to the disk-queue file, so it is not a big deal. Unfortunately, we could not deal with wrapped disk-queue files even before this patch set, so I did not fix it. We could write some heuristics, so that the first short read indicates that we should wrap instead, but I do not really want to complicate the wrap logic, if it is not absolutely necessary. Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
…buf-size Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
98929aa
to
d401346
Compare
Sorry for the accidental Close :S @alltilla do we still need the nightly random platform test for this? |
@Hofi No problem :) If I remember correctly every disk-buffer testdb tests are run in KIRA, too, so there is no need to run PE testdb tests. Stress ( == perf) testing would be nice, but I can see that it cannot run currently :( |
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
There is a bug in
dqtool
which was caused by 2 things:disk-buf-size()
.disk-buf-size()
in the disk-queue header.There is no way to tell where we should wrap without knowing the
disk-buf-size()
before loading the disk-queue.With this patch set we now store the initial
disk-buf-size()
in the header and use that for the wrap condition.Also we can now detect whether the user tries to change the
disk-buf-size()
of an existing disk-queue file. For now we will reject that change and go with the originaldisk-buf-size()
, which is a bugfix itself, as before this patch set we did not realize that with a config change and might have tried to overread/overwrite the disk-queue file.We can implement the
disk-buf-size()
change in the future in a safe way, however it is not the scope of this patch set.Signed-off-by: Attila Szakacs szakacs.attila96@gmail.com