New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
diskq: fix invalid read after disk_buf_size #3726
Conversation
8e79f3a
to
65decf5
Compare
Build FAILURE |
65decf5
to
214379d
Compare
214379d
to
4ea96d7
Compare
Build FAILURE |
4ea96d7
to
428d680
Compare
Build FAILURE |
@kira-syslogng do stresstest |
Can you explain the exact difference between |
file_size is the physical disk used by the disk-queue file. useful_file_size is the physical disk usage of the mmapped region and the log messages which are waiting to be processed (read or acked). Sorry if I am cryptic. Edit: This is not completely true, because we can have already handled messages in the start of the disk buffer... The useful_file_size variable is the theoretically minimal file size, if we would truncate every time. So it does not take into account the already handled messages in the end of the file, which could have been removed by truncate, but wasn't because of truncate-size-ratio. |
Kira-stress-test: Build SUCCESS |
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
It seems that I think the best would be if we could fix this issue without adding more "state". |
@kira-syslogng do stresstest |
Kira-stress-test: Build SUCCESS |
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
It turned out that the additional "state" is required, and it also has to be persisted (has to be added to the disk buffer header). This is because the truncation logic was not only used to reduce file size, but for 2 special operations as well:
Alternatively, we could truncate in the above 2 rare(?) cases unconditionally as we did before. That would save us from complicating things further. |
Kira-stress-test: Build SUCCESS |
Build SUCCESS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've just glimpsed into the PR to see what's changing.
This is a strange unit test, it was created to help to investigate a bug and validate its fix. I added this first, so one can checkout this commit, and see, that it fails with the current code. The details and the root cause of the bug will be in the next commit's message. Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
With the truncate-size-ratio() option, now we cannot base our read_head wrapping logic on self->file_size (see next commit). We can base it on disk_buf_size, if we guarantee, that the write_head does not overwrite disk_buf_size with more than 1 message. Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
We do not check beforehand, whether we can fit a new message to the disk-buffer, we decide to wrap the write_head after we wrote over the disk-buf-size limit. It is possible to be just before the disk-buf-size with the write_head and write a huge message, then wrap the write_head to the beginning. This will change self->file_size to be a large value. Let's say we set truncate-size-ratio(1), so we never truncate and never change the file_size from that large value. Imagine, that we handled every message, then wrote a lot of messages to the disk-buffer again, so we are just before the disk-buf-size limit. We write 2 small messages, one will end just after disk-buf-size but still before the huge self->file_size, the next one will start at the beginning, and end a bit after that. This is because we decide to wrap the write_head based on the position related to disk-buf-size (see `_is_qdisk_overwritten()`). When we are reading from the disk-buffer we currently decide to wrap the read_head based on whether it is after the self->file_size. We read the small message that ends just after disk-buf-size, and we decide not to wrap, and try to read one more message there, which is the middle of an old message. This commit matches the wrap logic between the write and read head, so they both measure their position relative to disk-buf-size. Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
This way we can change the disk-buffer version immediately at load, so it is easier to add a new disk-buffer version later. Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Follow-up for syslog-ng#3689. Signed-off-by: László Várady <laszlo.varady@protonmail.com>
As the truncation logic is now conditional, debug messages has to be moved inside _maybe_truncate_file(). Signed-off-by: László Várady <laszlo.varady@protonmail.com>
Signed-off-by: László Várady <laszlo.varady@protonmail.com>
Signed-off-by: László Várady <laszlo.varady@protonmail.com>
Signed-off-by: László Várady <laszlo.varady@protonmail.com>
The in-memory buffers of the non-reliable diskq is persisted when syslog-ng is restarted, which increases the size of the actual disk-buffer file. This test case tests the correctness of the non-reliable diskq when truncation is completely disabled. Signed-off-by: László Várady <laszlo.varady@protonmail.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
f3f985b
to
4a19167
Compare
@kira-syslogng do stresstest |
Build SUCCESS |
Kira-stress-test: Build SUCCESS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank You!
We do not check beforehand, whether we can fit a new message to the disk-buffer, we decide to wrap the
write_head
after we wrote over thedisk-buf-size
limit.It is possible to be just before the
disk-buf-size
with thewrite_head
and write a huge message, then wrap thewrite_head
to the beginning. This will changeself->file_size
to be a large value.Let's say we set
truncate-size-ratio(1)
, so we never truncate and never change thefile_size
from that large value.Imagine, that we handled every message, then wrote a lot of messages to the disk-buffer again, so we are just before the
disk-buf-size
limit.We write 2 small messages, one will end just after
disk-buf-size
but still before the hugeself->file_size
, the next one will start at the beginning, and end a bit after that.This is because we decide to wrap the
write_head
based on the position related to disk-buf-size (see_is_qdisk_overwritten()
).When we are reading from the disk-buffer we currently decide to wrap the
read_head
based on whether it is after theself->file_size
.We read the small message that ends just after
disk-buf-size
, and we decide not to wrap, and try to read one more message there, which is the middle of an old message.This PR matches the wrap logic between the write and read head, so they both measure their position relative to
disk-buf-size
.Signed-off-by: Attila Szakacs attila.szakacs@oneidentity.com