Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

package-indexer: add .gz, .xz and .bz2 compression for Packages #4313

Merged
merged 1 commit into from Feb 6, 2023
Merged

package-indexer: add .gz, .xz and .bz2 compression for Packages #4313

merged 1 commit into from Feb 6, 2023
+18 −0

Conversation

alltilla
Copy link
Collaborator

@alltilla alltilla commented Feb 6, 2023

Tested locally:

# apt update
Ign:1 http://127.0.0.1:8000/apt nightly InRelease
Get:2 http://127.0.0.1:8000/apt nightly Release [2497 B]                                      
Ign:3 http://127.0.0.1:8000/apt nightly Release.gpg                                           
Get:4 http://127.0.0.1:8000/apt nightly/ubuntu-jammy amd64 Packages [11.7 kB]                 
Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease                                        
Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Get:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [107 kB]
Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease                              
Fetched 121 kB in 7s (17.4 kB/s)                                                              
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
10 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@alltilla-Precision-5530:/etc/apt/sources.list.d# apt search syslog-ng-core
Sorting... Done
Full Text Search... Done
syslog-ng-core/nightly 4.0.1.116.gfdb88de.dirty-snapshot+20230206T094016 amd64
  Enhanced system logging daemon (core)


$ python3 -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
127.0.0.1 - - [06/Feb/2023 11:35:23] code 404, message File not found
127.0.0.1 - - [06/Feb/2023 11:35:23] "GET /apt/dists/nightly/InRelease HTTP/1.1" 404 -
127.0.0.1 - - [06/Feb/2023 11:35:23] "GET /apt/dists/nightly/Release HTTP/1.1" 200 -
127.0.0.1 - - [06/Feb/2023 11:35:23] code 404, message File not found
127.0.0.1 - - [06/Feb/2023 11:35:23] "GET /apt/dists/nightly/Release.gpg HTTP/1.1" 404 -
127.0.0.1 - - [06/Feb/2023 11:35:23] "GET /apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages.gz HTTP/1.1" 200 -

Note: The 404 comes from the repo not being signed and the InRelease and Release.gpg files are missing, which is due to the local testing.

Fixes #4309

Signed-off-by: Attila Szakacs szakacs.attila96@gmail.com

@alltilla
package-indexer: add .gz, .xz and .bz2 compression for Packages
e214f32 
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2023

No news file has been detected. Please write one, if applicable.

bazsi
bazsi approved these changes Feb 6, 2023
View reviewed changes
Copy link
Collaborator

@bazsi bazsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure how large those Packages files can become, but this will read everything into
a memory based buffer and then compress it in memory before writing it out.

As long as those files are tiny, I am ok, but this will eventually bite us. :)

With all that said, this is nice and simple and will probably work for years to come, so I am ok with this as implemented.

@bazsi
Copy link
Collaborator

bazsi commented Feb 6, 2023

The nightly is now green, please run nightly if you merge this to check if this break anything as it gets to master.

@alltilla
Copy link
Collaborator Author

alltilla commented Feb 6, 2023 

$ find -name "Packages*" | xargs ls -algh
-rw-rw-r-- 1 alltilla 61K febr   6 11:35 ./apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages
-rw-rw-r-- 1 alltilla 11K febr   6 11:35 ./apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages.bz2
-rw-rw-r-- 1 alltilla 12K febr   6 11:35 ./apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages.gz
-rw-rw-r-- 1 alltilla 11K febr   6 11:35 ./apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages.xz

I think we can get rid of the simple Packages file, if we are afraid of memory/bandwidth usage.
E.g.: http://hu.archive.ubuntu.com/ubuntu/dists/jammy/main/binary-amd64/

Storing the Packages file is not a big deal IMO, and from my quick test with apt, it ignored the Packages file, and downloaded Packages.gz automatically, so our bandwidth is reserved. Cannot comment on other clients, though.

@alltilla
Copy link
Collaborator Author

alltilla commented Feb 6, 2023

Oh, I have completely missed your point about storing it in memory while compressing it, sorry.
Yes that can be a problem, but I presume these files do not get that large (maybe a couple of MBs). We will see it in the future.

@bazsi
Copy link
Collaborator

bazsi commented Feb 6, 2023

agreed

@bazsi bazsi merged commit bfd2f3a into syslog-ng:master Feb 6, 2023
alltilla added a commit to alltilla/syslog-ng that referenced this pull request Feb 14, 2023
@alltilla
news: add entries for syslog-ng#4213, syslog-ng#4262, syslog-ng#4298, s…
2edaa35 
…yslog-ng#4302, syslog-ng#4305, syslog-ng#4313

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
MrAnno added a commit that referenced this pull request Feb 14, 2023
@MrAnno
Merge pull request #4326 from alltilla/missing-news-entries
91de539 
news: add entries for #4213, #4262, #4298, #4302, #4305, #4313
Genfood pushed a commit to Genfood/syslog-ng that referenced this pull request Jun 14, 2023
@alltilla @Genfood
news: add entries for syslog-ng#4213, syslog-ng#4262, syslog-ng#4298, s…
637e848 
…yslog-ng#4302, syslog-ng#4305, syslog-ng#4313

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bazsi bazsi bazsi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

apt-mirror can't open index file (Packages.gz missing)
2 participants
@alltilla @bazsi