Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ggml: Optimize Q4_0 into Q4_0_X_Y repack #10324

Merged
merged 1 commit into from
Nov 16, 2024
Merged

ggml: Optimize Q4_0 into Q4_0_X_Y repack #10324

merged 1 commit into from
Nov 16, 2024
+86 −28

Conversation

eddnjjn
Copy link
Contributor

@eddnjjn eddnjjn commented Nov 15, 2024

This pull request optimizes the code for repacking Q4_0 into Q4_0_4_4, Q4_0_4_8 and Q4_0_8_8.

@github-actions github-actions bot added the ggml changes relating to the ggml tensor library for machine learning label Nov 15, 2024
slaren
slaren approved these changes Nov 16, 2024
View reviewed changes
Copy link
Collaborator

@slaren slaren left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, for me this cuts the load time by 2/3 on x86, even more on M3 Max.

@slaren slaren merged commit 1e58ee1 into ggerganov:master Nov 16, 2024
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@slaren slaren slaren approved these changes

Assignees
No one assigned
Labels
ggml changes relating to the ggml tensor library for machine learning
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@eddnjjn @slaren