Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new feature: Checksum #5549

Open
1 task
Xuanwo opened this issue Jan 15, 2025 · 0 comments
Open
1 task

new feature: Checksum #5549

Xuanwo opened this issue Jan 15, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@Xuanwo
Copy link
Member

Xuanwo commented Jan 15, 2025

Feature Description

Checksum is extremely important for storage services. We currently support some checksum features in OpenDAL, but they are very limited: we only support setting checksum_algorithm for S3.

I hope we can introduce this feature across all services, supporting newer and faster checksum algorithms such as crc64-nvme or crc64-ecma.

Problem and Solution

  • Add checksum_algorithm for all supported services.
  • Add additional checksum_algorithm options for S3, such as crc64-nvme.
  • We can also add some slow checksums like sha1 and md5

Additional Context

  • Perhaps we should also expose this at the metadata level.
  • Should we verify the checksum during the reading process?
  • Do we need to introduce a new error type for this?
  • We should re-consider the current design.
    • Is it a good idea to add at service level?
    • Is it better to have a ChecksumLayer for this?

Are you willing to contribute to the development of this feature?

  • Yes, I am willing to contribute to the development of this feature.
@Xuanwo Xuanwo added the enhancement New feature or request label Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant