You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Checksum is extremely important for storage services. We currently support some checksum features in OpenDAL, but they are very limited: we only support setting checksum_algorithm for S3.
I hope we can introduce this feature across all services, supporting newer and faster checksum algorithms such as crc64-nvme or crc64-ecma.
Problem and Solution
Add checksum_algorithm for all supported services.
Add additional checksum_algorithm options for S3, such as crc64-nvme.
We can also add some slow checksums like sha1 and md5
Additional Context
Perhaps we should also expose this at the metadata level.
Should we verify the checksum during the reading process?
Do we need to introduce a new error type for this?
We should re-consider the current design.
Is it a good idea to add at service level?
Is it better to have a ChecksumLayer for this?
Are you willing to contribute to the development of this feature?
Yes, I am willing to contribute to the development of this feature.
The text was updated successfully, but these errors were encountered:
Feature Description
Checksum is extremely important for storage services. We currently support some checksum features in OpenDAL, but they are very limited: we only support setting checksum_algorithm for S3.
I hope we can introduce this feature across all services, supporting newer and faster checksum algorithms such as
crc64-nvme
orcrc64-ecma
.Problem and Solution
checksum_algorithm
for all supported services.checksum_algorithm
options for S3, such ascrc64-nvme
.Additional Context
ChecksumLayer
for this?Are you willing to contribute to the development of this feature?
The text was updated successfully, but these errors were encountered: