-
Notifications
You must be signed in to change notification settings - Fork 468
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(hash): Support hash field expiration #2402
base: unstable
Are you sure you want to change the base?
Conversation
As we specified in https://kvrocks.apache.org/community/data-structure-on-rocksdb#encoding-version-and-data-type, the first and second (and maybe more bits from the MSB) of So it should not be used for field expiration. (Also, not all data types need the field expiration feature.) Instead, you can add a new enum value of data type, or add a new field in the metadata of hash. |
@jjz921024 Thanks for your contributions. As @PragmaTwice mentioned, we cannot put the expired flag in the version field and we also need to adopt the SubkeyFilter in compact_filter.cc to recycle expired subkeys. |
@git-hulk Thanks for you reminding. I have added this feature in latest commit. About
I see this issues #2292. Maybe we should finish it first? |
No. It's too ad hoc and will introduce additional maintanance effort. (the encoding of HASH will be inconsistent with other types, which brings huge understanding, maintanance cost and additional code logic) And also, it breaks our whole design of "encoding version", e.g. HASH cannot have encoding version larger than 1. You can consider these options, as I mentioned before.
-1 for this change if no further commits. |
@PragmaTwice Got it, i will change it. |
@ jjz921024 Please let me know if I'm wrong. If the data structure contains at least 1 key that can be expired, we set a special bit to true and later check it. |
src/types/redis_hash.h
Outdated
|
||
private: | ||
rocksdb::Status GetMetadata(Database::GetOptions get_options, const Slice &ns_key, HashMetadata *metadata); | ||
static rocksdb::Status decodeFieldValue(const HashMetadata &metadata, std::string *value, uint64_t &expire); | ||
static rocksdb::Status encodeValueExpire(std::string *value, uint64_t expire); | ||
static bool isMeetCondition(HashFieldExpireType type, uint64_t new_expire, uint64_t old_expire); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we change the name of this function to describe more clearly what it checks? What question does it answer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. Due to the expiration time is encoded in the field, the time complexity of some commands changes from O(1) to O(n), such as |
@PragmaTwice Hi, I fix it about |
src/types/redis_hash.h
Outdated
|
||
private: | ||
rocksdb::Status GetMetadata(Database::GetOptions get_options, const Slice &ns_key, HashMetadata *metadata); | ||
static rocksdb::Status decodeFieldValue(const HashMetadata &metadata, std::string *value, uint64_t &expire); | ||
static rocksdb::Status encodeValueExpire(std::string *value, uint64_t expire); | ||
static bool isMeetCondition(HashFieldExpireType type, uint64_t new_expire, uint64_t old_expire); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@git-hulk Hi, I had renamed the |
Modification:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I mentioned before, we can add a config option (e.g. hash-field-expiration yes/no
) for users to decide if they want to enable field expiration.
Also our testing can benefit from this option since we can cover the old hash encoding.
@PragmaTwice Thank for you tip, I have finished this feature. please review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See comments : )
@git-hulk @PragmaTwice @torwig Sorry to bother your. I have changed the code as review required. May I ask your review it again? I think this is a very useful feature and hope it can be merge to kvrocks. |
I had fixed the gofmt error and passed all CI in my fork repo. https://github.com/jjz921024/kvrocks/actions/runs/10337719500 |
src/config/config.h
Outdated
@@ -167,6 +167,9 @@ struct Config { | |||
int json_max_nesting_depth = 1024; | |||
JsonStorageFormat json_storage_format = JsonStorageFormat::JSON; | |||
|
|||
// hash |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The variable name is self-explanatory so you can remove this comment. Otherwise, it would be best if you elaborated on the goal of this variable/option.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. I had polished the comment of here
Seems we need to solve the conflict first? |
rebase this pr on the latest commit of unstable branch. |
Seems there are failure in CI. I will check it up |
Quality Gate passedIssues Measures |
@jjz921024 I will take another pass in a few days, thanks for your contribution. |
This pr want to close #2269
Encoding
To implement hash expiration by field level, we encode the expiration time into the first 8 bytes of value.
For example:
And we use the second bit of the flags in metadata to indicate whether encode the expiration into value.
If all the fields in the hash have no encode expiration time, the
flags
will be1 0 0 0 | 0 0 1 0
.If any field in the hash is encoded into the expiration time, the
flags
will be1 1 0 0 | 0 0 1 0
.Flags conversion
For the HEXPIRE command, it will set an expiration on one or more fields.
If we execute hexpire command on a hash key that has not been set to expire, this will set the secode bit of
flags
and encode expiration for all fields (fields that not appear in the command will be set to 0)Add commands
According to https://redis.io/docs/latest/commands add a series of related commands:
HEXPIRE
,HPEXPIRE
,HEXPIREAT
,HPEXPIREAT
HEXPIRETIME
,HPEXPIRETIME
,HTTL
,HPTTL
HPERSIST