[kv] Support version merge engine #277

sunxiaojian · 2024-12-27T09:53:25Z

Purpose

Support version merge engine

Linked issue: close #213

Tests

com.alibaba.fluss.connector.flink.sink.FlinkTableSinkITCase#testVersionMergeEngineWithTypeBigint
com.alibaba.fluss.connector.flink.sink.FlinkTableSinkITCase#testVersionMergeEngineWithTypeTimestamp
com.alibaba.fluss.client.table.FlussTableITCase#testMergeEngineWithVersion

API and Format

No

Documentation

No

CLAassistant · 2024-12-27T09:53:30Z

All committers have signed the CLA.

sunxiaojian · 2024-12-28T01:21:24Z

@wuchong CLA has been sent, PTAL

wuchong · 2024-12-29T15:03:49Z

Thanks for the contribution @sunxiaojian ! Will take a look. cc @luoyuxia as well.

wuchong · 2024-12-29T15:04:22Z

Btw, the checkstyle is failed: https://github.com/alibaba/fluss/actions/runs/12515363835/job/34959455034?pr=277

sunxiaojian · 2024-12-30T02:22:25Z

Btw, the checkstyle is failed: https://github.com/alibaba/fluss/actions/runs/12515363835/job/34959455034?pr=277

fixed

luoyuxia

@sunxiaojian Thanks for the pr... I left some comments... PTAL

fluss-common/src/main/java/com/alibaba/fluss/config/MergeEngine.java

...-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/VersionMergeEngineWrapper.java

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/MergeEngineWrapper.java

...-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/VersionMergeEngineWrapper.java

fluss-server/src/test/java/com/alibaba/fluss/server/kv/KvTabletTest.java

...uss-connector-flink/src/main/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSink.java

sunxiaojian · 2025-01-02T02:06:19Z

@sunxiaojian Thanks for the pr... I left some comments... PTAL

@luoyuxia Thanks for the review. I will handle it as soon as possible

sunxiaojian · 2025-01-06T09:25:34Z

@luoyuxia PTAL again , thanks

luoyuxia

@sunxiaojian Thanks for update... Left some comments..

fluss-common/src/main/java/com/alibaba/fluss/metadata/MergeEngine.java

fluss-client/src/test/java/com/alibaba/fluss/client/table/FlussTableITCase.java

...nnector-flink/src/test/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSinkITCase.java

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/MergeEngineWrapper.java

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/VersionRowMergeEngine.java

sunxiaojian · 2025-01-08T02:59:50Z

@luoyuxia PTAL again, thanks.

luoyuxia

@sunxiaojian Thanks for updating again... I left some comments. PTAL. Should be look good to me in next iterations..

fluss-common/src/main/java/com/alibaba/fluss/metadata/MergeEngine.java

fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvTablet.java

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/VersionRowMergeEngine.java

luoyuxia · 2025-01-08T12:36:14Z

fluss-common/src/main/java/com/alibaba/fluss/metadata/TableDescriptor.java

@@ -281,7 +281,7 @@ public boolean isDataLakeEnabled() {
    }

    public @Nullable MergeEngine getMergeEngine() {
-        return configuration().get(ConfigOptions.TABLE_MERGE_ENGINE);
+        return MergeEngine.create(properties);


I mean in here to check the data type of version column of version merge engine..

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/RowMergeEngine.java

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/FirstRowMergeEngine.java

fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvTablet.java

luoyuxia · 2025-01-08T13:53:33Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvTablet.java

@@ -372,6 +380,23 @@ public LogAppendInfo putAsLeader(
                });
    }

+    private RowMergeEngine getRowMergeEngine(Schema schema) {


I append a commit about RowMergeEngine in my own branch luoyuxia@67f7605
There're two changes:

If no merge engine, I set the RowMergeEngine to null instead of creating a new RowMergeEngine.. Since using a RowMergeEngine for tablet without setting merge engine look strange to me.

if it's first row merge engine, don't to deserize the old value

You can have a look..Hope it can help

@luoyuxia About 1 ，Can refer to Paimon's default merge engine Deduplication (https://paimon.apache.org/docs/master/primary-key-table/merge-engine/overview/), which is similar to ours. I do not recommend returning null. By default, it can be understood as the last row merge engine. @wuchong What do you think of it?

TBH, althogth we haven't consider the default behavior of kv as Deduplication merge engine in our doc like Paimon and code design, I'm not too objection to condier the default behaviorv as another special merge engine in here since now it's only for internal use.

But I think consider the default behavior as a specical merge engine may make the code logic complex..May be I'm wong.

For example, in update case, default behavior will need to consider partial update/delete, but the first_row, version merge engine don't need, so you will need to introduce method ignoreDelete and ignorePartialUpdate in RowMergeEngine. Logically, RowMergeEngine should only care how how to mere two row, the logical of ignoreDelete and ignorePartialUpdate should be considered by caller..

Also, If we decide to consider the default as Deduplication , I'd like to suggest use a new class naed DeduplicateRowMergeEngine or KeepLastRowMergeEngine or any thing else instead of using anonymous class

sunxiaojian

@luoyuxia Thanks for review ... I left some comments. PTAL.

sunxiaojian · 2025-01-09T16:23:39Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvTablet.java

@@ -372,6 +380,23 @@ public LogAppendInfo putAsLeader(
                });
    }

+    private RowMergeEngine getRowMergeEngine(Schema schema) {


@luoyuxia About 1 ，Can refer to Paimon's default merge engine Deduplication (https://paimon.apache.org/docs/master/primary-key-table/merge-engine/overview/), which is similar to ours. I do not recommend returning null. By default, it can be understood as the last row merge engine. @wuchong What do you think of it?

fluss-common/src/main/java/com/alibaba/fluss/metadata/TableDescriptor.java

luoyuxia

@sunxiaojian Thanks again.. Left minor comments..

luoyuxia · 2025-01-10T04:25:00Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvTablet.java

@@ -372,6 +380,23 @@ public LogAppendInfo putAsLeader(
                });
    }

+    private RowMergeEngine getRowMergeEngine(Schema schema) {


TBH, althogth we haven't consider the default behavior of kv as Deduplication merge engine in our doc like Paimon and code design, I'm not too objection to condier the default behaviorv as another special merge engine in here since now it's only for internal use.

But I think consider the default behavior as a specical merge engine may make the code logic complex..May be I'm wong.

For example, in update case, default behavior will need to consider partial update/delete, but the first_row, version merge engine don't need, so you will need to introduce method ignoreDelete and ignorePartialUpdate in RowMergeEngine. Logically, RowMergeEngine should only care how how to mere two row, the logical of ignoreDelete and ignorePartialUpdate should be considered by caller..

Also, If we decide to consider the default as Deduplication , I'd like to suggest use a new class naed DeduplicateRowMergeEngine or KeepLastRowMergeEngine or any thing else instead of using anonymous class

fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvTablet.java

luoyuxia

@sunxiaojian Thanks.. I have no further comment exception for the argument in here about the abstract of RowMergeEngine in KvTablet. Btw, next time, please append a new commit instead of squash all into a commit so that reviewer can track what changes since last review instead of go through all files..
@wuchong Could you please have a final check?

fluss-server/src/main/java/com/alibaba/fluss/server/coordinator/CoordinatorService.java

luoyuxia · 2025-01-10T12:05:40Z

fluss-server/src/test/java/com/alibaba/fluss/server/kv/KvTabletTest.java

@@ -600,6 +603,80 @@ void testFirstRowMergeEngine(@TempDir File tempLogDir, @TempDir File tmpKvDir)
        checkEqual(actualLogRecords, expectedLogs);
    }

+    @Test
+    void testVersionRowMergeEngine(@TempDir File tempLogDir, @TempDir File tmpKvDir)


don't forget my comments in here.

sunxiaojian · 2025-01-11T12:11:33Z

@sunxiaojian Thanks.. I have no further comment exception for the argument in here about the abstract of RowMergeEngine in KvTablet. Btw, next time, please append a new commit instead of squash all into a commit so that reviewer can track what changes since last review instead of go through all files.. @wuchong Could you please have a final check?

@luoyuxia Fixed all, sorry, Because there was a conflict with the main branch, I accidentally merged everything. I will pay attention next time. Thanks again.

wuchong · 2025-01-15T05:57:15Z

Thanks for the awesome work @sunxiaojian @luoyuxia ! I will go through the code tomorrow to have a final check!

swuferhong · 2025-01-17T02:16:55Z

Hi, @sunxiaojian For some comments that have already been resolved and are no longer in dispute, can you click "resolve conversion" on your side? There are too many comments, and I don't know which ones have been resolved finally.

sunxiaojian · 2025-01-17T03:01:06Z

Hi, @sunxiaojian For some comments that have already been resolved and are no longer in dispute, can you click "resolve conversion" on your side? There are too many comments, and I don't know which ones have been resolved finally.

@swuferhong Fixed, PTAL again, thanks.

wuchong

@sunxiaojian thanks for the contribution. This PR is already in a good shape. The comments are mainly around row merge engine. If you don't mind, I can help to refine this PR and merge it.

wuchong · 2025-01-18T14:32:02Z

...nnector-flink/src/test/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSinkITCase.java

@@ -826,6 +826,111 @@ void testUnsupportedStmtOnFirstRowMergeEngine() {
                        tablePath);
    }

+    @Test
+    void testUnsupportedStmtOnVersionRowMergeEngine() {


Suggested change

void testUnsupportedStmtOnVersionRowMergeEngine() {

void testUnsupportedStmtOnVersionMergeEngine() {

wuchong · 2025-01-18T14:37:20Z

...nnector-flink/src/test/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSinkITCase.java

+                "create table log_sink (a int not null primary key not enforced, b string, ts bigint)");
+
+        JobClient insertJobClient =
+                tEnv.executeSql("insert into log_sink select * from merge_engine_with_version")


why insert into another pk table? If we want to subscribe the changelogs of merge_engine_with_version, we can directly iterate tEnv.executeSql("select * from merge_engine_with_version").collect() in streaming mode.

wuchong · 2025-01-18T14:40:14Z

...nnector-flink/src/test/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSinkITCase.java

+
+        // insert again, update id=3
+        tEnv.executeSql(
+                        "insert into merge_engine_with_version (a, b, ts) VALUES (3, 'v33', 1001), (4, 'v44', 1000)")


add a test data 2, v22, 1000 that has the same version to verify it should also be updated.

wuchong · 2025-01-18T14:41:14Z

...nnector-flink/src/test/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSinkITCase.java

+                                + "(1, 'v1', TIMESTAMP '2024-12-27 12:00:00.123'), "
+                                + "(2, 'v2', TIMESTAMP '2024-12-27 12:00:00.123'), "
+                                + "(1, 'v11', TIMESTAMP '2024-12-27 11:59:59.123'), "
+                                + "(3, 'v3', TIMESTAMP '2024-12-27 12:00:00.123');")


Add a test data that has the same timestamp to verify the later one should replace the previous one.

Suggested change

+ "(3, 'v3', TIMESTAMP '2024-12-27 12:00:00.123');")

+ "(3, 'v33', TIMESTAMP '2024-12-27 12:00:00.123');")

+ "(3, 'v3', TIMESTAMP '2024-12-27 12:00:00.123');")

wuchong · 2025-01-18T14:46:50Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/VersionRowMergeEngine.java

+    public static final Set<String> VERSION_MERGE_ENGINE_SUPPORTED_DATA_TYPES =
+            ImmutableSet.of(
+                    BigIntType.class.getName(),
+                    IntType.class.getName(),
+                    TimestampType.class.getName(),
+                    TimeType.class.getName(),
+                    LocalZonedTimestampType.class.getName());


Use DataTypeRoot instead.

wuchong · 2025-01-18T16:40:46Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvTablet.java

@@ -267,12 +271,27 @@ public LogAppendInfo putAsLeader(
                                KvRecordReadContext.createReadContext(kvFormat, fieldTypes);
                        ValueDecoder valueDecoder =
                                new ValueDecoder(readContext.getRowDecoder(schemaId));
-
+                        RowMergeEngine rowMergeEngine = getRowMergeEngine(schema);


This is a heavy operation, the row merger should be initialized when kv tablet is setup. And a dynamic PartialUpdateRowMerger will be used if the put is a partial update.

wuchong · 2025-01-18T16:41:45Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvTablet.java

                        int appendedRecordCount = 0;
                        for (KvRecord kvRecord : kvRecords.records(readContext)) {
                            byte[] keyBytes = BytesUtils.toArray(kvRecord.getKey());
                            KvPreWriteBuffer.Key key = KvPreWriteBuffer.Key.of(keyBytes);
                            if (kvRecord.getRow() == null) {
+                                // currently, all supported merge engine will ignore delete row.
+                                if (rowMergeEngine.ignoreDelete()) {


We still need the nullable MergeEngineType in the KvTablet to skip deleting if we introduce RowMerger without ignoreDelete().

wuchong · 2025-01-18T16:42:53Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/VersionRowMergeEngine.java

+        if (dataType instanceof TimestampType) {
+            return (left, right) ->
+                    ((TimestampNtz) left).getMillisecond()
+                            > ((TimestampNtz) right).getMillisecond();


Should use TimestampNtz#compareTo to consider nano seconds. Add tests for TIMESTAMP(9).

wuchong · 2025-01-18T16:43:35Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/VersionRowMergeEngine.java

+        }
+        if (dataType instanceof LocalZonedTimestampType) {
+            return (left, right) ->
+                    ((TimestampLtz) left).toEpochMicros() > ((TimestampLtz) right).toEpochMicros();


Should use TimestampLtz#compareTo to consider nano seconds. Add tests for TIMESTAMP_LTZ(9).

wuchong · 2025-01-18T16:43:56Z

fluss-server/src/main/java/com/alibaba/fluss/server/kv/mergeengine/VersionRowMergeEngine.java

+    }
+
+    interface ValueComparator {
+        boolean isGreaterThan(Object left, Object right);


Should return int or change to isGreaterOrEqualThan to handle equal case.

sunxiaojian force-pushed the support-merge-engine-version branch 2 times, most recently from 944bdb3 to 7f115bf Compare December 27, 2024 11:12

sunxiaojian force-pushed the support-merge-engine-version branch 2 times, most recently from b1363d6 to 5b67c20 Compare December 30, 2024 02:12

luoyuxia reviewed Dec 30, 2024

View reviewed changes

sunxiaojian force-pushed the support-merge-engine-version branch 12 times, most recently from 023351d to ef2eada Compare January 6, 2025 09:13

luoyuxia reviewed Jan 7, 2025

View reviewed changes

sunxiaojian force-pushed the support-merge-engine-version branch 2 times, most recently from 40b7987 to d037237 Compare January 8, 2025 02:57

sunxiaojian force-pushed the support-merge-engine-version branch from d037237 to fce5768 Compare January 8, 2025 03:09

luoyuxia reviewed Jan 8, 2025

View reviewed changes

sunxiaojian force-pushed the support-merge-engine-version branch 4 times, most recently from 5a9c7b7 to ef304c0 Compare January 9, 2025 16:43

sunxiaojian commented Jan 10, 2025

View reviewed changes

luoyuxia reviewed Jan 10, 2025

View reviewed changes

fluss-common/src/main/java/com/alibaba/fluss/metadata/TableDescriptor.java Outdated Show resolved Hide resolved

luoyuxia reviewed Jan 10, 2025

View reviewed changes

sunxiaojian force-pushed the support-merge-engine-version branch 3 times, most recently from a03e5de to bf696d9 Compare January 10, 2025 06:51

Introduce version merge engine for primary key table

ed96ec7

sunxiaojian force-pushed the support-merge-engine-version branch from 1895fe3 to ed96ec7 Compare January 10, 2025 06:53

luoyuxia reviewed Jan 10, 2025

View reviewed changes

fixed

ceb5448

wuchong reviewed Jan 18, 2025

View reviewed changes

	void testUnsupportedStmtOnVersionRowMergeEngine() {
	void testUnsupportedStmtOnVersionMergeEngine() {

	+ "(3, 'v3', TIMESTAMP '2024-12-27 12:00:00.123');")
	+ "(3, 'v33', TIMESTAMP '2024-12-27 12:00:00.123');")
	+ "(3, 'v3', TIMESTAMP '2024-12-27 12:00:00.123');")

[kv] Support version merge engine #277

Are you sure you want to change the base?

[kv] Support version merge engine #277

Conversation

sunxiaojian commented Dec 27, 2024 • edited Loading

Purpose

Tests

API and Format

Documentation

CLAassistant commented Dec 27, 2024 • edited Loading

sunxiaojian commented Dec 28, 2024

wuchong commented Dec 29, 2024

wuchong commented Dec 29, 2024

sunxiaojian commented Dec 30, 2024

luoyuxia left a comment

Choose a reason for hiding this comment

sunxiaojian commented Jan 2, 2025

sunxiaojian commented Jan 6, 2025

luoyuxia left a comment

Choose a reason for hiding this comment

sunxiaojian commented Jan 8, 2025

luoyuxia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunxiaojian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luoyuxia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luoyuxia left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunxiaojian commented Jan 11, 2025 • edited Loading

wuchong commented Jan 15, 2025

swuferhong commented Jan 17, 2025

sunxiaojian commented Jan 17, 2025

wuchong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunxiaojian commented Dec 27, 2024 •

edited

Loading

CLAassistant commented Dec 27, 2024 •

edited

Loading

luoyuxia left a comment •

edited

Loading

sunxiaojian commented Jan 11, 2025 •

edited

Loading