[WIP] Create Cassandra db schema on session initialization #5922

akstron · 2024-09-02T18:04:32Z

Create Schema (if not present) on Session Initialization

Once a session is established with cassandra db, the added code parses the template file containing queries for creating schema and create queries out of it. Post which it executes those queries to create the required types and tables.

Which problem is this PR solving?

Resolves #5797

Description of the changes

The PR includes the following changes:
1. Embedding template files into binary
1. Creation of database schema in initialization steps once session to database is established.

How was this change tested?

Schema rendering is being tested with unit test.
bash scripts/cassandra-integration-test.sh 4 v004 v2 -s

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: yarn lint and yarn test

plugin/storage/cassandra/factory.go

yurishkuro · 2024-09-02T21:54:36Z

plugin/storage/cassandra/factory.go

+	return result
+}
+
+func constructQueriesFromTemplateFiles(session cassandra.Session, params *StorageConfigParams) ([]cassandra.Query, error) {


Is cassandra.Session not able to execute multiple queries at once?

Are you talking about running individual queries in parallel or a batch query option? I didn't try executing parallel queries.

plugin/storage/cassandra/factory.go

cmd/jaeger/config-cassandra.yaml

pkg/cassandra/config/config.go

plugin/storage/cassandra/schema/schema.go

…ution for initialize database Signed-off-by: Alok Kumar Singh <[email protected]>

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro · 2024-10-28T16:07:39Z

pkg/cassandra/config/config.go

+	//Datacenter is the name for network topology
+	Datacenter string `mapstructure:"datacenter" valid:"optional"`
+	// TraceTTL is Time To Live (TTL) for the trace data in seconds
+	TraceTTL int `mapstructure:"trace_ttl" valid:"optional"`


can the type here be time.Duration so that the user could specify 72h?

I am assuming that there should be a validation to make sure that user can't specify something like "ms"?

that's the point of using strong type - the business logic does not need to worry about validations, they should happen separately during parsing. I believe if you simply change int to time.Duration and use 10ms as a value in YAML, the parser will do the right thing.

TraceTTL and DependecyTTL sets: default_time_to_live (https://cassandra.apache.org/doc/latest/cassandra/developing/cql/ddl.html) which should be supplied in seconds. Won't making it time.Duration and allowing users to use something like 10ms complicate it? Like we have to block more precise duration than seconds like millisecond or nanosecond?

the option is either we name the field trace_ttl_seconds and force the user to deal with ridiculous numbers like 1123200 (2w - can you tell? I can't without doing math), or we keep a clean name trace_ttl and let the user specify easy-to-read values like 24h or 14d

yurishkuro · 2024-10-28T16:08:33Z

pkg/cassandra/config/config.go

+	// CasVersion is version of cassandra used
+	CasVersion int `mapstructure:"cas_version" valid:"optional"`
+	// CompactionWindow of format "^[0-9]+[mhd]$" tells the compaction window of the db
+	CompactionWindow string `mapstructure:"compaction_window" valid:"optional"`


so this is defining a time interval? Can we then also use time.Duration type?

Only time precision form https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/TimeUnit.html are allowed as per https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/twcs.html.

Also, below is based on the current script used. Should we go ahead with breaking it?
// CompactionWindow of format "^[0-9]+[mhd]$" tells the compaction window of the db

we are creating new config, not bound by previous restrictions. If the destination API does not allow smaller units we can always add validation cw >= time.Minute

pkg/cassandra/config/config.go

plugin/storage/cassandra/schema/v004-go-tmpl-test.cql.tmpl

plugin/storage/cassandra/schema/schema.go

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro · 2024-10-31T02:09:50Z

plugin/storage/cassandra/schema/v004-go-tmpl-test.cql.tmpl

@@ -0,0 +1,43 @@
+-- There are total 4 queries here


what's the point of test schema?

Instead of testing on all the queries and manually creating expected result for all of them, I reduced the number of queries to ease the creation of expected result.

that doesn't really make sense to me. We can either test that the template is correct or that the code using the template is correct. You're not doing the former by having another template, and the latter you can do with the primary template. We don't need to validate that the output of running template is "as expected", that's like testing the Go template package.

We don't need to validate that the output of running template is "as expected", that's like testing the Go template package.
We are also removing comments and constructing individual query strings out of it by iterating over lines in the template output. The test checks this "individual query string" construction, which not just involves using the template package.

and the latter you can do with the primary template.
Sure, I think I can write an integration test for this.

Hi @yurishkuro , I have the integration test script ready. Should I merge it with the current cassandra-intergration-test.sh? This would require adding an additional 4th parameter skip_apply_schema for running the script

Signed-off-by: Alok Kumar Singh <[email protected]>

codecov · 2024-11-09T19:16:05Z

Codecov Report

Attention: Patch coverage is 62.70270% with 69 lines in your changes missing coverage. Please review.

Project coverage is 96.18%. Comparing base (640615e) to head (1dc16cd).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/cassandra/config/schema.go	57.44%	44 Missing and 16 partials ⚠️
pkg/cassandra/config/config.go	79.54%	6 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5922      +/-   ##
==========================================
- Coverage   96.41%   96.18%   -0.24%     
==========================================
  Files         353      355       +2     
  Lines       20135    20310     +175     
==========================================
+ Hits        19414    19535     +121     
- Misses        535      572      +37     
- Partials      186      203      +17

Flag	Coverage Δ
badger_v1	`8.16% <0.00%> (-0.16%)`	⬇️
badger_v2	`1.64% <0.00%> (-0.04%)`	⬇️
cassandra-4.x-v1	`15.25% <62.70%> (+0.85%)`	⬆️
cassandra-4.x-v2	`1.59% <0.00%> (-0.04%)`	⬇️
cassandra-5.x-v1	`15.25% <62.70%> (+0.85%)`	⬆️
cassandra-5.x-v2	`1.59% <0.00%> (-0.04%)`	⬇️
elasticsearch-6.x-v1	`18.27% <0.00%> (-0.23%)`	⬇️
elasticsearch-7.x-v1	`18.35% <0.00%> (-0.24%)`	⬇️
elasticsearch-8.x-v1	`18.51% <0.00%> (-0.24%)`	⬇️
elasticsearch-8.x-v2	`1.64% <0.00%> (-0.03%)`	⬇️
grpc_v1	`9.31% <0.00%> (-0.22%)`	⬇️
grpc_v2	`6.88% <0.00%> (-0.14%)`	⬇️
kafka-v1	`8.72% <0.00%> (-0.17%)`	⬇️
kafka-v2	`1.64% <0.00%> (-0.04%)`	⬇️
memory_v2	`1.64% <0.00%> (-0.03%)`	⬇️
opensearch-1.x-v1	`18.40% <0.00%> (-0.25%)`	⬇️
opensearch-2.x-v1	`18.39% <0.00%> (-0.25%)`	⬇️
opensearch-2.x-v2	`1.63% <0.00%> (-0.05%)`	⬇️
tailsampling-processor	`0.46% <0.00%> (-0.02%)`	⬇️
unittests	`94.86% <36.21%> (-0.47%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro · 2024-11-14T03:13:17Z

pkg/cassandra/config/config.go

 // NewSession creates a new Cassandra session
 func (c *Configuration) NewSession() (cassandra.Session, error) {
+	err := c.newSessionPrerequisites()


this is redundant. Create the session as before and call GenerateSchemaIfNotPresent just before returning from NewSession

yurishkuro · 2024-11-14T03:14:06Z

pkg/cassandra/config/v004-go-tmpl-test.cql.tmpl

@@ -0,0 +1,43 @@
+-- There are total 4 queries here


I already asked - why do we need this file?

yurishkuro · 2024-11-14T03:15:20Z

pkg/cassandra/config/config.go

@@ -58,6 +58,18 @@ type Schema struct {
 	// while connecting to the Cassandra Cluster. This is useful for connecting to clusters, like Azure Cosmos DB,
 	// that do not support SnappyCompression.
 	DisableCompression bool `mapstructure:"disable_compression"`
+	// Datacenter is the name for network topology


we need to add CreateSchema bool (default false) for backwards compatibility

yurishkuro · 2024-11-14T03:16:16Z

pkg/cassandra/config/schema.go

+	CompactionWindowUnit string `mapstructure:"compaction_window_unit" valid:"optional"`
+}
+
+func DefaultParams() TemplateParams {


not needed. There is DefaultConfig function somewhere, it should populate these settings there.

yurishkuro · 2024-11-14T03:19:22Z

pkg/cassandra/config/schema_test.go

+
+	expOutputQueries := []string{
+		`CREATE TYPE IF NOT EXISTS jaeger_v1_dc1.keyvalue (
+key             text,


all these details are not relevant for the unit test - after all we don't know if they are correct anyway since we're not testing against a real DB (but the integration test will test against it). So at most you can verify that CREATE commands are issued for expected db objects, but there's no need to match full queries.

yurishkuro · 2024-11-14T03:19:54Z

scripts/cassandra-integration-test.sh

+	s)
+    SKIP_APPLY_SCHEMA="true"
+		;;
+  *)
+		;;


please use two-space indentation, not tabs

yurishkuro reviewed Sep 2, 2024

View reviewed changes

yurishkuro mentioned this pull request Sep 7, 2024

[jaeger-v2] Align Cassandra Storage Config With OTEL #5928

Closed

6 tasks

mahadzaryab1 reviewed Sep 7, 2024

View reviewed changes

plugin/storage/cassandra/factory.go Outdated Show resolved Hide resolved

akstron force-pushed the create-database-scheme-cassandra branch 2 times, most recently from 69275fb to bcad4c0 Compare October 25, 2024 13:34

akstron marked this pull request as ready for review October 25, 2024 13:52

akstron requested a review from a team as a code owner October 25, 2024 13:52

akstron requested a review from albertteoh October 25, 2024 13:52

dosubot bot added area/storage storage/cassandra labels Oct 25, 2024

akstron requested a review from yurishkuro October 26, 2024 16:18

yurishkuro reviewed Oct 26, 2024

View reviewed changes

cmd/jaeger/config-cassandra.yaml Outdated Show resolved Hide resolved

pkg/cassandra/config/config.go Outdated Show resolved Hide resolved

pkg/cassandra/config/config.go Outdated Show resolved Hide resolved

plugin/storage/cassandra/schema/schema.go Outdated Show resolved Hide resolved

akstron force-pushed the create-database-scheme-cassandra branch from f9dd90e to 90368b1 Compare October 28, 2024 10:46

Embeded template file in binary and added query construction and exec…

afc786d

…ution for initialize database Signed-off-by: Alok Kumar Singh <[email protected]>

akstron force-pushed the create-database-scheme-cassandra branch from 90368b1 to afc786d Compare October 28, 2024 10:50

akstron added 3 commits October 28, 2024 16:45

Removed unnecessary SchemaConfig struct

3a82683

Signed-off-by: Alok Kumar Singh <[email protected]>

Added new schema configs in default config generator

b3e0370

Signed-off-by: Alok Kumar Singh <[email protected]>

Revert Keyspace removal

c35ef2d

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro reviewed Oct 28, 2024

View reviewed changes

akstron added 4 commits October 29, 2024 12:31

Bug fix while creating queries

18dcf1e

Signed-off-by: Alok Kumar Singh <[email protected]>

Improving test

b230412

Signed-off-by: Alok Kumar Singh <[email protected]>

Created new struct for derived params

9642e38

Signed-off-by: Alok Kumar Singh <[email protected]>

Remove fields from yaml file

32485af

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro reviewed Oct 31, 2024

View reviewed changes

Added integration test and minor changes

ed3e172

Signed-off-by: Alok Kumar Singh <[email protected]>

akstron changed the title ~~Create Cassandra db schema on session initialization~~ [WIP] Create Cassandra db schema on session initialization Nov 9, 2024

akstron added 4 commits November 9, 2024 23:34

Minor fixes

07ab247

Signed-off-by: Alok Kumar Singh <[email protected]>

Removing temp command

8e68b62

Signed-off-by: Alok Kumar Singh <[email protected]>

lint fixes

96025e2

Signed-off-by: Alok Kumar Singh <[email protected]>

Test fix

4595ca7

Signed-off-by: Alok Kumar Singh <[email protected]>

Script fix

840bdf8

Signed-off-by: Alok Kumar Singh <[email protected]>

akstron changed the title ~~[WIP] Create Cassandra db schema on session initialization~~ Create Cassandra db schema on session initialization Nov 9, 2024

Added default case for flag in script

1dc16cd

Signed-off-by: Alok Kumar Singh <[email protected]>

akstron changed the title ~~Create Cassandra db schema on session initialization~~ [WIP] Create Cassandra db schema on session initialization Nov 10, 2024

akstron marked this pull request as draft November 10, 2024 05:21

yurishkuro reviewed Nov 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Create Cassandra db schema on session initialization #5922

[WIP] Create Cassandra db schema on session initialization #5922

akstron commented Sep 2, 2024 •

edited

Loading

yurishkuro Sep 2, 2024

akstron Oct 25, 2024

yurishkuro Oct 28, 2024

akstron Oct 29, 2024 •

edited

Loading

yurishkuro Oct 31, 2024

akstron Nov 9, 2024 •

edited

Loading

yurishkuro Nov 14, 2024

yurishkuro Oct 28, 2024

akstron Nov 9, 2024

yurishkuro Nov 14, 2024

yurishkuro Oct 31, 2024

akstron Oct 31, 2024 •

edited

Loading

yurishkuro Oct 31, 2024

akstron Nov 1, 2024

akstron Nov 10, 2024

codecov bot commented Nov 9, 2024 •

edited

Loading

yurishkuro Nov 14, 2024

yurishkuro Nov 14, 2024

yurishkuro Nov 14, 2024

yurishkuro Nov 14, 2024

yurishkuro Nov 14, 2024

yurishkuro Nov 14, 2024

[WIP] Create Cassandra db schema on session initialization #5922

Are you sure you want to change the base?

[WIP] Create Cassandra db schema on session initialization #5922

Conversation

akstron commented Sep 2, 2024 • edited Loading

Which problem is this PR solving?

Description of the changes

How was this change tested?

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron Nov 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 9, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron commented Sep 2, 2024 •

edited

Loading

akstron Oct 29, 2024 •

edited

Loading

akstron Nov 9, 2024 •

edited

Loading

akstron Oct 31, 2024 •

edited

Loading

codecov bot commented Nov 9, 2024 •

edited

Loading