[WIP] Refactor storage factories to hold one configuration

[mahadzaryab1]

Which problem is this PR solving?

• Towards Investigate if ES Storage is passed a proper isArchive flag #6065

Description of the changes

How was this change tested?

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[mahadzaryab1]

@yurishkuro can you take another look at the changes? I minimized the code movements to simply remove the others field from Options. The problem now is that this isn't initializing the CLI flags for es-archive. Any thoughts on how to get around this?

[yurishkuro]

see #6156 (comment)

The query service instantiates just a single factory and casts it to ArchiveFactory. With your changes (which are in the right direction, but of insufficient scope) it never gets a chance to create archive CLI flags, because that has to happen via different factories.

[mahadzaryab1]

@yurishkuro is the archive factory only needed for the query service?

[yurishkuro]

yes

[yurishkuro]

need to be very careful about removing ArchiveFactory interface, because query service uses it via runtime cast, so unless we have integration tests (many of them disable archive tests as I recall) you can introduce a breaking change

jaeger/cmd/query/app/querysvc/query_service.go

Line 131 in 772d84c

archiveFactory, ok := storageFactory.(storage.ArchiveFactory)

[mahadzaryab1]

@yurishkuro yep - noted. I was thinking to avoid breaking changes we can remove the ArchiveFactory within this PR and use an archive storage factory wherever needed.

[mahadzaryab1]

https://github.com/jaegertracing/jaeger/blob/main/plugin/storage/integration/elasticsearch_test.go#L165 - ES doesn't skip the archive test

[mahadzaryab1]

@yurishkuro we've got 3 callsites for InitArchiveStorage that currently does the runtime cast. I had a couple of questions on how to move forward here and wanted to get your thoughts.

1. https://github.com/jaegertracing/jaeger/blob/main/cmd/jaeger/internal/extension/jaegerquery/server.go#L137-L139. The previous check to get the traces archive factory should be sufficient. If it exists, we can set the archive span reader and archive span writer in query options. Does that make sense?

2. jaeger/cmd/query/app/flags.go

   Line 139 in 772d84c

   func (qOpts *QueryOptions) BuildQueryServiceOptions(storageFactory storage.Factory, logger *zap.Logger) *querysvc.QueryServiceOptions {

   . This is called by cmd/query and cmd/all-in-one. Should we instantiate a new storage factory here that's the archive storage factory and pass that down to the function linked.
3. https://github.com/jaegertracing/jaeger/blob/main/cmd/remote-storage/app/server.go#L81. This is used in the remote storage in cmd/remote-storage. Should we do the same thing here as suggested in 2?

[yurishkuro]

I don't think you need to change (1), but we need to change InitArchiveStorage() not to cast but to use the storage directly

(2) yes

(3) I think you don't need to change it because if the caller needs an archive storage it should instantiate a different remote storage. As I understand it there's not a dedicated gRPC API for archive storage, which could go away the same way as ArchiveFactory.

[mahadzaryab1]

@yurishkuro Got it. For (2) - this is how the primary storage factory is initialized. How would we go about initializing the archive storage factory here to expose the CLI flags for storages that have archive flags (cassandra/es)

[mahadzaryab1]

@yurishkuro For v1, what do you think of passing the isArchive flag into https://github.com/jaegertracing/jaeger/blob/main/plugin/storage/factory.go#L116. This way, we can create a new archive storage using NewFactory, which we can pass down to es.NewFactory() and any other storage configs that need it.

[mahadzaryab1]

@yurishkuro any thoughts on how we should handle this case here? previously, this wouldn't initialize the archive storage if the factory didn't implement the ArchiveStorage interface but now it always will.

[codecov]

Codecov Report

Attention: Patch coverage is 98.75000% with 1 line in your changes missing coverage. Please review.

Project coverage is 96.44%. Comparing base (0a24f6d) to head (4194aab).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
plugin/storage/es/factory.go	97.56%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6156      +/-   ##
==========================================
- Coverage   96.47%   96.44%   -0.04%     
==========================================
  Files         354      354              
  Lines       20126    19997     -129     
==========================================
- Hits        19417    19286     -131     
- Misses        524      526       +2     
  Partials      185      185              

Flag	Coverage Δ
badger_v1	8.42% <0.00%> (+0.10%)	⬆️
badger_v2	1.70% <0.00%> (+0.02%)	⬆️
cassandra-4.x-v1	14.15% <35.00%> (-0.24%)	⬇️
cassandra-4.x-v2	1.64% <0.00%> (+0.02%)	⬆️
cassandra-5.x-v1	14.15% <35.00%> (-0.24%)	⬇️
cassandra-5.x-v2	1.64% <0.00%> (+0.02%)	⬆️
elasticsearch-6.x-v1	18.18% <38.75%> (-0.43%)	⬇️
elasticsearch-7.x-v1	18.26% <38.75%> (-0.43%)	⬇️
elasticsearch-8.x-v1	18.43% <38.75%> (-0.42%)	⬇️
elasticsearch-8.x-v2	1.69% <0.00%> (+0.01%)	⬆️
grpc_v1	?
grpc_v2	6.91% <3.75%> (-0.09%)	⬇️
kafka-v1	8.99% <0.00%> (+0.11%)	⬆️
kafka-v2	1.70% <0.00%> (+0.02%)	⬆️
memory_v2	1.70% <0.00%> (+0.02%)	⬆️
opensearch-1.x-v1	18.30% <38.75%> (-0.44%)	⬇️
opensearch-2.x-v1	18.31% <38.75%> (-0.43%)	⬇️
opensearch-2.x-v2	1.69% <0.00%> (+0.01%)	⬆️
tailsampling-processor	0.47% <0.00%> (+<0.01%)	⬆️
unittests	95.29% <83.75%> (-0.10%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mahadzaryab1]

@yurishkuro For ES Archive, it looks like the archive flag is used in two places:

• To choose the suffix for the index name (https://github.com/jaegertracing/jaeger/blob/main/plugin/storage/es/spanstore/reader.go#L167)
• In getSourceFn to add sorting and a search after clause if we're not querying the archive index (https://github.com/jaegertracing/jaeger/blob/main/plugin/storage/es/spanstore/reader.go#L211-L214)

For Cassandra, its a bit more straightforward:

• The only difference between the primary storage factory and the archive one is that the metrics factory is initialized with a different namespace (https://github.com/jaegertracing/jaeger/blob/main/plugin/storage/cassandra/factory.go#L137).

Do you have any thoughts on how we should proceed? Do we still want to expose an isArchive or setAsArchive flag?

[yurishkuro]

The metrics namespacing for Cassandra can be easily done elsewhere, does not need to be based on isArchive. It's only needed there now because primary/archive distinction is made internally in the factory.

For ES:

• "To choose the suffix for the index name" - not needed since the user can do that themselves. One significant change in v2 is that we cannot provide different defaults in the config for primary/archive, and having the same index prefix by mistake will be bad for the user. Maybe we can introduce additional validation for configs of the same type and catch that as a configuration error.
• "to add sorting and a search after clause if we're not querying the archive index" - I don't understand the purpose of that difference. Any ideas? Would it hurt if the logic for archive was the same as for primary?
• There is a 3rd, most important usage of isArchive - in the GetIndicesFn. That's the one where I wonder if we could replace isArchive with a different logic based on the lookback parameter.

[mahadzaryab1]

The metrics namespacing for Cassandra can be easily done elsewhere, does not need to be based on isArchive. It's only needed there now because primary/archive distinction is made internally in the factory.

With the setup of v2, how would we make that distinction?

For ES:

• "To choose the suffix for the index name" - not needed since the user can do that themselves. One significant change in v2 is that we cannot provide different defaults in the config for primary/archive, and having the same index prefix by mistake will be bad for the user. Maybe we can introduce additional validation for configs of the same type and catch that as a configuration error.

@yurishkuro Okay I see. But if the configurations are being held in different factories - how would we perform validation there?

• "to add sorting and a search after clause if we're not querying the archive index" - I don't understand the purpose of that difference. Any ideas? Would it hurt if the logic for archive was the same as for primary?

I was thinking the same as well. I'm guessing its an optimization to avoid sorting the archive storage which would be larger than the a non-archive storage. Here is the documentation for Search After. Would we have a performance degradation here if we enabled this for archive as well?

• There is a 3rd, most important usage of isArchive - in the GetIndicesFn. That's the one where I wonder if we could replace isArchive with a different logic based on the lookback parameter.

Ah yes. I was only looking at the reader. Are you referring to getSpanAndServiceIndexFn? This looks to once again be creating a different suffix based on whether the storage is archive or not (https://github.com/jaegertracing/jaeger/blob/main/plugin/storage/es/spanstore/writer.go#L104-L111). Can we not use the same approach here as the first point for the ES reader?

[yurishkuro]

With the setup of v2, how would we make that distinction?

the storage extension manages factories and knows storage names, it can use those names to bound MetricsFactory to have a specific label.

Okay I see. But if the configurations are being held in different factories - how would we perform validation there?

No, configuration are passed to factories, but held in a single place in storage extension, which can invoke additional validation here:

jaeger/cmd/jaeger/internal/extension/jaegerstorage/extension.go

Line 118 in a420fd9

for storageName, cfg := range s.config.Backends {

Would we have a performance degradation here if we enabled this for archive as well?

but the thing is, we never search for traces in archive storage, we only retrieve trace by ID, so sorting in this case would only apply to the spans within a trace - yes, could have overhead for very large trace, but still small.

[yurishkuro]

Are you referring to getSpanAndServiceIndexFn? This looks to once again be creating a different suffix

not just suffix, when it's primary storage with manually rotated indices they also have the date pattern in the name, but archive index never has that (because it doesn't grow large). One compromise we could do is recommend that users don't use archive storage with manually rotated indices, only with ILM. Unless there's another way that I am not seeing.

Btw, reader also has similar branching in index naming logic:

jaeger/plugin/storage/es/spanstore/reader.go

Line 174 in a420fd9

return addRemoteReadClusters(func(indexPrefix, _ /* indexDateLayout */ string, _ /* startTime */ time.Time, _ /* endTime */ time.Time, _ /* reduceDuration */ time.Duration) []string {

[mahadzaryab1]

With the setup of v2, how would we make that distinction?

the storage extension manages factories and knows storage names, it can use those names to bound MetricsFactory to have a specific label.

So this is where the Cassandra storage is initialized in the extension. Are you suggesting we can pass the label into the constructor here? If so, how would we make the distinction based on just the name?

Okay I see. But if the configurations are being held in different factories - how would we perform validation there?

No, configuration are passed to factories, but held in a single place in storage extension, which can invoke additional validation here:

Oh okay, I see. So whenever we're processing an ES config, we would go through all the other ones that exist and make sure that the index prefixes are not the same?

jaeger/cmd/jaeger/internal/extension/jaegerstorage/extension.go

Line 118 in a420fd9

for storageName, cfg := range s.config.Backends {

Would we have a performance degradation here if we enabled this for archive as well?

but the thing is, we never search for traces in archive storage, we only retrieve trace by ID, so sorting in this case would only apply to the spans within a trace - yes, could have overhead for very large trace, but still small.

Sounds good. I can remove the indirection here then.

[mahadzaryab1]

Are you referring to getSpanAndServiceIndexFn? This looks to once again be creating a different suffix

not just suffix, when it's primary storage with manually rotated indices they also have the date pattern in the name, but archive index never has that (because it doesn't grow large). One compromise we could do is recommend that users don't use archive storage with manually rotated indices, only with ILM. Unless there's another way that I am not seeing.

Btw, reader also has similar branching in index naming logic:

jaeger/plugin/storage/es/spanstore/reader.go

Line 174 in a420fd9

return addRemoteReadClusters(func(indexPrefix, _ /* indexDateLayout */ string, _ /* startTime */ time.Time, _ /* endTime */ time.Time, _ /* reduceDuration */ time.Duration) []string {

How would making that recommendation simplify the archive branching for us?

[mahadzaryab1]

@yurishkuro Regarding getSourceFn, the es archive integration tests seem to fail when the SearchAfter clause is added as it is unable to find the trace.