Store events in a Redis stream instead of a list #8
Conversation
jjmerchante
force-pushed
the
redis-stream
branch
2 times, most recently
from
153f64b to
6e5e0d9
Compare
jjmerchante
force-pushed
the
redis-stream
branch
4 times, most recently
from
a962b7d to
7cb5cee
Compare
jjmerchante
force-pushed
the
redis-stream
branch
from
7cb5cee to
3843158
Compare
jjmerchante
force-pushed
the
redis-stream
branch
from
3843158 to
7a47edc
Compare
jjmerchante
force-pushed
the
redis-stream
branch
from
7a47edc to
6365565
Compare
|
I updated the PR to include only the change that inserts data in the stream. I also removed the file grimoirelab-dev that was not being used because the contents were moved to runner/commands/run/ |
| @@ -83,6 +81,8 @@ def server(ctx: Context, devel: bool): | |||
| env["UWSGI_SINGLE_INTERPRETER"] = "true" | |||
|
|
|||
| # Run maintenance tasks | |||
| from grimoirelab.core.scheduler.scheduler import maintain_tasks | |||
There was a problem hiding this comment.
Why did you have to move this statement here?
There was a problem hiding this comment.
Because otherwise, it doesn't work. It has to initialize the Django module first, which is done here: https://github.com/chaoss/grimoirelab-core/blob/main/src/grimoirelab/core/runner/cmd.py#L58
If you try to do the import there you get django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
| Q_PERCEVAL_JOBS = os.environ.get('GRIMOIRELAB_Q_PERCEVAL_JOBS', 'default') | ||
| Q_STORAGE_ITEMS = os.environ.get('GRIMOIRELAB_Q_STORAGE_ITEMS', 'items') | ||
| Q_EVENTS = os.environ.get('GRIMOIRELAB_Q_EVENTS', 'events') | ||
| Q_EVENTIZER_JOBS = os.environ.get('GRIMOIRELAB_Q_EVENTIZER_JOBS', 'default') |
There was a problem hiding this comment.
I think we can use the prefix GRIMOIRELAB_ to all these variables that can be configured to make it clear they are part of this app and don't come from other.
| } | ||
|
|
||
| EVENTS_STREAM_NAME = os.environ.get('GRIMOIRELAB_EVENTS_STREAM_NAME', 'events') | ||
| EVENTS_STREAM_MAX_LENGTH = int(os.environ.get('GRIMOIRELAB_EVENTS_STREAM_MAX_LENGTH', 2 * 10 ** 6)) |
There was a problem hiding this comment.
This looks like a magic number. What is it and why is that the length?
There was a problem hiding this comment.
This is the stream's limit. Once the stream reaches this capacity, new items will replace older ones. On average, each Git event consumes approximately 2.6 Kb (calculated using MEMORY USAGE). While this size can be reduced, it is important for consumers to read from the stream before it reaches its limit; otherwise, items will be discarded.
I can lower this number, but it depends on the memory of the Redis server.
jjmerchante
force-pushed
the
redis-stream
branch
from
a88b68f to
175abeb
Compare
This commit updates the method of storing events in Redis for later consumption. Events are now added to a Redis Stream, allowing different types of consumers to process the items for various purposes. The trade-off is that the stream has a fixed length, causing older items to be deleted when new items are added. Signed-off-by: Jose Javier Merchante <[email protected]>
jjmerchante
force-pushed
the
redis-stream
branch
from
175abeb to
14084fd
Compare
This PR updates the method of storing events in Redis for later consumption.
Events are now added to a Redis Stream, allowing different types of consumers to process the items for various purposes. The trade-off is that the stream has a fixed length, causing older items to be deleted when new items are added.