[WIP] Add monocleaner #991
Conversation
| echo "Threshold is 0, skipping filtering" | ||
| cp "${output_prefix}.${lang}.rule-based.zst" "${output_prefix}.${lang}.zst" | ||
| else | ||
| # the model is 125MB, similar in size to the fastText one, so it's ok to download it here |
There was a problem hiding this comment.
@bhearsum do you think it's still worth extracting this download to a separate task? I see the models are 100-200Mb, similar to the fastText ones. Extracting would save on some ingress traffic, but would complicate the pipeline and model updates, similar to the bilceaner ones. The bicleaner models are a lot larger, so it's justified to cache them.
There was a problem hiding this comment.
I have no objections. IMO the more important downside is that not caching these means that you'll be more susceptible to failures when wherever is hosting them is down.
(And I could be wrong, but it looks like ingress is free)
There was a problem hiding this comment.
Ok, I'll skip it for now then. Let's see if we have issues with that. Github is hosting them, so I guess it should be reliable.
Let's start testing it first, then we can update docs with recommendations based on results.
closes #247
closes #476
closes #985
closes #789