Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove CentOS and RedHat from the Helix environment #3189

Closed
1 of 3 tasks
v-zheyan opened this issue Jun 27, 2024 · 7 comments
Closed
1 of 3 tasks

Remove CentOS and RedHat from the Helix environment #3189

v-zheyan opened this issue Jun 27, 2024 · 7 comments
Assignees
@dougbu
Labels
Ops - Compliance Operational issues related to MSFT process compliance. Ops - Spike Work items to be included in our Ops Spike

Comments

@v-zheyan
Copy link
Member

v-zheyan commented Jun 27, 2024 •
edited by dougbu
Loading

CentOS and RedHat meet the EOL date on 2024-06-30, once the customers stop using them, they should be removed.

Release Note Category

  • Feature changes/additions
  • Bug fixes
  • Internal Infrastructure Improvements

Release Note Description

Removed the queues for CentOS and RedHat i.e., all centos.7.amd64* and redhat.7.amd64* queues.
@v-zheyan v-zheyan added the Ops - Compliance Operational issues related to MSFT process compliance. label Jun 27, 2024
@ilyas1974
Copy link
Contributor

We need to work with the product teams to make sure workloads are moved to other queues before these are removed.

@dougbu
Copy link
Member

dougbu commented Jul 3, 2024

CentOS 7 is in a worse state than RedHat 7 b/c RedHat at least has an extended support option for RedHat 7. CentOS 7 is completely EOL as of 30 June '24. I'm pretty sure we can't create CentOS 7 images anymore. Will test…

A few details

Performance, runtime, and wcf runtimes still use the centos7.amd64[.open], even over the past week.

See Teams conversation and Delete CentOS 7 Dockerfiles PR reasoning. Note workarounds for missing package repo mirror choice site don't make new package updates available for the OS.

@dougbu
Copy link
Member

dougbu commented Jul 6, 2024

I'm pretty sure we can't create CentOS 7 images anymore. Will test…

Turns out image generation has no problems installing CentOS 7 packages though one of the RedHat changes to "incentivize" moving off that OS was to disable the service used to choose between package repository mirrors. See !40896, success of MakeImage operations in #20240706.01, and agent logs in Image Factory run 9b16eda7-4b1b-5f13-94c9-0592dab9de75.

In addition, execute in [Web] [Desktop] [cluster('engsrvprod.kusto.windows.net').database('engineeringdata')]

Jobs
| where QueueAlias has "centos" and Finished > ago(5d)
| project JobId, Source, Type, Queued, Started, Finished, QueueName, Repository, Branch

shows numerous repositories and their pipelines continue to test on CentOS queues.

@ilyas1974 ilyas1974 added Ops - P1 Operations task, priority 1 (highest priority) Ops - Spike Work items to be included in our Ops Spike and removed Ops - P1 Operations task, priority 1 (highest priority) labels Jul 24, 2024
@dougbu dougbu self-assigned this Aug 30, 2024
@dougbu
Copy link
Member

dougbu commented Sep 28, 2024

Above Kusto query mixed up QueueAlias and QueueName. Correct version is
Execute in [Web] [Desktop] [cluster('engsrvprod.kusto.windows.net').database('engineeringdata')]

Jobs
| where QueueName has "centos" and Finished > ago(30d)
| project JobId, Source, Type, QueueName, Repository, Branch, Queued, Started, Finished

or, for redhat:
Execute in [Web] [Desktop] [cluster('engsrvprod.kusto.windows.net').database('engineeringdata')]

Jobs
| where QueueName has "RedHat" and Finished > ago(30d)
| project JobId, Source, Type, QueueName, Repository, Branch, Queued, Started, Finished

Current data shows performance team has cleaned up their remaining use of the CentOS queue but RedHat use continues in at least aspnetcore (for release/6.0), arcade, and runtime pipelines.

Not sure how to resolve this w/o another discussion w/ .NET PMs or waiting for @ilyas1974 to return. One idea might be to remove the CentOS queue and start a conversation about RedHat. @garath and @markwilkie what are your thoughts❓

dougbu added a commit to dougbu/arcade that referenced this issue Oct 22, 2024
@dougbu
[release/6.0] Test on CentOS 7, not RedHat 7
30af457 
- dotnet/dnceng#4298 part 1/II
  - contributes to dotnet/dnceng#3189
- use Docker container for CentOS, mimicking dotnet/runtime#105063
@dougbu dougbu mentioned this issue Oct 22, 2024
Merged
1 task
dougbu added a commit to dougbu/aspnetcore that referenced this issue Oct 22, 2024
@dougbu
[release/6.0] Test on CentOS 7, not RedHat 7
bb38858 
- dotnet/dnceng#4298 part II/II
  - contributes to dotnet/dnceng#3189
- use Docker container for CentOS, mimicking dotnet/runtime#105063
@dougbu dougbu mentioned this issue Oct 22, 2024
Merged
4 tasks
dougbu added a commit to dougbu/aspnetcore that referenced this issue Oct 22, 2024
@dougbu
[release/6.0] Test on CentOS 7, not RedHat 7
97577c8 
- dotnet/dnceng#4298 part II/II
  - contributes to dotnet/dnceng#3189
- use Docker container for CentOS, mimicking dotnet/runtime#105063
dougbu added a commit to dotnet/arcade that referenced this issue Oct 22, 2024
@dougbu
[release/6.0] Test on CentOS 7, not RedHat 7 (#15193)
f74981a 
- dotnet/dnceng#4298 part I/II
  - contributes to dotnet/dnceng#3189
- use Docker container for CentOS, mimicking dotnet/runtime#105063
@dougbu
Copy link
Member

dougbu commented Oct 28, 2024

temporarily blocked while dotnet/aspnetcore#58561 is blocked

wtgodbe pushed a commit to dotnet/aspnetcore that referenced this issue Nov 5, 2024
@dougbu
[release/6.0] Test on CentOS 7, not RedHat 7 (#58561)
e185938 
- dotnet/dnceng#4298 part II/II
  - contributes to dotnet/dnceng#3189
- use Docker container for CentOS, mimicking dotnet/runtime#105063
@dougbu
Copy link
Member

dougbu commented Nov 7, 2024

all resolved. just waiting for #4334 fix to roll out

@dougbu
Copy link
Member

dougbu commented Nov 12, 2024

all done

@dougbu dougbu closed this as completed Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dougbu dougbu

Labels
Ops - Compliance Operational issues related to MSFT process compliance. Ops - Spike Work items to be included in our Ops Spike
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dougbu @ilyas1974 @v-zheyan