Skip to content
Snippets Groups Projects
Commit ff855b18 authored by Rastislav Kruták's avatar Rastislav Kruták Committed by Pavel Břoušek
Browse files

ci: added an MR bot running on schedule

- added a python script to be run periodically
- the script will crawl through projects and find projects waiting
for review and projects approved by enough people
- the summary of the search will be posted in configured slack channel
parent 2cb39dfd
Branches
No related tags found
1 merge request!4ci: added an MR bot running on schedule
Pipeline #405857 failed
...@@ -11,3 +11,12 @@ gitlab_event_bot: ...@@ -11,3 +11,12 @@ gitlab_event_bot:
- python gitlab_bots/gitlab_event_bot.py - python gitlab_bots/gitlab_event_bot.py
rules: rules:
- if: '$CI_PIPELINE_SOURCE == "trigger" && ($GITLAB_BOT_EVENT == "mr" || $GITLAB_BOT_EVENT == "comment")' - if: '$CI_PIPELINE_SOURCE == "trigger" && ($GITLAB_BOT_EVENT == "mr" || $GITLAB_BOT_EVENT == "comment")'
gitlab_scheduled_bot:
image: ${CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX}/bitnami/python:3.11
script:
- pip3 install python-gitlab~=4.4
- pip3 install slack-sdk~=3.27
- python gitlab_bots/gitlab_scheduled_bot.py
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
...@@ -13,4 +13,5 @@ Behavior of some jobs can be modified by CI variables, ...@@ -13,4 +13,5 @@ Behavior of some jobs can be modified by CI variables,
and all jobs can be modified by extending. and all jobs can be modified by extending.
## Gitlab bots ## Gitlab bots
This repository includes scripts to automate some merge request operations see [bots README](gitlab_bots/README.md).
\ No newline at end of file This repository includes scripts to automate some merge request operations see [bots README](gitlab_bots/README.md).
...@@ -21,7 +21,7 @@ The event bot is to be called from pipeline as reaction to webhooks and implemen ...@@ -21,7 +21,7 @@ The event bot is to be called from pipeline as reaction to webhooks and implemen
- All discussions resolved -> remove the requested changes label - All discussions resolved -> remove the requested changes label
- MR approved -> resolve all unresolved threads opened by the approver - MR approved -> resolve all unresolved threads opened by the approver
### Configuration ### Event bot configuration
Following webhooks need to be set up in gitlab: Following webhooks need to be set up in gitlab:
...@@ -37,9 +37,59 @@ Where: ...@@ -37,9 +37,59 @@ Where:
The pipeline then needs to be set up to just run the `gitlab_event_bot` script. The pipeline then needs to be set up to just run the `gitlab_event_bot` script.
In code set the In code set the `GITLAB_BASE_URL` constant to the url of the gitlab instance e.g. `https://gitlab.ics.muni.cz`
`GITLAB_BASE_URL` constant to the url of the gitlab instance e.g. `https://gitlab.ics.muni.cz`
## Scheduled bot
A script meant to be called periodically, will crawl through the configured repositories
and create a summary of merge requests:
- Missing a review - MRs that do not yet have the specified amount of reviewers sorted from the oldest
- (Possibly) ready to be merged - MRs that have at least the specified amount of reviewers, where all the
reviewers have approved the MR and all discussions have been resolved
The formatted summary will be sent as a message to the configured Slack channels.
### Scheduled bot configuration
First a Slack app representing this bot needs to be created with the scopes `app_mentions:read` and
`chat:write`. The minimal Slack app config can then look as follows:
```json
{
"display_information": {
"name": "MR bot"
},
"oauth_config": {
"scopes": {
"bot": [
"app_mentions:read",
"chat:write"
]
}
}
}
```
More information about the Slack app setup to be found [here](https://api.slack.com/tutorials/tracks/getting-a-token).
After creating the app set the Slack access token as environment variable `SLACK_TOKEN`.
Following constants then need to be set in the code:
- `GITLAB_BASE_URL` the url of the gitlab instance e.g. `https://gitlab.ics.muni.cz`
- `ROOT_GROUP_IDENTIFIER` the name of the group from which the script will recursively start its
crawl through the projects e.g. `perun`
- `CHANNEL_FILTERS_DICTIONARY` defines the mappings of type `channel -> filter` where `channel` is the name of the Slack channel to send the notifications about the MRs
fulfilling the conditions set by the `filter` to. The `filters` are either:
- List of regular expressions `[regex1, regex2, regex3...]` where each projects on
path matching at least one of the regexes are satisfied by the filter (meaning its MRs are checked).
- List of one regular expression starting with `!` like `[!regex]` where only projects
on path not matching the regex will be included.
- `WAITING_FOR_REVIEW_1_LABEL` the value of the label marking the MR as needing 1 review
- `WAITING_FOR_REVIEW_2_LABEL` the value of the label marking the MR as needing 2 reviews
- `NEEDED_REVIEWS_DEFAULT` how many reviews are needed by default if no label with this information is present
### Dependencies ### Dependencies
- [python-gitlab](https://python-gitlab.readthedocs.io/en/stable/index.html) - [python-gitlab](https://python-gitlab.readthedocs.io/en/stable/index.html)
- [slack-sdk](https://pypi.org/project/slack-sdk/)
from datetime import datetime
import gitlab
import os
import logging
import slack_sdk
import re
GITLAB_BASE_URL = "https://gitlab.ics.muni.cz"
# group to start the crawl at (name or id)
ROOT_GROUP_IDENTIFIER = "perun"
# key is the name of the Slack channels to send the notifications to
# value is a list of filters (regex) to choose to which projects to apply, use ! to negate the regex
CHANNEL_FILTERS_MAP = {"stribog_reviews": [".*test_subproj1.*", ".*test-subproj-2.*"],
"svarog_reviews": ["!.*test_group.*"]}
WAITING_FOR_REVIEW_1_LABEL = "waiting for review::1"
WAITING_FOR_REVIEW_2_LABEL = "waiting for review::2"
# reviews needed by default
NEEDED_REVIEWS_DEFAULT = 2
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
channel_mrs_all_done_map = {channel: [] for channel in CHANNEL_FILTERS_MAP.keys()}
channel_mrs_waiting_for_review_map = {channel: [] for channel in CHANNEL_FILTERS_MAP.keys()}
def is_service_account(name):
return False if name is None \
else bool(re.compile(r'service', re.IGNORECASE).search(name))
def path_match(path, regex):
return not bool(re.match(regex[1:], path, re.IGNORECASE)) \
if len(regex) > 0 and regex[0] == "!" \
else bool(re.match(regex, path, re.IGNORECASE))
def matching_channels(project_path):
matched_channels = set()
for channel, regexes in CHANNEL_FILTERS_MAP.items():
for regex in regexes:
if path_match(project_path, regex):
matched_channels.add(channel)
return matched_channels
def check_mrs(project_id, matched_channels):
if len(matched_channels) == 0:
return
project = gl.projects.get(project_id, lazy=True)
mrs = project.mergerequests.list(state='opened')
for mr in mrs:
if is_service_account(mr.author.get('name')):
continue
try:
reviews_needed = NEEDED_REVIEWS_DEFAULT
if WAITING_FOR_REVIEW_1_LABEL in mr.labels:
reviews_needed = 1
if WAITING_FOR_REVIEW_2_LABEL in mr.labels:
reviews_needed = 2
approver_ids = set(map(lambda approver: approver["user"]["id"], mr.approvals.get().approved_by))
reviewer_ids = set(map(lambda user: user["id"], mr.reviewers))
all_resolved = all(all(not note["resolvable"] or note["resolved"]
for note in discussion.attributes["notes"])
for discussion in mr.discussions.list(all=True))
if len(approver_ids) >= reviews_needed and reviewer_ids.issubset(approver_ids) and all_resolved:
for channel in matched_channels:
channel_mrs_all_done_map[channel].append(mr)
if len(approver_ids.union(reviewer_ids)) < reviews_needed:
for channel in matched_channels:
channel_mrs_waiting_for_review_map[channel].append(mr)
except Exception as ex:
logger.error(f"Checking the mr {mr.id} failed with {ex}")
def construct_slack_message(channel):
msg = ""
mrs_waiting_for_review = channel_mrs_waiting_for_review_map[channel]
mrs_all_done = channel_mrs_all_done_map[channel]
if len(mrs_waiting_for_review) > 0:
msg += "Following MRs are waiting for review:\n"
for mr in mrs_waiting_for_review:
mr.created_at = datetime.strptime(mr.created_at, '%Y-%m-%dT%H:%M:%S.%fZ')
for mr in sorted(mrs_waiting_for_review, key=lambda merge_req: merge_req.created_at):
days_since_opened = (datetime.now() - mr.created_at).days
msg += (f"<{mr.web_url}|{mr.title}> - opened {'<' if days_since_opened < 1 else ''}{days_since_opened}"
f" day{'s' if days_since_opened != 1 else ''} ago"
f" with {len(mr.reviewers)} review{'s' if len(mr.reviewers) != 1 else ''} so far\n")
msg += "\n"
if len(mrs_all_done) > 0:
msg += "Following MRs have necessary amount of reviewers and have been approved by them all:\n"
for mr in mrs_all_done:
msg += f"<{mr.web_url}|{mr.title}>\n"
return msg
def crawl_projects(group):
group = gl.groups.get(group.id)
projects = group.projects.list(all=True)
subgroups = group.subgroups.list(all=True)
for project in projects:
matched_channels = matching_channels(project.path_with_namespace)
check_mrs(project.id, matched_channels)
for subgroup in subgroups:
crawl_projects(subgroup)
access_token_gl = os.environ.get("GL_TOKEN") \
if (os.environ.get("GL_TOKEN") is not None) \
else os.environ.get("GITLAB_TOKEN")
if access_token_gl is None:
logger.error("Required env variable GL_TOKEN or GITLAB_TOKEN was not found.")
exit(1)
access_token_slack = os.environ.get("SLACK_TOKEN")
if access_token_slack is None:
logger.error("Required env variable SLACK_TOKEN was not found.")
exit(1)
gl = None
root_group = None
try:
gl = gitlab.Gitlab(GITLAB_BASE_URL, private_token=access_token_gl)
root_group = gl.groups.get(ROOT_GROUP_IDENTIFIER)
except Exception as e:
logger.error(f"Fetching the gitlab client and the root group failed with {e}")
exit(1)
slack_client = None
try:
slack_client = slack_sdk.WebClient(token=access_token_slack)
except Exception as e:
logger.error(f"Fetching the slack client failed with {e}")
exit(1)
crawl_projects(root_group)
# mapping between Slack channel name and the message to send to that channel
channel_msg_map = {channel: construct_slack_message(channel)
for channel in CHANNEL_FILTERS_MAP.keys()}
for channel, message in channel_msg_map.items():
if message == "":
continue
try:
slack_client.chat_postMessage(channel=channel, text=message)
except Exception as e:
print(f"Sending the slack message to channel {channel} failed with {e}")
exit(1)
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment