Repo and commit id whitelisting stopped working

[sagarvsh]

Hi @SimeonCloutier @denniskennedy and Team,

I quickly wanted to reach out to your guys and seek suggestions. We are using sedated extensively in our organization and we started facing issue from last Friday, where repo and commit id whitelisting stopped working. I had about 850 commit id's and 50 repos been whitelisted, but now new commit id's or repo's whitelisting does not take effect. As a pre-receive hook, its working fine but when something is blocked, unable to whitelist to move forward.

Note: We use whitelisting repo to get around the 5sec rule issue. When teams try to push a big change, they usually get blocked.

Can you help me troubleshoot this issue further. Thank you,

Sagar

[sclouts]

@sagarvsh Thanks for reaching out and sorry to hear you are facing this issue. There are a number of both internal and external factors that could be factoring into what you are experiencing. Can you confirm a few things:

1. What type of environment are you running in (github, gitlab, etc...)?
2. When you say you are unable to whitelist, can you elaborate on exactly what the end user experience is and what exactly do they see after you attempt to whitelist?
3. What version of SEDATED are you running (we just released v 1.2.0 last week)?
4. If you were to create a new empty repo, and just do a basic commit (no pull request or anything like that), and then whitelist that commit, does it work in this very basic scenario?
5. Is all whitelisting broken or just some?

[sagarvsh]

Hi @SimeonCloutier

Thank for your quick response. As I have troubleshooted more, it's taking longer time for the whitelisting changes to get reflected and not instant anymore. Below are the details.

1. What type of environment are you running in (github, gitlab, etc...)?
   GitHub 2.20.8

2. When you say you are unable to whitelist, can you elaborate on exactly what the end user experience is and what exactly do they see after you attempt to whitelist?
   This is especially an issue in 2 scenarios:
   a. When the sensitive data is found and push is blocked. Commit ID is whitelisted if its false positive to complete the push. After its whitelisted, user have to wait anywhere between 15 mins to an hour before they can push successfully.
   b. When developer is trying to push a bigger change and end up with "remote: pre-receive.sh: execution exceeded 5s timeout. Repo is temporarily whitelisted, but have to wait for 15 mins to 1 hour, to successful push.

This was instant earlier.

3. What version of SEDATED are you running (we just released v 1.2.0 last week)?
   v1.1.3

4. If you were to create a new empty repo, and just do a basic commit (no pull request or anything like that), and then whitelist that commit, does it work in this very basic scenario?
   yes

5. Is all whitelisting broken or just some?
   Only the latests one.

[sclouts]

@sagarvsh Thanks for the additional info. Sounds like whitelisting is still working for you but just very slow and delayed. I have some thoughts, let me know your outcome on the below.

Github has a replication service that occurs every time you make a change to the SEDATED® repo (ie. whitelisting file). Sometimes this service can be slow or delayed. In lieu of this, and how we created SEDATED®, there are some things you should check.

• Is your github instance experiencing abnormally degraded performance. Have your github admins do a general health check and engage Github support as necessary.

• Were there any recent changes made to your Github instance(s) and/or supporting networks that could potentially be interfering and causing this slowness.

• Because SEDATED® is replicated, it's important to keep the repo and it's contents to a minimum. Do not store any additional files (text, images, binary files, etc..) within the SEDATED® repo as this could potentially interfere with Github's ability to quickly replicate the repo.

Lastly, coincidently we experienced a similar issue a couple weeks ago. Our whitelisting was taking nearly 24 hours to be reflected for a couple days. We engaged Github support but didn't receive much insight, however we did happen to notice 1 thing in particular and it resolved the issue for us. In the github admin console for our SEDATED® repo, there is an option to reindex the repos (screen shot below). Prior to doing this, we engaged Github support to better understand what happens when doing this, afterwhich we did the reindex and everything started working again just fine (without delays or slowness) and has been fine since then. So this is certainly something you can check into as well however please understand I am not a github admin expert or anything so I do recommend you get the necessary consulting/advice before taking this step.

Let us know how you make out!

[sagarvsh]

Hi @SimeonCloutier

The issue actually died down yesterday without any action, so keeping a close eye if this issue reoccur. On your analysis, yes, we see a degraded performance in GitHub, that might be the root cause of this issue. I am keeping the repo reindex option handy and will try it when this issue reoccur. I have reached out to GitHub Support to ensure there is no downside for this task.

Again, thank you very much for your quick support and response. Will keep you updated on how we progress.

[sclouts]

@sagarvsh Great to hear! Keep us posted if the issues surfaces again and your outcome. We've only experienced this once in the many years we've been running SEDATED® so hopefully you don't find this is an outgoing issue.

[sagarvsh]

@SimeonCloutier

Encountered the issue again today, found that the issue was linked with GitHub replication issue. When the process of replication had issue, sedated whitelisting also had issues. As soon as the replication issue was resolved, the sedated whitelisting feature was back functioning. Is there any relationship with sedated repo being not replicated to replication site.

[sagarvsh]

This is what GitHub shared, might be useful if others face similar issue. Thank you,

Response from GitHub Support:
As the pre-receive hook scripts and configuration files are in a repository which gets replicated, replication issues could impact your pre-receive hooks. While we haven't directly worked with SEDATED, we have seen replication issues cause slowness and degradation in other tickets.

[sclouts]

@sagarvsh In that case, if the Github server is not replicating properly then yes it would impact SEDATED white/repo list updates. Ok, that is properly something like a "Known limitation/issue" of sorts and we could maybe create an FAQ and include that. I'll get together with @denniskennedy on this. Thanks for the info.