Skip to content
This repository has been archived by the owner on Jun 24, 2024. It is now read-only.

Import resiliency (3/3) #1385

Merged
merged 35 commits into from
Jul 12, 2018
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
9c9914b
Fixed actioncable consumer logic
BenEmdon Jul 3, 2018
f8401ee
Added custom job queues
BenEmdon Jul 5, 2018
f59a44d
Configured Redis for ActionCable
BenEmdon Jul 5, 2018
6ef3480
Add status column to invitation
BenEmdon Jul 6, 2018
83d09b1
Broadcast hash structured objects
BenEmdon Jul 6, 2018
b416808
Added :pending status and made status readable
BenEmdon Jul 6, 2018
9f8a77c
Changed invitation status to an ActiveRecord enum
BenEmdon Jul 6, 2018
10f21be
Added new behavior behind beta flag
BenEmdon Jul 7, 2018
2c602f1
added progress route
BenEmdon Jul 7, 2018
2512e81
Added front end with polling
BenEmdon Jul 9, 2018
baeca08
gave progress route the ability to retry the job on error
BenEmdon Jul 9, 2018
90bf12d
Update broacasting with status field
BenEmdon Jul 9, 2018
e6fa568
Fixed create_github_repository_job spec by adding status to broadcasts
BenEmdon Jul 9, 2018
54d035c
Added #create route to kick off repo creation
BenEmdon Jul 10, 2018
5178c4f
Added setupv2 spec
BenEmdon Jul 10, 2018
2de8ff1
Please the hound
BenEmdon Jul 10, 2018
e2e5774
Fix cassets
BenEmdon Jul 10, 2018
33ae03e
Merge branch 'master' into new-repo-creation-flow
BenEmdon Jul 10, 2018
7d0a98e
Fix cassets (fr this time)
BenEmdon Jul 10, 2018
2514169
Please the hound
BenEmdon Jul 10, 2018
addbf3d
Please hound
BenEmdon Jul 10, 2018
109a9e0
Removed extra newlines
BenEmdon Jul 11, 2018
3fb3909
Added extra stat recording
BenEmdon Jul 11, 2018
3ab5158
Added specs for stat reporting
BenEmdon Jul 11, 2018
3cde0ff
Please the hound
BenEmdon Jul 11, 2018
bbb7eda
Roll back migration in favor of #1386
BenEmdon Jul 11, 2018
15a110d
Merge branch 'master' into new-repo-creation-flow
BenEmdon Jul 12, 2018
6700963
Merge branch 'master' into new-repo-creation-flow
BenEmdon Jul 12, 2018
eb1d7c9
Remove redundant poll
BenEmdon Jul 12, 2018
0ebd551
Moved poll interval out into a constant
BenEmdon Jul 12, 2018
945f5de
Change #create to #create_repo
BenEmdon Jul 12, 2018
7a06406
Merge branch 'master' into new-repo-creation-flow
BenEmdon Jul 12, 2018
ca3081f
Revert to ES5
BenEmdon Jul 12, 2018
3c6ecf4
Remove eslint rule
BenEmdon Jul 12, 2018
cc81287
Hit /create_repo from js
BenEmdon Jul 12, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion app/assets/javascripts/cable.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
//= require_tree ./channels

(function() {
if (this.App) this.App = {};
if (!this.App) this.App = {};

App.cable = ActionCable.createConsumer();

Expand Down
134 changes: 134 additions & 0 deletions app/assets/javascripts/setupv2.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
/*jshint esversion: 6 */
(function() {
var ready, check_progress, show_success, invitation_path, progress_path, success_path, display_progress, indicate_completion, indicate_failure, indicate_in_progress;

indicate_completion = function(step_indicator) {
$(step_indicator).addClass("border-green bg-green-light");

var status, completion_indicator;
status = $(step_indicator).find(".status");
if (status.hasClass("d-none")) {
status.removeClass("d-none");
}

completion_indicator = $(status).find(".complete-indicator");
failure_indicator = $(status).find(".failure-indicator");
spinner_indicator = $(status).find(".spinner");

$(completion_indicator).show();
$(failure_indicator).hide();
$(spinner_indicator).hide();
};

indicate_failure = function(step_indicator) {
$(step_indicator).addClass("border-red bg-red-light");

var status, failure_indicator;
status = $(step_indicator).find(".status");
if (status.hasClass("d-none")) {
status.removeClass("d-none");
}

completion_indicator = $(status).find(".complete-indicator");
failure_indicator = $(status).find(".failure-indicator");
spinner_indicator = $(status).find(".spinner");

$(failure_indicator).show();
$(completion_indicator).hide();
$(spinner_indicator).hide();
};

indicate_in_progress = function(step_indicator) {
let status;
status = $(step_indicator).find(".status");
if (status.hasClass("d-none")) {
status.removeClass("d-none");
}

completion_indicator = $(status).find(".complete-indicator");
failure_indicator = $(status).find(".failure-indicator");
spinner_indicator = $(status).find(".spinner");

$(step_indicator).addClass("border-green");

$(spinner_indicator).show();
$(failure_indicator).hide();
$(completion_indicator).hide();
};

display_progress = function(progress) {
let create_repo_progress_indicator = $("#create-repo-progress");
let import_repo_progress_indicator = $("#import-repo-progress");
switch(progress.status) {
case "creating_repo":
indicate_in_progress(create_repo_progress_indicator);
break;
case "importing_starter_code":
indicate_completion(create_repo_progress_indicator);
indicate_in_progress(import_repo_progress_indicator);
break;
case "errored":
indicate_failure(create_repo_progress_indicator);
indicate_failure(import_repo_progress_indicator);
break;
case "completed":
indicate_completion(create_repo_progress_indicator);
indicate_completion(import_repo_progress_indicator);
setTimeout(show_success, 500);
break;
}
};

success_path = function () {
return invitation_path() + "/success";
};

progress_path = function () {
return invitation_path() + "/progress";
};

job_path = function () {
return invitation_path();
};

invitation_path = function () {
const pathname = window.location.pathname;
let path_components = pathname.split("/");
path_components.pop();
return path_components.join("/");
};

show_success = function() {
location.href = success_path();
};

check_progress = function() {
const path = progress_path();

$.ajax({type: "GET", url: path}).done(function(data) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style nit, do we need a newline here?

display_progress(data);
setTimeout(check_progress, 3000);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS here looks good 👍

The only thing I'd suggest is abstracting the two uses of 3000 out to a constant at the top of the file so that if we want to adjust the polling rate in the future, we don't need to go through the whole file looking for where it is.

});
};

start_job = function() {
const path = job_path();

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same style nit here

$.ajax({type: "POST", url: path}).done(function(data) {

display_progress(data);
setTimeout(check_progress, 3000);
});
};

ready = (function() {
const setup_progress = $(".setupv2");
if (setup_progress.length !== 0) {
start_job();
check_progress();
}
});

$(document).ready(ready);
}).call(this);
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,9 @@ def repo_setup_enabled?
logged_in? && current_user.feature_enabled?(:repo_setup)
end
helper_method :repo_setup_enabled?

def import_resiliency_enabled?
logged_in? && current_user.feature_enabled?(:import_resiliency)
end
helper_method :import_resiliency_enabled?
end
74 changes: 68 additions & 6 deletions app/controllers/assignment_invitations_controller.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# frozen_string_literal: true

# rubocop:disable ClassLength
class AssignmentInvitationsController < ApplicationController
include InvitationsControllerMethods
include RepoSetup
Expand All @@ -8,19 +9,79 @@ class AssignmentInvitationsController < ApplicationController
before_action :ensure_submission_repository_exists, only: %i[setup setup_progress success]
before_action :ensure_authorized_repo_setup, only: %i[setup setup_progress]

# rubocop:disable PerceivedComplexity
# rubocop:disable MethodLength
# rubocop:disable AbcSize
# rubocop:disable CyclomaticComplexity
def accept
create_submission do
GitHubClassroom.statsd.increment("exercise_invitation.accept")
if current_submission.starter_code_repo_id
redirect_to setup_assignment_invitation_path
else
redirect_to success_assignment_invitation_path
if import_resiliency_enabled?
result = current_invitation.redeem_for(current_user, import_resiliency: import_resiliency_enabled?)
case result.status
when :success
GitHubClassroom.statsd.increment("exercise_invitation.accept")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of re-using the old metric key here, what do you think about using a new one? v2_excercise_invitation.accept, for example.

This'll give us a little bit more granularity in our metrics when trying to see how the new flow is performing.

if current_submission.starter_code_repo_id
redirect_to setup_assignment_invitation_path
else
current_invitation.completed!
redirect_to success_assignment_invitation_path
end
when :pending
GitHubClassroom.statsd.increment("exercise_invitation.accept")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment about metric granularity here

redirect_to setupv2_assignment_invitation_path
when :error
GitHubClassroom.statsd.increment("exercise_invitation.fail")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment about metric granularity here

current_invitation.errored!

flash[:error] = result.error
redirect_to assignment_invitation_path(current_invitation)
end
else
create_submission do
GitHubClassroom.statsd.increment("exercise_invitation.accept")
if current_submission.starter_code_repo_id
redirect_to setup_assignment_invitation_path
else
redirect_to success_assignment_invitation_path
end
end
end
end
# rubocop:enable PerceivedComplexity
# rubocop:enable MethodLength
# rubocop:enable AbcSize
# rubocop:enable CyclomaticComplexity

def setup; end

def setupv2
render status: 404 unless import_resiliency_enabled?
end

# rubocop:disable MethodLength
def create
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused by this route, what's happening here? By Rails convention, a create action in the AssignmentInvitationsController would be creating and saving an assignment invitation, but that doens't look like what's happening here.

Copy link
Contributor Author

@BenEmdon BenEmdon Jul 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. create doesn't make sense. What are your thoughts on these actions instead:

patch :create_assignment
patch :create_repo
patch :create_assignment_repo

if import_resiliency_enabled?
job_started = false
if current_invitation.accepted? || current_invitation.errored?
AssignmentRepo::CreateGitHubRepositoryJob.perform_later(current_assignment, current_user)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we ensure that the job is a unique job before we kick it off? Since jobs retry with exponential backoff we could end up in a situation where another job is kicked off while another is about to be retired.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, yes.

Copy link
Contributor Author

@BenEmdon BenEmdon Jul 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default Sidekiq doesn't have this feature. It does in enterprise, but we aren't on that licensed plan.

Another option would be to use mhenrixon/sidekiq-unique-jobs. This would require us to move away from ActiveJob and to using straight Sidekiq.

Possibly worth investigating in a follow up PR.

job_started = true
end
render json: {
job_started: job_started
}
else
render status: 404, json: {}
end
end
# rubocop:enable MethodLength

def progress
if import_resiliency_enabled?
render json: { status: current_invitation.status }
else
render status: 404, json: {}
end
end

def setup_progress
perform_setup(current_submission, classroom_config) if configurable_submission?

Expand Down Expand Up @@ -104,3 +165,4 @@ def required_scopes
GitHubClassroom::Scopes::ASSIGNMENT_STUDENT
end
end
# rubocop:enable ClassLength
36 changes: 23 additions & 13 deletions app/jobs/assignment_repo/create_github_repository_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -19,32 +19,25 @@ class CreateGitHubRepositoryJob < ApplicationJob
def perform(assignment, user)
start = Time.zone.now

assignment.invitation&.creating_repo!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this method defined? I wasn't able to find it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah it's an enum helper, nevermind!


creator = Creator.new(assignment: assignment, user: user)

creator.verify_organization_has_private_repos_available!

ActionCable.server.broadcast(
RepositoryCreationStatusChannel.channel(user_id: user.id),
text: CREATE_REPO
text: CREATE_REPO,
status: assignment.invitation&.status
)

assignment_repo = assignment.assignment_repos.build(
github_repo_id: creator.create_github_repository!,
user: user
)

ActionCable.server.broadcast(
RepositoryCreationStatusChannel.channel(user_id: user.id),
text: ADDING_COLLABORATOR
)

creator.add_user_to_repository!(assignment_repo.github_repo_id)

ActionCable.server.broadcast(
RepositoryCreationStatusChannel.channel(user_id: user.id),
text: IMPORT_STARTER_CODE
)

creator.push_starter_code!(assignment_repo.github_repo_id) if assignment.starter_code?

begin
Expand All @@ -56,12 +49,29 @@ def perform(assignment, user)
duration_in_millseconds = (Time.zone.now - start) * 1_000
GitHubClassroom.statsd.timing("exercise_repo.create.time", duration_in_millseconds)

PorterStatusJob.perform_later(assignment_repo, user)
if assignment.starter_code?
assignment.invitation&.importing_starter_code!
ActionCable.server.broadcast(
RepositoryCreationStatusChannel.channel(user_id: user.id),
text: IMPORT_STARTER_CODE,
status: assignment.invitation&.status
)
PorterStatusJob.perform_later(assignment_repo, user)
else
assignment.invitation&.completed!
ActionCable.server.broadcast(
RepositoryCreationStatusChannel.channel(user_id: user.id),
text: Creator::REPOSITORY_CREATION_COMPLETE,
status: assignment.invitation&.status
)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be recording stats about successful repo creation on this vs the old way? if so using GitHubClassroom.statsd?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely, we should be recording metrics wherever we think might help us be confident the new porter logic is better and more successful than the old one. I said this in an earlier comment, but we'll also want to have granular metrics so we can distinguish imports made using old logic and new logic.

end
rescue Creator::Result::Error => err
creator.delete_github_repository(assignment_repo.try(:github_repo_id))
assignment.invitation&.errored!
ActionCable.server.broadcast(
RepositoryCreationStatusChannel.channel(user_id: user.id),
text: err
text: err,
status: assignment.invitation&.status
)
raise err
end
Expand Down
8 changes: 6 additions & 2 deletions app/jobs/assignment_repo/porter_status_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,19 @@ def perform(assignment_repo, user)

case result
when Creator::REPOSITORY_STARTER_CODE_IMPORT_FAILED
assignment_repo.assignment.invitation&.errored!
ActionCable.server.broadcast(
RepositoryCreationStatusChannel.channel(user_id: user.id),
text: result
text: result,
status: assignment_repo.assignment.invitation&.status
)
logger.warn result.to_s
when Creator::REPOSITORY_CREATION_COMPLETE
assignment_repo.assignment.invitation&.completed!
ActionCable.server.broadcast(
RepositoryCreationStatusChannel.channel(user_id: user.id),
text: result
text: result,
status: assignment_repo.assignment.invitation&.status
)
end
end
Expand Down
22 changes: 20 additions & 2 deletions app/models/assignment_invitation.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
class AssignmentInvitation < ApplicationRecord
include ShortKey

enum status: %i[unaccepted accepted creating_repo importing_starter_code completed errored]

default_scope { where(deleted_at: nil) }

update_index("stafftools#assignment_invitation") { self }
Expand All @@ -20,12 +22,18 @@ class AssignmentInvitation < ApplicationRecord

after_initialize :assign_key

after_initialize :set_defaults, unless: :persisted?

delegate :title, to: :assignment

# Public: Redeem an AssignmentInvtiation for a User invitee.
#
# Returns a AssignmentRepo::Creator::Result.
def redeem_for(invitee)
#
# rubocop:disable MethodLength
# rubocop:disable AbcSize
def redeem_for(invitee, import_resiliency: false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using a kwarg here, can we just directly check the feature flag?

Copy link
Contributor Author

@BenEmdon BenEmdon Jul 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't use the feature dependency here because we are outside of a controller (feature 🚩s are defined on ApplicationController).

accepted!
if (repo_access = RepoAccess.find_by(user: invitee, organization: organization))
assignment_repo = AssignmentRepo.find_by(assignment: assignment, repo_access: repo_access)
return AssignmentRepo::Creator::Result.success(assignment_repo) if assignment_repo.present?
Expand All @@ -36,8 +44,14 @@ def redeem_for(invitee)

return AssignmentRepo::Creator::Result.failed("Invitations for this assignment have been disabled.") unless enabled?

AssignmentRepo::Creator.perform(assignment: assignment, user: invitee)
if import_resiliency
AssignmentRepo::Creator::Result.pending
else
AssignmentRepo::Creator.perform(assignment: assignment, user: invitee)
end
end
# rubocop:enable MethodLength
# rubocop:enable AbcSize

def to_param
key
Expand All @@ -52,4 +66,8 @@ def enabled?
def assign_key
self.key ||= SecureRandom.hex(16)
end

def set_defaults
self.status ||= :unaccepted
end
end
Loading