Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FYST-1089] refactor twilio service #4944

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
7ebf9aa
change the structure of the credentials file (just dev for now) to ha…
jnf Nov 1, 2024
a914b96
an idea of how to evolve the twilio service, presented as a v2 for ea…
jnf Nov 1, 2024
505cdfc
tweak how the multi tenant service fetches twilio creds
jnf Nov 4, 2024
623d5dd
bring the v2 version of TwilioClient into the original object so i ca…
jnf Nov 5, 2024
ae26344
updates TwilioService specs to conform to new API
jnf Nov 5, 2024
c12385b
don't need to clear a class variable in the mock twilio object anymore
jnf Nov 5, 2024
530dcf4
instantiate TwilioService as a gyr messenger unless caller says other…
jnf Nov 5, 2024
eacfb9d
updates callsite and specs for twilio webhooks controller; this is th…
jnf Nov 5, 2024
ceee3f7
updates callsites and specs for sms sending in the bulk signup
jnf Nov 5, 2024
077654d
updates the other #get_metadata callsite and associated specs
jnf Nov 5, 2024
64d8be2
update the twilio status backfiller to the new TwilioService api. the…
jnf Nov 5, 2024
2a1a4cd
updates verification code job to use new TwilioService api. also upda…
jnf Nov 5, 2024
ee2f22b
updated TwilioService callsites in the CtcSignupMessage, but it doesn…
jnf Nov 5, 2024
bb05cb2
updates Signup to use updated TwilioService api. updates specs too. i…
jnf Nov 5, 2024
40e4dbd
updates TwilioService callsite in the send_outgoing_text_message_job …
jnf Nov 5, 2024
db5747d
updates the gyr incoming text message job/handling/specs to use the u…
jnf Nov 5, 2024
6812f1e
update code in and specs for TextMessageVerificationCodeService to us…
jnf Nov 5, 2024
4bd6e7b
updates the client login spec to work with the updated TwilioService
jnf Nov 5, 2024
288ac36
updates the login spec to work with the new TwilioService api
jnf Nov 5, 2024
64c28b9
removes the 'v2' twilio service as the changes proposed have been ref…
jnf Nov 5, 2024
4f92fea
updates the hub outbound call from to use TwilioService rather than r…
jnf Nov 7, 2024
fa01ca8
updates TwilioService, MultiTenantService, dev credentials, and assoc…
jnf Nov 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion app/controllers/twilio_webhooks_controller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,6 @@ def outbound_call_connect
private

def validate_twilio_request
return head 403 unless TwilioService.valid_request?(request)
return head 403 unless TwilioService.new.valid_request?(request)
end
end
5 changes: 2 additions & 3 deletions app/forms/hub/outbound_call_form.rb
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@ def initialize(attrs = {}, **kwargs)
def dial
return false unless valid?

twilio_client = Twilio::REST::Client.new(EnvironmentCredentials.dig(:twilio, :account_sid),
EnvironmentCredentials.dig(:twilio, :auth_token))
twilio_client = TwilioService.new(:gyr).client
OutboundCall.transaction do
@outbound_call = @client.outbound_calls.create(
user_id: @user.id,
Expand Down Expand Up @@ -64,4 +63,4 @@ def twiml
twiml.to_xml
end
end
end
end
6 changes: 3 additions & 3 deletions app/jobs/bulk_action/send_one_bulk_signup_message_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ class SendOneBulkSignupMessageJob < ApplicationJob
include Rails.application.routes.url_helpers

def perform(signup, bulk_signup_message)
service_info = MultiTenantService.new(signup.class.name == "CtcSignup" ? :ctc : :gyr)
if bulk_signup_message.message_type == "sms"
line_type = TwilioService.get_metadata(phone_number: signup.phone_number)&.dig("type")
line_type = TwilioService.new(service_info.service_type).get_metadata(phone_number: signup.phone_number)&.dig("type")
if line_type == "landline"
DatadogApi.increment("twilio.outgoing_text_messages.bulk_signup_message_not_sent_landline")
return
Expand All @@ -19,14 +20,13 @@ def perform(signup, bulk_signup_message)
message_id =
case bulk_signup_message.message_type
when "sms"
TwilioService.send_text_message(
TwilioService.new(service_info.service_type).send_text_message(
to: signup.phone_number,
body: bulk_signup_message.message,
status_callback: twilio_update_status_url(outgoing_message_status.id, locale: nil),
outgoing_text_message: outgoing_message_status
)&.sid
when "email"
service_info = MultiTenantService.new(signup.class.name == "CtcSignup" ? :ctc : :gyr)
SignupFollowupMailer.new.followup(email_address: signup.email_address, message: OpenStruct.new(
email_body: bulk_signup_message.message,
service_type: service_info.service_type,
Expand Down
2 changes: 1 addition & 1 deletion app/jobs/get_phone_metadata_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ class GetPhoneMetadataJob < ApplicationJob
def perform(intake)
return if intake.phone_number.blank?

metadata = TwilioService.get_metadata(phone_number: intake.phone_number)
metadata = TwilioService.new(:ctc).get_metadata(phone_number: intake.phone_number)

return if metadata.blank?

Expand Down
4 changes: 2 additions & 2 deletions app/jobs/request_verification_code_for_login_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,11 @@ def perform(email_address: nil, phone_number: nil, locale:, visitor_id:, service
when :statefile
I18n.t("state_file.intake_logins.no_match_sms", url: url, locale: locale)
end
TwilioService.send_text_message(
TwilioService.new(service_type).send_text_message(
to: phone_number,
body: body
)
end
end
end
end
end
2 changes: 1 addition & 1 deletion app/jobs/send_outgoing_text_message_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ class SendOutgoingTextMessageJob < ApplicationJob
def perform(outgoing_text_message_id)
outgoing_text_message = OutgoingTextMessage.find(outgoing_text_message_id)
begin
message = TwilioService.send_text_message(
message = TwilioService.new.send_text_message( # TODO: set explicit service scope; is this job for :ctc or :gyr?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[DUST] again, would like to remove comments before merging

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

absolutely. do you know if/how this job is aware of it's tenant service/environment?

to: outgoing_text_message.to_phone_number,
body: outgoing_text_message.body,
status_callback: outgoing_text_message_url(outgoing_text_message, locale: nil),
Expand Down
20 changes: 9 additions & 11 deletions app/jobs/send_outgoing_text_message_without_client_job.rb
Original file line number Diff line number Diff line change
@@ -1,17 +1,15 @@
class SendOutgoingTextMessageWithoutClientJob < ApplicationJob
def perform(phone_number:, body:)
begin
TwilioService.send_text_message(
to: phone_number,
body: body,
)
TwilioService.new(:gyr).send_text_message(
to: phone_number,
body: body,
)

DatadogApi.increment("twilio.outgoing_text_messages.sent")
rescue Twilio::REST::TwilioError => e
if e.cause == Net::OpenTimeout
DatadogApi.increment("twilio.outgoing_text_messages.failure.timeout")
retry_job(outgoing_text_message_id)
end
DatadogApi.increment("twilio.outgoing_text_messages.sent")
rescue Twilio::REST::TwilioError => e
if e.cause == Net::OpenTimeout
DatadogApi.increment("twilio.outgoing_text_messages.failure.timeout")
retry_job(outgoing_text_message_id)
end
end

Expand Down
7 changes: 4 additions & 3 deletions app/lib/ctc_signup_message.rb
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def self.send(count)
BODY

if ctc_signup.phone_number.present?
TwilioService.send_text_message(
TwilioService.new(:ctc).send_text_message(
to: ctc_signup.phone_number,
body: (body % { name: ctc_signup.name }),
)
Expand All @@ -55,11 +55,12 @@ def self._send_one_launch_announcement(ctc_signup)
es_sms = '¡Gracias por registrarse para recibir las actualizaciones de GetCTC! ¡GetCTC está oficialemente disponible y a su disposición para presenter su declaración de impuestos de forma simplificada para reclamara su Crédito Tributario por Hijos y los pagos de estímulo! Ingrese a GetCTC.org y haga clic en “Declare de forma simplificada ahora” para iniciar.'

if ctc_signup.phone_number.present?
TwilioService.send_text_message(
ts = TwilioService.new(:ctc)
ts.send_text_message(
to: ctc_signup.phone_number,
body: en_sms,
)
TwilioService.send_text_message(
ts.send_text_message(
to: ctc_signup.phone_number,
body: es_sms,
)
Expand Down
2 changes: 1 addition & 1 deletion app/models/signup.rb
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ def self.send_message(message_name, batch_size=nil, after: nil)
end

if signup.phone_number.present?
TwilioService.send_text_message(to: signup.phone_number, body: message.sms_body)
TwilioService.new(:ctc).send_text_message(to: signup.phone_number, body: message.sms_body)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is actually not just a CTC thing! see: https://www.getyourrefund.org/en/sign-up

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that makes me moderately unhappy. but it's kinda sorta ok as right now ctc and gyr seem to be sharing the same twilio setup.

signup.touch(sent_at_column)
end
end
Expand Down
2 changes: 1 addition & 1 deletion app/services/incoming_text_message_service.rb
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def self.process(params)
DatadogApi.increment("twilio.incoming_text_messages.#{event_name}")

# process attachments once
attachments = TwilioService.parse_attachments(params)
attachments = TwilioService.new.parse_attachments(params)

clients.map do |client|
documents = attachments.map do |attachment|
Expand Down
8 changes: 8 additions & 0 deletions app/services/multi_tenant_service.rb
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,14 @@ def backtax_years(now = DateTime.now)
filing_years(now).without(current_tax_year)
end

def twilio_creds
@_twlio_creds ||= {
account_sid: EnvironmentCredentials.dig(:twilio, service_type, :account_sid),
auth_token: EnvironmentCredentials.dig(:twilio, service_type, :auth_token),
messaging_service_sid: EnvironmentCredentials.dig(:twilio, service_type, :messaging_service_sid)
}
end

class << self
def ctc
new(:ctc)
Expand Down
4 changes: 2 additions & 2 deletions app/services/text_message_verification_code_service.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ def initialize(phone_number:, locale: :en, visitor_id:, client_id: nil, service_
def request_code
verification_code, access_token = TextMessageAccessToken.generate!(sms_phone_number: @phone_number, client_id: @client_id)
outgoing_message_status = OutgoingMessageStatus.find_or_create_by!(message_type: :sms, parent: access_token)
twilio_response = TwilioService.send_text_message(
twilio_response = TwilioService.new(@service_data.service_type).send_text_message(
to: @phone_number,
body: I18n.t("verification_code_sms.with_code",
service_name: @service_data.service_name,
Expand All @@ -35,4 +35,4 @@ def request_code
def self.request_code(**args)
new(**args).request_code
end
end
end
172 changes: 85 additions & 87 deletions app/services/twilio_service.rb
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
class TwilioService
FAILED_STATUSES = ["undelivered", "failed", "delivery_unknown", "twilio_error"].freeze
SUCCESSFUL_STATUSES = ["sent", "delivered"].freeze
IN_PROGRESS_STATUSES = ["accepted", "queued", "sending", nil].freeze
FAILED_STATUSES = %w(undelivered failed delivery_unknown twilio_error)
SUCCESSFUL_STATUSES = %w(sent delivered)
IN_PROGRESS_STATUSES = %w(accepted queued sending) << nil
ALL_KNOWN_STATUSES = FAILED_STATUSES + SUCCESSFUL_STATUSES + IN_PROGRESS_STATUSES
ORDERED_STATUSES = [nil, "twilio_error"] + %w[
ORDERED_STATUSES = %w(
twilio_error
queued
accepted
sending
Expand All @@ -12,104 +13,101 @@ class TwilioService
delivered
undelivered
failed
].freeze
).unshift(nil) # why do we need nil in this list, and why must it be first?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[DUST] i assume you're planning to remove this comment before merging? would prefer not to leave stuff like this around the codebase

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes! for sure. i literally don't know the answer but would like to. 😅


class << self
def valid_request?(request)
validator = Twilio::Security::RequestValidator.new(EnvironmentCredentials.dig(:twilio, :auth_token))
validator.validate(
request.url,
request.POST,
request.headers["X-Twilio-Signature"],
)
end
attr_reader :client, :messaging_service_sid, :auth_token

def fetch_attachment(url)
begin
response = Net::HTTP.get_response(URI(url)) # first we get a redirect from Twilio to S3
response = Net::HTTP.get_response(URI(response['location'])) # then we get a redirect from S3 to S3
response = Net::HTTP.get_response(URI(response['location'])) # finally we should get a 200 OK with the file
filename_from_s3 = response['content-disposition'].split('"').last # S3 gives us the original filename
{
filename: filename_from_s3,
body: response.body,
}
rescue ArgumentError => e
Rails.logger.error("Error getting attachment from Twilio: #{url}: #{response&.code}: #{response&.to_hash}")
{
filename: "unknown-file",
body: nil
}
end
end
def initialize(service_type = :gyr)
creds = MultiTenantService.new(service_type).twilio_creds
@messaging_service_sid = creds[:messaging_service_sid]
@auth_token = creds[:auth_token]
@client = Twilio::REST::Client.new(creds[:account_sid], auth_token)
end

def parse_attachments(params)
num_media = params["NumMedia"].to_i
def send_text_message(to:, body:, status_callback: nil, outgoing_text_message: nil)
arguments = {
messaging_service_sid: ENV['MESSAGING_SERVICE_SID'] || messaging_service_sid, # why do we check the environment for this??
to: to,
body: body
}
arguments[:status_callback] = status_callback if status_callback.present?

(0..(num_media - 1)).map do |i|
content_type = params["MediaContentType#{i}"]
attachment = fetch_attachment(params["MediaUrl#{i}"])
DatadogApi.increment("twilio.outgoing_text_messages.sent")

if FileTypeAllowedValidator.mime_types(Document).include?(content_type) && !attachment[:body].empty?
{
content_type: params["MediaContentType#{i}"],
filename: attachment[:filename],
body: attachment[:body]
}
else
{
content_type: "text/plain;charset=UTF-8",
filename: "invalid-#{attachment[:filename]}.txt",
body: <<~TEXT
Unusable file with unknown or unsupported file type.
File name: #{attachment[:filename]}
File type: #{content_type}
File size: #{attachment[:body].size} bytes
TEXT
}
end
client.messages.create(**arguments)
rescue Twilio::REST::RestError => e
status_key =
if outgoing_text_message.is_a?(OutgoingMessageStatus)
:delivery_status
else
:twilio_status
end
end
outgoing_text_message&.update(status_key => "twilio_error")

def client
@@_client ||= Twilio::REST::Client.new(
EnvironmentCredentials.dig(:twilio, :account_sid),
EnvironmentCredentials.dig(:twilio, :auth_token)
)
unless e.code == 21211 # Invalid 'To' Phone Number https://www.twilio.com/docs/api/errors/21211
raise # should we include the original exception here (e)??
end

def send_text_message(to:, body:, status_callback: nil, outgoing_text_message: nil)
arguments = {
messaging_service_sid: ENV['MESSAGING_SERVICE_SID'] || EnvironmentCredentials.dig(:twilio, :messaging_service_sid),
to: to,
body: body
}
arguments[:status_callback] = status_callback if status_callback.present?
nil
end

DatadogApi.increment("twilio.outgoing_text_messages.sent")
def get_metadata(phone_number:)
client.lookups.v2.phone_numbers(phone_number).fetch(fields: 'line_type_intelligence').line_type_intelligence
rescue Twilio::REST::RestError
{}
end

client.messages.create(**arguments)
rescue Twilio::REST::RestError => e
status_key =
if outgoing_text_message.is_a?(OutgoingMessageStatus)
:delivery_status
else
:twilio_status
end
outgoing_text_message&.update(status_key => "twilio_error")
def valid_request?(request)
validator = Twilio::Security::RequestValidator.new(auth_token)
validator.validate(
request.url,
request.POST,
request.headers["X-Twilio-Signature"],
)
end

unless e.code == 21211 # Invalid 'To' Phone Number https://www.twilio.com/docs/api/errors/21211
raise
end
def fetch_attachment(url)
response = Net::HTTP.get_response(URI(url)) # first we get a redirect from Twilio to S3
response = Net::HTTP.get_response(URI(response['location'])) # then we get a redirect from S3 to S3
response = Net::HTTP.get_response(URI(response['location'])) # finally we should get a 200 OK with the file
filename_from_s3 = response['content-disposition'].split('"').last # S3 gives us the original filename
{
filename: filename_from_s3,
body: response.body,
}
rescue ArgumentError => e
Rails.logger.error("Error getting attachment from Twilio: #{url}: #{response&.code}: #{response&.to_hash}")
{
filename: "unknown-file",
body: nil
}
end

nil
end
def parse_attachments(params)
num_media = params["NumMedia"].to_i

def get_metadata(phone_number:)
client.lookups.v2.phone_numbers(phone_number).fetch(fields: 'line_type_intelligence').line_type_intelligence
(0...num_media).map do |i|
content_type = params["MediaContentType#{i}"]
attachment = fetch_attachment(params["MediaUrl#{i}"])

rescue Twilio::REST::RestError
{}
if FileTypeAllowedValidator.mime_types(Document).include?(content_type) && !attachment[:body].empty?
{
content_type: params["MediaContentType#{i}"],
filename: attachment[:filename],
body: attachment[:body]
}
else
{
content_type: "text/plain;charset=UTF-8",
filename: "invalid-#{attachment[:filename]}.txt",
body: <<~TEXT
Unusable file with unknown or unsupported file type.
File name: #{attachment[:filename]}
File type: #{content_type}
File size: #{attachment[:body].size} bytes
TEXT
}
end
end
end
end
2 changes: 1 addition & 1 deletion config/credentials/development.yml.enc

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion lib/tasks/outgoing_messages.rake
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ namespace :outgoing_messages do
end

def backfill_twilio_statuses(limit: 1000)
twilio_client = TwilioService.client
twilio_client = TwilioService.new.client
OutgoingTextMessage
.in_progress
.where(created_at: ...4.hours.ago).order(created_at: :desc).limit(limit).each do |outgoing_text_message|
Expand Down
Loading
Loading