Getting "The request signature we calculated does not match the signature you provided." when creating subscriptions using aws-appsync-subscription-link and apollo client v3

[lyne-jy]

Note: If your issue/feature-request/question is regarding the AWS AppSync service, please log it in the
official AWS AppSync forum

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Recently we upgraded from apollo client v2 to v3 so we also upgraded aws-appsync-auth-link and aws-appsync-subscription-link. When we use apollo client v3 and aws-appsync-subscription-link to create subscriptions, we get this error:

The request signature we calculated does not match the signature you provided. 
Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'POST
/graphql

accept:*/*
content-type:application/json; charset=UTF-8
host:xxx.appsync-api.us-west-2.amazonaws.com
x-amz-date:20230821T214203Z
x-amz-security-token:xxx

accept;content-type;host;x-amz-date;x-amz-security-token
990ebd3e582fa2e4a448047e52ebaa031c56d86b468af9fe4adfea2687c60cd3'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20230821T214203Z
20230821/us-west-2/appsync/aws4_request
5ebe6e4c844b8a6aa3333a373feef69d913d2facb886fc24a9ea55ed5a58f19c'

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
This is how we create the apollo client and the necessary links, basically following the README:

const url = credentials.graphQLEndpoint;

const auth = {
  type: AUTH_TYPE.AWS_IAM as const,
  credentials: credentials
};

const httpLink = new HttpLink({ uri: url });

const errorLink = onError(({ graphQLErrors, networkError }) => {
  if (graphQLErrors) {
    graphQLErrors.forEach(({ message, locations, path }) =>
      console.log(`[GraphQL error]: Message: ${message}, Location: ${locations}, Path: ${path}`)
    );
  }

  if (networkError) {
    console.log(`[Network error]: ${networkError}`);
  }
  this.handleAppSyncStatusChange(AppSyncStatus.ERROR);
});
const link = ApolloLink.from([
  errorLink,
  createAuthLink({ url: credentials.graphQLEndpoint, region: credentials.region, auth: auth }),
  createSubscriptionHandshakeLink(
    { url: credentials.graphQLEndpoint, region: credentials.region, auth: auth },
    httpLink
  )
]);

const appSyncClient = new ApolloClient({
  link,
  cache: new InMemoryCache()
});

When we use this client to create subscriptions, it returns the error "The request signature we calculated does not match the signature you provided." We can confirm that the credentials are correct. We also have an Amplify client that also uses the same GraphQL API, but that's working fine.

What is the expected behavior?
Return the subscription response without errors.

Which versions and which environment (browser, react-native, nodejs) / OS are affected by this issue? Did this work in previous versions?
We use:

"@apollo/client": "3.3.21",
"aws-appsync-auth-link": "3.0.7",
"aws-appsync-subscription-link": "3.1.2"
"graphql": "^15.3.0",

We've tried apollo client v2 + aws-appsync-auth-link v2.x + aws-appsync-subscription-link v2.x and this combo worked.
We tried aws-appsync v4.1.9 which also worked.
However these are in maintenance mode.

[dax-hurley]

I can confirm that I am also experiencing the same behavior. My client setup is very similar and looks very close to the readme:

import {
  ApolloClient,
  InMemoryCache,
  ApolloLink,
  HttpLink,
} from "@apollo/client";
import { createAuthLink, AuthOptions } from "aws-appsync-auth-link";
import { createSubscriptionHandshakeLink } from "aws-appsync-subscription-link";

const createClient = (url: string) => {
  const auth = {
    type: "AWS_IAM",
    credentials: async () => {
      return {
        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
        sessionToken: process.env.AWS_SESSION_TOKEN!,
      };
    },
  } as AuthOptions;

  const httpLink = new HttpLink({ uri: url });
  const region = process.env.AWS_REGION!;
  const config = {
    url,
    region,
    auth,
  };
  const link = ApolloLink.from([
    createAuthLink(config),
    createSubscriptionHandshakeLink(config, httpLink),
  ]);

  const apolloClient = new ApolloClient({
    link,
    cache: new InMemoryCache(),
  });

  return apolloClient;
};

export { createClient };

I am getting my AWS credentials out of my Lambda function's environment variables.

[danrivett]

@dax-hurley @lyne-jy Did either of you manage to resolve this?

We recently migrated off of aws-appsync and onto Apollo Client v3 with aws-appsync-auth-link and the vast majority of Lambda-based queries and mutations are working, but a particular Query is failing repeatedly, and I'm at loss to understand why that is.

The query itself is auto-generated through AWS Amplify codegen like all our other queries, so there shouldn't be any syntax errors.

[danrivett]

Ok, so I finally solved my issue, and it was trivial in the end, but the error returned by AWS so very misleading. I'm posting it here as although it's not related to AppSync Subscriptions (I was issuing a Query), if others search for the The request signature we calculated does not match the signature you provided. error it may help them:

It turns out my variables object I was passing in was incorrect, and I didn't get a GraphQL error to tell me that, but an AWS signature error. e.g. I had previously:

apolloClient.query({
  query: gql(myQuery),
  variables: {
    id: sharedInstanceId,
  },
});

And it should have been (as my query's input variable for this query was sharedInstanceId and not id:

apolloClient.query({
  query: gql(myQuery),
  variables: {
    sharedInstanceId,
  },
});

It was a bug introduced in a refactoring a while back but only caught when switching from aws-appsync to Apollo Client v3 (it was previously silently failing as I also wasn't checking the errors on the response 🤦‍♂️

Lesson learned.

Still that AWS signature error was incredibly misleading, so hopefully this helps someone else out.

[svidgen]

Thanks for the repro @danrivett!

Using incorrect top level keys in variables does indeed reproduce the problem. And in watching the requests over the wire, it was actually pretty easy to identify a thread to tug on -- any/all "unexpected" keys in variables simply go missing from the requests.

With a tiny bit of extra logging, it becomes clear that AuthLink is eagerly signing the request + variables as-provided before handing off to the subsequent link. Down the line, it looks like Apollo strips "unused" keys from the variables object. This immediately makes the signature invalid, but Apollo has no idea what it's done! So, the request is rejected before it can even get to AppSync execution/validation layer. Hence, the only error AppSync knows to give is an unhelpful signing error.

I don't have a quick fix for this, unfortunately. I'll speak with the team internally to decide how best to address this. But, with that understanding of the problem, we at least have something concrete to discuss. And in the meantime, future readers can know what to look for!

TLDR; For now, if you see unexpected IAM signing errors, confirm that your requests over the wire match your code, and that your code matches your schema!

---

(And thank-you again for finding a repro, @danrivett!)

[danrivett]

Thanks for investigating this further @svidgen. That's a great explanation of why this happening, that makes sense.

It of course would be great to find a solution to this, but until then it's definitely a step forward to understand why it's happening at least.