Enhancement: Amazon Bedrock

[Manouchehri]

Contact Details

No response

What features would you like to see added?

Getting access to Anthropic's APIs via Amazon Bedrock is much easier. In the time it took me to write this ticket, Amazon already approved access to Claude 2 on my company AWS account. 🙂

More details

Pricing seems to be exactly the same.

https://www-files.anthropic.com/production/images/model_pricing_july2023.pdf
https://aws.amazon.com/bedrock/pricing/

Which components are impacted by your request?

No response

Pictures

Code of Conduct

• I agree to follow this project's Code of Conduct

[Manouchehri]

I would recommend using aws4fetch, or the official node package if size doesn't matter.

https://docs.aws.amazon.com/bedrock/latest/APIReference/welcome.html

https://docs.aws.amazon.com/bedrock/latest/userguide/api-methods-run-inference.html

[Manouchehri]

https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html

Looks relatively easy to use. :)

[Manouchehri]

fetch("https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-v2/invoke-with-response-stream", {
  "headers": {
    "accept": "*/*",
    "accept-language": "en-US,en-CA;q=0.9,en;q=0.8",
    "amz-sdk-invocation-id": "removed-removed-removed-removed-removed",
    "amz-sdk-request": "attempt=1; max=0",
    "authorization": "AWS4-HMAC-SHA256 Credential=removed/20231104/us-east-1/bedrock/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent;x-amzn-bedrock-accept, Signature=removed",
    "content-type": "application/json",
    "sec-ch-ua": "\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Google Chrome\";v=\"120\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"macOS\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "cross-site",
    "x-amz-content-sha256": "removed",
    "x-amz-date": "20231104T162611Z",
    "x-amz-security-token": "removed",
    "x-amz-user-agent": "aws-sdk-js/1.0.0 os/macOS/10.15.7 lang/js md/browser/Chrome_120.0.0.0 api/bedrock_runtime/1.0.0 Bedrock",
    "x-amzn-bedrock-accept": "*/*"
  },
  "referrer": "https://us-east-1.console.aws.amazon.com/",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": "{\"prompt\":\"\\n\\nHuman: Tell me about your day.\\n\\nAssistant:\",\"max_tokens_to_sample\":2048,\"temperature\":1,\"top_k\":250,\"top_p\":0.999,\"stop_sequences\":[\"\\n\\nHuman:\"],\"anthropic_version\":\"bedrock-2023-05-31\"}",
  "method": "POST",
  "mode": "cors",
  "credentials": "include"
});

[Manouchehri]

LangChain supports Bedrock too. https://js.langchain.com/docs/integrations/chat/bedrock

There's now Llama 2 13B support in Bedrock!

https://aws.amazon.com/blogs/aws/amazon-bedrock-now-provides-access-to-llama-2-chat-13b-model/

[danny-avila]

I can see this being useful for SDXL but makes sense to use LiteLLM for Bedrock access.

[stirredo]

@danny-avila Can you reconsider your decision now that AWS bedrock supports Claude Opus?

[danny-avila]

I will get to it eventually, but it's not a priority at the moment. PRs welcome.

[nacartwright]

I have AWS customers wanting to build their own ChatGPT like experience on Bedrock. Having Bedrock support in Librechat would allow customers to do that. So, would love to see Bedrock LLM support as well as Bedrock KB.

What would be even better is support for the Langchain LLMs so that you support every LLM that Langchain can support.