Skip to content

feat: Integrate Bitget for Rate Fetching & Median Calculation Across Multiple Providers #441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 57 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
f0185fa
feat: implement multi-provider rate fetching system with test
sundayonah Feb 13, 2025
469f49f
feat(rates): enhance rate fetching with detailed metadata
sundayonah Feb 13, 2025
9d00340
test: modify test and move external_market file to utils
sundayonah Feb 14, 2025
7f72384
refactor(config): move API URLs to environment variables
sundayonah Feb 14, 2025
bd3c348
refactor(utils): overhaul external market rate fetching implementation
sundayonah Feb 14, 2025
31e3a32
refactor: (utils) replace FetchBitgetRate endpoint, remove /api/v1 fr…
sundayonah Mar 11, 2025
59e047c
fix: resolve rebase issues and reset branch to stable state
sundayonah Mar 20, 2025
fe0cdce
refactor(config): remove Bitget API keys from AuthConfiguration
sundayonah Mar 20, 2025
4864dea
Merge branch 'main' into biget-rates-market-refactor
sundayonah Mar 20, 2025
65c7c58
fix: update function name for fetching Quidax rates
sundayonah Mar 20, 2025
0a46c74
Merge branch 'main' into biget-rates-market-refactor
sundayonah Apr 3, 2025
b9116be
Merge branch 'main' into biget-rates-market-refactor
sundayonah Apr 20, 2025
984efba
Merge branch 'main' into biget-rates-market-refactor
chibie Apr 22, 2025
88f601e
feat: add HTTPS URL validation for host identifier in UpdateProviderP…
sundayonah Apr 10, 2025
4cba6d3
test: add vtest cases for HostIdentifier HTTPS validation
sundayonah Apr 10, 2025
3412c04
fix(tests): update HostIdentifier to use HTTPS and improve assertions…
sundayonah Apr 21, 2025
a0108f6
fix: setup repo
Atanda1 Mar 27, 2025
7959afd
docs: update README to specify Sender API usage with sandbox API Key …
chibie Mar 27, 2025
fda12a3
fix: update SQL dump and user operation logic
chibie Mar 28, 2025
ae9148b
fix: update SQL dump for provider profile host identifier
chibie Mar 28, 2025
3fabc7f
chore: remove unnecessary print statements
chibie Apr 22, 2025
cd9feb2
fix: Handle amounts below minimum bucket
sundayonah Mar 15, 2025
bf6c876
test: Add unit tests for minimum bucket handling
sundayonah Mar 15, 2025
a0896a6
fix: remove duplicate token import
sundayonah Mar 15, 2025
dd14d9b
feat: modify provision bucket call to capture unused isLessThanMin an…
sundayonah Mar 18, 2025
d0ba090
fix(indexer): improve error handling in CreateLockPaymentOrder and en…
chibie Apr 21, 2025
b87ad09
fix(indexer): enhance error message in CreateLockPaymentOrder to incl…
chibie Apr 24, 2025
f0e4e88
fix(indexer): change error handling for token fetch in CreateLockPaym…
chibie Apr 24, 2025
61beac8
Provider Token Rate Slippage Configuration (#415)
sundayonah Apr 25, 2025
7161a65
feat: enhance logging with contextual information across services (#459)
onahprosper Apr 25, 2025
4823053
refactor(logging): standardize error logging format and enhance conte…
chibie Apr 28, 2025
6893643
refactor(profile): update provider rate calculation logic in profile …
chibie Apr 29, 2025
c966593
refactor(logging): standardize error logging format across indexer an…
chibie Apr 30, 2025
c5bdb59
refactor(indexer, priority_queue, order): enforce settlement address …
chibie May 3, 2025
2007b52
fix(profile): handle rate slippage assignment in provider profile upd…
chibie May 3, 2025
f4a9137
fix(priority_queue): change error handling in matchRate to continue o…
chibie May 3, 2025
eed9fb0
refactor(provider): improve stats calculation and error handling in P…
chibie May 4, 2025
4f8bcb3
refactor(provider): streamline stats calculation for USD and local st…
chibie May 4, 2025
8a2baff
refactor(provider): update stats query to filter by USD token for acc…
chibie May 4, 2025
aa985f7
refactor(provider): simplify total fiat volume calculation in Stats m…
chibie May 4, 2025
e0aa4c8
refactor(sender, indexer): enhance institution retrieval logic and im…
chibie May 4, 2025
eae210f
refactor(provider): enhance Stats method to filter out USD token and …
chibie May 4, 2025
eca9a66
refactor(sender): remove debug print statements from Stats method for…
chibie May 4, 2025
861a467
feat(migrations): add TZS and UGX bank institutions
sundayonah Apr 25, 2025
e5702f9
fix: remove provision buckets from UGX and TZS bank institution migra…
sundayonah May 2, 2025
21525d6
refactor(profile): update profile test and API response messages for …
chibie May 5, 2025
0a8b789
refactor(indexer): commented out the split lock payment order functio…
chibie May 5, 2025
e198963
chore(migrations): updated versioning of TZS and UGX bank institution…
chibie May 5, 2025
94ee460
fix: timezone error in Slack notifications with additional test cases
sundayonah May 5, 2025
4a7dd78
feat: support ~all countries in KYC verification (#457)
sundayonah May 9, 2025
5659d9c
refactor: update KYC verification structure and remove deprecated files
chibie May 10, 2025
0166b0a
feat: add metadata field to payment order and recipient models (#467)
chibie May 11, 2025
d3f80fd
feat(ci): add Atlas database migrations workflow (#468)
chibie May 11, 2025
674b9e2
refactor: enhance error logging in FulfillOrder and CancelOrder contr…
onahprosper May 11, 2025
99575f2
refactor(utils): overhaul external market rate fetching implementation
sundayonah Feb 14, 2025
ba44653
fix: resolve rebase issues and reset branch to stable state
sundayonah Mar 20, 2025
5257afa
fix: remove duplicate type definitions
sundayonah May 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -566,8 +566,6 @@ github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkU
github.com/spf13/cast v1.5.1 h1:R+kOtfhWQE6TVQzY+4D7wJLBgkdVasCEFxSUBYBYIlA=
github.com/spf13/cast v1.5.1/go.mod h1:b9PdjNptOpzXr7Rq1q9gJML/2cdGQAo69NKzQ10KN48=
github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
github.com/spf13/cobra v1.7.0 h1:hyqWnYt1ZQShIddO5kBpj3vu05/++x6tJ6dg8EC572I=
github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0=
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk=
github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo=
Expand Down
90 changes: 1 addition & 89 deletions tasks/tasks.go
Original file line number Diff line number Diff line change
Expand Up @@ -1078,94 +1078,6 @@ func SubscribeToRedisKeyspaceEvents() {
go ReassignStaleOrderRequest(ctx, orderRequestChan)
}

// fetchExternalRate fetches the external rate for a fiat currency
func fetchExternalRate(currency string) (decimal.Decimal, error) {
currency = strings.ToUpper(currency)
supportedCurrencies := []string{"KES", "NGN", "GHS", "TZS", "UGX", "XOF"}
isSupported := false
for _, supported := range supportedCurrencies {
if currency == supported {
isSupported = true
break
}
}
if !isSupported {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: currency not supported")
}

// Fetch rates from third-party APIs
var price decimal.Decimal
if currency == "NGN" {
res, err := fastshot.NewClient("https://www.quidax.com").
Config().SetTimeout(30*time.Second).
Build().GET(fmt.Sprintf("/api/v1/markets/tickers/usdt%s", strings.ToLower(currency))).
Retry().Set(3, 5*time.Second).
Send()
if err != nil {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: %w", err)
}

data, err := utils.ParseJSONResponse(res.RawResponse)
if err != nil {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: %w %v", err, data)
}

price, err = decimal.NewFromString(data["data"].(map[string]interface{})["ticker"].(map[string]interface{})["buy"].(string))
if err != nil {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: %w", err)
}
} else {
res, err := fastshot.NewClient("https://p2p.binance.com").
Config().SetTimeout(30*time.Second).
Header().Add("Content-Type", "application/json").
Build().POST("/bapi/c2c/v2/friendly/c2c/adv/search").
Retry().Set(3, 5*time.Second).
Body().AsJSON(map[string]interface{}{
"asset": "USDT",
"fiat": currency,
"tradeType": "SELL",
"page": 1,
"rows": 20,
}).
Send()
if err != nil {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: %w", err)
}

resData, err := utils.ParseJSONResponse(res.RawResponse)
if err != nil {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: %w", err)
}

// Access the data array
data, ok := resData["data"].([]interface{})
if !ok || len(data) == 0 {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: No data in the response")
}

// Loop through the data array and extract prices
var prices []decimal.Decimal
for _, item := range data {
adv, ok := item.(map[string]interface{})["adv"].(map[string]interface{})
if !ok {
continue
}

price, err := decimal.NewFromString(adv["price"].(string))
if err != nil {
continue
}

prices = append(prices, price)
}

// Calculate and return the median
price = utils.Median(prices)
}

return price, nil
}

// ComputeMarketRate computes the market price for fiat currencies
func ComputeMarketRate() error {
// ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
Expand All @@ -1183,7 +1095,7 @@ func ComputeMarketRate() error {

for _, currency := range currencies {
// Fetch external rate
externalRate, err := fetchExternalRate(currency.Code)
externalRate, err := utils.FetchQuidaxRates(currency.Code)
if err != nil {
continue
}
Expand Down
7 changes: 0 additions & 7 deletions tasks/tasks_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ import (
"github.com/paycrest/aggregator/types"
"github.com/paycrest/aggregator/utils"
"github.com/paycrest/aggregator/utils/test"
"github.com/shopspring/decimal"
"github.com/stretchr/testify/assert"
)

Expand Down Expand Up @@ -169,10 +168,4 @@ func TestTasks(t *testing.T) {

assert.Equal(t, hook.Status, webhookretryattempt.StatusExpired)
})

t.Run("fetchExternalRate", func(t *testing.T) {
value, err := fetchExternalRate("KSH")
assert.Error(t, err)
assert.Equal(t, value, decimal.Zero)
})
}
22 changes: 22 additions & 0 deletions types/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -646,6 +646,28 @@ type LinkedAddressTransactionList struct {
Transactions []LinkedAddressTransaction `json:"transactions"`
}

// BitgetResponse is the response for Bitget advertisement endpoint
type BitgetResponse struct {
Code string `json:"code"`
Data BitgetData `json:"data"`
Msg string `json:"msg"`
}

// BitgetData is the struct for Bitget advertisement data
type BitgetData struct {
DataList []BitgetAd `json:"dataList"`
}

// BitgetAd is the struct for Bitget advertisement data item
type BitgetAd struct {
Price string `json:"price"`
CoinCode string `json:"coinCode"`
FiatCode string `json:"fiatCode"`
Amount string `json:"amount"`
MinAmount string `json:"minAmount"`
MaxAmount string `json:"maxAmount"`
}

// SupportedTokenResponse represents the structure for supported tokens
type SupportedTokenResponse struct {
Symbol string `json:"symbol"`
Expand Down
217 changes: 217 additions & 0 deletions utils/external_market.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
package utils

import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"

fastshot "github.com/opus-domini/fast-shot"
"github.com/paycrest/aggregator/types"
"github.com/shopspring/decimal"
)

var (
BitgetAPIURL = "https://www.bitget.com"
BinanceAPIURL = "https://p2p.binance.com"
QuidaxAPIURL = "https://www.quidax.com"
)

// FetchExternalRate fetches the external rate for a fiat currency
func FetchExternalRate(currency string) (decimal.Decimal, error) {
currency = strings.ToUpper(currency)
supportedCurrencies := []string{"KES", "NGN", "GHS", "TZS", "UGX", "XOF"}
isSupported := false
for _, supported := range supportedCurrencies {
if currency == supported {
isSupported = true
break
}
}
if !isSupported {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: currency not supported")
}

var prices []decimal.Decimal

// Fetch rates based on currency
if currency == "NGN" {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is not enough.
The first check should includes what we have here

if currency == "NGN" || currency == "KES" || currency == "GHS"
// get market price from each Bitget, Binance, and Quidax
// calculate the median

Copy link
Collaborator Author

@sundayonah sundayonah Apr 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is not enough. The first check should includes what we have here

if currency == "NGN" || currency == "KES" || currency == "GHS"
// get market price from each Bitget, Binance, and Quidax
// calculate the median

based on this part:

GIVEN the currency is NGN
WHEN fetching rates for USDT/NGN
THEN the system should fetch rates from Quidax and Bitget

GIVEN the currency is not NGN
WHEN fetching rates for USDT/
THEN the system should fetch rates from Binance and Bitget

if currency == NGN use FetchQuidaxRates and FetchBitgetRates
else if currency is ... use FetchBinanceRates and FetchBitgetRates

         if currency == "NGN" {
		quidaxRate, err := FetchQuidaxRates(currency)
		if err == nil {
			prices = append(prices, quidaxRate)
		}
	} else {
		binanceRates, err := FetchBinanceRates(currency)
		if err == nil {
			prices = append(prices, binanceRates...)
		}
	}

	// Fetch Bitget rates for all supported currencies
	bitgetRates, err := FetchBitgetRates(currency)
	if err == nil {
		prices = append(prices, bitgetRates...)
	}
	```

quidaxRate, err := FetchQuidaxRates(currency)
if err == nil {
prices = append(prices, quidaxRate)
}
} else {
binanceRates, err := FetchBinanceRates(currency)
if err == nil {
prices = append(prices, binanceRates...)
}
}

// Fetch Bitget rates for all supported currencies
bitgetRates, err := FetchBitgetRates(currency)
if err == nil {
prices = append(prices, bitgetRates...)
}

if len(prices) == 0 {
return decimal.Zero, fmt.Errorf("ComputeMarketRate: no valid rates found")
}

// Return the median price
return Median(prices), nil
}

// FetchQuidaxRate fetches the USDT exchange rate from Quidax (NGN only)
func FetchQuidaxRates(currency string) (decimal.Decimal, error) {
url := fmt.Sprintf("/api/v1/markets/tickers/usdt%s", strings.ToLower(currency))

res, err := fastshot.NewClient(QuidaxAPIURL).
Config().SetTimeout(30*time.Second).
Build().GET(url).
Retry().Set(3, 5*time.Second).
Send()
if err != nil {
return decimal.Zero, fmt.Errorf("FetchQuidaxRate: %w", err)
}

data, err := ParseJSONResponse(res.RawResponse)
if err != nil {
return decimal.Zero, fmt.Errorf("FetchQuidaxRate: %w", err)
}

price, err := decimal.NewFromString(data["data"].(map[string]interface{})["ticker"].(map[string]interface{})["buy"].(string))
if err != nil {
return decimal.Zero, fmt.Errorf("FetchQuidaxRate: %w", err)
}

return price, nil
}

// FetchBinanceRates fetches USDT exchange rates from Binance P2P
func FetchBinanceRates(currency string) ([]decimal.Decimal, error) {
res, err := fastshot.NewClient(BinanceAPIURL).
Config().SetTimeout(30*time.Second).
Header().Add("Content-Type", "application/json").
Build().POST("/bapi/c2c/v2/friendly/c2c/adv/search").
Retry().Set(3, 5*time.Second).
Body().AsJSON(map[string]interface{}{
"asset": "USDT",
"fiat": currency,
"tradeType": "SELL",
"page": 1,
"rows": 20,
}).
Send()
if err != nil {
return nil, fmt.Errorf("FetchBinanceRates: %w", err)
}

resData, err := ParseJSONResponse(res.RawResponse)
if err != nil {
return nil, fmt.Errorf("FetchBinanceRates: %w", err)
}

data, ok := resData["data"].([]interface{})
if !ok || len(data) == 0 {
return nil, fmt.Errorf("FetchBinanceRates: no data in response")
}

var prices []decimal.Decimal
for _, item := range data {
adv, ok := item.(map[string]interface{})["adv"].(map[string]interface{})
if !ok {
continue
}

price, err := decimal.NewFromString(adv["price"].(string))
if err != nil {
continue
}

prices = append(prices, price)
}

if len(prices) == 0 {
return nil, fmt.Errorf("FetchBinanceRates: no valid prices found")
}

return prices, nil
}

// FetchBitgetRates fetches USDT exchange rates from Bitget P2P listings
func FetchBitgetRates(currency string) ([]decimal.Decimal, error) {
payload := map[string]interface{}{
"side": 2,
"pageNo": 1,
"pageSize": 20,
"coinCode": "USDT",
"fiatCode": currency,
"languageType": 0,
}

payloadBytes, err := json.Marshal(payload)
if err != nil {
return nil, fmt.Errorf("FetchBitgetRates: failed to marshal payload: %w", err)
}

client := &http.Client{Timeout: 30 * time.Second}
req, err := http.NewRequest("POST", BitgetAPIURL+"/v1/p2p/pub/adv/queryAdvList", bytes.NewBuffer(payloadBytes))
if err != nil {
return nil, fmt.Errorf("FetchBitgetRates: failed to create request: %w", err)
}
req.Header.Set("Content-Type", "application/json")

var resp *http.Response
err = Retry(3, 5*time.Second, func() error {
var retryErr error
resp, retryErr = client.Do(req)
return retryErr
})
if err != nil {
return nil, fmt.Errorf("FetchBitgetRates: failed to send request after retries: %w", err)
}
defer resp.Body.Close()

bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("FetchBitgetRates: failed to read response body: %w", err)
}

var resData types.BitgetResponse
err = json.Unmarshal(bodyBytes, &resData)
if err != nil {
fmt.Printf("FetchBitgetRates: failed to parse response, raw body: %s\n", string(bodyBytes))
return nil, fmt.Errorf("FetchBitgetRates: failed to parse response: %w", err)
}

if resData.Code != "00000" {
return nil, fmt.Errorf("FetchBitgetRates: API error: %s", resData.Msg)
}

if len(resData.Data.DataList) == 0 {
return nil, fmt.Errorf("FetchBitgetRates: no sell ads found for %s/USDT", currency)
}

var prices []decimal.Decimal
for i, ad := range resData.Data.DataList {
if ad.CoinCode != "USDT" || ad.FiatCode != currency {
continue
}
price, err := decimal.NewFromString(ad.Price)
if err != nil {
fmt.Printf("FetchBitgetRates: skipping ad at index %d with invalid price '%s': %v\n", i, ad.Price, err)
continue
}
prices = append(prices, price)
}

if len(prices) == 0 {
return nil, fmt.Errorf("FetchBitgetRates: no valid sell ads found for %s/USDT", currency)
}

return prices, nil
}

Loading
Loading