Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Candles HLOC Data #1887

Merged
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
0c5bec0
Add getOrderBookMidPrice function
adamfraser Jun 27, 2024
1e9ccd7
Add orderbook mid open and mid close prices to candles model
adamfraser Jul 10, 2024
4ebf9be
mend
adamfraser Jul 10, 2024
c104358
Merge branch 'main' into adam/ct-973-add-midbookopen-and-midbookclose…
adamfraser Jul 10, 2024
9ec0e6f
Github workflow tweak
adamfraser Jul 10, 2024
2cd954b
Fix typo on column name
adamfraser Jul 10, 2024
464d7ea
Set orderbook mid price to be required but nullable
adamfraser Jul 11, 2024
99dfeab
Add athena table changes
adamfraser Jul 11, 2024
1e055c3
Get tests building
adamfraser Jul 11, 2024
1bd9a8f
WIP: Tests passing
adamfraser Jul 17, 2024
a079b29
Add helper function to reset orderbook levels cache
adamfraser Jul 17, 2024
e850980
Add tests for orderbook price updates
adamfraser Jul 17, 2024
8b259a1
Return candle updates as well as new candles
adamfraser Jul 18, 2024
1187de5
Add functionality to use orderbook for HLOC in candles API response
adamfraser Jul 18, 2024
9346dc2
Add logging for unexpected orderbook level values
adamfraser Jul 23, 2024
acb65b7
Deploy to Dev2
adamfraser Jul 23, 2024
b0c7e81
Fix typo in db migration
adamfraser Jul 23, 2024
a5c78be
Comments
adamfraser Jul 23, 2024
297d4d5
Athena db column changes
adamfraser Jul 23, 2024
b2b708f
Revert "Add functionality to use orderbook for HLOC in candles API re…
adamfraser Jul 23, 2024
3470736
Remove unused comments
adamfraser Jul 23, 2024
af54233
Update Candles DB migration file
adamfraser Jul 24, 2024
78cdcc5
Remove unnecessary Promise.all
adamfraser Jul 24, 2024
1b534fd
Test name tweak
adamfraser Jul 24, 2024
0ee1a17
Remove unnecessary redis helper
adamfraser Jul 24, 2024
9055013
Match types to codebase style
adamfraser Jul 24, 2024
33d678f
Revert "Github workflow tweak"
adamfraser Jul 24, 2024
6652e22
Update candle model jsonSchema
adamfraser Jul 25, 2024
5037908
orderbookMidPrices arent required fields
adamfraser Jul 25, 2024
3903935
Wait for candle updates to run before checking for updates
adamfraser Jul 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions indexer/packages/postgres/__tests__/helpers/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,8 @@ export const defaultCandle: CandleCreateObject = {
usdVolume: '2200000',
trades: 300,
startingOpenInterest: '200000',
orderbookMidPriceOpen: '11500',
orderbookMidPriceClose: '12500',
};

export const defaultCandleId: string = CandleTable.uuid(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,11 @@ describe('CandleTable', () => {
const updatedCandle: CandleUpdateObject = {
id: defaultCandleId,
open: '100',
orderbookMidPriceClose: '200',
orderbookMidPriceOpen: '300',
};

await CandleTable.update({
id: defaultCandleId,
open: '100',
});
await CandleTable.update(updatedCandle);

const candle: CandleFromDatabase | undefined = await CandleTable.findById(
defaultCandleId,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import * as Knex from 'knex';

export async function up(knex: Knex): Promise<void> {
return knex
.schema
.alterTable('candles', (table) => {
table.decimal('orderbookMidPriceOpen', null);
table.decimal('orderbookMidPriceClose', null);
});
Comment on lines +3 to +9
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specify precision and scale for decimal columns.

It's a good practice to specify the precision and scale for decimal columns to avoid potential issues with data storage and retrieval.

-  table.decimal('orderbookMidPriceOpen', null);
-  table.decimal('orderbookMidPriceClose', null);
+  table.decimal('orderbookMidPriceOpen', 18, 8);
+  table.decimal('orderbookMidPriceClose', 18, 8);
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
export async function up(knex: Knex): Promise<void> {
return knex
.schema
.alterTable('candles', (table) => {
table.decimal('orderbookMidPriceOpen', null);
table.decimal('orderbookMidPriceClose', null);
});
export async function up(knex: Knex): Promise<void> {
return knex
.schema
.alterTable('candles', (table) => {
table.decimal('orderbookMidPriceOpen', 18, 8);
table.decimal('orderbookMidPriceClose', 18, 8);
});

}

export async function down(knex: Knex): Promise<void> {
return knex
.schema
.alterTable('candles', (table) => {
table.dropColumn('orderbookMidPriceOpen');
table.dropColumn('orderbookMidPriceClose');
});
}
4 changes: 4 additions & 0 deletions indexer/packages/postgres/src/models/candle-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,4 +77,8 @@ export default class CandleModel extends Model {
trades!: number;

startingOpenInterest!: string;

orderbookMidPriceOpen?: string;

orderbookMidPriceClose?: string;
}
4 changes: 4 additions & 0 deletions indexer/packages/postgres/src/types/candle-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ export interface CandleCreateObject {
usdVolume: string;
trades: number;
startingOpenInterest: string;
orderbookMidPriceOpen: string | undefined;
orderbookMidPriceClose: string | undefined;
}

export interface CandleUpdateObject {
Expand All @@ -24,6 +26,8 @@ export interface CandleUpdateObject {
usdVolume?: string;
trades?: number;
startingOpenInterest?: string;
orderbookMidPriceOpen?: string;
orderbookMidPriceClose?: string;
}

export enum CandleResolution {
Expand Down
2 changes: 2 additions & 0 deletions indexer/packages/postgres/src/types/db-model-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,8 @@ export interface CandleFromDatabase extends IdBasedModelFromDatabase {
usdVolume: string;
trades: number;
startingOpenInterest: string;
orderbookMidPriceOpen?: string | null;
orderbookMidPriceClose?: string | null;
}

export interface PnlTicksFromDatabase extends IdBasedModelFromDatabase {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
deleteZeroPriceLevel,
getLastUpdatedKey,
deleteStalePriceLevel,
getOrderBookMidPrice,
} from '../../src/caches/orderbook-levels-cache';
import { OrderSide } from '@dydxprotocol-indexer/postgres';
import { OrderbookLevels, PriceLevel } from '../../src/types';
Expand Down Expand Up @@ -685,4 +686,93 @@ describe('orderbookLevelsCache', () => {
expect(size).toEqual('10');
});
});

describe('getMidPrice', () => {
beforeEach(() => {
jest.restoreAllMocks();
jest.restoreAllMocks();
});
afterEach(() => {
jest.restoreAllMocks();
jest.restoreAllMocks();
});

it('returns the correct mid price', async () => {
await Promise.all([
updatePriceLevel({
ticker,
side: OrderSide.BUY,
humanPrice: '45200',
sizeDeltaInQuantums: '2000',
client,
}),
updatePriceLevel({
ticker,
side: OrderSide.BUY,
humanPrice: '45100',
sizeDeltaInQuantums: '2000',
client,
}),
updatePriceLevel({
ticker,
side: OrderSide.BUY,
humanPrice: '45300',
sizeDeltaInQuantums: '2000',
client,
}),
updatePriceLevel({
ticker,
side: OrderSide.SELL,
humanPrice: '45500',
sizeDeltaInQuantums: '2000',
client,
}),
updatePriceLevel({
ticker,
side: OrderSide.SELL,
humanPrice: '45400',
sizeDeltaInQuantums: '2000',
client,
}),
updatePriceLevel({
ticker,
side: OrderSide.SELL,
humanPrice: '45600',
sizeDeltaInQuantums: '2000',
client,
}),
]);

const midPrice = await getOrderBookMidPrice(ticker, client);
expect(midPrice).toEqual(45350);
});

});

it('returns undefined if there are no bids or asks', async () => {
await updatePriceLevel({
ticker,
side: OrderSide.SELL,
humanPrice: '45400',
sizeDeltaInQuantums: '2000',
client,
});

const midPrice = await getOrderBookMidPrice(ticker, client);
expect(midPrice).toBeUndefined();
});

it('returns undefined if humanPrice is NaN', async () => {
await updatePriceLevel({
ticker,
side: OrderSide.SELL,
humanPrice: 'nan',
sizeDeltaInQuantums: '2000',
client,
});

const midPrice = await getOrderBookMidPrice(ticker, client);

expect(midPrice).toBeUndefined();
});
});
35 changes: 35 additions & 0 deletions indexer/packages/redis/src/caches/orderbook-levels-cache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -529,3 +529,38 @@ function convertToPriceLevels(
};
});
}

export async function getOrderBookMidPrice(
ticker: string,
client: RedisClient,
): Promise<number | undefined> {
const levels = await getOrderBookLevels(ticker, client, {
removeZeros: true,
sortSides: true,
uncrossBook: true,
limitPerSide: 1,
});

if (levels.bids.length === 0 || levels.asks.length === 0) {
const message: string = `Orderbook bid length: ${levels.bids.length}, ask length: ${levels.asks.length}. Expected > 0`;
logger.error({
at: 'orderbook-levels-cache#getOrderBookMidPrice',
message,
});
return undefined;
}

const bestAsk = Number(levels.asks[0].humanPrice);
const bestBid = Number(levels.bids[0].humanPrice);

if (bestAsk === undefined || bestBid === undefined) {
const message: string = `Orderbook bid or ask failed to parse to Number, bid: ${levels.bids[0]}, ask: ${levels.asks[0]}`;
logger.error({
at: 'orderbook-levels-cache#getOrderBookMidPrice',
message,
});
return undefined;
}

return bestBid + (bestAsk - bestBid) / 2;
}
8 changes: 8 additions & 0 deletions indexer/services/comlink/public/api-documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -692,6 +692,8 @@ fetch(`${baseURL}/candles/perpetualMarkets/{ticker}?resolution=1MIN`,
"usdVolume": "string",
"trades": 0.1,
"startingOpenInterest": "string",
"orderbookMidPriceOpen": "string",
"orderbookMidPriceClose": "string",
"id": "string"
}
]
Expand Down Expand Up @@ -3575,6 +3577,8 @@ This operation does not require authentication
"usdVolume": "string",
"trades": 0.1,
"startingOpenInterest": "string",
"orderbookMidPriceOpen": "string",
"orderbookMidPriceClose": "string",
"id": "string"
}

Expand All @@ -3595,6 +3599,8 @@ This operation does not require authentication
|usdVolume|string|true|none|none|
|trades|number(double)|true|none|none|
|startingOpenInterest|string|true|none|none|
|orderbookMidPriceOpen|string¦null|false|none|none|
|orderbookMidPriceClose|string¦null|false|none|none|
|id|string|true|none|none|

## CandleResponse
Expand All @@ -3619,6 +3625,8 @@ This operation does not require authentication
"usdVolume": "string",
"trades": 0.1,
"startingOpenInterest": "string",
"orderbookMidPriceOpen": "string",
"orderbookMidPriceClose": "string",
"id": "string"
}
]
Expand Down
8 changes: 8 additions & 0 deletions indexer/services/comlink/public/swagger.json
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,14 @@
"startingOpenInterest": {
"type": "string"
},
"orderbookMidPriceOpen": {
"type": "string",
"nullable": true
},
"orderbookMidPriceClose": {
"type": "string",
"nullable": true
},
"id": {
"type": "string"
}
Expand Down
18 changes: 18 additions & 0 deletions indexer/services/ender/__tests__/helpers/redis-helpers.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import { OrderSide } from '@dydxprotocol-indexer/postgres';
import {
NextFundingCache,
OrderbookLevelsCache,
StateFilledQuantumsCache,
} from '@dydxprotocol-indexer/redis';
import Big from 'big.js';
Expand Down Expand Up @@ -29,3 +31,19 @@ export async function expectStateFilledQuantums(
expect(stateFilledQuantums).toBeDefined();
expect(stateFilledQuantums).toEqual(quantums);
}

export async function updatePriceLevel(
adamfraser marked this conversation as resolved.
Show resolved Hide resolved
ticker: string,
price: string,
side: OrderSide,
): Promise<void> {
const quantums: string = '30';

await OrderbookLevelsCache.updatePriceLevel({
ticker,
side,
humanPrice: price,
sizeDeltaInQuantums: quantums,
client: redisClient,
});
}
Loading
Loading