guofu
/
dolphin-dev
1
0
Fork 0
Code Issues 3 Pull Requests Packages Projects Releases 1 Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '1f53e14564'
${ noResults }
dolphin-dev/ipynb/ddb.ipynb

1857 lines
55 KiB
Raw Normal View History Unescape

1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
{
"cells": [
{
"cell_type": "code",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
"execution_count": 2,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"id": "7246e0c8-61cd-4cbf-a978-aa0dc0172d6d",
"metadata": {},
"outputs": [],
"source": [
"import dolphindb as ddb"
]
},
{
"cell_type": "code",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
"execution_count": 3,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"id": "5d0f471e-682e-43cc-abdb-7e52f3bbd707",
"metadata": {},
"outputs": [],
"source": [
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
"sess = ddb.session('localhost', 8848)\n",
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"sess.login('admin', '123456')"
]
},
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
{
"cell_type": "code",
"execution_count": null,
"id": "fca58bdc-2aa1-4610-9a94-67d55f97a6e1",
"metadata": {},
"outputs": [],
"source": [
"sess.run(\"\"\"\n",
" migrate('/data/dolphindb/backup/', \"dfs://hft_stock_ts\", \"OrderPartitioned\")\n",
"\"\"\")"
]
},
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
{
"cell_type": "code",
Update README.md
2 years ago
"execution_count": 60,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"id": "ed17fd0b-9b36-47e4-9ab6-11459a3621fb",
"metadata": {},
Update README.md
2 years ago
"outputs": [
{
"data": {
"text/plain": [
"115548"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"source": [
Update README.md
2 years ago
"sess = ddb.session('192.168.1.7', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"# backup(backup_path, sql_obj, force, parallel)\n",
"code = \"\"\"\n",
Update README.md
2 years ago
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://hft_stock_ts\", \"TickPartitioned\")>, true, true)\n",
"\"\"\"\n",
"\n",
"sess.run(code)"
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
]
},
{
"cell_type": "code",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"execution_count": 32,
"id": "c8d07fc8-d80c-490f-9220-0d3e8e4c72a4",
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"50"
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
]
},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"execution_count": 32,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"sess = ddb.session('localhost', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"# backup(backup_path, sql_obj, force, parallel)\n",
"code = \"\"\"\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"bs_common_ori\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"bs_common_adj\")>, true, false);\n",
" \n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"cfs_common_ori\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"cfs_common_adj\")>, true, false);\n",
" \n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"is_common_adj\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"is_common_adj\")>, true, false);\n",
" \n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"divident\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"earnings_preannouncement\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"preliminary_earnings_estimate\")>, true, false);\n",
"\"\"\"\n",
"\n",
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"sess.run(code)"
]
},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
{
"cell_type": "code",
"execution_count": 33,
"id": "8b7dae3d-aef1-4c50-92b2-460d4fea0a96",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>tableName</th>\n",
" <th>physicalIndex</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>daily_kline</td>\n",
" <td>uoH</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>hft_daily_factor</td>\n",
" <td>u6J</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" tableName physicalIndex\n",
"0 daily_kline uoH\n",
"1 hft_daily_factor u6J"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" listTables(\"dfs://daily_stock_ts\");\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "8af3381d-9fdc-4cb0-b3a9-d6538db97476",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"50"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://daily_stock_ts\", \"daily_kline\") >, true, false);\n",
"\"\"\")"
]
},
1. `DDBDailyFactor.py` => `DDBFactor.py` 2. 在`DDBLoader.py`中增加了`DDBPITLoader`类，专门用于向DolphinDB里面添加PIT数据 3. 其他一些优化
2 years ago
{
"cell_type": "code",
"execution_count": 69,
"id": "bf3bc38e-74cb-4549-bdca-6a3d2a601488",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2287.0"
]
},
"execution_count": 69,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"114350 / 50"
]
},
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
{
"cell_type": "code",
"execution_count": 6,
"id": "11275975-af43-4123-973a-f75096eb4e43",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2311.0"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"115550 / 50"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "155735c7-bdad-4c64-a2a1-514fe54e088c",
"metadata": {},
"outputs": [],
"source": [
"import dolphindb as ddb\n",
"\n",
"sess = ddb.session('localhost', 8848)\n",
"sess.login('admin', '123456')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a2e95bc5-c1d5-47f5-9488-9b8702947afc",
"metadata": {},
"outputs": [],
"source": [
"# migrate(backupDir, [backupDBPath], [backupTableName], [newDBPath=backupDBPath], [newTableName=backupTableName])\n",
"code = \"\"\"\n",
" migrate('/data/dolphindb/backup/',\"dfs://hft_stock_ts\", \"KLinePartitioned\")\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8a6198ef-e7f1-4f58-ba24-32dc429c9803",
"metadata": {},
"outputs": [],
"source": [
"sess.run(code)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "60f71516-33fc-4147-9740-c493ccb6b6a8",
"metadata": {},
"outputs": [],
"source": [
"def make_kurto_code_partition(partition_num):\n",
" code = \"\"\"\n",
" tbl = loadTable(\"dfs://hft_stock_ts\", \"KLinePartitioned\");\n",
" ret = select \n",
" code, m_nDate, eachPre(\\, m_nClose)-1.0 as ret \n",
" from tbl \n",
" where partition(code, {partition_num}) \n",
" context by m_nDate;\n",
" kurto = select \n",
" code, m_nDate, sqrt(239) * sum(pow(ret, 3)) / pow(sum(pow(ret, 2)), 1.5) as kurto \n",
" from ret \n",
" group by code, m_nDate;\n",
" \"\"\".format(\n",
" partition_num = partition_num\n",
" )\n",
" sess.run(code)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "1397c1b7-f81a-465e-868a-894b3ef818fc",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>count</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000400.SZ</td>\n",
" <td>2305</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000558.SZ</td>\n",
" <td>2063</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000602.SZ</td>\n",
" <td>47</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000677.SZ</td>\n",
" <td>2009</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000731.SZ</td>\n",
" <td>2311</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>94</th>\n",
" <td>688039.SH</td>\n",
" <td>624</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>688052.SH</td>\n",
" <td>52</td>\n",
" </tr>\n",
" <tr>\n",
" <th>96</th>\n",
" <td>688091.SH</td>\n",
" <td>198</td>\n",
" </tr>\n",
" <tr>\n",
" <th>97</th>\n",
" <td>688368.SH</td>\n",
" <td>656</td>\n",
" </tr>\n",
" <tr>\n",
" <th>98</th>\n",
" <td>688396.SH</td>\n",
" <td>575</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>99 rows × 2 columns</p>\n",
"</div>"
],
"text/plain": [
" code count\n",
"0 000400.SZ 2305\n",
"1 000558.SZ 2063\n",
"2 000602.SZ 47\n",
"3 000677.SZ 2009\n",
"4 000731.SZ 2311\n",
".. ... ...\n",
"94 688039.SH 624\n",
"95 688052.SH 52\n",
"96 688091.SH 198\n",
"97 688368.SH 656\n",
"98 688396.SH 575\n",
"\n",
"[99 rows x 2 columns]"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" select code, count(*) from kurto group by code\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "59758901-3728-4355-bbb8-15b3eb537cdf",
"metadata": {},
"outputs": [],
"source": [
"def create_daily_ddb_database():\n",
" code = \"\"\"\n",
" daily_stock_ts = database(\n",
" \"dfs://daily_stock_ts\", \n",
" HASH, [SYMBOL, 50], \n",
" engine = 'TSDB'\n",
" )\n",
" \"\"\"\n",
" sess.run(code)"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "d8fa94ae-6ca4-41f3-8b13-47ab8876ba95",
"metadata": {},
"outputs": [],
"source": [
"def load_daily_ddb_database():\n",
" code = \"\"\"\n",
" daily_stock_ts = database(\"dfs://daily_stock_ts\")\n",
" \"\"\"\n",
" sess.run(code)\n",
"\n",
"load_daily_ddb_database()"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "84eb237e-959c-419f-bce1-6779acda86d4",
"metadata": {},
"outputs": [],
"source": [
"def create_kurto_partition_table():\n",
" # createPartitionedTable(dbHandle, table, tableName, [partitionColumns], [compressMethods], [sortColumns], [keepDuplicates=ALL], [sortKeyMappingFunction])\n",
" code = \"\"\"\n",
" dropTable(daily_stock_ts, \"kurto_partitioned\");\n",
" kurto_partitioned = createPartitionedTable(\n",
" daily_stock_ts,\n",
" kurto, \n",
" \"kurto_partitioned\",\n",
" partitionColumns = 'code',\n",
" compressMethods = {'m_nDate' : 'delta'},\n",
" sortColumns = `code`m_nDate\n",
" );\n",
" \"\"\"\n",
" sess.run(code)\n",
" \n",
"create_kurto_partition_table()"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "0d93b3d1-39b7-4e23-8abf-eda86dcd2a23",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>m_nDate</th>\n",
" <th>kurto</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-0.317991</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000558.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-15.387820</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000677.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-15.459625</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000731.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>15.443339</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000752.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>15.396262</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>000789.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-14.801720</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>000903.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-15.405085</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>000912.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>14.755241</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>000935.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>15.413214</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>000970.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>15.458413</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" code m_nDate kurto\n",
"0 000400.SZ 2013-01-04 -0.317991\n",
"1 000558.SZ 2013-01-04 -15.387820\n",
"2 000677.SZ 2013-01-04 -15.459625\n",
"3 000731.SZ 2013-01-04 15.443339\n",
"4 000752.SZ 2013-01-04 15.396262\n",
"5 000789.SZ 2013-01-04 -14.801720\n",
"6 000903.SZ 2013-01-04 -15.405085\n",
"7 000912.SZ 2013-01-04 14.755241\n",
"8 000935.SZ 2013-01-04 15.413214\n",
"9 000970.SZ 2013-01-04 15.458413"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" kurto_partitioned.tableInsert(kurto);\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "eea5cb0b-8452-41a9-a0ad-7cf52c856c25",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>count</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>154953</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" count\n",
"0 154953"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" select count(*) from kurto_partitioned;\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "9e7bc856-517b-4bf9-866d-4fe85b9705a3",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'partitionType': 5,\n",
" 'partitionColumnType': 17,\n",
" 'partitionColumnIndex': 0,\n",
" 'chunkPath': None,\n",
" 'colDefs': name typeString typeInt comment\n",
" 0 code SYMBOL 17 \n",
" 1 m_nDate DATE 6 \n",
" 2 kurto DOUBLE 16 ,\n",
" 'chunkGranularity': 'TABLE',\n",
" 'partitionTypeName': 'HASH',\n",
" 'keepDuplicates': 'ALL',\n",
" 'engineType': 'TSDB',\n",
" 'partitionColumnName': 'code',\n",
" 'partitionSchema': 50,\n",
" 'sortColumns': array(['code', 'm_nDate'], dtype=object),\n",
" 'partitionSites': None}"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" schema(loadTable(\"dfs://daily_stock_ts\", \"kurto_partitioned\"));\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "94ae00b5-ba42-48b9-b114-012cf306d871",
"metadata": {},
"outputs": [],
"source": [
"sess.run(\"m_table = table(kurto)\")"
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "e30134f9-5110-421d-8f71-1bef25e7ef17",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>m_nDate</th>\n",
" <th>kurto</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-0.317991</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-07</td>\n",
" <td>0.790872</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-08</td>\n",
" <td>0.655005</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-09</td>\n",
" <td>0.965054</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-10</td>\n",
" <td>-0.116488</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154948</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-04</td>\n",
" <td>-15.420192</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154949</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-05</td>\n",
" <td>-15.426922</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154950</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-06</td>\n",
" <td>-15.389434</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154951</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-07</td>\n",
" <td>-15.411602</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154952</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-08</td>\n",
" <td>-15.401453</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>154953 rows × 3 columns</p>\n",
"</div>"
],
"text/plain": [
" code m_nDate kurto\n",
"0 000400.SZ 2013-01-04 -0.317991\n",
"1 000400.SZ 2013-01-07 0.790872\n",
"2 000400.SZ 2013-01-08 0.655005\n",
"3 000400.SZ 2013-01-09 0.965054\n",
"4 000400.SZ 2013-01-10 -0.116488\n",
"... ... ... ...\n",
"154948 688396.SH 2022-07-04 -15.420192\n",
"154949 688396.SH 2022-07-05 -15.426922\n",
"154950 688396.SH 2022-07-06 -15.389434\n",
"154951 688396.SH 2022-07-07 -15.411602\n",
"154952 688396.SH 2022-07-08 -15.401453\n",
"\n",
"[154953 rows x 3 columns]"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"m_table\")"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "947e3346-15f6-4fb5-b2d6-b97b013a3b36",
"metadata": {},
"outputs": [],
"source": [
"code = \"\"\"\n",
" // 保证创建新的分区表不会和已经存在表冲突\n",
" if (existsTable(\"{ddb_daily_path}\", \"{partition_tbl_name}\")) {{\n",
" dropTable({ddb_daily_dbname}, \"{partition_tbl_name}\");\n",
" }}\n",
"\n",
" {partition_tbl_name} = createPartitionedTable(\n",
" dbHandle = {ddb_daily_dbname},\n",
" table = {memory_tbl_name}, \n",
" \"{partition_tbl_name}\",\n",
" partitionColumns = 'code',\n",
" compressMethods = {{'m_nDate' : 'delta'}},\n",
" sortColumns = `code`m_nDate\n",
" );\n",
"\"\"\".format(\n",
" ddb_daily_path = \"A\",\n",
" ddb_daily_dbname = \"B\",\n",
" partition_tbl_name = \"C\",\n",
" memory_tbl_name = \"D\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "5d758c68-301a-47a7-9cd2-b7dd7edf1a55",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"('\\n'\n",
" ' // 保证创建新的分区表不会和已经存在表冲突\\n'\n",
" ' if (existsTable(\"A\", \"C\")) {\\n'\n",
" ' dropTable(B, \"C\");\\n'\n",
" ' }\\n'\n",
" '\\n'\n",
" ' C = createPartitionedTable(\\n'\n",
" ' dbHandle = B,\\n'\n",
" ' table = D, \\n'\n",
" ' \"C\",\\n'\n",
" \" partitionColumns = 'code',\\n\"\n",
" \" compressMethods = {'m_nDate' : 'delta'},\\n\"\n",
" ' sortColumns = `code`m_nDate\\n'\n",
" ' );\\n')\n"
]
}
],
"source": [
"from pprint import pprint\n",
"pprint(code)"
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "400424d9-9c58-4417-94e4-fe1569522b03",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'chunkGranularity': 'TABLE',\n",
" 'chunkPath': None,\n",
" 'colDefs': name typeString typeInt comment\n",
"0 code SYMBOL 17 \n",
"1 m_nDate DATE 6 \n",
"2 kurto DOUBLE 16 ,\n",
" 'engineType': 'TSDB',\n",
" 'keepDuplicates': 'ALL',\n",
" 'partitionColumnIndex': 0,\n",
" 'partitionColumnName': 'code',\n",
" 'partitionColumnType': 17,\n",
" 'partitionSchema': 50,\n",
" 'partitionSites': None,\n",
" 'partitionType': 5,\n",
" 'partitionTypeName': 'HASH',\n",
" 'sortColumns': array(['code', 'm_nDate'], dtype=object)}\n"
]
}
],
"source": [
"pprint(\n",
" sess.run(\"\"\"\n",
" schema(loadTable(\"dfs://daily_stock_ts\", \"hft_daily_factor\"));\n",
" \"\"\")\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "68e74de8-359c-4c47-a361-2c006896211c",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>count</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>309906</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" count\n",
"0 309906"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" select count(*) from loadTable(\"dfs://daily_stock_ts\", \"hft_daily_factor\");\n",
"\"\"\")"
]
},
{
"cell_type": "code",
reorganize the directory structure
2 years ago
"execution_count": 21,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"id": "b1bd7605-c347-44f2-a90f-bc949cde0ef0",
"metadata": {},
reorganize the directory structure
2 years ago
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>report_period</th>\n",
" <th>appear_in_period</th>\n",
" <th>appear_at_date</th>\n",
" <th>TOT_OPER_REV</th>\n",
" <th>OPER_REV</th>\n",
" <th>INT_INC</th>\n",
" <th>INSUR_PREM_UNEARNED</th>\n",
" <th>HANDLING_CHRG_COMM_INC</th>\n",
" <th>NET_INC_OTHER_OPS</th>\n",
" <th>...</th>\n",
" <th>OTHER_INCOME</th>\n",
" <th>MEMO</th>\n",
" <th>ASSET_DISPOSAL_INCOME</th>\n",
" <th>CONTINUED_NET_PROFIT</th>\n",
" <th>END_NET_PROFIT</th>\n",
" <th>CREDIT_IMPAIRMENT_LOSS</th>\n",
" <th>RD_EXPENSE</th>\n",
" <th>STMNOTE_FINEXP</th>\n",
" <th>FIN_EXP_INT_INC</th>\n",
" <th>TOT_OPER_COST2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000677.SZ</td>\n",
" <td>2009-12-31</td>\n",
" <td>2009-12-31</td>\n",
" <td>2010-04-26</td>\n",
" <td>3.514495e+09</td>\n",
" <td>3.514495e+09</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>26392229.68</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>nan</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>3.474455e+09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000677.SZ</td>\n",
" <td>2009-12-31</td>\n",
" <td>2009-12-31</td>\n",
" <td>2011-12-10</td>\n",
" <td>3.514495e+09</td>\n",
" <td>3.514495e+09</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>26392229.68</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>nan</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>3.460335e+09</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>2 rows × 69 columns</p>\n",
"</div>"
],
"text/plain": [
" code report_period appear_in_period appear_at_date TOT_OPER_REV \\\n",
"0 000677.SZ 2009-12-31 2009-12-31 2010-04-26 3.514495e+09 \n",
"1 000677.SZ 2009-12-31 2009-12-31 2011-12-10 3.514495e+09 \n",
"\n",
" OPER_REV INT_INC INSUR_PREM_UNEARNED HANDLING_CHRG_COMM_INC \\\n",
"0 3.514495e+09 0.0 0.0 0.0 \n",
"1 3.514495e+09 0.0 0.0 0.0 \n",
"\n",
" NET_INC_OTHER_OPS ... OTHER_INCOME MEMO ASSET_DISPOSAL_INCOME \\\n",
"0 26392229.68 ... 0.0 nan 0.0 \n",
"1 26392229.68 ... 0.0 nan 0.0 \n",
"\n",
" CONTINUED_NET_PROFIT END_NET_PROFIT CREDIT_IMPAIRMENT_LOSS RD_EXPENSE \\\n",
"0 0.0 0.0 0.0 0.0 \n",
"1 0.0 0.0 0.0 0.0 \n",
"\n",
" STMNOTE_FINEXP FIN_EXP_INT_INC TOT_OPER_COST2 \n",
"0 0.0 0.0 3.474455e+09 \n",
"1 0.0 0.0 3.460335e+09 \n",
"\n",
"[2 rows x 69 columns]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import dolphindb as ddb \n",
"\n",
"sess = ddb.session('192.168.1.167', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"sess.run(\"\"\"\n",
" is_common_adj = loadTable(\"dfs://pit_stock_ts\", \"is_common_adj\")\n",
" is_common_ori = loadTable(\"dfs://pit_stock_ts\", \"is_common_ori\")\n",
" select * from is_common_ori where code='000677.SZ' and report_period=2009.12.31\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "f0b3928d-c06f-4976-b568-091d98293a6e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>report_period</th>\n",
" <th>appear_in_period</th>\n",
" <th>appear_at_date</th>\n",
" <th>TOT_OPER_REV</th>\n",
" <th>OPER_REV</th>\n",
" <th>INT_INC</th>\n",
" <th>INSUR_PREM_UNEARNED</th>\n",
" <th>HANDLING_CHRG_COMM_INC</th>\n",
" <th>NET_INC_OTHER_OPS</th>\n",
" <th>...</th>\n",
" <th>OTHER_INCOME</th>\n",
" <th>MEMO</th>\n",
" <th>ASSET_DISPOSAL_INCOME</th>\n",
" <th>CONTINUED_NET_PROFIT</th>\n",
" <th>END_NET_PROFIT</th>\n",
" <th>CREDIT_IMPAIRMENT_LOSS</th>\n",
" <th>RD_EXPENSE</th>\n",
" <th>STMNOTE_FINEXP</th>\n",
" <th>FIN_EXP_INT_INC</th>\n",
" <th>TOT_OPER_COST2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000677.SZ</td>\n",
" <td>2009-12-31</td>\n",
" <td>2010-12-31</td>\n",
" <td>2011-04-29</td>\n",
" <td>3.514495e+09</td>\n",
" <td>3.514495e+09</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>26392229.68</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>nan</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>3.474455e+09</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>1 rows × 69 columns</p>\n",
"</div>"
],
"text/plain": [
" code report_period appear_in_period appear_at_date TOT_OPER_REV \\\n",
"0 000677.SZ 2009-12-31 2010-12-31 2011-04-29 3.514495e+09 \n",
"\n",
" OPER_REV INT_INC INSUR_PREM_UNEARNED HANDLING_CHRG_COMM_INC \\\n",
"0 3.514495e+09 0.0 0.0 0.0 \n",
"\n",
" NET_INC_OTHER_OPS ... OTHER_INCOME MEMO ASSET_DISPOSAL_INCOME \\\n",
"0 26392229.68 ... 0.0 nan 0.0 \n",
"\n",
" CONTINUED_NET_PROFIT END_NET_PROFIT CREDIT_IMPAIRMENT_LOSS RD_EXPENSE \\\n",
"0 0.0 0.0 0.0 0.0 \n",
"\n",
" STMNOTE_FINEXP FIN_EXP_INT_INC TOT_OPER_COST2 \n",
"0 0.0 0.0 3.474455e+09 \n",
"\n",
"[1 rows x 69 columns]"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"select * from is_common_adj where code='000677.SZ' and report_period=2009.12.31\")"
]
},
{
"cell_type": "code",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"execution_count": 29,
reorganize the directory structure
2 years ago
"id": "d68ea326-82c3-4a7c-97cf-c04dd8aee56b",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
" <th>sum_cnt</th>\n",
reorganize the directory structure
2 years ago
" </tr>\n",
" </thead>\n",
" <tbody>\n",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
" <tr>\n",
" <th>0</th>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
" <td>1468390810</td>\n",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
" </tr>\n",
reorganize the directory structure
2 years ago
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
" sum_cnt\n",
"0 1468390810"
reorganize the directory structure
2 years ago
]
},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"execution_count": 29,
reorganize the directory structure
2 years ago
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess = ddb.session('192.168.1.7', 8848)\n",
"sess.login('admin', '123456')\n",
"sess.run(\"\"\"\n",
"tbl = loadTable(\"dfs://hft_stock_ts\", 'TickPartitioned');\n",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
"select sum(cnt) from (select count(*) as cnt from tbl map);\n",
"\"\"\")"
]
},
{
"cell_type": "code",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"execution_count": 38,
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
"id": "7cbc4906-7756-424a-9ce5-9d2b6d1bab4b",
"metadata": {},
"outputs": [
{
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sum_cnt</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4991926704</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sum_cnt\n",
"0 4991926704"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
}
],
"source": [
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"sess = ddb.session('192.168.1.7', 8848)\n",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
2 years ago
"sess.login('admin', '123456')\n",
"sess.run(\"\"\"\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
" tbl = loadTable(\"dfs://hft_stock_ts\", 'TickPartitioned');\n",
" select sum(cnt) from (select count(*) as cnt from tbl map);\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "e9ab5e57-dce5-4426-9bac-4238cd067197",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sum_cnt</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>9388749</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sum_cnt\n",
"0 9388749"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" select sum(cnt) from (select count(*) as cnt from tbl where code='002182.SZ' map);\n",
"\"\"\")"
]
},
{
"cell_type": "code",
Update README.md
2 years ago
"execution_count": 100,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"id": "4ba45027-bbb5-4b27-99da-3452cc8d2f1c",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>count</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
Update README.md
2 years ago
" <td>2227</td>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" count\n",
Update README.md
2 years ago
"0 2227"
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
]
},
Update README.md
2 years ago
"execution_count": 100,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" select count(*) from (\n",
Update README.md
2 years ago
" select code, m_nDate, count(*) as cnt from tbl where code='002459.SZ' group by code, m_nDate map\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
" );\n",
reorganize the directory structure
2 years ago
"\"\"\")"
]
},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
{
"cell_type": "code",
Update README.md
2 years ago
"execution_count": 103,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"id": "29ab8af5-e571-4064-b691-a186d9fb4d08",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
Update README.md
2 years ago
" <th>count</th>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
" </tr>\n",
" <tr>\n",
" <th>m_nDate</th>\n",
Update README.md
2 years ago
" <th></th>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Empty DataFrame\n",
Update README.md
2 years ago
"Columns: [count]\n",
"Index: []"
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
]
},
Update README.md
2 years ago
"execution_count": 103,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = sess.run(\"\"\"\n",
Update README.md
2 years ago
" select m_nDate, count(*) from tbl where code='002458.SZ' group by m_nDate having count(*) > 5000 map;\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"\"\"\").set_index('m_nDate')\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "013d191b-2a41-4d7b-8382-b6ee8764c2d0",
"metadata": {},
"outputs": [],
"source": [
"import sqlalchemy as sa\n",
"engine = sa.create_engine(\n",
" 'mssql+pyodbc://sa:passw0rd!@192.168.1.7/master?driver=ODBC+Driver+18+for+SQL+Server',\n",
" connect_args = {\n",
" \"TrustServerCertificate\": \"yes\"\n",
" }, echo=False)"
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "c90e8e9d-cde0-4a7a-afd3-ea771483f001",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'20130104'"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with engine.connect() as conn:\n",
" rows = conn.execute(\"select * from [Level2BytesTick].dbo.[Tick] where S_INFO_WINDCODE='002182.SZ'\").fetchall()\n",
" rows = [row for row in rows]\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 58,
"id": "525dfcf1-720a-4b65-90b5-28d0c2118cf5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"dt = pd.to_datetime(rows[0][1], format='%Y%m%d')\n",
"dt in df.index"
]
},
reorganize the directory structure
2 years ago
{
"cell_type": "code",
"execution_count": 31,
"id": "0e6e949b-ff25-40eb-a70a-72a3a6639aec",
"metadata": {},
"outputs": [
{
"ename": "RuntimeError",
"evalue": "<Exception> in run: Failed to read response header from the socket with IO error type 13",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"Input \u001b[0;32mIn [31]\u001b[0m, in \u001b[0;36m<cell line: 15>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m code \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;124mtbl = loadTable(\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdfs://hft_stock_ts\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m, \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mTickPartitioned\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m)\u001b[39m\n\u001b[1;32m 3\u001b[0m \n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[38;5;124m}\u001b[39m\n\u001b[1;32m 14\u001b[0m \u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[0;32m---> 15\u001b[0m \u001b[43msess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcode\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/.venv/tinysoft/lib/python3.8/site-packages/dolphindb/session.py:161\u001b[0m, in \u001b[0;36msession.run\u001b[0;34m(self, script, *args, **kwargs)\u001b[0m\n\u001b[1;32m 159\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfetchSize\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m kwargs\u001b[38;5;241m.\u001b[39mkeys():\n\u001b[1;32m 160\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m BlockReader(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcpp\u001b[38;5;241m.\u001b[39mrunBlock(script, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs))\n\u001b[0;32m--> 161\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcpp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mscript\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mRuntimeError\u001b[0m: <Exception> in run: Failed to read response header from the socket with IO error type 13"
]
}
],
"source": [
"code = \"\"\"\n",
"tbl = loadTable(\"dfs://hft_stock_ts\", 'TickPartitioned')\n",
"\n",
"for (code in ['000666.SZ' , '000667.SZ', '000668.SZ', '000669.SZ']) {\n",
"\tdel_date = 2013.01.01\n",
"\t//end_date = 2013.01.31\n",
"\tdo {\n",
"\t\t//print('Deleting ' + code + ' from ' + start_date + ' to ' + end_date);\n",
"\t\tprint('Deleting ' + code + ' of ' + del_date);\n",
"\t\tdelete from tbl where m_nDate=del_date, code=code;\n",
"\t\tdel_date = temporalAdd(del_date, 1, \"d\");\n",
"\t} while (del_date <= 2022.12.31)\n",
"}\n",
"\"\"\"\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"# 这段代码运行好赶紧注释掉，非常危险\n",
"# sess.run(code)"
reorganize the directory structure
2 years ago
]
},
{
"cell_type": "code",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"execution_count": 37,
reorganize the directory structure
2 years ago
"id": "ea0501e7-d416-45ce-add5-e443c55f158c",
"metadata": {},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
2 years ago
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>industry_code</th>\n",
" <th>industry_name</th>\n",
" <th>industry_level</th>\n",
" <th>enter_date</th>\n",
" <th>exit_date</th>\n",
" <th>is_new</th>\n",
" <th>industry_class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310000</td>\n",
" <td>申万交运设备</td>\n",
" <td>1</td>\n",
" <td>1992-03-31</td>\n",
" <td>2011-10-10</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310300</td>\n",
" <td>申万非汽车交运设备</td>\n",
" <td>2</td>\n",
" <td>1992-03-31</td>\n",
" <td>2011-10-10</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310301</td>\n",
" <td>申万摩托车</td>\n",
" <td>3</td>\n",
" <td>1992-03-31</td>\n",
" <td>2011-10-10</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310000</td>\n",
" <td>申万交运设备</td>\n",
" <td>1</td>\n",
" <td>2011-10-10</td>\n",
" <td>2014-02-21</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310300</td>\n",
" <td>申万非汽车交运设备</td>\n",
" <td>2</td>\n",
" <td>2011-10-10</td>\n",
" <td>2014-02-21</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY260202</td>\n",
" <td>申万工程机械</td>\n",
" <td>3</td>\n",
" <td>2000-10-12</td>\n",
" <td>2014-02-21</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>96</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY640000</td>\n",
" <td>申万机械设备</td>\n",
" <td>1</td>\n",
" <td>2014-02-21</td>\n",
" <td>2021-12-13</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>97</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY640200</td>\n",
" <td>申万专用设备</td>\n",
" <td>2</td>\n",
" <td>2014-02-21</td>\n",
" <td>2021-12-13</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>98</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY640201</td>\n",
" <td>申万工程机械</td>\n",
" <td>3</td>\n",
" <td>2014-02-21</td>\n",
" <td>2021-12-13</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>99</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY640000</td>\n",
" <td>申万机械设备</td>\n",
" <td>1</td>\n",
" <td>2021-12-13</td>\n",
" <td>NaT</td>\n",
" <td>1</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>100 rows × 8 columns</p>\n",
"</div>"
],
"text/plain": [
" code industry_code industry_name industry_level enter_date \\\n",
"0 000017.SZ SWHY310000 申万交运设备 1 1992-03-31 \n",
"1 000017.SZ SWHY310300 申万非汽车交运设备 2 1992-03-31 \n",
"2 000017.SZ SWHY310301 申万摩托车 3 1992-03-31 \n",
"3 000017.SZ SWHY310000 申万交运设备 1 2011-10-10 \n",
"4 000017.SZ SWHY310300 申万非汽车交运设备 2 2011-10-10 \n",
".. ... ... ... ... ... \n",
"95 000157.SZ SWHY260202 申万工程机械 3 2000-10-12 \n",
"96 000157.SZ SWHY640000 申万机械设备 1 2014-02-21 \n",
"97 000157.SZ SWHY640200 申万专用设备 2 2014-02-21 \n",
"98 000157.SZ SWHY640201 申万工程机械 3 2014-02-21 \n",
"99 000157.SZ SWHY640000 申万机械设备 1 2021-12-13 \n",
"\n",
" exit_date is_new industry_class \n",
"0 2011-10-10 0 SWHY \n",
"1 2011-10-10 0 SWHY \n",
"2 2011-10-10 0 SWHY \n",
"3 2014-02-21 0 SWHY \n",
"4 2014-02-21 0 SWHY \n",
".. ... ... ... \n",
"95 2014-02-21 0 SWHY \n",
"96 2021-12-13 0 SWHY \n",
"97 2021-12-13 0 SWHY \n",
"98 2021-12-13 0 SWHY \n",
"99 NaT 1 SWHY \n",
"\n",
"[100 rows x 8 columns]"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
"tbl = loadTable(\"dfs://info_stock_ts\", 'info_industry');\n",
"select top 100 * from tbl map;\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "16dd860d-049f-420e-8d04-d1f8969e5ed1",
"metadata": {},
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
2 years ago
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}