New capital storage v.145 MPORTANT

[robcarver17]

When you get v1.45 you will find that the capital storage has been changed (from native mongo to arctic). This was to fix #716

rob@TradingPC2:~/pysystemtrade/sysproduction/linux/scripts$ . update_total_capital 

2022-08-23 11:06:21 {'type': 'Update-Total-Capital', 'component': 'mongoIbBrokerClientIdData'}  Locked IB client ID 1421
2022-08-23 11:06:27 {'type': 'Update-Total-Capital', 'broker': 'IB', 'clientid': 1421}  Broker account value is 371226.830000
2022-08-23 11:06:28 {'type': 'Update-Total-Capital', 'broker': 'IB', 'clientid': 1421} *CRITICAL* Error No capital in database - can't update whilst updating total capital; you may have to use update_capital_manual script or function OR IF YOU HAVEN'T DONE SO ALREADY YOU MUST RUN sysdata/production/TEMP_capital_transfer.py from the command line to transfer your old capital
2022-08-23 11:06:30 {'type': 'Update-Total-Capital', 'broker': 'IB', 'clientid': 1421}  Sent email subject *CRITICAL* ERROR: {'type': 'Update-Total-Capital', 'broker': 'IB', 'clientid': 1421}: Error No capital in database - can't update whilst updating total capital; you may have to use update_capital_manual script or function OR IF YOU HAVEN'T DONE SO ALREADY YOU MUST RUN sysdata/production/TEMP_capital_transfer.py from the command line to transfer your old capital

To transfer your old capital data across:

rob@TradingPC2:~/pysystemtrade/sysdata/production$ python3 TEMP_capital_transfer.py 

One off utility to transform old total capital storage into new
Strongly suggest you have backups and run this when nothing else is running!

### (At a minimum, you must stop anything that modifies capital from running)

Continue?yes
Original capital
                            Actual       Max  Accumulated     Broker
2014-04-08 00:00:00.000  600000.00  600000.0      2455.30  609168.47
2014-04-09 00:00:00.000  600000.00  600000.0      2455.30  609168.47
2014-04-10 00:00:00.000  600000.00  600000.0      6336.76  613049.93
2014-04-11 00:00:00.000  598915.82  600000.0       622.44  607335.61
2014-04-14 00:00:00.000  594625.14  600000.0     -1522.90  605190.27
...                            ...       ...          ...        ...
2022-08-23 01:00:52.869  370346.07  400000.0    759732.58  371124.56
2022-08-23 03:00:47.927  370346.07  400000.0    759732.58  371124.56
2022-08-23 05:00:51.304  370145.64  400000.0    759532.15  370924.13
2022-08-23 07:00:52.248  370088.68  400000.0    759475.19  370867.17
2022-08-23 09:00:54.914  370129.85  400000.0    759516.36  370908.34

[6449 rows x 4 columns]
Happy to delete current 'new' capital and replace with transferred 'old'?y
New capital
                            Actual       Max  Accumulated     Broker
index                                                               
2014-04-08 00:00:00.000  600000.00  600000.0      2455.30  609168.47
2014-04-09 00:00:00.000  600000.00  600000.0      2455.30  609168.47
2014-04-10 00:00:00.000  600000.00  600000.0      6336.76  613049.93
2014-04-11 00:00:00.000  598915.82  600000.0       622.44  607335.61
2014-04-14 00:00:00.000  594625.14  600000.0     -1522.90  605190.27
...                            ...       ...          ...        ...
2022-08-23 01:00:52.869  370346.07  400000.0    759732.58  371124.56
2022-08-23 03:00:47.927  370346.07  400000.0    759732.58  371124.56
2022-08-23 05:00:51.304  370145.64  400000.0    759532.15  370924.13
2022-08-23 07:00:52.248  370088.68  400000.0    759475.19  370867.17
2022-08-23 09:00:54.914  370129.85  400000.0    759516.36  370908.34

[6449 rows x 4 columns]

#### I do a visual check that the new capital matches the old

Happy to delete old global capital?y  

## now for strategies

{'dynamic_TF_carry': 2021-10-07 15:27:39.678    371408.095476
2021-10-08 01:00:36.548    371771.399142
2021-10-11 01:00:38.792    373174.378830
2021-10-12 01:00:22.710    378191.157102
2021-10-13 01:00:27.619    378292.486968
                               ...      
2022-08-16 01:01:05.562    368242.560000
2022-08-17 01:01:11.787    366206.320000
2022-08-18 01:01:14.405    367290.670000
2022-08-19 01:01:18.841    368676.710000
2022-08-23 01:01:10.476    370346.070000
Name: capital_value, Length: 231, dtype: float64, 'example': 179285.17, 'medium_speed_TF_carry': 2020-05-27 12:50:48.728    348897.650000
2020-05-28 06:48:25.020    348897.650000
2020-05-28 06:48:53.988    362058.620000
2020-05-29 10:13:28.492    361564.370000
2020-06-01 14:03:06.953    363061.150000
                               ...      
2022-03-20 11:34:24.386        37.839827
2022-03-21 01:00:41.971        37.980416
2022-03-22 01:00:40.509        38.491843
2022-03-23 01:00:48.716        38.312050
2022-03-24 01:00:41.883        38.875056
Name: capital_value, Length: 563, dtype: float64}
Transferring dynamic_TF_carry
Original capital
2021-10-07 15:27:39.678    371408.095476
2021-10-08 01:00:36.548    371771.399142
2021-10-11 01:00:38.792    373174.378830
2021-10-12 01:00:22.710    378191.157102
2021-10-13 01:00:27.619    378292.486968
                               ...      
2022-08-16 01:01:05.562    368242.560000
2022-08-17 01:01:11.787    366206.320000
2022-08-18 01:01:14.405    367290.670000
2022-08-19 01:01:18.841    368676.710000
2022-08-23 01:01:10.476    370346.070000
Name: capital_value, Length: 231, dtype: float64
Happy to delete current 'new' capital and replace with transferred 'old'?y
New capital
                         capital_value
index                                 
2021-10-07 15:27:39.678  371408.095476
2021-10-08 01:00:36.548  371771.399142
2021-10-11 01:00:38.792  373174.378830
2021-10-12 01:00:22.710  378191.157102
2021-10-13 01:00:27.619  378292.486968
...                                ...
2022-08-16 01:01:05.562  368242.560000
2022-08-17 01:01:11.787  366206.320000
2022-08-18 01:01:14.405  367290.670000
2022-08-19 01:01:18.841  368676.710000
2022-08-23 01:01:10.476  370346.070000

[231 rows x 1 columns]
Happy to delete old capital?y

### sometimes you get junk but it deals with it

Transferring example
Original capital
179285.17
Happy to delete current 'new' capital and replace with transferred 'old'?y
Problem with capital adding 179285.17 - probably a data artifact can ignore
New capital
missing data
Happy to delete old capital?y
Transferring medium_speed_TF_carry
Original capital
2020-05-27 12:50:48.728    348897.650000
2020-05-28 06:48:25.020    348897.650000
2020-05-28 06:48:53.988    362058.620000
2020-05-29 10:13:28.492    361564.370000
2020-06-01 14:03:06.953    363061.150000
                               ...      
2022-03-20 11:34:24.386        37.839827
2022-03-21 01:00:41.971        37.980416
2022-03-22 01:00:40.509        38.491843
2022-03-23 01:00:48.716        38.312050
2022-03-24 01:00:41.883        38.875056
Name: capital_value, Length: 563, dtype: float64
Happy to delete current 'new' capital and replace with transferred 'old'?y
New capital
                         capital_value
index                                 
2020-05-27 12:50:48.728  348897.650000
2020-05-28 06:48:25.020  348897.650000
2020-05-28 06:48:53.988  362058.620000
2020-05-29 10:13:28.492  361564.370000
2020-06-01 14:03:06.953  363061.150000
...                                ...
2022-03-20 11:34:24.386      37.839827
2022-03-21 01:00:41.971      37.980416
2022-03-22 01:00:40.509      38.491843
2022-03-23 01:00:48.716      38.312050
2022-03-24 01:00:41.883      38.875056

[563 rows x 1 columns]
Happy to delete old capital?y

Everything should now work fine.

These temporary files, and a couple of others required to support them, will vanish in a future version.

[VMatthijs]

Hmm. The recent changes to update capital seem to have broken things for me. Even after running this script, I get an error when updating strategy capital:
2022-08-27 08:45:28 {'type': 'Update-Strategy-Capital'} *CRITICAL* Error ['named_object' object has no attribute 'squeeze'] whilst allocating strategy capital
After this, system backtests do not run anymore:

Traceback (most recent call last):
  File "/home/matthijs/pysystemtrade/sysproduction/linux/scripts/run.py", line 65, in <module>
    func(*args, **kwargs)
  File "/home/matthijs/pysystemtrade/sysproduction/update_system_backtests.py", line 29, in update_system_backtests
    system_backtest_runner.run_strategy_method()
  File "/home/matthijs/pysystemtrade/syscontrol/strategy_tools.py", line 41, in run_strategy_method
    method()
  File "/home/matthijs/pysystemtrade/sysproduction/strategy_code/run_system_classic.py", line 53, in run_backtest
    base_currency, notional_trading_capital = self._get_currency_and_capital()
  File "/home/matthijs/pysystemtrade/sysproduction/strategy_code/run_system_classic.py", line 69, in _get_currency_and_capital
    notional_trading_capital = capital_data.get_current_capital_for_strategy(strategy_name)
  File "/home/matthijs/pysystemtrade/sysproduction/data/capital.py", line 142, in get_current_capital_for_strategy
    capital_value = self.db_capital_data.get_current_capital_for_strategy(
  File "/home/matthijs/pysystemtrade/sysdata/production/new_capital.py", line 176, in get_current_capital_for_strategy
    return capital_series.iloc[-1, 0]
AttributeError: 'named_object' object has no attribute 'iloc'

Does anyone else have this issue?

[VMatthijs]

(By the way, I fixed this on my local branch by manually updating the strategy capital, but I felt like I should still let you know that this seemed to cause problems, at least in my setup. If I create new strategies, it still continues to cause problems for me, until I manually assign some suitable strategy capital dataframe to get it started.)

[robcarver17]

I can't reproduce this issue; but it looks like the transfer code didn't properly recreate your strategy capital and so returns missing data (named_object type). You could try rerunning the transfer code and pasting the trace of that.