merge main back into testnet7 (#3230)

* Added new issue templates (#3033)

* added new issue templates for support and requestes

* spelling is hard...

* Add lock, keep cache consistent (#3051)

* execute task decorator

* use blockchain lock

* indentation

* lint

* execute_task

* Ms.empty blocks3 (#3064)

* Fix empty blocks

* Remove error log

* Missing imports

* .header_hash instead of .prev_hash in simulator

* Fix typo (#3069)

* Use secure coin ids (#3095)

* Add announcements for standard tx (#3023)

* changed bahviour of wallet and wallet tools for standard solutions

* black formatting wallet_tools

* added a test for hypothetical stolen zero_output coin

* swap to sha256 from sha256tree for message

* fix wallet_tools version, address lint issues

* correctly int_from_bytes

* fix uninstantiated key in dict

* Comment out broken test

* Fix types (used SerializedProgram)

Co-authored-by: Mariano <sorgente711@gmail.com>

* cache VDF validation results (per process) (#3110)

* Update GUI for plot refresh issue (#3135)

Co-authored-by: Adam Kelly <aqk@aqk.im>

* remove commit and not needed query (#3148)

* Copyright 2021 Chia Network in LICENSE (#3153)

* just filter limit (#3152)

* fix plot dupe (#3154)

* increase ratio (#3155)

* wait to end of process to avoid zombies (#3160)

* fix takes 1 positional argument but 2 were given (#3202)

* Update CLI tools to use EiB when appropriate. (#3117)

* Improve streamable (#3031)

* Avoid importing `test_constants` as it takes a long time.

* Factor out `parse_*` functions.

* First crack at refactoring `Streamable.parse`.

* Don't add `_parse_functions` attribute to `Streamable`.

This no longer requires an extra `_parse_functions` attribute on a
`Streamable`, as it may be confusing serializers or other functions
that use `__annotations__`.

* Fix lint problems with `black`.

* Fix `parse_tuple`.

* Defer some parsing failures to parse time rather than class-creation time.

* Tidy up & remove some obsolete stuff.

* Decorate `RequestBlocks` as `streamable`.

* Fix wrong uses of Streamable class

Revert an earlier commit and error out on class creation in case a
Streamable subclass is instantiated incorrectly, e.g. containing a
non-serializable member.

Fix cases where Streamable parent class was forgotten.

* Fix wrong types when creating DerivationRecord and WalletCoinRecord

* additional unit tests for streamable parsers

* add type annotations (#3222)

Co-authored-by: Rostislav <rostislav@users.noreply.github.com>
Co-authored-by: arvidn <arvid@libtorrent.org>

* Tests skipping mempool (#3065)

* Avoid importing `test_constants` as it takes a long time.

* Some new tests that circumvents mempool.

* Fix lint problems.

Co-authored-by: J Eckert <sargonas@users.noreply.github.com>
Co-authored-by: Yostra <straya@chia.net>
Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com>
Co-authored-by: Antonio Borrero Granell <me@antoniobg.com>
Co-authored-by: matt-o-how <48453825+matt-o-how@users.noreply.github.com>
Co-authored-by: Mariano <sorgente711@gmail.com>
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
Co-authored-by: Adam Kelly <aqk@aqk.im>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Joel <34619326+joelcho@users.noreply.github.com>
Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com>
Co-authored-by: Richard Kiss <him@richardkiss.com>
Co-authored-by: Rostislav <rostislav@users.noreply.github.com>

chia/cmds/netspace_funcs.py

@@ -60,8 +60,13 @@ async def netstorge_async(rpc_port: int, delta_block_height: str, start: str) ->
                 f"Header Hash:      0x{newer_block_header.header_hash}\n"
             )
             network_space_terabytes_estimate = network_space_bytes_estimate / 1024 ** 4
-            if network_space_terabytes_estimate > 1024:
-                print(f"The network has an estimated {network_space_terabytes_estimate / 1024:.3f} PiB")
+            if network_space_terabytes_estimate >= 1024:
+                network_space_terabytes_estimate = network_space_terabytes_estimate / 1024
+                if network_space_terabytes_estimate >= 1024:
+                    network_space_terabytes_estimate = network_space_terabytes_estimate / 1024
+                    print(f"The network has an estimated {network_space_terabytes_estimate:.3f} EiB")
+                else:
+                    print(f"The network has an estimated {network_space_terabytes_estimate:.3f} PiB")
             else:
                 print(f"The network has an estimated {network_space_terabytes_estimate:.3f} TiB")
 

chia/cmds/show.py

@@ -88,7 +88,11 @@ async def show_async(
                 network_space_human_readable = blockchain_state["space"] / 1024 ** 4
                 if network_space_human_readable >= 1024:
                     network_space_human_readable = network_space_human_readable / 1024
-                    print(f"{network_space_human_readable:.3f} PiB")
+                    if network_space_human_readable >= 1024:
+                        network_space_human_readable = network_space_human_readable / 1024
+                        print(f"{network_space_human_readable:.3f} EiB")
+                    else:
+                        print(f"{network_space_human_readable:.3f} PiB")
                 else:
                     print(f"{network_space_human_readable:.3f} TiB")
                 print(f"Current difficulty: {difficulty}")

chia/daemon/server.py

@@ -506,6 +506,7 @@ async def _start_plotting(self, id: str, loop: asyncio.AbstractEventLoop, queue:
         finally:
             if current_process is not None:
                 self.services[service_name].remove(current_process)
+                current_process.wait()  # prevent zombies
             self._run_next_serial_plotting(loop, queue)
 
     async def start_plotting(self, request: Dict[str, Any]):

chia/plotting/plot_tools.py

@@ -178,9 +178,12 @@ def process_file(filename: Path) -> Tuple[int, Dict]:
                     log.error(f"Failed to open file {filename}. {e}")
                     return 0, new_provers
                 if stat_info.st_mtime == provers[filename].time_modified:
-                    new_provers[filename] = provers[filename]
                     with plot_ids_lock:
+                        if provers[filename].prover.get_id() in plot_ids:
+                            log.warning(f"Have multiple copies of the plot {filename}, not adding it.")
+                            return 0, new_provers
                         plot_ids.add(provers[filename].prover.get_id())
+                    new_provers[filename] = provers[filename]
                     return stat_info.st_size, new_provers
             try:
                 prover = DiskProver(str(filename))

chia/protocols/full_node_protocol.py

@@ -76,6 +76,7 @@ class RejectBlock(Streamable):
 
 
 @dataclass(frozen=True)
+@streamable
 class RequestBlocks(Streamable):
     start_height: uint32
     end_height: uint32

chia/protocols/timelord_protocol.py

@@ -74,7 +74,7 @@ class NewEndOfSubSlotVDF(Streamable):
 
 @dataclass(frozen=True)
 @streamable
-class RequestCompactProofOfTime:
+class RequestCompactProofOfTime(Streamable):
     new_proof_of_time: VDFInfo
     header_hash: bytes32
     height: uint32
@@ -83,7 +83,7 @@ class RequestCompactProofOfTime:
 
 @dataclass(frozen=True)
 @streamable
-class RespondCompactProofOfTime:
+class RespondCompactProofOfTime(Streamable):
     vdf_info: VDFInfo
     vdf_proof: VDFProof
     header_hash: bytes32

chia/rpc/full_node_rpc_api.py

@@ -61,7 +61,7 @@ async def _state_changed(self, change: str) -> List[WsRpcMessage]:
             return payloads
         return []
 
-    async def get_initial_freeze_period(self):
+    async def get_initial_freeze_period(self, _: Dict):
         freeze_period = self.service.constants.INITIAL_FREEZE_END_TIMESTAMP
         return {"INITIAL_FREEZE_END_TIMESTAMP": freeze_period}
 

chia/types/generator_types.py

@@ -21,8 +21,7 @@ class GeneratorArg(Streamable):
 
 
 @dataclass(frozen=True)
-@streamable
-class CompressorArg(Streamable):
+class CompressorArg:
     """`CompressorArg` is used as input to the Block Compressor"""
 
     block_height: uint32

chia/util/initial-config.yaml

@@ -267,9 +267,9 @@ full_node:
   # How often to initiate outbound connections to other full nodes.
   peer_connect_interval: 30
   # Accept peers until this number of connections
-  target_peer_count: 60
+  target_peer_count: 80
   # Initiate outbound connections until this number is hit.
-  target_outbound_peer_count: 10
+  target_outbound_peer_count: 8
   # Accept at most # of inbound connections for different node types.
   max_inbound_wallet: 20
   max_inbound_farmer: 10

chia/util/keychain.py

@@ -135,7 +135,7 @@ def _get_service(self):
 
     def _get_pk_and_entropy(self, user: str) -> Optional[Tuple[G1Element, bytes]]:
         """
-        Returns the keychain conntents for a specific 'user' (key index). The contents
+        Returns the keychain contents for a specific 'user' (key index). The contents
         include an G1Element and the entropy required to generate the private key.
         Note that generating the actual private key also requires the passphrase.
         """

chia/util/streamable.py

@@ -7,7 +7,7 @@
 import pprint
 import sys
 from enum import Enum
-from typing import Any, BinaryIO, Dict, List, Tuple, Type
+from typing import Any, BinaryIO, Dict, List, Tuple, Type, Callable, Optional
 
 from blspy import G1Element, G2Element, PrivateKey
 
@@ -125,6 +125,9 @@ def recurse_jsonify(d):
     return d
 
 
+PARSE_FUNCTIONS_FOR_STREAMABLE_CLASS = {}
+
+
 def streamable(cls: Any):
     """
     This is a decorator for class definitions. It applies the strictdataclass decorator,
@@ -152,80 +155,121 @@ def streamable(cls: Any):
     """
 
     cls1 = strictdataclass(cls)
-    return type(cls.__name__, (cls1, Streamable), {})
+    t = type(cls.__name__, (cls1, Streamable), {})
+
+    parse_functions = []
+    try:
+        fields = cls1.__annotations__  # pylint: disable=no-member
+    except Exception:
+        fields = {}
+
+    for _, f_type in fields.items():
+        parse_functions.append(cls.function_to_parse_one_item(f_type))
+
+    PARSE_FUNCTIONS_FOR_STREAMABLE_CLASS[t] = parse_functions
+    return t
+
+
+def parse_bool(f: BinaryIO) -> bool:
+    bool_byte = f.read(1)
+    assert bool_byte is not None and len(bool_byte) == 1  # Checks for EOF
+    if bool_byte == bytes([0]):
+        return False
+    elif bool_byte == bytes([1]):
+        return True
+    else:
+        raise ValueError("Bool byte must be 0 or 1")
+
+
+def parse_optional(f: BinaryIO, parse_inner_type_f: Callable[[BinaryIO], Any]) -> Optional[Any]:
+    is_present_bytes = f.read(1)
+    assert is_present_bytes is not None and len(is_present_bytes) == 1  # Checks for EOF
+    if is_present_bytes == bytes([0]):
+        return None
+    elif is_present_bytes == bytes([1]):
+        return parse_inner_type_f(f)
+    else:
+        raise ValueError("Optional must be 0 or 1")
+
+
+def parse_bytes(f: BinaryIO) -> bytes:
+    list_size_bytes = f.read(4)
+    assert list_size_bytes is not None and len(list_size_bytes) == 4  # Checks for EOF
+    list_size: uint32 = uint32(int.from_bytes(list_size_bytes, "big"))
+    bytes_read = f.read(list_size)
+    assert bytes_read is not None and len(bytes_read) == list_size
+    return bytes_read
+
+
+def parse_list(f: BinaryIO, parse_inner_type_f: Callable[[BinaryIO], Any]) -> List[Any]:
+    full_list: List = []
+    # wjb assert inner_type != get_args(List)[0]
+    list_size_bytes = f.read(4)
+    assert list_size_bytes is not None and len(list_size_bytes) == 4  # Checks for EOF
+    list_size = uint32(int.from_bytes(list_size_bytes, "big"))
+    for list_index in range(list_size):
+        full_list.append(parse_inner_type_f(f))
+    return full_list
+
+
+def parse_tuple(f: BinaryIO, list_parse_inner_type_f: List[Callable[[BinaryIO], Any]]) -> Tuple[Any, ...]:
+    full_list = []
+    for parse_f in list_parse_inner_type_f:
+        full_list.append(parse_f(f))
+    return tuple(full_list)
+
+
+def parse_size_hints(f: BinaryIO, f_type: Type, bytes_to_read: int) -> Any:
+    bytes_read = f.read(bytes_to_read)
+    assert bytes_read is not None and len(bytes_read) == bytes_to_read
+    return f_type.from_bytes(bytes_read)
+
+
+def parse_str(f: BinaryIO) -> str:
+    str_size_bytes = f.read(4)
+    assert str_size_bytes is not None and len(str_size_bytes) == 4  # Checks for EOF
+    str_size: uint32 = uint32(int.from_bytes(str_size_bytes, "big"))
+    str_read_bytes = f.read(str_size)
+    assert str_read_bytes is not None and len(str_read_bytes) == str_size  # Checks for EOF
+    return bytes.decode(str_read_bytes, "utf-8")
 
 
 class Streamable:
     @classmethod
-    def parse_one_item(cls: Type[cls.__name__], f_type: Type, f: BinaryIO):  # type: ignore
+    def function_to_parse_one_item(cls: Type[cls.__name__], f_type: Type):  # type: ignore
+        """
+        This function returns a function taking one argument `f: BinaryIO` that parses
+        and returns a value of the given type.
+        """
         inner_type: Type
         if f_type is bool:
-            bool_byte = f.read(1)
-            assert bool_byte is not None and len(bool_byte) == 1  # Checks for EOF
-            if bool_byte == bytes([0]):
-                return False
-            elif bool_byte == bytes([1]):
-                return True
-            else:
-                raise ValueError("Bool byte must be 0 or 1")
+            return parse_bool
         if is_type_SpecificOptional(f_type):
             inner_type = get_args(f_type)[0]
-            is_present_bytes = f.read(1)
-            assert is_present_bytes is not None and len(is_present_bytes) == 1  # Checks for EOF
-            if is_present_bytes == bytes([0]):
-                return None
-            elif is_present_bytes == bytes([1]):
-                return cls.parse_one_item(inner_type, f)  # type: ignore
-            else:
-                raise ValueError("Optional must be 0 or 1")
+            parse_inner_type_f = cls.function_to_parse_one_item(inner_type)
+            return lambda f: parse_optional(f, parse_inner_type_f)
         if hasattr(f_type, "parse"):
-            return f_type.parse(f)
+            return f_type.parse
         if f_type == bytes:
-            list_size_bytes = f.read(4)
-            assert list_size_bytes is not None and len(list_size_bytes) == 4  # Checks for EOF
-            list_size: uint32 = uint32(int.from_bytes(list_size_bytes, "big"))
-            bytes_read = f.read(list_size)
-            assert bytes_read is not None and len(bytes_read) == list_size
-            return bytes_read
+            return parse_bytes
         if is_type_List(f_type):
             inner_type = get_args(f_type)[0]
-            full_list: List[inner_type] = []  # type: ignore
-            # wjb assert inner_type != get_args(List)[0]  # type: ignore
-            list_size_bytes = f.read(4)
-            assert list_size_bytes is not None and len(list_size_bytes) == 4  # Checks for EOF
-            list_size = uint32(int.from_bytes(list_size_bytes, "big"))
-            for list_index in range(list_size):
-                full_list.append(cls.parse_one_item(inner_type, f))  # type: ignore
-            return full_list
+            parse_inner_type_f = cls.function_to_parse_one_item(inner_type)
+            return lambda f: parse_list(f, parse_inner_type_f)
         if is_type_Tuple(f_type):
             inner_types = get_args(f_type)
-            full_list = []
-            for inner_type in inner_types:
-                full_list.append(cls.parse_one_item(inner_type, f))  # type: ignore
-            return tuple(full_list)
+            list_parse_inner_type_f = [cls.function_to_parse_one_item(_) for _ in inner_types]
+            return lambda f: parse_tuple(f, list_parse_inner_type_f)
         if hasattr(f_type, "from_bytes") and f_type.__name__ in size_hints:
             bytes_to_read = size_hints[f_type.__name__]
-            bytes_read = f.read(bytes_to_read)
-            assert bytes_read is not None and len(bytes_read) == bytes_to_read
-            return f_type.from_bytes(bytes_read)
+            return lambda f: parse_size_hints(f, f_type, bytes_to_read)
         if f_type is str:
-            str_size_bytes = f.read(4)
-            assert str_size_bytes is not None and len(str_size_bytes) == 4  # Checks for EOF
-            str_size: uint32 = uint32(int.from_bytes(str_size_bytes, "big"))
-            str_read_bytes = f.read(str_size)
-            assert str_read_bytes is not None and len(str_read_bytes) == str_size  # Checks for EOF
-            return bytes.decode(str_read_bytes, "utf-8")
-        raise RuntimeError(f"Type {f_type} does not have parse")
+            return parse_str
+        raise NotImplementedError(f"Type {f_type} does not have parse")
 
     @classmethod
     def parse(cls: Type[cls.__name__], f: BinaryIO) -> cls.__name__:  # type: ignore
-        values = []
-        try:
-            fields = cls.__annotations__  # pylint: disable=no-member
-        except Exception:
-            fields = {}
-        for _, f_type in fields.items():
-            values.append(cls.parse_one_item(f_type, f))  # type: ignore
+        values = [parse_f(f) for parse_f in PARSE_FUNCTIONS_FOR_STREAMABLE_CLASS[cls]]
         return cls(*values)
 
     def stream_one_item(self, f_type: Type, item, f: BinaryIO) -> None:

chia/wallet/derivation_record.py

@@ -4,13 +4,11 @@
 
 from chia.types.blockchain_format.sized_bytes import bytes32
 from chia.util.ints import uint32
-from chia.util.streamable import Streamable, streamable
 from chia.wallet.util.wallet_types import WalletType
 
 
 @dataclass(frozen=True)
-@streamable
-class DerivationRecord(Streamable):
+class DerivationRecord:
     """
     These are records representing a puzzle hash, which is generated from a
     public key, derivation index, and wallet type. Stored in the puzzle_store.