Hello ๐ folks, Iโm Kevkevin. Iโm an open-source developer and reporter for Insider Edition. Last week I reviewed several pull requests from the Bitcoin Core repo. On Windows, bitcoin.exe used to start the child and return before that process finished. A caller waiting on the wrapper saw exit code 0. util::ExecVp now uses _spawnvp(_P_NOWAIT) and _cwait, and _exit forwards the child status. Another merge parses each JSON-RPC batch element into its own JSONRPCRequest, so a non-object element no longer answers with the previous id.
Thursdayโs #bitcoin-core-dev meeting covered the working groups. The kernel note is an encapsulation RFC. The benchmarking note is a reindex-chainstate experiment that connects blocks out of order, with no precomputed hints file and no assumevalid. QA has more than 300 surviving mutants in src/validation.cpp.
Merged PRโs
Every week, several changes are officially added to Bitcoin Core. This week, 35 changes were merged. Here are some I found interesting this week.
On Windows, util::ExecVp in src/util/exec.cpp used _execvp, which starts the child and exits the parent right away. Anything waiting on the bitcoin wrapper saw exit code 0 and empty stdout, whatever the child did. It now starts the child with _spawnvp(_P_NOWAIT), waits on it with _cwait, and forwards the childโs status through _exit. _P_WAIT was not used because a child exit of -1 would look the same as a failed spawn.
ExecCommand in src/bitcoin.cpp now throws std::system_error with std::generic_category(), which matches the POSIX errno code ExecVp sets. tool_bitcoin.py no longer skips Windows. interface_gui.py still skips on vcpkg Qt builds (is_qt_vcpkg()), where the headless minimal Qt platform plugin is unusable.
ExecuteHTTPRPC in src/httprpc.cpp reused one JSONRPCRequest for a whole batch. Parsing wrote the id and JSON-RPC version into that object, so the next element inherited both. An element that is not an object was answered with the previous elementโs id, and a notification followed by โnot an objectโ lost the error, because it was treated as the notificationโs (nonexistent) reply.
Each element now gets its own copy, JSONRPCRequest elem_jreq{jreq}, before parsing. The new test_batch_invalid_element_reply in interface_rpc.py checks that [{โjsonrpcโ:โ2.0โ,โmethodโ:โgetblockcountโ,โidโ:โXโ}, โnot an objectโ] returns the result with id X plus an Invalid Request with id None, and that a notification plus โnot an objectโ returns only that error, with HTTP 200.
There are always changes being updated and reviewed in real-time. Here are some notable PRโs that are still up and looking for reviews.
CScriptWitness stores a witness stack as a vector of vectors: a 24-byte vector object plus one allocation per element. That lets a transaction take a lot of memory after deserialization and makes it slow to deserialize. This PR keeps the stack in its serialized form in a single allocation (none for an empty stack) with a small header, so CScriptWitness shrinks from 24 bytes to 8 and CTxIn from 112 to 96. A witness can be replaced but not edited in place, and element access is O(n).
Later commits reject, during deserialization, transactions with more inputs or outputs than a valid transaction can have, and, on P2P only, transactions with 0 inputs or 0 outputs and transactions or blocks over the 4M weight limit. A P2P deserialization error now just drops the message. At Thursdayโs meeting sedited asked for comments on the encapsulation RFC #36842, which lists this PR as one of its open threads.
With it, the worst-case memory of a valid deserialized block drops from 106.8 MiB to 6.0 MiB.
IRC meeting notes
Every week on Thursday, there is an IRC meeting. Here are some short notes from that meeting.
--- Topic 1 ---
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: #topic Fuzzing WG Update (eugenesiegel, marcofleon)
๐ฒ๐๐ด๐ฒ๐ป๐ฒ๐๐ถ๐ฒ๐ด๐ฒ๐น: no update
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: I think they are not here, I will call wg that are around then
--- Topic 2 ---
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: #topic addr-relay WG update (stratospher)
๐๐๐ฟ๐ฎ๐๐ผ๐๐ฝ๐ต๐ฒ๐ฟ: no update. weโve been thinking more about addr-relay protections.
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: any prโs to review there or write ups?
๐๐๐ฟ๐ฎ๐๐ผ๐๐ฝ๐ต๐ฒ๐ฟ: coming soon
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: nice :)
--- Topic 3 ---
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: #topic Kernel WG Update (sedited)
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: purpleKarrot posted another RFC issue on encapsulation: #36842 .
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: Iโd like to encourage people to comment there.
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: Iโve explored down some rabbit holes, but nothing to present here yet.
--- Topic 4 ---
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: #topic Benchmarking WG Update (l0rinc)
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: I noticed some activity on benchcoin, anything interesting to share @andrewtoth?
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: l0rinc isnโt here?
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: yeah, i think he is travelling.
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: well sedited shared a branch he was working on, which he managed to reindex-chainstate in 20 minutes or so :rocket:
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: I managed it in under an hour on my machine, others have similar speeds
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: I believe itโs an amalgamation of many different techniques that have been experimented with in the past, notably sipaโs suggestion to use a negative utxo set
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: but really sedited can speak more to it
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: thatโs all Iโve got on it :)
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: @sedited anything to add there?
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: thanks @andrewtoth
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: yes, the core of its architecture is inspired by a comment sipa left on delving some time ago: https://delvingbitcoin.org/t/libbitcoin-for-core-people/1222/36?u=thecharlatan
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: i think it might be a promising direction to take things, but just exploring for now.
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: Is the speedup just from arch change, or we were able to max cpu?
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: it does finally seem to bind validation performance to mostly the CPU on all its cores.
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: i think the arch change allows the max cpu
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: the main gist is that it allows parallel connecting of blocks on multiple threads
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: swiftsync i believe would allow something like that as well
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: but this method does not need a precomputed hints file
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: nice thats a +
--- Topic 5 ---
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: #topic QA WG Update (brunoerg)
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: and works without assumevalid.
๐๐ฎ๐ป๐ฐ๐: im not sure what is meant by parallel connecting of blocks
๐๐ฎ๐ป๐ฐ๐: parsing a block in multiple threads?
๐ฎ๐ป๐ฑ๐ฟ๐ฒ๐๐๐ผ๐๐ต: rather out-of-order connecting yancy
๐ฏ๐ฟ๐๐ป๐ผ๐ฒ๐ฟ๐ด: I ran mutation testing for src/validation.cpp and got over 300 surviving mutants, in case anyone wants to see, theyโre all on mutanthub now: https://mutanthub.space/mutants?q=&project=&mutationStatus=SURVIVED&reviewStatus=&operator=&contributor=&file=validation.cpp&commit=&since=&until=&drift=&superseded=&sort=
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: by connecting here, you mean validation and adding to the chain i presume?
๐ฏ๐ฟ๐๐ป๐ผ๐ฒ๐ฟ๐ด: Iโm reviewing them again that might be equivalent or not useful - but so far looks like there are many gaps to be addressed in the tests
๐ฏ๐ฟ๐๐ป๐ผ๐ฒ๐ฟ๐ด: thatโs all
๐๐ฒ๐ฑ๐ถ๐๐ฒ๐ฑ: sounds like we need to brace for a lot of PRs for that ^^
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: brunoerg: Do you have to manually validate each pr that the mutant was indeed killed? or its automated? then drahdbot link for that will be useful?
๐ฏ๐ฟ๐๐ป๐ผ๐ฒ๐ฟ๐ด: Iโm trying to automate but would be helpful if people put in the PR description the link to the mutant from mutanthub.space and/or the git diff of the mutant. Recently, I had to deal with a PR where the mutants were printscreen haha
๐ฏ๐ฟ๐๐ป๐ผ๐ฒ๐ฟ๐ด: but yes, most of the time I validate manually, especially because LLMs can write test cases to kill mutants but it doesnโt mean they are done correctly or in a good shape
--- Topic 6 ---
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: #topic QML GUI WG Update (johnny9dev)
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: jarolrod handed off a document to us with 300 findings against gui-qml after spending a lot of time doing exhaustive QA with his clankers.
๐๐ฎ๐ป๐ฐ๐: grep screen :P
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: this certainly pushes back my schedule for opening up the PR to bitcoin/bitcoin
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: the good thing is that he canโt seem to find any more issues and we are feature complete
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: so while it will take us longer to get a PR ready, weโll be much better after satisfying those findings
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: his experience was that having his agents model the software and investigate teh model uncovers a lot more
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: โcreate an arc42 documentโ or โcreate a state machine with all of the implicit and explicit states in a single documentโ seemed to be juicy prompts
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: @johnny9dev, maybe a good idea to use scan with project loupe?
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: i think we already have reached out
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: so that will happen too
๐ท๐ผ๐ต๐ป๐ป๐๐ต๐ฑ๐ฒ๐: anyway, thats all for this week. working through issues and feedback
--- Topic 7 ---
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: #topic re-activate Erlay WG and Signal chat (or alternative) (sr_gi)
๐๐ฟ_๐ด๐ถ: Just wanted to say that given the interest shown in the last two coredevs and the state of the reworked approach for Erlay, I wanted to re-activate our old Signal group for discussions (or find an alternative that works for people who are interested) and also re-activate the WG (will do that myself on GH).
๐๐ฟ_๐ด๐ถ: Those who are interested in joining, let me know so I can add you to the chat. That will be mostly to discuss some of the pending changes and open questions.
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: Okay I will move it from the archive to active wg, starting next week, you should be pinged for update
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: If there are updates already, we can discuss now too?
๐๐ฟ_๐ด๐ถ: Not really, I rebased the PR on master and Iโm working on addressing some of the points that I brought up during coredev
--- Topic 8 ---
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: #topic Erlay WG Update (sr_gi)
๐๐ฟ_๐ด๐ถ: Something that came up recently on another PR (cannot recall the number now) is that the Erlay handshake is special and now we have better ways to do it with BIP 434, so given BIP-330 is in draft mode, it may make sense to get rid of the custom negotiation and move to BIP-434 for that
๐ ๐๐ฟ๐ฐ๐ต[๐บ]: That sounds reasonable
๐๐ฟ_๐ด๐ถ: Besides that, just working on addressing the points brought up at coredev, will have a todo list and report on that regularly
๐๐ฟ_๐ด๐ถ: Thatโs it
๐ฎ๐ฏ๐๐ฏ๐ฎ๐ธ๐ฎ๐ฟ๐๐ฎ๐ฑ๐ถ๐พ: Anything else to discuss?
Releases
There were no new releases this week.
Thank you for reading. Be sure to tune in again next week for your updates on Bitcoin Core!
If there are any comments, suggestions, or errors, do not hesitate to reach out or comment.