net.mbedtls: enable MBEDTLS_THREADING_C on Windows and macOS#27437
Conversation
mbedtls' internal locking (around the shared RNG, RSA key blinding and the library globals) was enabled only on Linux/FreeBSD/OpenBSD and disabled on Windows and macOS, because V only wired up mbedtls' pthread mutex backend and Windows (MSVC) has no pthreads. This blocks any design that shares an mbedtls config/RNG across threads (e.g. parallel TLS server handshakes). - macOS: it has pthreads, so add it to the gate (it was simply omitted). - Windows: enable MBEDTLS_THREADING_ALT and supply the four mutex callbacks over a Win32 CRITICAL_SECTION (vlib/net/mbedtls/ mbedtls_threading.h + thirdparty threading_alt.h), installed once via mbedtls_threading_set_alt() from the net.mbedtls module init() before any TLS use. The setup is a no-op on non-Windows builds. Folded into mbedtls.patch so it survives the next mbedtls upgrade. Note: changing the threading config alters mbedtls struct layouts, so an existing local build must rebuild the mbedtls objects (touch the sources or delete the cached .o); fresh/CI builds are unaffected. Verified on Windows: net.http TLS handshakes work with threading enabled (the mutex callbacks are exercised on every TLS operation); the only server_tls_test failures are the pre-existing net.shutdown idle-interrupt ones, unchanged from master. macOS to be verified on Apple Silicon. Closes vlang#27436 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
76cffd6 to
8853eed
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8853eed2cc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
Re: the Codex P2 about stale cached mbedtls objects — it's a real issue and I'd like the team's call on how to address it. Root cause. Flipping the threading macros lives in The PR currently documents the manual workaround (
I'm happy to implement whichever you pick (1 and 2 are both ready to go). My weak preference is 1 as the correct long-term fix, possibly split into its own PR per option 3 so this one stays scoped — but it's your call. |
|
For me, option 1 seems like the best long-term fix, but I think it should cover both current thirdparty object paths. This is just my personal opinion: For the V1 builder, fixing only the regular C path would not be enough. The MSVC path also has its own cached For this PR, the scope can stay focused: invalidate mbedTLS objects from the source file, sibling headers and the relevant mbedTLS include/config headers, especially I would also like to add another important point in my opinion, version 2 of the V compiler. If V2 eventually owns that path later, the cache key should include source/header/config dependencies plus the target/build context. ( Still, this should be validated with Medvednikov first ). Of course, not necessarily for this PR, but it could make the future V1 -> V2 migration smoother. I hope this opinion is useful. |
|
I agree with @GGRei that option 1 would be best. Objects should always be rebuilt from header only changes, as that type of change can affect the .c files, too. |
|
Thanks @GGRei @JalonSolov — going with option 1 in this PR, scoped as you described, and covering both thirdparty object paths through a single shared decision so they can't drift. Confirming the plan after reading the builder:
V2 cache work is out of scope here per @GGRei — agreed; that's for a separate discussion with Medvednikov. Since this turns into a general builder change (cache invalidation for all thirdparty objects, not just mbedtls), I'll call it out as a distinct improvement in the PR description so it gets the right review attention even though it's folded in here. Implementing now. |
Cached thirdparty objects were only invalidated against the source `.c` and its sibling headers in the source directory (e.g. thirdparty/mbedtls/library/). A change to a config header elsewhere in the module — notably thirdparty/mbedtls/include/mbedtls/mbedtls_config.h, which alters mbedtls struct layouts — left stale objects cached, so an existing checkout silently reused the old layout (link errors on MSVC, missing locking on macOS) until `v wipe-cache` was run by hand. Worse, the MSVC path reused a cached `.obj` whenever it merely existed, with no mtime comparison against the source or any header at all — the root of the long-standing "-cc msvc: delete library/*.obj by hand" workaround. Both paths now share one decision via thirdparty_deps_mtime(), which folds the newest `.h`/`.hpp` mtime under the whole thirdparty module root (include/ included) into the cache key. The recursive header scan is memoized per module root, so it runs once per module rather than once per object. Source-less bundled objects keep their reuse-if-present behavior. Verified (tcc): editing include/mbedtls/mbedtls_config.h now rebuilds the mbedtls objects, an unchanged tree rebuilds nothing, and the build re-stabilizes afterwards. Co-Authored-By: WOZCODE <contact@withwoz.com>
|
Done — pushed in b96251d (its own commit). Both paths now go through a shared Verified on tcc: editing |
|
Looks good to me. :) This is only my opinion so it is up to you, one small regression test would make this cache fix safer long term. A fake If I were you and when you are done, chain Good job! |
|
Sorry about that, I was only giving an opinion and accidentally triggered Codex. ^^ |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…/ tree
Regression guard for the cache-invalidation fix: thirdparty objects must
rebuild when any header in the module changes, including config headers
under include/ (e.g. mbedtls_config.h), not just siblings of the .c.
- test_thirdparty_module_root_spans_library_and_include: the pure path
helper resolves a library/ source to the whole thirdparty/<mod> root,
normalizes separators, and falls back to the source dir off-tree.
- test_thirdparty_deps_mtime_includes_module_include_headers: a temp
thirdparty/<mod>/{library,include} tree where an include/ header is
newer than the .c; thirdparty_deps_mtime must reflect it. Verified to
fail against the old source-dir-only scan. This helper is shared by the
C and MSVC object paths, so it guards both.
Co-Authored-By: WOZCODE <contact@withwoz.com>
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
Enables mbedtls' internal locking (
MBEDTLS_THREADING_C) on Windows and macOS;previously it was on only for Linux/FreeBSD/OpenBSD.
Why
V wired up only mbedtls' pthread mutex backend, gated to
__linux__ || __FreeBSD__ || __OpenBSD__, and#undef'd threading everywhere else. Windows(MSVC) has no
pthread.h, and macOS — which does have pthreads — was simplynot in the gate. With threading off, mbedtls does not mutex-protect its shared
state (the
ctr_drbgRNG, RSA key blinding, library globals), so sharing anmbedtls config/RNG across threads is unsafe on those platforms. That blocks, for
example, parallelizing TLS server handshakes across workers (the deferred
item #3 from #27433).
What
__APPLE__to the pthread gate — it has pthreads, soMBEDTLS_THREADING_PTHREADworks as-is.MBEDTLS_THREADING_ALTand supply the four mutex callbacksover a Win32
CRITICAL_SECTION:thirdparty/mbedtls/include/mbedtls/threading_alt.h— thembedtls_threading_mutex_ttype.vlib/net/mbedtls/mbedtls_threading.h— theinit/free/lock/unlockcallbacks + a
v_mbedtls_threading_setup()that callsmbedtls_threading_set_alt(). A no-op on every other platform.net.mbedtlsmoduleinit(), before any TLS use.thirdparty/mbedtls/mbedtls.patchso it survives the next mbedtlsupgrade.
Builder: auto-invalidate thirdparty objects on header changes
The threading flip lives in
include/mbedtls/mbedtls_config.h, but V's thirdpartyobject cache only invalidated against the source
.cand headers in the sourcedirectory (
thirdparty/mbedtls/library/) — so an existing checkout silentlyreused objects built with the old struct layout. The MSVC path was worse: it
reused a cached
.objwhenever it merely existed, with no mtime check at all(the root of the long-standing "
-cc msvc: deletelibrary/*.objby hand"workaround).
Per the discussion below (thanks @GGRei @JalonSolov), this is fixed generically:
both the C and MSVC paths now share one
thirdparty_deps_mtime()that folds thenewest
.h/.hppmtime under the whole thirdparty module root (include/included) into the cache key, memoized per module root. This is a general builder
improvement (it applies to every thirdparty module, not just mbedtls) — called
out here so it gets distinct review attention even though it's folded into this
PR. It is its own commit.
Verification
THREADING_ALT;net.httpTLS handshakes work (the mutex callbacks are exercised on every TLS op);server_tls_test.vunchanged from master (only the pre-existingnet.shutdownidle-interrupt timeouts)THREADING_ALT; TLS handshake (new_ssl_conn) runsserver_tls_test.vpassesserver_tls_test.vpassesBuilder cache invalidation (both object paths): editing
include/mbedtls/mbedtls_config.hrebuilds the 112 mbedtls objects and anunchanged tree rebuilds nothing — verified on tcc (C path) and msvc (the
.objpath; confirmed via the per-objecttrace_thirdparty_obj_filessignal,since MSVC compiles through
@response.rspfiles). gcc/clang share the tcc C path.Note for reviewers / local builds
Changing the threading config alters mbedtls struct layouts, so existing local
checkouts must rebuild the mbedtls objects. With the builder fix above this is now
automatic — pulling this branch updates
mbedtls_config.h's mtime, whichinvalidates the cached objects on the next build.
v wipe-cacheis no longerrequired; it remains a safe manual fallback. Fresh clones and CI are unaffected.
This is a prerequisite for safe parallel TLS handshakes on Windows/macOS (#27433
item #3); it fixes no current bug on its own (today's single-context-per-thread
usage is already safe) but unblocks that work.
Closes #27436.
🤖 Generated with Claude Code