Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Syrus Akbary
- Sep 11, 2023, 12:30 PM (6 days ago)
- Note: Wasmer just offered a $5,000 bounty on this issue.
- Requirements
- The bounty would be considered achieved when:
- All C examples from wasix-libc can be compiled correctly from Zig directly into Wasm/WASIX: https://github.com/wasix-org/wasix-libc/tree/main/examples
- Zig files can be compiled correctly to Wasm/WASIX, with custom examples working
- Sockets
- Process spawning
- Piping
- Info: while ideally the PR should be merged into Zig master, is not a necessary requirement to receive the bounty. As long as a fork exists that fulfills the requirements laid out before, the bounty will be awarded
- ---
- Rohlem
- Sep 11, 2023, 2:37 PM (6 days ago)
- I'll be taking a look at this.
- Anyone interested in collaborating feel free to let me know.
- ---
- Pratham
- Sep 11, 2023, 4:44 PM (6 days ago)
- /attempt #17115
- ---
- Pratham
- Sep 11, 2023, 6:15 PM (6 days ago)
- Is it correct to assume that the first step should be to integrate https://github.com/wasix-org/wasix-libc into wasi_libc.zig, then get C programs working, and then work on porting the wasm-related zig stdlib code to wasix-libc?
- ---
- Syrus Akbary
- Sep 11, 2023, 6:38 PM (6 days ago)
- Yup, that seems like a reasonable approach @git-bruh .
- ---
- Rohlem
- Sep 11, 2023, 6:38 PM (6 days ago)
- @git-bruh That sounds like a reasonable approach.
- I've been trying the opposite, to start with manually specifying wasix-libc using the --libc CLI argument with a finished build, however I'm a bit stuck on runtime errors in the startup code of the resulting wasm module.
- It's possible that some flags or known workarounds in the wasi_libc.zig logic resolve those issues (and/or my wasix build got messed up for other reasons).
- ---
- Aaron Housh
- Sep 11, 2023, 8:15 PM (6 days ago)
- Question for your last part, wasix can run UDP in the browser, or is that just worded in a way I am confusing. I thought even with wasm it wasn't possible to use UDP from the browser
- ---
- Pratham
- Sep 11, 2023, 9:45 PM (6 days ago)
- Got a super hacky PoC working -- had to stub out a bunch of files that don't compile (related to pthreads and processes)
- 2023-09-12T00:14:15,531194583+05:30
- ---
- Rohlem
- Sep 11, 2023, 9:45 PM (6 days ago)
- Alright, I've got something working, but it might not be worthwhile progress yet.
- I'm still hoping that wasi_libc.zig solves some of these issues, but it's probably not the only missing piece.
- zig cc <C source file> --sysroot=...: Fails with undeclared function errors, I might be doing it wrong.
- This should be the easiest, because the wasix readme basically says clang is supposed to work / the only supported compiler.
- The --sysroot argument is recognized (in that it doesn't lead to the error that unrecognized arguments do), but I keep getting undefined function errors regardless, so I assume Zig keeps choosing the packaged libc instead.
- zig build-exe <C source file> --libc=....: Works with tweaks / if code doesn't use TLS.
- The current Makefile allows choosing THREAD_MODEL as posix or single,
- however it also contains wildcards and find invocations which always end up including the pthread-dependent files anyway.
- FWICT this Makefile logic needs to be fixed to re-enable single-threaded builds of wasix-libc; seems like that hasn't been tested in a while.
- When compiling with THREAD_MODEL=posix, the resulting executables trigger Wasmer to emit error: RuntimeError: out of bounds memory access:
- any time they access __wasilibc_pthread_self, which is in ____wasi_init_tp during startup and throughout the library, f.e. in getrandom.
- I assume this might be a linker bug related to TLS, but not sure.
- also once in __wasilibc_init_ssp. There's no TLS usage here, so I'm not sure what the cause might be.
- Maybe global initialization / CRT constructor calls happen too soon for some reason?
- Without much prior knowledge of linkers and WASM I might poke around with binaryen's wasm-dis and wasm-as to see what the issue is.
- zig build-exe <Zig source file> --libc=...: Fails due to strange lookup failure of symbol main.
- libc.a somehow depends on main but doesn't find it. This might have to do with exporting behavior, not sure.
- Should probably look for differences between wasix-libc and the bundled wasi-libc, since the latter works.
- @Dispersia How I understand it is that WASIX is a standardization effort of features beyond WASI, like UDP sockets.
- For now (only?) Wasmer supports the standard, but the goal would be for other environments (like Browsers) to also support it in the future.
- ---
- Pratham
- Sep 11, 2023, 9:49 PM (6 days ago)
- @rohlem Some of the errors at runtime when using threads/processes are probably an upstream thing, they happen for me even when compiling everything using the official wasi-sdk toolchain
- 2023-09-12T00:18:58,997149583+05:30
- ---
- Rohlem
- Sep 11, 2023, 9:57 PM (6 days ago)
- @git-bruh Great progress!
- Interesting, from looking at the Makefile it seemed like at some point a single-threaded build without TLS should have worked by just excluding those files.
- ---
- Pratham
- Sep 11, 2023, 9:59 PM (6 days ago)
- Yeah many parts of the Makefile are broken, like the globs unconditionally include threading-related code even if it's disabled
- ---
- Isaac Freund
- Sep 11, 2023, 10:09 PM (6 days ago)
- Before people put large amounts of effort into implementing this, let's make clear that this is a feature request/proposal which has not (yet?) been accepted by the Zig leadership.
- As for my personal take, I see WASIX as an effort to port legacy POSIX APIs (including many with questionable design such as fork()) to WASM. This of course makes it technically easier to port existing programs relying on these legacy POSIX APIs to WASM but I worry about how runtimes could possibly implement this API surface securely while maintaining reasonable performance. It frankly seems quite like "containers, but with extra steps"
- Furthermore, It appears to be driven by a single company which is also responsible for the only implementation. I don't any information on the standardization process and can only assume that it is not open. Without any 3rd party implementations including support in browsers, I think adding support to the Zig standard library would be quite premature.
- Overall, I do not see how supporting WASIX aligns with the goals and values of the Zig project.
- ---
- Rohlem
- Sep 11, 2023, 10:18 PM (6 days ago)
- @ifreund The way I see it, if WASIX is a functional libc, and Zig allows users to provide a custom libc, then there shouldn't be harm in supporting it, right?
- Zig is of course free to decide not to ship with it, and not to mention it in the official std module.
- But a third-party package should ultimately be able to offer all of the same functionality via the Bring-Your-Own-OS mechanism, right?
- (Only difference being users have to add the corresponding entry to their zig.zon.)
- ---
- Syrus Akbary
- Sep 12, 2023, 3:09 AM (5 days ago)
- Furthermore, It appears to be driven by a single company which is also responsible for the only implementation. I don't any information on the standardization process and can only assume that it is not open
- You may have missed this, posting here in case is useful: https://wasix.org/docs/community-and-governance
- Note: WASIX governance is completely open up for any developer and/or company that wants to contribute or collaborate (including yourself, if you feel that being part of WASIX could be useful for you!) :)
- Without any 3rd party implementations including support in browsers
- WASIX programs can fully run in the browser (as reflected in the wasix.org homepage).
- If you want to see a very cool demo, please visit: https://wasmer.io/ and type wasmer run python! 🚀
- I do not see how supporting WASIX in upstream Zig aligns with the goals and values of the Zig project
- I hope you will eventually be able to see the long-term value that WASIX could provide to Zig, and hopefully, the work done here by some of the contributors may help to showcase the value of WASIX to the Zig community.
- Here are a few examples that would be enabled: being able to run Zig compiler anywhere (without restrictions, including Zig+LLVM zig cc running on the browser), or using Zig a package in other ecosystems (including JS!), allowing any Zig program runnable anywhere and so on...
- As previously commented, the reward will be awarded to the developer who fulfills the requirements laid out on the same thread.
- We would really love if Zig adopted WASIX upstream (I believe we will have very cool demos to showcase why this may be useful for others!), but merging upstream is not a requirement for awarding the $5,000 sponsorship.
- Keep up the good work!
- ---
- Sebastian
- Sep 12, 2023, 3:43 AM (5 days ago)
- Just wanted to give my two cents on why I will not be participating in this bounty. I contacted Syrus over the weekend about a post he made in the zig discord a couple months ago about how wasmer were interested in zig compiler support for wasix and were interested in sponsoring the necessary work. I told him I was interested in a sponsorship & he responded Monday morning with a link to a newly created bounty. (This feels like slimy behaviour to me; why turn something that should be a very straightforward agreement into some stressful rush for cash? Is this a PR move to try and create excitement for wasix, or a way to get the time of multiple developers while only paying one?). Upon further thought I realised that it didn't make sense for it to be in the zig compiler anyway, and his company wouldn't have the ability to maintain such a fork long term (& likely not the budget to do so either). Unless they have the developer time to take ownership of maintaining it for years then it's a largely pointless endeavour. The final nail in the coffin for me is this article - even if some of this is true I would highly recommend not working with the wasmer company or Syrus https://mnt.io/2021/10/04/i-leave-wasmer/
- ---
- Syrus Akbary
- Sep 12, 2023, 4:22 AM (5 days ago)
- I told him I was interested in a sponsorship
- This issue was directly incentivized by your private request @Sobeston, so I’m quite surprised by your response. Specially, so because the sponsorship reward was actually suggested by me after of your request for guidance to work freely on adding support for WASIX to zig stdlib (happy to attach any screenshots of the conversation if you don’t recall this)
- None of the community things that we do at Wasmer is in private. So yes, once we offer a sponsorship is on a public realm so everyone from the community can work in it, and is not a deal closed behind private doors. I’m sorry you didn’t like this, but I strongly feel this is the right way of approaching sponsorships in OSS projects.
- Honestly, I find sad that to see your reaction, and your attempt to stop other people to try to add support for WASIX in a move I believe would benefit greatly the Zig ecosystem which I’m a great fan of. Even if you didn’t like that the sponsorship was open for everyone to fulfill I sincerely think you could have done better on your response.
- Either way, I appreciate if we can bring points into this issue that are no political and/or personal but centers the conversation on the technical points, so those who want to bring WASIX support in Zig can move it forward. The community will benefit from such an approach.
- Thanks :)
- ---
- Pratham
- Sep 12, 2023, 8:56 AM (5 days ago)
- One hard to solve issue I do see with the bounty approach (if my understanding of the system is correct) is that multiple devs would be working on their own private forks, not open sourcing them until everything is done, since otherwise someone else can just take their work and "claim" the bounty. This will lead to a lot of duplicated effort, then there's also a ton of edge cases that cannot be solved by a single person alone (eg. the threading related issues mentioned above), so no individual dev's work would ever be polished enough to qualify for the bounty, which would lead to stagnation of the actual proposal.
- ---
- Stephen Gutekanst
- Sep 12, 2023, 9:30 AM (5 days ago)
- Problem statement
- Overall I think the problem statement (or lack of) is valid: WASI as it stands today is quite weak, there's a lot it can't do, and that is harming WASM adoption overall. I've heard this from my younger brother and others who work professionally using WASM at various CDNs using it for edge functions. It's a real problem, and needs a solution.
- Solutions
- There are three solutions I see:
- Wait for WASI / the Bytecode Alliance to propose more WASI semantics. So far this has been a.. rather slow, non-existent even, process? But 5 days ago (I wonder if that's a coincidence?), they began work on WASI preview 2 - so although I have no clue what they're going to do with it, the lifeless corpse is twitching a bit?
- Adopt WASIX. Putting aside the controversy that seems to follow Wasmer, I just don't know that a lot of languages/ecosystems are actually going to adopt this - and WASM is nothing if not 'somewhat' standardized. I'm sure Wasmer is aware of this, which is in part why they are trying to push this proposal so hard.
- Create our own WASI-like specification, or better see if Bytecode Alliance folks would be open to figuring this out. Zig could actually be a good, strong enough WASM community to propose this sort of thing - and an unbiased enough entity to make it have some teeth - but I imagine this won't happen for the same reason #1 is the case today: few people have time/interest to do this work. I imagine this would look like 'solve the same problem WASIX aims to, but not by just exposing syscalls to WASM that are hard to secure / problematic, rather by figuring out the proper WASM APIs for those and writing a spec - experiment and iterate on it, prove it out, and make a proposal'.. a slow, arduous process.
- Overall I think the situation is 'unfortunate'. It seems unlikely that WASI will improve soon, even more unlikely that WASIX will get any broad adoption by other WASM communities, and it seems unlikely to me 'our own WASI-like specification' would be good long term either (but who knows.)
- It may be best to just wait.
- Proposal
- I expect Zig proposals to follow more of what I wrote above: problem statements, possible solutions. If a company wants to sponsor development of an issue, that's great and reasonable in my view - but the way it was done here was pretty damn poor.
- I also see your company using it as a marketing opportunity on Twitter and Reddit[0][1] - I'm not against corporate sponsorship of development, and also not against companies or people publicizing they're sponsoring development. Heck, I've done this. But it feels a bit wrong and distasteful given the proposal is to implement your specification, it wasn't even discussed first, and is structured as some sort of bum rush to implement it and get $5k, no?
- You could have made the proposal, mentioned sponsorship being something you are interested in, see what the community said - and then later on publicize the great work a community member did for you after the fact. That would've felt a lot more like 'company works with community to move software forward' rather than 'company pays bounty in hopes of pushing something through and/or getting a marketing opportunity'
- (P.S. Is it that I've made a habit of calling out Wasmer/Syrus over the years for their iffy behavior(1)? Or does it just follow me? blah. I can't believe I took the time to write this.)
- ---
- Rohlem
- Sep 12, 2023, 11:50 AM (5 days ago)
- Best-case scenario everyone should be happy with:
- @git-bruh implements the requested changes (in a fork)
- in a cost-effective amount of time (i.e. more $/h than other opportunities) and gets the money,
- but doesn't foster too much emotional attachment to the project,
- so doesn't volunteer too much (or any) unpaid free time to maintain anything beyond that.
- Along the way this results in a reduction and fix to the runtime out-of-bounds errors triggered by C-language TLS currently in wasix-libc.
- (To me seems like a Zig bug / room for improved diagnostics during compilation.)
- @syrusakbary happily pays the money as promised.
- Zig decides whether to upstream / bundle support (reasonably probably no),
- otherwise this integration will only be accessible via the package manager.
- Other programs using POSIX now have a lazier path to being ported to WASIX,
- but this doesn't overly distract them from the better long-term decision of
- refactoring away from old APIs, towards new, fundamentally better APIs
- (i.e. doesn't slow down development of main WASI, and doesn't lead to any vendor lock-in).
- @git-bruh I agree that cooperation around a single reward becomes difficult. I'm personally happy to "forfeit", since in my eyes you're currently in the lead, and I'm probably stopping experimentation until I can use your wasi_libc.zig setup, explicitly to avoid wasted duplicate effort.
- (Generally I think you could change the LICENSE and put a note into the readme of your fork to state that all commits past X are all-rights-reserved-proprietary, but that leads to an argument if a bad actor shows up and copies those changes anyway.)
- ---
- Sebastian
- Sep 12, 2023, 4:48 PM (5 days ago)
- @rohlem I'm fine, there is no issue for me with someone completing this bounty (but thanks for the thought regardless); personally I feel like I might have dodged a bucket. I just don't want people to end up burning time on and being upset by this bounty, and thought it was best that people are warned about what they might be going into. I also agree that this is not a place for competition, and it's cool to see you happy to forfeit to someone else.
- ---
- Till Schneidereit
- Sep 13, 2023, 11:58 AM (4 days ago)
- @slimsag
- Wait for WASI / the Bytecode Alliance to propose more WASI semantics. So far this has been a.. rather slow, non-existent even, process? But 5 days ago (I wonder if that's a coincidence?), they began work on WASI preview 2 - so although I have no clue what they're going to do with it, the lifeless corpse is twitching a bit?
- Just as a quick heads-up from within the twitching corpse: WASI Preview2 has been in the works for a long time, is largely done at this point, and a perfectly reasonable thing to target right now.
- See this blog post on the Preview 2 roadmap for more details.
- ---
- Syrus Akbary
- Sep 13, 2023, 5:12 PM (4 days ago)
- it feels a bit wrong and distasteful given the proposal is to implement your specification
- It's quite tiring to continuously having to defend oneself from acts that evidently good intended. If you don't agree that WASIX can bring a positive change and don't think Zig will benefit from it, then do not contribute to it, I get completely it.
- But channeling a well-intended move (especially this being a sponsorship) as “wrong and distateful” because it moves the world towards the place we want to go as a company is skewed to say at least.
- Having said that, haters gonna hate, so I’ll stop trying to placate hate and make sure we are spending our time on building, which seems we are way better at! No apologies will be given, as we have done nothing wrong here.
- And an assurance to everyone on this thread. Wasmer will continue pushing for WASIX forward, and Zig will also be a big part of this picture (at least for us!). Hopefully, once demos are placed people will start seeing the value of it!
- The $5,000 reward is still very active, and it does not require the PR to be merged on Zig upstream to be awarded (as long as technical requirements are fulfilled).
- All the best to everyone ❤️
- ---
- Aaron Housh
- Sep 14, 2023, 1:20 AM (3 days ago)
- Ya, this thread seems a bit ridiculous to me and is making me lose faith in the Zig membership. Is it really being complained that the person that made a standard is paying for others to... help push that forward? That seems like the most obvious reason that anyone would ever sponsor work, like that is almost the legitimate reason that people pay to sponsor work. On top of that, it was stated in the beginning that it did not have to be merged to master, just even having some way of doing it was enough, so it's not even trying to force itself into standard. Let someone do the work, see how it pans out, see if the community likes it and then move forward if so with concrete plans to get it merged into main.
- I don't understand almost any point that is being posted against this right now. I understand that the person might have gotten hurt about the private bounty going public, and I can't see the full story of that as obviously it happened in private, but I see no issue with someone pushing this forward in the capacity being requested.
- And if one person writing an article about not liking working there deters you from ever wanting to work with a company... you need to get rid of every machine and piece of software you've ever written.
- ---
- githubuser6000
- Sep 14, 2023, 5:42 AM (3 days ago)
- I'm just dropping a comment to say that there seems to be a motive to possibly generate a lot of hype and consequently a lot of requests / pressure on the zig team / community to support wasix, even before any official confirmation from the project leads.
- Links: 1, 2, and on the homepage too
- It just leaves a bad taste in my mouth. Doesn't scream collaboration in any sense of the word.
- ---
- Syrus Akbary
- Sep 14, 2023, 10:04 AM (3 days ago)
- Can someone from @ziglang leadership explain why this comment from @Dispersia was removed from this thread? @andrewrk @kristoff-it @kubkon
- Screenshot 2023-09-14 at 8 49 46 AM
- Transcription:
- Ya, this thread seems a bit ridiculous to me and is making me lose faith in the Zig membership. Is it really being complained that the person that made a standard is paying for others to... help push that forward? That seems like the most obvious reason that anyone would ever sponsor work, like that is almost the legitimate reason that people pay to sponsor work. On top of that, it was stated in the beginning that it did not have to be merged to master, just even having some way of doing it was enough, so it's not even trying to force itself into standard. Let someone do the work, see how it pans out, see if the community likes it and then move forward if so with concrete plans to get it merged into main.
- I don't understand almost any point that is being posted against this right now. I understand that the person might have gotten hurt about the private bounty going public, and I can't see the full story of that as obviously it happened in private, but I see no issue with someone pushing this forward in the capacity being requested.
- And if one person writing an article about not liking working there deters you from ever wanting to work with a company... you need to get rid of every machine and piece of software you've ever written.
- seems to be a motive to possibly generate a lot of hype and consequently a lot of requests / pressure on the zig team / community to support wasix, even before any official confirmation from the project leads
- This is getting absurd. Yes, we like to hype things to get attention on them so the sponsorship can move fast, so what? The assumption you concluded has no sense whatsoever.
- Let me expand on why we want WASIX support in Zig (technically, not even upstream) to help people understand how ridiculous all this argument is. We (Wasmer) want Bun (written in Zig, and recently reached 1.0) running on WASIX asap, a move which is high-priority for a few internal reasons at Wasmer.
- It’s incredibly tiring, as I previously commented, having to defend oneself just because people keep assuming bad intentions (for what reason exactly? We always treated the Zig community with great respect). I explicitly stated that the work did not need to be merged upstream, and that the sponsorship would be awarded as soon as the technical requirements were met.
- ---
- Andrew Kelley
- Sep 14, 2023, 10:10 AM (3 days ago)
- I'll address this during business hours tomorrow, for now please calm down while we sleep
- ---
- Andrew Kelley
- Sep 14, 2023, 11:20 PM (3 days ago)
- Alright. @kristoff-it and I took a break from working on our other tasks today to address this issue, and in fact we wrote a blog post about it.
- As a reminder, the Zig issue tracker is not a platform for free speech, for discussing bug bounties, for discussing other people's reputations, or for archiving Internet history. It is only, exclusively, and for no other purpose than for working on the Zig compiler and related tools. Any content that is not doing that will be removed, and any users who I subjectively predict will create more of such content in the future will be blocked.
- Let us have no more discussion of anything in this topic other than WASIX and its technical merits, or lackthereof.
- ---
- Leron Gray
- Sep 15, 2023, 3:55 AM (2 days ago)
- Random internet person here.
- Personally, I would love if Zig (or some additional C compatible languages like Nim or V) supported WASI/WASIX directly.
- As primarily a Python dev, I know of an entire dev community that would benefit from having an accessible low level language like Zig with WASI implementations. But I also get the points made about WASM/WASI standards vs how WASIX comes across.
- Regardless of how the bounty stuff came across, I hope y'all implement it. WASI would be a great pivot for me to use Zig. 🙏🏿
- ---
- githubuser6000
- Sep 15, 2023, 4:28 AM (2 days ago)
- @daddycocoaman Unless I'm mistaken, there is (some?) support already for WASI
Advertisement
Add Comment
Please, Sign In to add comment