update Yosys and GHDL #7569
update Yosys and GHDL #7569
Conversation
|
After discussing it with @danyeaw, Therefore, this PR is now ready for review. |
|
I don't like asking anyone to use PIP. Ever. Can we do something better? |
@mingwandroid can you explain, why |
|
@mingwandroid, please see #7580 and the conversation in January 2, 2021 2:01 AM and January 2, 2021 7:35 PM. TL;DR it's not possible to distribute pyGHDL as a MINGW package without packaging other at least 3 packages which are not available in MINGW. It is neither acceptable to install those dependencies through pip when the package is built. Hence, I don't think we can do anything better. Yet, I'm open to suggestions. |
|
I don't know how much you know about our fork of python, how pip works or compiler compatibility issues so its difficult for me know how much to say. Does debian recommend you use pip for us python packages? Does arch Linux? No! Why? Because then the native package manager cannot keep track of the files (and thus the environment - bad for reproducibility). It is commonly accepted in the circles that i run in that complete system package managers are far better than language package managers (why limit your package manager to site-packagss? Is that the only place python cares about? No far from it). PIP install binary wheels. Those are compiled with msvc. That is not compatible with mingw.
Too long didn't read or a bit tricky so didn't try? What is it about these three packages that make them incompatible with mingw-w64 (please avoid crediting the wrong team here!) Remember that at one point python was incompatible with mingw-w64 until Alexey and I fixed up Roumen Petrov's work on this! Nothing worked until we fixed it! I am not opposed to providing PIP but only with a warning personally that it shouldn't be used esp. to install wheels. Cheers. |
|
@mingwandroid, it seems that you misclicked and unwantedly closed this PR, instead of posting a comment only. Can you please reopen as I'm not shown the option to do so? |
I find this slightly disrespectful... With 'TL;DR' I meant "in case you don't want to read the references, because they are too long, let me summarise for you". I did not only read everything (since I took part in the conversations), but I also implemented the MINGW package for distributing pyGHDL before opening this PR. In fact, that was the state of this PR until 11h ago. Please, read the parts of the first comment above which were striked out earlier today. Also, please, see this previous run: https://github.com/msys2/MINGW-packages/actions/runs/457717819. You will find that three packages were built on each environment: GHDL, pyGHDL and Yosys. There is a comment in the TEST stage about the dependencies of pyGHDL: https://github.com/msys2/MINGW-packages/runs/1636659003?check_suite_focus=true#step:7:11. You can find other CI runs in my fork: https://github.com/umarcor/MINGW-packages/actions. After successfully implementing the MINGW package based solution, and before adding the remaining dependencies as MINGW packages, I discussed the use case with other contributors in the chat and the outcome was to use pip (see the reference above). Should there be conflicting/opposing POVs/practices among maintainers and frequent contributors in this regard, I believe it'd be desirable to have a discussion and document some guidelines for new contributors.
I think this is a misunderstanding. I used 'MINGW' to tell packages apart from 'MSYS2' and 'CLANG' packages, which are the names used in the repos of this org. Both packages (GHDL and Yosys) were already upstreamed to this repo. This is an update PR! NOTE: we did have some issues because a contributor was using pacman package
AFAIAA there is no technical reason preventing those packages from being compatible with mingw-w64. Those are Python only packages which should work nice and clean. In fact, they work perfectly if installed through pip, as you can see in older CI runs of this PR. It is also shown in https://github.com/umarcor/MINGW-packages/runs/1654428699?check_suite_focus=true#step:7:3, which is a commit ahead of this PR (for adding the tests). The main motivations for avoiding the addition of 3+ extra Python packages to MINGW-packages are following:
The only motivation for adding pyGHDL as a MINGW-package (generated in the same PKGBUILD recipe as GHDL) was ensuring that users got compatible binaries and bindings without manually checking the version. Since ghdl/ghdl#1582 was added to GHDL, the problem of getting out of sync is now solved (for MSYS2 or any other platform).
In the current state of this PR, MSYS2 is neither providing nor suggesting PIP to be used. This PR updates GHDL, which is a tool written in Ada, and Yosys, written in C/C++. Nevertheless, as said in #7580, I think that clarifications, such as why wheels should not be installed through PIP, woul be very helpful for users to understand how to proceed.
Is that true for Cheers |
|
Clearly I know what TL;DR means and I was bending the definition for comedic effect. I also stand by what I wrote FWIW. Please try to be less sensitive around me if you can. I have troubles like you would not believe. Anyway I'm here to talk ABI matters; hard facts on the ground we cannot ignore. Facts that make using PIP for compiled packages simply impossible (or at best dangerous). I don't have time to spend reading loads of issues all over the open source ecosystem making the point that binary wheels on Windows Py 2.7 link to Visual Studio 2008 and binary wheels on Windows Py 3.x link to the UCRT. The UCRT is incompatible with MSVCRT. Therefore binary Windows wheels should never be used by our Python. Trust me on this; I made this thing work in the first place and I also maintain a large swathe of the core packages in Anaconda Distribution.. So I do not care (one jot) for what 'upstream' project developers say, just because they are upstream and said it. I only care about the validity and veracity of what is said, irrespective of source. I don't have time specifically because I am dying from cancer and life's too short. If I have managed to educate you a bit around wheels and when you need to ignore what others say about them, then great. The python-ecosystem's isolationist approach (they key exponent of which is pip and site-packages, the second of which is the recommendation to link everything statically) is based around their wish that their wheels continuing to work in the face of a changing environment round them. Linux distros and in particular rolling distros (and MSYS2) embrace this chaos and pull out surf-boards to ride the wave to ride the wave of chaos. The MinGW-w64 project would prefer if people do not call their efforts MINGW. I am only passing that along as I also dislike to see this and do consider it a mis-credit each time I read it (unless it is explicitly referencing some variable name or other). |
|
BTW you did just admit in your TL;DR that the only preventative issue was that it was effort. I was hardly calling anything out that you were trying to hide here.
|
This is far from the point. If we tell people "PIP is OK" which is what we do when we use it then they will expect that PIP is OK. And it really isn't. |
|
And it is beholden on old-timers of MSYS2 to make sure that new people do not come along and break things (I appreciate your efforts I really do and it's not your fault we didn't document that PIP is currently inadvisable). |
The PyPA (Python Packaging Authority) is in control of everything to do with PIP. We don't attempt to use it, fix it, and we do not recommend it. I will state it again, there are 2 diametrically opposed views on how Python tools, libs and apps should be packaged. On one side we have a few Python core developers and python library/tool creators who love PIP and promote it when they can (IMHO in many cases, their fandom of their favorite language clouds the purity of what an ideal installer would actually look like and it will never be something that can handle only one language, software stacks are composed of libs and tools across many languages). On the other side, we have people who work on software packaging outside of Python/PIP but who still need to provide Python/PIP to their end users. I have yet to find anyone on that side of things recommending PIP because, well, they have their own package managers that in general they wrote themselves (sometimes this will be a system PM - any linux distro more or less - sometimes not - MSYS2 and conda) and since it's tightly integrated into the system itself it can know everything it needs to about the system and can therefore manage it holistically in the context of all the other other languages and dependencies a given installation requires. At Anaconda, we ignore utterly the 'please link statically' advice from PIP/PyPA because it is a dangerous recommendation. You are baking exploits in a compiler-oven right into your software. Exploits that will remain there for years as ticking time-bombs for hackers. Shared is far better (but more work, you need to keep up with ABIs, semver-like considerations and rebuilds), but hey our users avoid getting hacked. |
|
I appreciate that you took your time to partially explain your point of view, and I'm really sorry to know about your health issues. Unfortunately, I don't have time to deal with this derailed discussion, and with apparently off-topic condescendence. Should you want to talk about updating Yosys and GHDL packages, I opened #7606. "For system packaging fanatics versus Python fanboys", we already have #7580. |
It's not my point of view I'm afraid. Is just the facts about pip and msys2. I find this statement unfortunately dismissive of both my long experience as a software packager and the considerable amount of work i put into making MSYS2 succeed in its earliest days. I took the time, a considerable amount of it, to explain to you fully why PIP doesn't work with MSYS2. These are technical matters purely. Your first response to me regarded offense at an obvious joke (and a valid point, perhaps with attempted sugar in the delivery; you even said yourself that you didn't want to go that route as it was more of an effort .. then later decided to clarify that to you are wanting to avoid effort for Lazka - I strongly doubt moving some file around is a lot of effort and thats likely all you'd save here, even that's likely automated) and this latest comment seems just generally dismissive, and unpleasent as if the matter didn't err, matter. It does. With regard to cancer please understand that MSYS2 is probably the largest party of my life's legacy (certainly the bit i feel most proud of) and i want to see it curated thoughtfully by people aware of all the technical facts once I've carked it Regarding package managers, clearly the guy who picked pacman (after studying in detail all the alternatives) is going to prefer that to other package managers. There's also those very valid technical details I have already that I'll not repeat. . Also so giving me links to where i should communicate about these matters. I don't appreciate it. The main thing MSYS2 is is a team of mainly friendly helpful people working towards a common goal. It was started as something fun for Alexey and I and to fill a gap in Windows Open Source. |
|
And i will note you address exactly none of my technical points regarding pip. Also for the record I believe you have been highly condescending toward me so far. But whatever let's leave it here. Life is too short. |
|
@mingwandroid, please don't take me wrong. I ordered the following reply so important things come first.
I believe the work Alexey and you did is astonishing. Full stop. That is something you certainly need to be proud of. All the technical decisions which I am able to understand so far are plainly brilliant. Picking pacman as a packager manager is probably one of the best assets of this project, as it allows almost effortless migration of the shell scripts that most developers start with. It is the de facto standard for developers using Unix syntax/tooling to work on Windows (and to produce Windows executables), regardless of many efforts to drive the attention elsewhere. Again, something to really be proud of. I do have absolutely nothing to discuss about the technical details out of my scope. I do believe that all the points you made about PIP, wheels, and Python packaging are valid. And I didn't nor will suggest that the overall point you are making is not legit; it's probably the best in a not so distant horizon. I find myself everyday trying to understand what the hell are wheels and what's about conda. My very personal perception is that trying to distribute scripts together with pre-builts whatever is a broken approach. I expect system packages to be related to compiled languages, and I expect them to include binaries, shared libraries and assets. Then, scripts might be distributed either together or independently because scripts should be interpreted files, by definition, not dependent on any binary format, but just the standard C interfaces (in the runtime) for interacting with shared libs on the system. This is precisely the proposal in this PR. GHDL is distributed as a pacman package on MSYS2 (as an rpm on Fedora, a deb on Debian/Ubuntu, etc.). It provides a binary (ghdl), a couple of shared libraries (libghdl and libghdlvpi) and a bunch of assets (standard and IEEE VHDL libraries). Assets are pre-built if LLVM or GCC backends are used (thus, included as Now, there are some Python scripts in a directory named pyGHDL which users can use for interacting with libghdl (the one installed through pacman) from Python. Those Python scripts are currently not distributed through MSYS2. This is problematic because GHDL is a rapidly rolling project, where each new commit is a potential release. Therefore, MSYS2 cannot keep up with it. Users cloning the master branch of GHDL's repo do get mismatching ASTs (APIs). That is the only problem we want to solve. pyGHDL is NOT distributed through pip and I don't personally have any interest on doing so until it is stabilised. Due to GHDL's nature, cloning the git repo and checking out an specific commit is the right thing to do at the moment. How to know which commit to check out is the problem we wanted to solve. Distributing Python scripts through pacman was a possible solution, but I honestly think that allowing to do it through git is a better approach given the context. My perception is that you thought we were talking about distributing GHDL either as a pacman package or as a PIP package. That was never the case. Someone might want to follow that path, but as I already said it's a broken solution for me. My whole point is about distributing scripts, and only scripts, which happen to be written in Python. From this perspective, do you think that using PIP for distributing Python only packages is wrong? Would you be so kind to explain why? If scripts are pre-built to some internal format before being uploaded to PyPIP, I would really want to know whether that can be prevented. I'd like to use PIP for distributing a single package which works on any platform, precisely because all platform specific details are handled by platform specific package managers (paman, apt, dnf...). Adding Yosys and GHDL to MSYS2 took several weeks/months of work in half a dozen different repositories with so many different maintainers. This PR fixes some issues that may annoy users of those tools on MSYS2. That is the point of this thread. Adding or not adding Python bindings was (is) an irrelevant feature in this PR, which I just wanted to handle now in order to reduce lazka's work afterwards. GHDL allows analysis, compilation, simulation and synthesis regardless of Python. This PR does not add a single Python line. The outcome is that this PR was closed and the discussion is plainly ignoring GHDL and Yosys. I tried to make my point about that by stating explicitly that pyGHDL can be used without PIP and without pacman. If you ignored the comments about PYTHONPATH, I feel I can be unhappy if a maintainer closes a PR, drives the discussion off-topic, ignores a request for it to be reopened and keeps ignoring the points about the tools on topic. I do respect your work, your dedication and all the time and effort you put into this project. Please, do respect my time and work too.
As I said, I don't see the link with this PR and with using Python sources on MSYS2. That's why I said it was a 'partial point of view'. Absolutely no mean to dismiss neither your point of view, the facts nor your knowledge. Yet, bringing your background repeatly are ad hominens from a logical point of view. As much as I respect you, I cannot take those as valid arguments. It can perfectly be my own ignorance kicking in. Nonetheless, the outcome is that I don't understand where's the issue with the following procedure: pacman -S mingw-w64-x86_64-python-pytest mingw-w64-x86_64-python-ghdl
git clone https://github.com/ghdl/ghdl
cd ghdl
git reset --hard $(ghdl version hash)
PYTHONPATH=$(pwd) ...Or where is the issue with using CPython and PyCharm with binaries built on MSYS2 from Ada/C sources. Where/how are wheels, PIP, UCRT, MSVCRT, etc. involved in these use cases?
I do appreciate that a lot, and I'll be glad to keep talking about it somewhere where my unrelated and ready-to-be-merged work is not hidden and ignored. I do think it is critical to clarify the guidelines for Python packaging on MSYS2. Communication by different maintainers/contributors is clearly misleading, as is being seen in #7580.
Not at all, mate! We are grown up enough to read between words! Your joke implied that I had not done my homework and that I am trying to avoid doing an/any effort. At the same time, it was obvious that you had not read the PR before, nor the related discussion in the chat. Then, you claimed that you don't need to do your homework of knowing the context. That makes the joke inadequate, still obvious. I meant no offense, but to let you know that you were (I think you still are) out of context.
I already explained (and provided references) about having implemented it and I have shown my willingness to implement other packages, if it's needed. Your continuous attempts at dressing me in straw do little help to your argumentation. I ask you to please stop it. I explicity said that I stopped before implementing the other packages because I was told so. In fact, I would have loved if you had stepped in the conversation with Dan. I wish you still did so. Moreover, it is obvious to any regular viewer that MSYS2 cannot keep up with increasing Python package updates, as it can hardly keep up with updating regular pacman packages. Your strong doubt about how much effort it takes for lazka to review and approve new packages/updates is really dismissive of his work. That was purposely the very last point in my list and I brought it explicitly because it is a relevant limitation of this project (as most open source projects). Having packages updated on MSYS2 does not depend on contributors opening PRs and lazka merging them only. The autobuilder takes time and needs to be babysitted. Packages need to be then signed and uploaded, and AFAIAA David is almost the only one doing it ATM. Therefore, it is very unrealistic to have pacman packages updated in less than at least a week since upstream maintainers publish a release. I know you are aware of this and I'm not being condescendent. This is for any reader of this dialogue to know why I think that pacman packging in MSYS2 cannot currently replace pulling releases from git repositories. It's a complement, and it will lag behind (even if NOTE: I am absolutely ok with MSYS2 being updated once a week, a month, or every few months. I am not criticising the project at all, specially since the improvements in the last year were awesome in this regard. This is pertinent because pyGHDL would probably never be updated faster than GHDL. That is, it would be ok to package pyGHDL in the PKGBUILD recipe of this repository (as I did until a couple of days ago). That might work per se, even though it would require users to install dependencies on their own. However, the dependencies of pyGHDL which I avoided packaging are likely to move (much) faster than MSYS2 can keep up with. EDA tools are very fast moving targets and there is a natural friction in distributing them through MSYS2. Since there is no yaourt/yay/ equivalent in MSYS2 AFAIAA, I find of utmost importance to properly ponderate the effort of all the parties involved in the process.
Please, do try to focus on the matter then. Try not to argue about your background or the amount of work that you or me are willing to put into packaging. Unless proven otherwise, we both wish the best to MSYS2, and to each other.
I did not address your technical points regarding PIP because I ain't any expert and I have nothing to discuss in that regard. I'm a user or MSYS2, bash and git for open source EDA tooling regarding hardware designs written in VHDL. If using PIP is ok, let it be. If it is not, let it not be. I have zero feelings about it.
That was a pure reaction to you closing this PR and neither replying to my request for it to be reopened nor replying to my points about the tools at matter. In the issue I referenced (and also in the chat) I did already explain many of the content which I now repeated here. I was not telling you where to communicate, but where relevant communication had happened before (cause life's too short to repeat ourselves). WRT condencesdence, I was replying to an apparently uninformed maintainer calling me lazy (as a joke); call it a mirror.
Let's keep it like that! I used 'fanatics' and 'fanboys' jokingly because I'm a viewer in the match. I just want to understand what are the technical limitations for allowing users of GHDL on MSYS2 to make the best of the environment. I won't accept neither ad hominems nor comments/jokes dismissing my work or others. Other than that, I'm honestly glad we had this dialogue, because there is much information scattered among unfortunately inadequate miscommunication. |
|
Thanks for the detailed response @umacor. I'll be brief as I agree with most everything you say, still I'm somewhat the pedant and cannot help myself ;-) I decided I didn't need to read very much of the linked to PRs and comments because I felt that I needed to nip any "let's use PIP" stuff in the bud (and explain why) and also it seemed you did a good job of communicating the pertinent points. I am also constantly ridiculously busy (as I'm sure many are).
The closure of this PR was done in error, sorry about that. I totally didn't intend to do that and should've been clear about it. I cannot now re-open it as github tells me you have another PR to this repo.. no idea why that matters mind you! I have multiple PRs to multiple repos. Maybe we've tweaked a config var somewhere?
.. I'm a bit lost by now :-) Is there any points about any MSYS2 tooling that you'd like clarified?
Really honestly that wasn't the intention, it's quite a big leap IMHO from "Looks tricky so didn't try" to "You are lazy". As I say I wasn't being dismissive and was hoping to encourage you to package up all dependencies as pacman packages. Thanks again for your contributions and willigness to engage in the face of perceived rudeness from myself! |
I don't think it does, no. When I tried installing numpy, it picked up a zip package and started to compile it. Finally, it creates a wheel named something as Another thing, which can be done is creating a updator like thing for |
|
Oh one thing of technical note I skipped, for MSYS2 (as per ArchLinux), in general we want installation to be as simple as possible, therefore: .. seems very complicated and non-standard. How would any MSYS2 user know that they need to do this when they install everything else with only the pacman command? Presumably you would have to add a news entry for Regarding the versions getting out of sync, you can (and likely should) generate all the requisite packages in this GHDL recipe.
Pacman allows for version constraints, https://wiki.archlinux.org/index.php/PKGBUILD (search for "Dependencies") But the best approach when you have two things sharing a common resource (in this case, the GHDL AST I think!) is for that common resource to be put into a separate shared dependency used by both GHDL and pyGHDL. Regarding @lazka's workload, I hope he will reach out for help from others if it's getting too much, but honestly, I suspect that he's automated all the tricky bits. One this score, it would be good to know what's involved in package uploading these days. |
You might have found out already that you met your match here 😉
I felt that, and I am very grateful for you taking your time in this dialogue. I'm just sorry we didn't get to clarify the different topics in the discussion slightly before.
No worries and no regret about it. Shit happens sometimes, and we dealt with it in 24h total. Good enough! Plus, now we have a better background in our luggage.
That was my bad. I opened another PR (#7606) because I wasn't sure about being able to bring your attention into it. Nevermind; we can now focus on understanding and communicating/documenting Python and PIP stuff on MSYS2 only.
I'm sorry about the misunderstanding. I thought you should have read that I was/am willing to engage, despite it being tricky. That's why I took it pejoratively.
It is always good to exercise our (written) social skills from time to time 😆
My main (and probably only) question for now is: can we make anything for letting I would classify all existing Python packages in three groups:
Then, I would love for pacman and
Furthermore and ideally, should I have a requirements file with the following content: and I tried to execute Given the current state, users need to have multiple requirements files:
Knowing which dependencies to put in each file or which must be upstreamed is a manual and very time consuming task.
I'm funnily sorry because that was a very unfortunate typo on my side 😆 I should have been Anyway, I agree it seems very complicated and non-standard. But... don't you know where we are coming from! GHDL was packaged for Debian many years ago; then it was removed due to licensing issues with IEEE libs; a workaround was found; it came back; in dec 2019 IEEE released new libs as open source... During the last two years, synthesis features were added, which lead to several projects building and distributing GHDL using manual procedures. During that period, packages on Debian, Fedora and Arch started to be consistently updated, but not coherently yet. Building GHDL on Windows was very problematic (and it is still). 6 different variants exist, and 2 of them are supported only for now. This is to say that
The target audience are users of GHDL on Windows for synthesis and simulation, which are used to vendor products. Therefore, the most relevant feature is that they can do Users of the Python bindings must already be aware that they are using an experimental feature. The only useful tool at the moment is a language server. That needs a matching client (for VSCode, emacs, vim...), which is available in a separated repository and which users need to install/setup manually. Therefore, executing 4-5 easy to understand git commands is not a great deal. When the features based on Python bindings evolve in the future, we might want to rethink packaging so that the language server and the clients are provided as a separated package. We are not there yet.
There is a section in the docs (https://ghdl.github.io/ghdl/) about how to get/build GHDL on different platforms. And we are discussing how to update it for recommending system packages, since the ecosystem is starting to be stable enough.
That was exactly what I did in this PR until a couple of days ago. That is, I declared python as a dependency, and I built pyGHDL with
Yes. That's what I did, and I did it for that reason.
I am aware of that, but the versioning of GHDL is another point of friction. It is tagged once a year, but that's clearly not enough for packagers. Therefore, packagers in each distro pick arbitrary commits for bumping. We are about to roll out a monthly automatic tagging workflow in GHDL, in the hope to reduce the dispersion.
The point is that the AST is internal to the Ada codebase, where both GHDL and libghdl are created. The shared library (libghdl) is independent from the binary (ghdl) per se; however, in practice, libghdl needs access to all the assets (VHDL sources) that are distributed with GHDL. Therefore, the approach you propose might grow out of control:
Overall, GHDL is a tiny beast, and we are making our best to handle it within our knowledge and time constraints.
See the explanations below for a brief perspective on the future challenges I see.
@naveen521kk, I think that the point here is that MSYS2 is a rolling project. Therefore, it is difficult to guarantee that The problem exists with pacman packages too, which might be unrelated to Python. For example, I bumped and updated KiCad recently because it had went out of sync with some Qt DLLs. However, pacman provides mechanisms to know that, and updating dependency trees could be done automatically (if it wasn't for CI time and space constraints). Therefore, I believe that @mingwandroid's point about the issues of distributing pre-built packages through PIP is very relevant. Maintainers of projects would need to build their wheels on MSYS2 using
For context, until approximately 6 months ago, MSYS2 was updated manually almost exclusively by Alexey. He built all the packages locally, signed them and updated them. That's why major releases were done once a year, since maintaining packages took all his time. There have been some huge enhancements during the last year for having most of the process automated, which reduced average package update time to about a week. At the same time, "major" releases are being published once a month on average. This is to say that MSYS2 is being completely rebumped in this regard. Yet, I think that an automatic updater for python-* packages should not be a priority at the moment. The risk with MSYS2 moving too fast is that packages which receive less love can be rotten. There is currently no CI workflow for testing that oldest packages are still buildable. I believe that would be a better usage of available CI resources. That is, let's ensure that all the ecosystem is moving forward together, instead of letting some packages rot behind while priotising bleeding-edge projects. FTR, in hdl/MINGW-packages I created a weekly workflow for building a set of packages and preventing them from rotting: https://github.com/hdl/MINGW-packages/actions?query=workflow%3ABuild. I believe that's an interesting strategy because it off-loads MSYS2 org's CI. Unfortunately, there is no mechanism for reporting results back. Therefore, manual intervention is required for communicating which packages are currently broken on MSYS2: https://github.com/hdl/MINGW-packages/actions/runs/469249827. Therefore, you can prototype the idea of a automatic Python updater in a fork of yours. Just copy https://github.com/hdl/MINGW-packages/blob/main/.github/workflows/build.yml or the workflow in this repo, and adapt it to your needs. You might have PRs created against this repo automatically, but I discourage doing so; some minimal human intervention is desirable. Please, don't take me wrong. I have also thought about how to automate updating all pacman packages. That is, having a
Just for pedantic clarification, I believe that @mingwandroid would be ok with using pip, if it wasn't because he is aware of how many things can go wrong (because he already saw them go very badly several times). By the same token, I don't have a need for not maintaining packages, but I have the need to make the best of everyone's time and effort. |
I have a working prototype of doing this, where multiple handlers can be created for Github/Gitlab and query their api for the latest release, and I made a PR with that few days before, but simply because I forgot about dependency resolution and the way people add new dependency to a project I had to close that, unless it has a dependency resolver(not much atleast lists and check the dependencies for python ones).
That's kinda OK. The human intervention, can be done after creating the PR also right?
I would be asking how to do that? Simply, we can add a warning to our CI checks for PR.
That would simply be over complicated and as a user I would hate it. Also, that would mean that we should completely rewrite
I think Github Actions provide room for testing these things, what can be done is once a month trigger a workflow, which runs in chains one after other building each and every package out there.
I don't think we would need to reduce the load of or something like that, because Github Action is fully free, and makes no sense doing so. But the idea of testing things is great. |
|
Also, my suggestion is make a new issue for this and move it there, things should be more organised then :) |
Sure, but the impact of a mistake is a couple of orders of magnitude smaller if done in an "unknown" fork, rather than in this repo with over a hundred watchers. If the mistake involves creating a single PR, it's not a great deal. If the mistake involves creating dozens of wrong PRs, the noise might be significant.
I know CI reads the depends field of PKGBUILD recipes. If you update several packages in the same PR and one depends on another, CI will build them in order. See https://github.com/msys2/MINGW-packages/blob/master/.ci/ci-library.sh#L117-L124 and https://github.com/msys2/MINGW-packages/blob/master/.ci/ci-library.sh#L78-L92. I believe that the autobuilder uses a different procedure for getting a dependency graph of the tasks. But that might be provided by the API at packages.msys2.org. See also https://www.msys2.org/docs/package-management/#finding-dependencies-of-a-package.
I'm open to absolutely any suggestion which solves the problem. I don't have any strong feeling about it. However, stating that you hate something is not helpful. Stating that you would like it to be an opt-in/opt-out feature would have been more polite, provided it's the only proposed solution so far. I'm specially interested in knowing why would that imply "completely" rewriting Regarding other tools which depend on pip breaking, what's the difference between failing before installing a package or installing it knowing that it's broken?
Unfortunately, I don't think that's realistic. GitHub Actions is free, not unlimited. There are 4140 packages at the moment. Let's say it takes 15 min to build each of them. That's 96 packages a day. It would take ~43 days to build all the packages. GitHub Actions allows up to 20 workflows/jobs to be executed at the same time in all the repos of an organisation. We might use up to 15-20 workflows in parallel to reduce it downto ~2-3 days. That's a very best case scenario, because we might not be able to schedule all the threads properly. In practice, issues will arise, many packages take much longer than 15min, or run out space, some packages will be updated meanwhile... Therefore, it wouldn't be a monthly run, but a continuous task. That's why I proposed a more conservative approach: checking packages which are older than 3 months? Alternatively, off-loading the work to other namespaces. If your target is to update 250-500 packages each month, that might be feasible.
Unfortunately, I cannot take care of the discussion at the moment, so I'd prefer not to be the OP. Still, I agree that creating a new issue would be useful. |
Created a discussion #7619 |
A fix was pushed to Yosys, which is related to building the ghdl-yosys-plugin frontend: makefile: fix GHDL vars, replace GHDL_DIR with GHDL_PREFIX YosysHQ/yosys#2515. That's not critical for the yosys package on MSYS2, but it's desirable to bump it.
Python bindings in GHDL were reorganised and reworked to improve usability. Hence, the GHDL package is bumped.
There is currently an annoying constraint that prevents users/contributors of MSYS2 from building the Yosys PKGBUILD alone. At the moment, it must be built at the same location where GHDL was built. Since GHDL is/was built in CI, Yosys can "only" be built in CI (unless GHDL is built and installed locally before). Find further details in libghdl.link contains hardcoded paths from the build environment ghdl/ghdl#1570. This PR patches
libghdl.linkfor preventing that constraint.In GHDL's repo, the LLVM backend is built with clang. There is work in progress for using LLVM on MINGW32 too, but it's not there yet: Test MSYS2 build variants ghdl/ghdl#1547. In this PR, the MINGW64 package is built with clang.
GHDL's Python bindings (pyGHDL) were not included in MSYS2 packages yet. They can be used in the repo (through PYTHONPATH) or installed from sources. However, the version of the repo that users checkout should match the version the package was built from. That's because there is a tight relation between the internal AST and the bindings. To avoid users manually finding and retrieving matching git versions, in this PR the GHDL PKGBUILD recipe is extended for generatingmingw-w64-x86_64-python-pyGHDLandmingw-w64-i686-python-pyGHDL.TEST, is for testing purposes only and it'll be removed when the PR is ready to be merged.setup.py, namedghdl-ls. That entrypoint is properly created whenmingw-w64-*-python-pyGHDLis installed (see https://github.com/umarcor/MINGW-packages/runs/1635326267?check_suite_focus=true#step:7:2885).[ ] pyGHDL has two Python dependencies:pydecorandpyVHDLModel. Ideally, those should be installed automatically whenmingw-w64-*-python-pyGHDLis installed throughpacman. Those are Python-only packages that can be installed through pip, and which are defined in thesetup.py. However, it seems not to work as I'd expect, because those packages need to be installed explicitly. I was told in the chat thatpydecorandpyVHDLModelmight need to be packaged as MINGW packages too. Shall I do that?/cc @Paebbels