DarwinException: Attempting to copy two files

[marcelotduarte]

@cainesi Please see this: https://github.com/marcelotduarte/cx_Freeze/discussions/912
I already commented about this error in #887 (comment)
I think this is introduced in PR #590
Using macOS Github Action I have the same error with the sample https://github.com/marcelotduarte/cx_Freeze/tree/main/cx_Freeze/samples/pillow
This is an annoying error because the developer has no possibilities to workaround, then we have to choose the library or give an option to choose.
In the issue reported, the library to choose is a PIL library like in the sample.
Can you take a look?

[cainesi]

I remember reflecting on this back when I did PR #590. The basic issue is that the Freeze objects only tracks where files are going (in .files_copied) rather than where they came from. So, before the PR (and I think still for files other than Mach-O files on Darwin), if the Freeze tried to copy a second file to the same spot as a file had already been copied, it would just assume the file was previously copied and continue on. The PR changed that to report an error in those circumstances.

We could easily add a new option for build_exe to make it ignore the problem again (or make that the default behaviour?). If it is really just two copies of the same library, presumably it would still work. If we go the new option route, I could also change the error message to suggest trying that option?

[marcelotduarte]

I am not very familiar with the code of this PR, I just did basic tests and accepted what you told me, which are advances. And sometimes they need some adjustments.
I know why we have two libraries: one is installed on the system and the other by the package (in this case PIL distributes it on the wheel). This goes for other systems, on Linux and windows.
I think the correct thing is to choose the one distributed with the PIL, that is, the one distributed with the package.
The metadata for the package has this information. Maybe I can change this on module detection and you can use this information in darwintools.

[heinzchristen]

I just reinstalled python with homebrew, now i have version 3.9.1.
Same error, even the one line python program

#!/usr/local/bin/python
import sys

will produce the error "Attempting to copy two files". Then i checked the size of the 2 .dylib files:
source 1: 488 /usr/local/lib/python3.9/site-packages/PIL/.dylibs/liblzma.5.dylib
source 2: 320 /usr/local/Cellar/xz/5.2.5/lib/liblzma.5.dylib
that means they are not identical and both must be present.

[cainesi]

The sizes are 488 and 320 kb? I wonder if the difference is that the big one contains 32-bit and 64-bit code.

Anyway, I'm confused why the 1-line program would be trying to grab anything from the PIL package. What is in the setup.py script? First step is probably to figure out why that is happening.

[heinzchristen]

Sorry, my mistake: the 1-line program test.py is "import matplotlib" , with import sys it works.
The sizes are source1: 250kb, source2: 164kb (488,320 are results from ls -s)
My setup script is attached.

setup.txt

[cainesi]

Thanks. That makes more sense then.

Just tried, and it looks like it does actually load two copies of "liblzma.5.dylib" (also "libz.1.2.11.dylib") when you run the underlying program. (I think this is different from Windows, where you could only get one library with a given file name, as I understand it.) So to be totally safe, we would need to copy both libraries into the frozen application, which I think would require mangling the filenames or putting the "extra" copies in new a new subdirectory(ies). Or we could just hope that the libraries are interchangeable and only copy one (which was how things worked before PR #590).

In case it is helpful, here is a program that runs a specified python script and prints a list of the modules / dynamics libraries that are (in fact) loaded (so far only for OSX). Good for sanity checking with cx_Freeze is doing. Marcelo--this will appear in a PR with some other tools at some point when I get around to it.

script_runner.py.zip

Also, out of curiosity, was there a reason you explicitly included the "os" package in your setup script? (Just wondering if there was some problem otherwise.)

[heinzchristen]

Thank you for help.
No reason for explicitly including the "os" package, i just used the original setup.py from your documentation web site.
Here is my output of your script_runner.
result_of scriipt_runner.txt.zip

[cainesi]

I think the correct thing is to choose the one distributed with the PIL, that is, the one distributed with the package.
The metadata for the package has this information. Maybe I can change this on module detection and you can use this information in darwintools.

Hi Marcelo-- I (belatedly!) think that you were correct above. Since the dynamic link is to a file that is already being included in the frozen application (as part of the package) we should just detect that and update the link so that it points to the file in the package.

This isn't going to work if the package is being zip_included, or if the two libraries are outside packages, and in that case we would still have the error, or need to do name mangling, etc.

Would it be hard to make information available about what files are in copied modules, and where those files ended up?

[cainesi]

Thank you for help.
No reason for explicitly including the "os" package, i just used the original setup.py from your documentation web site.
Here is my output of your script_runner.
result_of scriipt_runner.txt.zip

The idea with script_runner.py would be to run it on the program being frozen (i.e., "test.py", in your case), rather than the setup script.

Sorry for the confusion.

[marcelotduarte]

This isn't going to work if the package is being zip_included

When the package goes to the zip, the extensions (.so, .pyd, etc) are renamed. For dependencies of an extension like that of PIL, it remains in the same folder (PIL/.dylibs).

Would it be hard to make information available about what files are in copied modules, and where those files ended up?

It is a possibility that I have in mind. I think that is not hard.

For now, I think we should bypass this error, and implement a better approach in the future. Bypassing the error gives a chance to the developer to substitute the library in a post build process.

There is a package delocate that finds dynamic libraries imported from python extensions. Do you know it?
It works with wheels, and I think we should do similar approach in cx-freeze.

[heinzchristen]

Here again my output of script_runner, this time i hope the correct one.
result2_of_script_runner.txt.zip

[cainesi]

There is a package delocate that finds dynamic libraries imported from python extensions. Do you know it?
It works with wheels, and I think we should do similar approach in cx-freeze.

I did not know about that one. Thanks. I'll take a look.

[cainesi]

Here again my output of script_runner, this time i hope the correct one.
result2_of_script_runner.txt.zip

Oddly, the report there shows only one copy of liblzma being loaded.

[heinzchristen]

I deleted the source 2 file and ran my setup script again. Nevertheless it appears again.

imac-von-heinz:lib christen$ cd /Users/christen/Desktop/xtest/
imac-von-heinz:xtest christen$ python setup.py bdist_mac
running bdist_mac
running build
running build_exe
.
.
copying /usr/local/lib/python3.9/site-packages/PIL/.dylibs/liblzma.5.dylib -> build/exe.macosx-11-x86_64-3.9/liblzma.5.dylib
.
.
cx_Freeze.darwintools.DarwinException: Attempting to copy two files to 'build/exe.macosx-11-x86_64-3.9/liblzma.5.dylib'
source 1: '/usr/local/lib/python3.9/site-packages/PIL/.dylibs/liblzma.5.dylib' (real: '/usr/local/lib/python3.9/site-packages/PIL/.dylibs/liblzma.5.dylib')
source 2: '/usr/local/opt/xz/lib/liblzma.5.dylib' (real: '/usr/local/Cellar/xz/5.2.5/lib/liblzma.5.dylib')
(This may be caused by including modules in the zip file that rely on binary libraries with the same name.)
imac-von-heinz:xtest christen$
imac-von-heinz:xtest christen$ ls /usr/local/Cellar/xz/5.2.5/lib/liblzma.5.dylib
ls: /usr/local/Cellar/xz/5.2.5/lib/liblzma.5.dylib: No such file or directory
imac-von-heinz:xtest christen$

Source 2 is no longer available, but it still appears in the log output. How is this possible? Is it hidden in a zip file?

[cainesi]

Hi @heinzchristen.
That is the expected behaviour. On Darwin, paths to the dependencies are stored in the library files. Depending on the nature of the paths, the error above might be triggered before cx_Freeze has actually confirmed that the second source file exists.

As a quick fix, could you try with PR #915? That change converts the error to a warning, so the freeze will proceed. The frozen application will only include the first version of the library in question, so behaviour might be different (or maybe not).

[heinzchristen]

I would really like to test the new version 915, but I don't know how to load it. I don't know enough about github. I simply installed cx_freeze with pip install cx_freeze.
Can you please help me? Thank you

[cainesi]

You can download the code here: https://github.com/cainesi/cx_Freeze/archive/darwin_dylib_collision_fix.zip

Download it, unzip it, go into the directory, and then run "pip install ."

That should install a copy of cx_Freeze based on the downloaded source. You need to have a C-compiler to build cx_Freeze, but I think if you're using homebrew then you would have already installed apple Command Line Tools?

[heinzchristen]

Thank you very much. This fixed my problem and my programs can be freezed.
I really appreciate your fast and efficient support.

[cainesi]

No problem. Just to confirm, did you check that the frozen program runs? (This fix is a hack, so it's possible that the frozen program would not run, it would run differently from the original unfrozen script)

[heinzchristen]

Yes, my programs run, even the biggest project (using biopython, reportlab, matplotlib and a lot of others) run perfectly. So i am very happy. Thanks.

[marcelotduarte]

@cainesi Thanks!

To document, you can suggest to others to test your PR directly (without additional work):

pip install -U git+https://github.com/marcelotduarte/cx_Freeze.git@refs/pull/915/head

[cainesi]

Thanks. I didn't know you could do that with pip. Next time.

I thought a bit more about how to fix this. I would propose the following:

1. Have cx-Freeze make a big list of all the files it intends to copy (basically, all the places where the Freezer currently calls _CopyFile(), would be replaced with adding the desired file to a list).
2. Once that list is done, go through and determine what extra files are needed as dependencies-- at this point, we would know if the required dynamics libraries are being copied as part of a package, so we could avoid duplicating them. We would not need extra code to tell us about the package contents, we would have that information "automatically" after _WriteModules() added the files to the copy list.
3. Copy all the files, all at once.

As a side benefit, this would allow some simplifications in the darwintools.py code.

If that sounds like it might be acceptable, I can do a PR.

[marcelotduarte]

@cainesi Sorry for the late response.
I have the impression that what you propose is already done, or should be done. However, I need to separate this code, because it is very mixed with what it is for Windows, Linux and mac and sometimes confuses a lot when analyzing bugs. So please wait until I can look at this and then you improve this code.
In the meantime, we will keep the palliative.

[cainesi]

Hi @marcelotduarte.

Thanks. I've wished that the Windows/Linux/Mac code was better separated for a while, but have not had the time to do it. I was thinking it would be better if we had a sub-class of Freezer for each platform, which implement all the necessary platform-specific adjustments--was that what you had in mind?

For the file copying suggestion, I've done some very preliminary work along the lines I was suggesting. I'll upload it to a draft PR, in case it's any help.

Are you going to put #915 into the next version, pending a more permanent fix?

[marcelotduarte]

sub-class of Freezer for each platform

Exactly.

Are you going to put #915 into the next version, pending a more permanent fix?

Yes.

[cainesi]

That's great, and will make things much easier. I can just add most of the darwintools stuff into the DarwinFreezer subclass.

I did some very preliminary work on implementing file copying changes I suggested earlier. It is here, if you are interested: cainesi@894f52c (at the moment it creates two copies of the frozen executable, to check that the old and new code were giving consistent results). I'll hold off on doing any more work on that until Freezer gets split.

Also, I took the draft off #915 if you want to use it.

[marcelotduarte]

cx_Freeze 6.6 has just been released.
Assuming this has been resolved.
If you had issues please open a new issue.