Ensembles 1.7 multiple problems and disabled functionality on Xcode 9

[keremerkan]

Ensembles 1.7 is unable to load its omo on Xcode 9, resulting in completely disabled functionality. Firstly it gives the error:

CoreData: annotation: Failed to load optimized model at path '/var/containers/Bundle/Application/46C61E42-DA78-460D-82B1-2B780511C9D8/MyApp.app/CDEEventStoreModel.momd/CDEEventStoreModel_2.omo'

And then we see

CoreData: error: -addPersistentStoreWithType:Binary configuration:(null) URL:file:///var/mobile/Containers/Data/Application/FBE36631-CF92-420E-A4D3-5EBA07B2447B/Library/Application%20Support/com.myapp.MyApp/com.mentalfaculty.ensembles.eventdata/transitcache/MyAppSync/download/236_159945E3-FA38-46B2-9BC3-394C2B07FD10-960-00000081E9732578_A23EF461.cdeevent options:{ NSInferMappingModelAutomaticallyOption = 1; NSMigratePersistentStoresAutomaticallyOption = 1; } ... returned error Error Domain=NSCocoaErrorDomain Code=259 "The file couldn’t be opened because it isn’t in the correct format." UserInfo={NSUnderlyingException=Can't read binary data from file, NSUnderlyingError=0x1c0647440 {Error Domain=NSCocoaErrorDomain Code=259 "The file “236_159945E3-FA38-46B2-9BC3-394C2B07FD10-960-00000081E9732578_A23EF461.cdeevent” couldn’t be opened because it isn’t in the correct format." UserInfo={NSFilePath=/var/mobile/Containers/Data/Application/FBE36631-CF92-420E-A4D3-5EBA07B2447B/Library/Application Support/com.myapp.MyApp/com.mentalfaculty.ensembles.eventdata/transitcache/MyAppSync/download/236_159945E3-FA38-46B2-9BC3-394C2B07FD10-960-00000081E9732578_A23EF461.cdeevent, NSUnderlyingException=value for key 'NS.objects' was of unexpected class 'CDEPropertyChangeValue'. Allowed classes are '{( NSNumber, NSUUID, NSOrderedSet, NSDate, NSSet, NSDictionary, NSNull, NSDictionaryMapNode, NSData, NSArray, NSString, NSDecimalNumber, NSURL )}'.}}} with userInfo dictionary { NSUnderlyingError = "Error Domain=NSCocoaErrorDomain Code=259 \"The file \U201c236_159945E3-FA38-46B2-9BC3-394C2B07FD10-960-00000081E9732578_A23EF461.cdeevent\U201d couldn\U2019t be opened because it isn\U2019t in the correct format.\" UserInfo={NSFilePath=/var/mobile/Containers/Data/Application/FBE36631-CF92-420E-A4D3-5EBA07B2447B/Library/Application Support/com.myapp.MyApp/com.mentalfaculty.ensembles.eventdata/transitcache/MyAppSync/download/236_159945E3-FA38-46B2-9BC3-394C2B07FD10-960-00000081E9732578_A23EF461.cdeevent, NSUnderlyingException=value for key 'NS.objects' was of unexpected class 'CDEPropertyChangeValue'. Allowed classes are '{(\n NSNumber,\n NSUUID,\n NSOrderedSet,\n NSDate,\n NSSet,\n NSDictionary,\n NSNull,\n NSDictionaryMapNode,\n NSData,\n NSArray,\n NSString,\n NSDecimalNumber,\n NSURL\n)}'.}"; NSUnderlyingException = "Can't read binary data from file"; }

I am not sure how I can proceed here, so I wanted to open an issue for this.

[drewmccormack]

How are you linking in Ensembles? Are you building yourself in your project? Cocoapods? Precompiled binaries?

[keremerkan]

Cocoapods. Completed the handbook and it has been working until Xcode 9. I think it has to do something with CDEPropertyChangeValue being an NSObject instead of the new allowed classes. So it cannot read the database which was created using Xcode 8 compiled app. With a brand new app that does not have any remnant iCloud files, I think the error was different. I'll need to test that again. The failing of optimized model is always there though.

[keremerkan]

Checked it now, when I am using an app that is compiled in Xcode 9 and has not done any syncing before, the app fails to load the model file and after that it cannot do anything else related to Ensembles.

[drewmccormack]

It is pretty clear the problem is loading the model file. Without that, it will not know about CDEPropertyChangeValue class, giving the other errors.

I guess a solution may be for me to add a new version compiled with Xcode 9. You could perhaps test if this will work by building Ensembles separate to your project, grabbing the CDEEventStoreModel.momd out of it, and replacing the Cocoapods one with that.

Drew

[keremerkan]

Hmm, OK, I'll try to do that now.

[drewmccormack]

I must say, it seems odd. I would think the file format would be back compatible. Are you sure it isn't just a glitch with the file in Cocoapods? Does it help to clean the caches and completely install again?

[keremerkan]

Well, I tried everything. I think it is not about Cocoapods. I encountered this after creating a new app with Xcode 9, so there were no caches. Then I tried to see if this was about that app only and then I saw that my earlier app was having problems when compiled on Xcode 9. When I switch to Xcode 8, everything works fine.

[drewmccormack]

Have you tried just creating an app without Cocoapods, dragging the Ensembles Xcode project in, and linking that way? Still a problem?

[keremerkan]

I am currently trying exactly that. I need to drag Ensembles iOS project to the workspace and link the framework, right?

[drewmccormack]

Yep. See README.

…

On 15 Aug 2017, at 10:29, keremerkan ***@***.***> wrote: I am currently trying exactly that. I need to drag Ensembles iOS project to the workspace and link the framework, right? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#259 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEuAOfXnZ_ocu6T0jHgftg315iVfTkoks5sYVbqgaJpZM4O3ScG>.

[keremerkan]

Can now load the model file, but the other problem still exists. It looks like the sync file is created but as CDEPropertyChangeValueis of type NSObject, SQLite rejects reading it. This seems to be over my head currently :)

[drewmccormack]

The CDEPropertyChangeValue uses NSCoding. SQL should never see it. Core Data should be transforming those change values into data, and storing that.

Is your app in Swift? Just wondering if there is some problem with it is looking in the wrong place for the module.

Could the problem be due to old data in the cloud? Maybe that is now corrupted. Be worth clearing the cloud data.

[keremerkan]

This is still with the pristine app. Removed everything in the cloud and compiled and ran the app. I am using Objective-C.

Here is the error when I enable syncing and add a record on the other device:

-[CDEEventMigrator migrateEventsInFromFiles:completion:]_block_invoke line 180: Failed to migrate modification events: Error Domain=NSCocoaErrorDomain Code=259 "The file couldn’t be opened because it isn’t in the correct format." UserInfo={NSUnderlyingException=Can't read binary data from file, NSUnderlyingError=0x1c4247410 {Error Domain=NSCocoaErrorDomain Code=259 "The file “0_A3FFF4B8-703D-4D2B-992B-FEB59F6CE43B-401-000000196B8033A8_5FB34B05.cdeevent” couldn’t be opened because it isn’t in the correct format." UserInfo={NSFilePath=/var/mobile/Containers/Data/Application/A9082474-E266-4F26-A3BB-EC0756DA454A/Library/Application Support/com.myapp.MyApp/com.mentalfaculty.ensembles.eventdata/transitcache/ListsSync/download/0_A3FFF4B8-703D-4D2B-992B-FEB59F6CE43B-401-000000196B8033A8_5FB34B05.cdeevent, NSUnderlyingException=value for key 'NS.objects' was of unexpected class 'CDEPropertyChangeValue'. Allowed classes are '{( NSDecimalNumber, NSURL, NSNumber, NSUUID, NSOrderedSet, NSDate, NSSet, NSDictionary, NSNull, NSDictionaryMapNode, NSData, NSArray, NSString )}'.}}}

And when I deleech, there are many errors on I/O with the events.sqlite It may be a bug in iOS 11 SQLite implementation, I am not sure.

[drewmccormack]

Seems as if for some reason either the proper keyed-archive transformation is not working (you could check the CDEEventStore model and see if that looks OK), or for some reason the CDEPropertyChangeValue class is not properly loaded from the library, which would be odd.

I will try to test this myself in the coming days and see what I can find.

[drewmccormack]

FWIW I have reproduced this problem using the Idiomatic sample. I suspect a bug in Core Data's binary store code when using transformables. I will investigate this further and see if I can make a simple example exhibiting the problem for Apple.

Note that Ensembles 2 does not use the Core Data binary format, it uses JSON instead, so this issue only affects Ensembles version 1.x in conjunction with Xcode 9.

[keremerkan]

I hope that it can be fixed. I love Ensembles and hope to migrate to Ensembles 2 soon, but currently it is not feasible.

[drewmccormack]

It seems to definitely be a Core Data bug. I have put in radar 33900451. An example of the problem is in this project (which doesn't even include Ensembles).

https://www.dropbox.com/s/htg0m0fq4asqz8f/BinaryStoreWithTransformableProperty.zip?dl=0

If you run that app once to create a store, then kill the app, and run it again, you get the same error. Seems to be that transformable properties aren't working with binary stores.

Ensembles 2 will work. You can try it out for free by taking a subscription, and cancelling before the first month is up.

[drewmccormack]

OK, got clarification from Apple. It is a very fresh change to the Core Data APIs. I will need to add metadata to workaround it.

/*
 Allows developers to provide an additional set of classes (which must implement NSSecureCoding) that should be used while
 decoding a binary store.
 Using this option is preferable to using NSBinaryStoreInsecureDecodingCompatibilityOption.
 */
COREDATA_EXTERN NSString * const NSBinaryStoreSecureDecodingClasses API_AVAILABLE(macosx(10.13),ios(11.0),tvos(11.0),watchos(4.0));

/*
 Indicate that the binary store should be decoded insecurely. This may be necessary if a store has metadata or transformable
 properties containing non-standard classes. If possible, developers should use the NSBinaryStoreSecureDecodingClasses option
 to specify the contained classes, allowing the binary store to to be securely decoded.
 Applications linked before the availability date will default to using this option.
 */
COREDATA_EXTERN NSString * const NSBinaryStoreInsecureDecodingCompatibilityOption API_AVAILABLE(macosx(10.13),ios(11.0),tvos(11.0),watchos(4.0));

Will try to do this in coming days. If you are in a hurry, you could try adding the second one to the metadata where the binary store is imported.

[keremerkan]

Well, hit the "Undefined symbols" error. It should be defined in Core Data but looks like it is not, yet. Or I am doing something wrong.

Undefined symbols for architecture arm64: "_NSBinaryStoreInsecureDecodingCompatibilityOption", referenced from: ___51-[CDEPersistentStoreImporter importWithCompletion:]_block_invoke in CDEPersistentStoreImporter.o

[drewmccormack]

I was told they only added it on the beta yesterday (6?) Kind regards, Drew

…

On 15 Aug 2017, at 22:24, keremerkan ***@***.***> wrote: Well, hit the "Undefined symbols" error. It should be defined in Core Data but looks like it is not, yet. Or I am doing something wrong. Undefined symbols for architecture arm64: "_NSBinaryStoreInsecureDecodingCompatibilityOption", referenced from: ___51-[CDEPersistentStoreImporter importWithCompletion:]_block_invoke in CDEPersistentStoreImporter.o — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[keremerkan]

I found that it was added on beta 5. Xcode recognises and autofills the option, but fails when building. And it does not yet have a beta 6 (even though iOS has one), so I think we will have to wait.

[drewmccormack]

I was told it is only for apps targeting iOS 11, so maybe just targeting iOS 10 may avoid it.

…

On 15 Aug 2017, at 22:32, keremerkan ***@***.***> wrote: I found that it was added on beta 5. Xcode recognises and autofills the option, but fails when building. And it does not yet have a beta 6 (even though iOS has one), so I think we will have to wait. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[keremerkan]

Unfortunately, this app has to target iOS 11. And the older one already targets iOS 9 and later, maybe I can be able to use Xcode 8 for that one for some time until this is fixed. But I am unable to use syncing for the new app for the time being. It cannot be released at the time being anyway as I'll need to wait for iOS 11 to be released. I hope everything will work correctly by then. :)

[drewmccormack]

I was told by the Core Data manager that this functionality was already in the beta, but very recent. I will try it, and if it isn’t, will get back to him about it. Drew

[keremerkan]

Great! I will check here for future updates.

[drewmccormack]

Apple says the SDK is not up to date, but the system is, or something like that. In other words, the option should work, but you can't use the variable. Apparently we can just use the string

"_NSBinaryStoreInsecureDecodingCompatibilityOption"

I have just tested my little test app with this string as key, and indeed it works. I will push it to the master of GitHub, so you could use that with Cocoapods directly (won't make a release just yet).

[drewmccormack]

Yep, just tested this in Idiomatic, and seems to solve the issue. Have pushed to master.

[keremerkan]

Yes, it is fixed. Now I need to find why I get libsqlite3 errors when deleeching :) I think this may indeed be a bug in iOS.

[logging] BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use: /private/var/mobile/Containers/Data/Application/9D5FA7A6-65CF-4BC5-98BF-772CB6A93C92/Library/Application Support/com.myapp.MyApp/com.mentalfaculty.ensembles.eventdata/ListsSync/events.sqlite [logging] invalidated open fd: 9 (0x11) [logging] BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use: /private/var/mobile/Containers/Data/Application/9D5FA7A6-65CF-4BC5-98BF-772CB6A93C92/Library/Application Support/com.myapp.MyApp/com.mentalfaculty.ensembles.eventdata/ListsSync/events.sqlite-shm [logging] invalidated open fd: 11 (0x11) [logging] BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use: /private/var/mobile/Containers/Data/Application/9D5FA7A6-65CF-4BC5-98BF-772CB6A93C92/Library/Application Support/com.myapp.MyApp/com.mentalfaculty.ensembles.eventdata/ListsSync/events.sqlite-wal [logging] invalidated open fd: 10 (0x11) [error] error: (6922) I/O error for database at /var/mobile/Containers/Data/Application/9D5FA7A6-65CF-4BC5-98BF-772CB6A93C92/Library/Application Support/com.myapp.MyApp/com.mentalfaculty.ensembles.eventdata/ListsSync/events.sqlite. SQLite error code:6922, 'disk I/O error'

[drewmccormack]

Yes, I saw those errors too. I think it is because deleeching deletes the sqlite store that is used internally by Ensembles. Should be harmless, because you don't need that anymore, but if the message persists, I will look into it more.

[keremerkan]

@drewmccormack I just added pull request #260 to fix this issue. The code works correctly on both iOS 11 and iOS 10. Apps on both iOS versions can now sync correctly with each other.

[yogesh-yudiz]

I am getting same issue in iOS 11 what want to do?

[drewmccormack]

The serious issue is fixed in the master branch already. You can use that until we make a new release (should be soon).

[yogesh-yudiz]

Yeah, already used update code, but its not work, I changes manually in my code now its work fine for me.

[drewmccormack]

Pushed some stuff now that may help. (Still not new version, so Cocoapods will not update yet).

[chadseld]

Thanks @drewmccormack, I set CocoaPods to use master and that fixed it for me.

[fabcan1]

I think I have the same issues.
In my project I have only that in my pod file:
pod 'Ensembles'
How to set CocoaPods to use master??
Thanks.

[drewmccormack]

You shouldn't need to use master now. Just get CocoaPods to update the code. It should get the new release with the fix.

[fabcan1]

So I have updated CocoaPods, Remove the Ensembles pod, install it again and I still have this issues:

ad/51_D3424470-92A2-498E-80E6-6F27AAF9D1A9-3319-0000049580B14474_C2180E91.cdeevent, NSUnderlyingException=value for key 'NS.objects' was of unexpected class 'CDEPropertyChangeValue'. Allowed classes are '{(
    NSString,
    NSSet,
    NSDictionary,
    NSURL,
    NSDecimalNumber,
    NSNull,
    NSData,
    NSUUID,
    NSNumber,
    NSDate,
    NSArray,
    NSOrderedSet,
    NSDictionaryMapNode
)}'.}}} with userInfo dictionary {
    NSUnderlyingError = "Error Domain=NSCocoaErrorDomain Code=259 \"The file \U201c51_D3424470-92A2-498E-80E6-6F27AAF9D1A9-3319-0000049580B14474_C2180E91.cdeevent\U201d couldn\U2019t be opened because it isn\U2019t in the correct format.\"

[drewmccormack]

In all likelihood it is a Cocoapods problem. I'm not expert with Cocoapods. Never use it myself.

I would check if the installed version of Ensembles is 1.7.1. That is the latest release. You should be able to see that in one of the Cocoapods files that it produces to show what the resolved versions are.

(Cocoapods does a heap of caching. Often you think you have removed things, but you haven't.)

[fabcan1]

I think you are right. In the terminal I can see when installing:
Installing Ensembles (1.7)
So apparently I don't have the latest release (1.7.1) and don't know how to have it?

[drewmccormack]

You tried pod update Ensembles?

Best to google ways to clear Cocoapods and update. This is unlikely to be directly related to Ensembles.

[fabcan1]

I have successfully update Cocoapods with
pod repo update
Then remove and install again Ensembles and now I have the 1.7.1 and that WORKS!
Thank you Drew.