Add civiimport core extension with Import api code for viewing import tables

[eileenmcnaughton]

Overview

Add civiimport core extension with Import api code for viewing import tables.

This PR is primary about adding new pseudo-crud api for interacting with the data import tables. The data import tables are short-lived (default is a week, although we do not yet have a good cleanup process) tables that hold the data the user is trying to import plus some status & informational fields. The goal of adding an API is to get logic out of quickforms & associated classes & standardise it and to get the ability to leveerage search kit, afform etc.

With this PR each import UserJob is not passed will be visible as an api pseudoentity (similar to eck) and will support crud actions per below. Note that the fields available are based on the field names in the user's data source - so in the example the user had a column 'Amount Given' (or similar).

Because of the above editable search kit displays can be added - although I have opted to keep the commit that adds them for a follow up PR to keep this focussed on api rather than ui.

The intent of putting the code in an extension is that it feels like the import code should be separated into a core extension as a code management consideration - but it is not intended to be an optional extension (maybe one day in the far far future). I need to work through the steps of making it always enabled & hidden and that tests run on it - but those are not blocking on merging this & generally I find a few steps are required AFTER merging any new extension.

I also have the intention to follow up with some additional apis / actions available to search-kit - most obviously
validate and import which will allow a user to edit-in-place a row that failed & re-try it. However, I'm also expecting some metadata functions to facilitate getting more logic out of the forms & potentially replacing some chunks of forms with angular (the relevant forms are quite modular)

Before

No api support for interacting with Import pseudo-entities, not available in search kit

After

   $importFields = Import::getFields($userJobID)->execute();

    Import::create($userJobID)->setValues([
      'external_identifier' => 678,
      'amount_given' => 80,
      '_status' => 'NEW',
    ])->execute();

  $import = Import::get($userJobID)->setSelect(['external_identifier', 'amount_given', '_status'])->execute()->first();
   

    Import::update($userJobID)->setValues([
      'amount_given' => NULL,
      '_id' => $import['_id'],
      '_status' => 'IMPORTED',
    ])->execute();


    Import::save($userJobID)->setRecords([
      [
        'external_identifier' => 999,
        'amount_given' => 9,
        '_status' => 'ERROR',
      ],
    ])->execute();

Technical Details

We do a couple of checks before exposing an Import to the api

• the table can be determined and exists
• the expires_date is not in the past - in future we will have a cleanup routine to get rid of tables with a past expires_date so it will be equivalent to the first condition

Permissioning is based on checking if the UserJob in question will load for the relevant user - https://github.com/civicrm/civicrm-core/pull/24230/files#diff-4077b8d652237c281456106d3f92c6e6a1cc3ec3f8dd3da117b58fc102c94d37R89-R98

Comments

I spun off a couple of PRs #24285 with the changes that are outside the extension #24275

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[colemanw]

This is kinda ugly and not the right way to use ts(), is there no user-facing label for these temp tables?

[colemanw]

Guess there's no label field? That could be a problem, how is the user supposed to choose the correct import job entity in SearchKit?

[eileenmcnaughton]

I think I could append the user display name - ie import2-by-Coleman Watts

[colemanw]

@eileenmcnaughton - doesn't have to be in this PR but maybe the label could be combination of {entity} {date} {user} like "Contact Import 8/15/2022 by Coleman Watts"...

Alternatively, (maybe better) we could add a column for _label and populate it with the name of the file.

[eileenmcnaughton]

@colemanw hmm ok - I'll ponder those ideas - I am inclined to do it as a follow up (if this passes)

[colemanw]

@eileenmcnaughton there's more overiding than necessary going on in the APIv4 classes. Ideally they shouldn't need to do anything special. #24290 ought to make that possible.

[eileenmcnaughton]

CiviApiImportTest::testApiActions
Error: Call to undefined method Civi\Api4\Import::create()

/home/jenkins/bknix-dfl/build/core-24230-7aju4/web/sites/all/modules/civicrm/ext/civiimport/tests/phpunit/CiviApiImportTest.php:76
/home/jenkins/bknix-dfl/extern/phpunit8/phpunit8.phar:1721

[colemanw]

@eileenmcnaughton I thought create wasn't allowed? Isn't this an update-only entity?

[eileenmcnaughton]

@colemanw yeah - I don't see a downside to having a CREATE api but not a current need (other than to create in the test) either. We wouldn't want to expose create to afform though

[colemanw]

Ok. I'll add it back. I think we need a way to tell afform what actions are allowed per entity, because I'm also working on CiviCase for Afform and that only supports create not update.

[eileenmcnaughton]

don't we mark some 'read-only' for that reason?

[eileenmcnaughton]

Hmm nope

CiviApiImportTest::testApiActions
CRM_Core_Exception: Import records can only be updated not created; "_id" is required.

[colemanw]

don't we mark some 'read-only' for that reason?

"Read-only" impies no create AND no update. We want to allow one not the other.

[eileenmcnaughton]

@colemanw it passed