diff --git a/CHANGES.md b/CHANGES.md
new file mode 100644
index 0000000..89ce2fd
--- /dev/null
+++ b/CHANGES.md
@@ -0,0 +1,112 @@
+# Version 2.0
+
+Massive changes to accomodate Mailchimp Api3 which is completely different, and
+automated testing capability.
+
+An upgrade hook is added to migrate from versions using Api <3. This must be run
+while Api2 is still working, i.e in 2016 according to Mailchimp.
+
+These changes have been made by Rich Lott / artfulrobot.uk with thanks to the
+*Sumatran Organutan Society* for funding a significant chunk of the work.
+
+Added this markdown changelog :-)
+
+## Contact and email selection
+
+Contacts must now
+
+- have an email available
+- not be deceased (new)
+- not have `is_opt_out` set
+- not have `do_not_email` set
+
+The system will prefer the bulk email address instead of the primary one.
+If no bulk one is available, then it will pick the primary, or if that's not
+there either (?!) it will pick any.
+
+## CiviCRM post hook changes
+
+The *post hook* used to fire API calls for all GroupContact changes. I've changed
+this to only do so when there is only one contact affected. This hook could be
+called with 1000 contacts which would have fired 1000 API calls one after
+another, so for stability I removed that 'feature' and for clarity I chose 1 as
+the maximum number of contacts allowed.
+
+
+## Identifying contacts in CiviCRM from Mailchimp
+
+Most of the fields in the tmp tables are now *`NOT NULL`*. Having nulls just made
+things more complex and we don't need to distinguish different types of
+not-there data.
+
+A new method is added to identify the CiviCRM contact Ids from Mailchimp details
+that looks to the subscribers we're expecting to find. This solves the issue
+when two contacts (e.g. related) are in CiviCRM with the same email, but only
+one of them is subscribed to the list - now it will pick the subscriber. This
+test ought to be the fastest of the methods, so it is run first.
+
+The email-is-unique test to identify a contact has been modified such that if
+the email is unique to a particular contact, we guess that contact. Previously
+the email had to be unique in the email table, which excludes the case that
+someone has the same email in several times (e.g. once as a billing, once as a
+bulk...).
+
+The email and name SQL for 'guessing' the contact was found buggy by testing so
+has been rewritten - see tests.
+
+
+## Group settings page
+
+Re-worded integration options for clarity. Added fixup checkbox, default ticked.
+On saving the form, if this is ticked, CiviCRM will ensure the webhook settings
+are correct at Mailchimp.
+
+## Mailchimp Settings Page
+
+Checks all lists including a full check on the webhook config.
+
+## Changes of email from Mailchimp's 'upemail' webhook
+
+Previously we found the first email match from the email table and changed that.
+This does not allow for the same email being in the table multiple times.
+
+This *should* only happen to people we know are subscribed to the list. Also,
+there's the case that a user has a primary email of personal@example.com and
+wanted to change it at Mailchimp to mybulkmail@example.com.
+
+So now what we do is:
+
+1. find the email. Filter for contacts we know to be subscribed.
+
+2. if this *is* their bulk email, just change it.
+
+3. if it's *not* their bulk email, do they have a bulk email?
+
+ Yes: change that.
+ No: create that with the new email.
+
+Ideally we'd have staff notified to check the emails, possibly in the 3:No case,
+set the email to on hold. But without further human interaction it's safest to
+do as outlined above.
+
+The upemail will change *all* emails found, not just the first, so long as they
+belong to a single contact on the list. So if the email is in CiviCRM against a
+different contact who is not in the mailchimp list, that will be left unchanged.
+
+## Changes to response to Mailchimp's 'cleaned' webhook
+
+Previously the first matching was found and put on hold.
+
+Cleaned comes in two flavours: hard (email keeps bouncing) and abuse (they don't
+like you anymore).
+
+If the email is bouncing for mailchimp, in all their deliverability might, it's
+almost definitely going to bounce for us. So in this case we put all matching
+emails on hold.
+
+In the case of 'abuse' we limit the action to email(s) belonging to contacts on
+this list only, since it might be to do with that list.
+
+
+
+
diff --git a/CRM/Mailchimp/Api3.php b/CRM/Mailchimp/Api3.php
new file mode 100644
index 0000000..3dba271
--- /dev/null
+++ b/CRM/Mailchimp/Api3.php
@@ -0,0 +1,453 @@
+api_key = $settings['api_key'];
+
+ // Set URL based on datacentre identifier at end of api key.
+ preg_match('/^.*-([^-]+)$/', $this->api_key, $matches);
+ if (empty($matches[1])) {
+ throw new InvalidArgumentException("Invalid API key - could not extract datacentre from given API key.");
+ }
+
+ if (!empty($settings['log_facility'])) {
+ $this->setLogFacility($settings['log_facility']);
+ }
+
+ $datacenter = $matches[1];
+ $this->server = "https://$datacenter.api.mailchimp.com/3.0";
+ }
+ /**
+ * Sets the log_facility to a callback
+ */
+ public function setLogFacility($callback) {
+ if (!is_callable($callback)) {
+ throw new InvalidArgumentException("Log facility callback is not callable.");
+ }
+ $this->log_facility = $callback;
+ }
+
+ /**
+ * Perform a GET request.
+ */
+ public function get($url, $data=null) {
+ return $this->makeRequest('GET', $url, $data);
+ }
+
+ /**
+ * Perform a POST request.
+ */
+ public function post($url, Array $data) {
+ return $this->makeRequest('POST', $url, $data);
+ }
+
+ /**
+ * Perform a PUT request.
+ */
+ public function put($url, Array $data) {
+ return $this->makeRequest('PUT', $url, $data);
+ }
+
+ /**
+ * Perform a PATCH request.
+ */
+ public function patch($url, Array $data) {
+ return $this->makeRequest('PATCH', $url, $data);
+ }
+
+ /**
+ * Perform a DELETE request.
+ */
+ public function delete($url, $data=null) {
+ return $this->makeRequest('DELETE', $url);
+ }
+
+ /**
+ * Perform a /batches POST request and sit and wait for the result.
+ *
+ * It quicker to run small ops directly for <15 items.
+ *
+ */
+ public function batchAndWait(Array $batch, $method=NULL) {
+ // This can take a long time...
+ set_time_limit(0);
+
+ if ($method === NULL) {
+ // Automatically determine fastest method.
+ $method = (count($batch) < 15) ? 'multiple' : 'batch';
+ }
+ elseif (!in_array($method, ['multiple', 'batch'])) {
+ throw new InvalidArgumentException("Method argument must be mulitple|batch|NULL, given '$method'");
+ }
+
+ // Validate the batch operations.
+ foreach ($batch as $i=>$request) {
+ if (count($request)<2) {
+ throw new InvalidArgumentException("Batch item $i invalid - at least two values required.");
+ }
+ if (!preg_match('/^get|post|put|patch|delete$/i', $request[0])) {
+ throw new InvalidArgumentException("Batch item $i has invalid method '$request[0]'.");
+ }
+ if (substr($request[1], 0, 1) != '/') {
+ throw new InvalidArgumentException("Batch item $i has invalid path should begin with /. Given '$request[1]'");
+ }
+ }
+
+ // Choose method and submit.
+ if ($method == 'batch') {
+ // Submit a batch request and wait for it to complete.
+ $batch_result = $this->makeBatchRequest($batch);
+
+ do {
+ sleep(3);
+ $result = $this->get("/batches/{$batch_result->data->id}");
+ } while ($result->data->status != 'finished');
+
+ // Now complete.
+ // Note: we have no way to check the errors. Mailchimp make a downloadable
+ // .tar.gz file with one file per operation available, however PHP (as of
+ // writing) has a bug (I've reported it
+ // https://bugs.php.net/bug.php?id=72394) in its PharData class that
+ // handles opening of tar files which means there's no way we can access
+ // that info. So we have to ignore errors.
+ return $result;
+ }
+ else {
+ // Submit the requests one after another.
+ foreach ($batch as $item) {
+ $method = strtolower($item[0]);
+ $path = $item[1];
+ $data = isset($item[2]) ? $item[2] : [];
+ try {
+ $this->$method($path, $data);
+ }
+ catch (CRM_Mailchimp_RequestErrorException $e) {
+ // Here we ignore exceptions from Mailchimp not because we want to,
+ // but because we have no way of handling such errors when done for
+ // 15+ items in a proper batch, so we don't handle them here either.
+ }
+ }
+ }
+ }
+ /**
+ * Sends a batch request.
+ *
+ * @param array batch array of arrays which contain three values: the method,
+ * the path (e.g. /lists) and the data describing a set of requests.
+ */
+ public function makeBatchRequest(Array $batch) {
+ $ops = [];
+ foreach ($batch as $request) {
+ $op = ['method' => strtoupper($request[0]), 'path' => $request[1]];
+ if (!empty($request[2])) {
+ if ($op['method'] == 'GET') {
+ $op['params'] = $request[2];
+ }
+ else {
+ $op['body'] = json_encode($request[2]);
+ }
+ }
+ $ops []= $op;
+ }
+ $result = $this->post('/batches', ['operations' => $ops]);
+
+ return $result;
+ }
+ /**
+ * Setter for $network_enabled.
+ */
+ public function setNetworkEnabled($enable=TRUE) {
+ $this->network_enabled = (bool) $enable;
+ }
+ /**
+ * Provide mock for curl.
+ *
+ * The callback will be called with the
+ * request object in $this->request. It must return an array with optional
+ * keys:
+ *
+ * - exec the mocked output of curl_exec(). Defaults to '{}'.
+ * - info the mocked output of curl_getinfo(), defaults to an array:
+ * - http_code => 200
+ * - content_type => 'application/json'
+ *
+ * Note the object must be operating with network-enabled for this to be
+ * called; it exactly replaces the curl work.
+ *
+ * @param null|callback $callback. If called with anything other than a
+ * callback, this functionality is disabled.
+ */
+ public function setMockCurl($callback) {
+ if (is_callable($callback)) {
+ $this->mock_curl = $callback;
+ }
+ else {
+ $this->mock_curl = NULL;
+ }
+ }
+ /**
+ * All request types handled here.
+ *
+ * Set up all parameters for the request.
+ * Submit the request.
+ * Return the response.
+ *
+ * Implemenations should call this first, then do their curl-ing (or not),
+ * then return the response.
+ *
+ * @throw InvalidArgumentException if called with a url that does not begin
+ * with /.
+ * @throw CRM_Mailchimp_NetworkErrorException
+ * @throw CRM_Mailchimp_RequestErrorException
+ */
+ protected function makeRequest($method, $url, $data=null) {
+ if (substr($url, 0, 1) != '/') {
+ throw new InvalidArgumentException("Invalid URL - must begin with root /");
+ }
+ $this->request = (object) [
+ 'id' => static::$request_id++,
+ 'created' => microtime(TRUE),
+ 'completed' => NULL,
+ 'method' => $method,
+ 'url' => $this->server . $url,
+ 'headers' => ["Content-Type: Application/json;charset=UTF-8"],
+ 'userpwd' => "dummy:$this->api_key",
+ // Set ZLS for default data.
+ 'data' => '',
+ // Mailchimp's certificate chain does not include trusted root for cert for
+ // some popular OSes (e.g. Debian Jessie, April 2016) so disable SSL verify
+ // peer.
+ 'verifypeer' => FALSE,
+ // ...but we can check that the certificate has the domain we were
+ // expecting.@see http://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html
+ 'verifyhost' => 2,
+ ];
+
+ if ($data !== null) {
+ if ($this->request->method == 'GET') {
+ // For GET requests, data must be added as query string.
+ // Append if there's already a query string.
+ $query_string = http_build_query($data);
+ if ($query_string) {
+ $this->request->url .= ((strpos($this->request->url, '?')===false) ? '?' : '&')
+ . $query_string;
+ }
+ }
+ else {
+ // Other requests have it added as JSON
+ $this->request->data = json_encode($data);
+ $this->request->headers []= "Content-Length: " . strlen($this->request->data);
+ }
+ }
+
+ // We set up a null response.
+ $this->response = (object) [
+ 'http_code' => null,
+ 'data' => null,
+ ];
+
+ if ($this->network_enabled) {
+ $this->sendRequest();
+ }
+ else {
+ // We're not going to send a request.
+ // So this is our chance to log something.
+ $this->log();
+ }
+ return $this->response;
+ }
+ /**
+ * Send the request and prepare the response.
+ */
+ protected function sendRequest() {
+ if (!$this->mock_curl) {
+ $curl = curl_init();
+ curl_setopt($curl, CURLOPT_CUSTOMREQUEST, $this->request->method);
+ curl_setopt($curl, CURLOPT_POSTFIELDS, $this->request->data);
+ curl_setopt($curl, CURLOPT_HTTPHEADER, $this->request->headers);
+ curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
+ curl_setopt($curl, CURLOPT_USERPWD, $this->request->userpwd);
+ curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, $this->request->verifypeer);
+ curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, $this->request->verifyhost);
+ curl_setopt($curl, CURLOPT_URL, $this->request->url);
+ curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
+ $result = curl_exec($curl);
+ $info = curl_getinfo($curl);
+ curl_close($curl);
+ }
+ else {
+ $callback = $this->mock_curl;
+ $output = $callback($this->request);
+ // Apply defaults to result.
+ $output += [
+ 'exec' => '{}',
+ 'info' => [],
+ ];
+ $output['info'] += [
+ 'http_code' => 200,
+ 'content_type' => 'application/json',
+ ];
+ $result = $output['exec'];
+ $info = $output['info'];
+ }
+
+ return $this->curlResultToResponse($info, $result);
+ }
+
+ /**
+ * For debugging purposes.
+ *
+ * Does nothing without $log_facility being set to a callback.
+ *
+ */
+ protected function log() {
+ if (!$this->log_facility) {
+ return;
+ }
+
+ $msg = "Request #{$this->request->id}\n=============================================\n";
+ if (!$this->network_enabled) {
+ $msg .= "Network : DISABLED\n";
+ }
+ $msg .= "Method : {$this->request->method}\n";
+ $msg .= "Url : {$this->request->url}\n";
+
+ if (isset($this->request->created)) {
+ $msg .= "Took : " . round((microtime(TRUE) - $this->request->created), 2) . "s\n";
+ }
+ $msg .= "Response Code: "
+ . (isset($this->response->http_code) ? $this->response->http_code : 'NO RESPONSE HTTP CODE')
+ . "\n";
+
+ $msg .= "Request Body : " . str_replace("\n", "\n ",
+ var_export(json_decode($this->request->data), TRUE)) . "\n";
+ $msg .= "Response Body: " . str_replace("\n", "\n ",
+ var_export($this->response->data, TRUE));
+ $msg .= "\n\n";
+
+ // Log response.
+ $callback = $this->log_facility;
+ $callback($msg);
+ }
+ /**
+ * Prepares the response object from the result of a cURL call.
+ *
+ * Public to allow testing.
+ *
+ * @return Array response object.
+ * @throw CRM_Mailchimp_RequestErrorException
+ * @throw CRM_Mailchimp_NetworkErrorException
+ * @param array $info output of curl_getinfo().
+ * @param string|null $result output of curl_exec().
+ */
+ public function curlResultToResponse($info, $result) {
+
+ // Check response.
+ if (empty($info['http_code'])) {
+ $this->log();
+ throw new CRM_Mailchimp_NetworkErrorException($this);
+ }
+
+ // Check response object is set up.
+ if (!isset($this->response)) {
+ $this->response = (object) [
+ 'http_code' => null,
+ 'data' => null,
+ ];
+ }
+
+ // Copy http_code into response object. (May yet be used by exceptions.)
+ $this->response->http_code = $info['http_code'];
+
+ // was JSON returned, as expected?
+ $json_returned = isset($info['content_type'])
+ && preg_match('@^application/(problem\+)?json\b@i', $info['content_type']);
+
+
+ if (!$json_returned) {
+ // According to Mailchimp docs it may return non-JSON in event of a
+ // timeout.
+ $this->log();
+ throw new CRM_Mailchimp_NetworkErrorException($this);
+ }
+
+ $this->response->data = $result ? json_decode($result) : null;
+ $this->log();
+
+ // Check for errors and throw appropriate CRM_Mailchimp_ExceptionBase.
+ switch (substr((string) $this->response->http_code, 0, 1)) {
+ case '4': // 4xx errors
+ throw new CRM_Mailchimp_RequestErrorException($this);
+ case '5': // 5xx errors
+ throw new CRM_Mailchimp_NetworkErrorException($this);
+ }
+
+ // All good return response as a convenience.
+ return $this->response;
+ }
+}
diff --git a/CRM/Mailchimp/DuplicateContactsException.php b/CRM/Mailchimp/DuplicateContactsException.php
new file mode 100644
index 0000000..64421da
--- /dev/null
+++ b/CRM/Mailchimp/DuplicateContactsException.php
@@ -0,0 +1,13 @@
+contacts = $contacts;
+ parent::__construct("Duplicate Contacts found.");
+ }
+}
diff --git a/CRM/Mailchimp/Exception.php b/CRM/Mailchimp/Exception.php
new file mode 100644
index 0000000..958071e
--- /dev/null
+++ b/CRM/Mailchimp/Exception.php
@@ -0,0 +1,26 @@
+request = isset($api->request) ? clone($api->request) : NULL;
+ $this->response = isset($api->response) ? clone($api->response) : NULL;
+
+ if (isset($this->response->data->title)) {
+ $message = $message_prefix . 'Mailchimp API said: ' . $this->response->data->title;
+ }
+ else {
+ $message = $message_prefix . 'No data received, possibly a network timeout';
+ }
+ parent::__construct($message, $this->response->http_code);
+ }
+
+}
diff --git a/CRM/Mailchimp/Form/Pull.php b/CRM/Mailchimp/Form/Pull.php
index 41afcac..9187b18 100644
--- a/CRM/Mailchimp/Form/Pull.php
+++ b/CRM/Mailchimp/Form/Pull.php
@@ -21,7 +21,11 @@ function preProcess() {
}
$output_stats = array();
+ $this->assign('dry_run', $stats['dry_run']);
foreach ($groups as $group_id => $details) {
+ if (empty($details['list_name'])) {
+ continue;
+ }
$list_stats = $stats[$details['list_id']];
$output_stats[] = array(
'name' => $details['civigroup_title'],
@@ -29,24 +33,71 @@ function preProcess() {
);
}
$this->assign('stats', $output_stats);
+
+ // Load contents of mailchimp_log table.
+ $dao = CRM_Core_DAO::executeQuery("SELECT * FROM mailchimp_log ORDER BY id");
+ $logs = [];
+ while ($dao->fetch()) {
+ $logs []= [
+ 'group' => $dao->group_id,
+ 'email' => $dao->email,
+ 'name' => $dao->name,
+ 'message' => $dao->message,
+ ];
+ }
+ $this->assign('error_messages', $logs);
}
}
public function buildQuickForm() {
- // Create the Submit Button.
- $buttons = array(
- array(
- 'type' => 'submit',
- 'name' => ts('Import'),
- ),
- );
- // Add the Buttons.
- $this->addButtons($buttons);
+
+ $groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), null, $membership_only = TRUE);
+ $will = '';
+ $wont = '';
+ if (!empty($_GET['reset'])) {
+ foreach ($groups as $group_id => $details) {
+ $description = ""
+ . "CiviCRM group $group_id: "
+ . htmlspecialchars($details['civigroup_title']) . "";
+
+ if (empty($details['list_name'])) {
+ $wont .= "
" . ts('The following lists will be synchronised') . "
$will
";
+
+ // Create the Submit Button.
+ $buttons = array(
+ array(
+ 'type' => 'submit',
+ 'name' => ts('Sync Contacts'),
+ ),
+ );
+
+ $this->addElement('checkbox', 'mc_dry_run',
+ ts('Dry Run? (if ticked no changes will be made to CiviCRM or Mailchimp.)'));
+
+ $this->addButtons($buttons);
+ }
+ if ($wont) {
+ $msg .= "
" . ts('The following lists will be NOT synchronised') . "
The following list(s) no longer exist at Mailchimp.
$wont
";
+ }
+ $this->assign('summary', $msg);
+
}
public function postProcess() {
$setting_url = CRM_Utils_System::url('civicrm/mailchimp/settings', 'reset=1', TRUE, NULL, FALSE, TRUE);
- $runner = self::getRunner();
+ $vals = $this->_submitValues;
+ $runner = self::getRunner(FALSE, !empty($vals['mc_dry_run']));
+ // Clear out log table.
+ CRM_Mailchimp_Sync::dropLogTable();
if ($runner) {
// Run Everything in the Queue via the Web.
$runner->runAllViaWeb();
@@ -55,7 +106,7 @@ public function postProcess() {
}
}
- static function getRunner($skipEndUrl = FALSE) {
+ public static function getRunner($skipEndUrl = FALSE, $dry_run = FALSE) {
// Setup the Queue
$queue = CRM_Queue_Service::singleton()->create(array(
'name' => self::QUEUE_NAME,
@@ -64,8 +115,8 @@ static function getRunner($skipEndUrl = FALSE) {
));
// reset pull stats.
- CRM_Core_BAO_Setting::setItem(Array(), CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'pull_stats');
- $stats = array();
+ $stats = ['dry_run' => $dry_run];
+ CRM_Core_BAO_Setting::setItem($stats, CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'pull_stats');
// We need to process one list at a time.
$groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), null, $membership_only=TRUE);
@@ -76,6 +127,11 @@ static function getRunner($skipEndUrl = FALSE) {
// Each list is a task.
$listCount = 1;
foreach ($groups as $group_id => $details) {
+ if (empty($details['list_name'])) {
+ // This list has been deleted at Mailchimp, or for some other reason we
+ // could not access its name. Best not to sync it.
+ continue;
+ }
$stats[$details['list_id']] = array(
'mc_count' => 0,
'c_count' => 0,
@@ -88,8 +144,8 @@ static function getRunner($skipEndUrl = FALSE) {
$task = new CRM_Queue_Task(
array ('CRM_Mailchimp_Form_Pull', 'syncPullList'),
- array($details['list_id'], $identifier),
- "Preparing queue for $identifier"
+ array($details['list_id'], $identifier, $dry_run),
+ "$identifier: collecting data from CiviCRM..."
);
// Add the Task to the Queue
@@ -97,7 +153,7 @@ static function getRunner($skipEndUrl = FALSE) {
}
// Setup the Runner
$runnerParams = array(
- 'title' => ts('Import From Mailchimp'),
+ 'title' => ($dry_run ? ts('Dry Run: ') : '') . ts('Mailchimp Pull Sync: update CiviCRM from Mailchimp'),
'queue' => $queue,
'errorMode'=> CRM_Queue_Runner::ERROR_ABORT,
'onEndUrl' => CRM_Utils_System::url(self::END_URL, self::END_PARAMS, TRUE, NULL, FALSE),
@@ -112,194 +168,113 @@ static function getRunner($skipEndUrl = FALSE) {
return $runner;
}
- static function syncPullList(CRM_Queue_TaskContext $ctx, $listID, $identifier) {
+ public static function syncPullList(CRM_Queue_TaskContext $ctx, $listID, $identifier, $dry_run) {
+ // Add the CiviCRM collect data task to the queue
+ // It's important that this comes before the Mailchimp one, as some
+ // fast contact matching SQL can run if it's done this way.
+ $ctx->queue->createItem( new CRM_Queue_Task(
+ array('CRM_Mailchimp_Form_Pull', 'syncPullCollectCiviCRM'),
+ array($listID),
+ "$identifier: Fetched data from CiviCRM, fetching from Mailchimp..."
+ ));
+
// Add the Mailchimp collect data task to the queue
$ctx->queue->createItem( new CRM_Queue_Task(
array('CRM_Mailchimp_Form_Pull', 'syncPullCollectMailchimp'),
array($listID),
- "$identifier: Fetching data from Mailchimp (can take a mo)"
+ "$identifier: Fetched data from Mailchimp. Matching..."
));
- // Add the CiviCRM collect data task to the queue
+ // Add the slow match process for difficult contacts.
$ctx->queue->createItem( new CRM_Queue_Task(
- array('CRM_Mailchimp_Form_Pull', 'syncPullCollectCiviCRM'),
+ array('CRM_Mailchimp_Form_Pull', 'syncPullMatch'),
array($listID),
- "$identifier: Fetching data from CiviCRM"
+ "$identifier: Matched up contacts. Comparing..."
));
- // Remaining people need something updating.
$ctx->queue->createItem( new CRM_Queue_Task(
- array('CRM_Mailchimp_Form_Pull', 'syncPullUpdates'),
+ array('CRM_Mailchimp_Form_Pull', 'syncPullIgnoreInSync'),
array($listID),
- "$identifier: Updating contacts in CiviCRM"
+ "$identifier: Ignored any in-sync contacts. Updating CiviCRM with changes."
+ ));
+
+ // Add the Civi Changes.
+ $ctx->queue->createItem( new CRM_Queue_Task(
+ array('CRM_Mailchimp_Form_Pull', 'syncPullFromMailchimp'),
+ array($listID, $dry_run),
+ "$identifier: Completed."
));
return CRM_Queue_Task::TASK_SUCCESS;
}
/**
- * Collect Mailchimp data into temporary working table.
+ * Collect CiviCRM data into temporary working table.
*/
- static function syncPullCollectMailchimp(CRM_Queue_TaskContext $ctx, $listID) {
+ public static function syncPullCollectCiviCRM(CRM_Queue_TaskContext $ctx, $listID) {
- // Shared process.
- $count = CRM_Mailchimp_Form_Sync::syncCollectMailchimp($listID);
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Pull syncPullCollectMailchimp = $count', $count);
- static::updatePullStats(array( $listID => array('mc_count'=>$count)));
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $stats[$listID]['c_count'] = $sync->collectCiviCrm('pull');
+ CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Pull syncPullCollectCiviCRM $stats[$listID][c_count]', $stats[$listID]['c_count']);
+ static::updatePullStats($stats);
return CRM_Queue_Task::TASK_SUCCESS;
}
/**
- * Collect CiviCRM data into temporary working table.
+ * Collect Mailchimp data into temporary working table.
*/
- static function syncPullCollectCiviCRM(CRM_Queue_TaskContext $ctx, $listID) {
-
- // Shared process.
- $stats[$listID]['c_count'] = CRM_Mailchimp_Form_Sync::syncCollectCiviCRM($listID);
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Pull syncPullCollectCiviCRM $stats[$listID][c_count]', $stats[$listID]['c_count']);
+ public static function syncPullCollectMailchimp(CRM_Queue_TaskContext $ctx, $listID) {
- // Remove identicals
- $stats[$listID]['in_sync'] = CRM_Mailchimp_Form_Sync::syncIdentical();
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Pull syncPullCollectCiviCRM $stats[$listID][in_sync]', $stats[$listID]['in_sync']);
+ // Nb. collectCiviCrm must have run before we call this.
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $stats[$listID]['mc_count'] = $sync->collectMailchimp('pull');
+ CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Pull syncPullCollectMailchimp count=', $stats[$listID]['mc_count']);
static::updatePullStats($stats);
+ return CRM_Queue_Task::TASK_SUCCESS;
+ }
+ /**
+ * Do the difficult matches.
+ */
+ public static function syncPullMatch(CRM_Queue_TaskContext $ctx, $listID) {
+
+ // Nb. collectCiviCrm must have run before we call this.
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $c = $sync->matchMailchimpMembersToContacts();
+ CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Pull syncPullMatch count=', $c);
return CRM_Queue_Task::TASK_SUCCESS;
}
/**
- * New contacts from Mailchimp need bringing into CiviCRM.
+ * Remove anything that's the same.
*/
- static function syncPullUpdates(CRM_Queue_TaskContext $ctx, $listID) {
- // Prepare the groups that we need to update
- $stats[$listID]['added'] = $stats[$listID]['removed'] = 0;
-
- // We need the membership group and any groups mapped to interest groupings with the allow MC updates option set.
- $membership_group_id = FALSE;
- $updatable_grouping_groups = array();
- foreach (CRM_Mailchimp_Utils::getGroupsToSync(array(), $listID) as $groupID=>$details) {
- if (!$details['grouping_id']) {
- $membership_group_id = $groupID;
- }
- elseif ($details['is_mc_update_grouping']) {
- // This group is one that we allow Mailchimp to update CiviCRM with.
- $updatable_grouping_groups[$groupID] = $details;
- }
- }
+ public static function syncPullIgnoreInSync(CRM_Queue_TaskContext $ctx, $listID) {
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Pull syncPullIgnoreInSync $listID= ', $listID);
- // First update the first name and last name of the contacts we
- // already matched. See issue #188.
- CRM_Mailchimp_Utils::updateGuessedContactDetails();
-
- // all Mailchimp table
- $dao = CRM_Core_DAO::executeQuery( "SELECT m.*, c.groupings c_groupings
- FROM tmp_mailchimp_push_m m
- LEFT JOIN tmp_mailchimp_push_c c ON m.email = c.email
- ;");
-
- // Loop the $dao object creating/finding contacts in CiviCRM.
- $groupContactRemoves = $groupContact = array();
- while ($dao->fetch()) {
- $params = array(
- 'FNAME' => $dao->first_name,
- 'LNAME' => $dao->last_name,
- 'EMAIL' => $dao->email,
- );
- if (empty($dao->cid_guess)) {
- // We don't know yet who this is.
- // Update/create contact.
- $contact_id = CRM_Mailchimp_Utils::updateContactDetails($params);
- }
- else {
- $contact_id = $dao->cid_guess;
- }
- if($contact_id) {
-
- // Ensure the contact is in the membership group.
- if (!$dao->c_groupings) {
- // This contact was not found in the CiviCRM table.
- // Therefore they are not in the membership group.
- // (actually they could have an email problem as well, but that's OK).
- // Add them into the membership group.
- $groupContact[$membership_group_id][] = $contact_id;
- $civi_groupings = array();
- $stats[$listID]['added']++;
- }
- else {
- // This contact is in C and MC, but has differences.
- // unpack the group membership from CiviCRM.
- $civi_groupings = unserialize($dao->c_groupings);
- }
- // unpack the group membership reported by MC
- $mc_groupings = unserialize($dao->groupings);
-
- // Now sort out the grouping_groups for those we are supposed to allow updates for
- foreach ($updatable_grouping_groups as $groupID=>$details) {
- // Should this person be in this grouping:group according to MC?
- if (!empty($mc_groupings[ $details['grouping_id'] ][ $details['group_id'] ])) {
- // They should be in this group.
- if (empty($civi_groupings[ $details['grouping_id'] ][ $details['group_id'] ])) {
- // But they're not! Plan to add them in.
- $groupContact[$groupID][] = $contact_id;
- }
- }
- else {
- // They should NOT be in this group.
- if (!empty($civi_groupings[ $details['grouping_id'] ][ $details['group_id'] ])) {
- // But they ARE. Plan to remove them.
- $groupContactRemoves[$groupID][] = $contact_id;
- }
- }
- }
- }
- }
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $stats[$listID]['in_sync'] = $sync->removeInSync('pull');
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Pull syncPullIgnoreInSync in-sync= ', $stats[$listID]['in_sync']);
+ static::updatePullStats($stats);
- // And now, what if a contact is not in the Mailchimp list? We must remove them from the membership group.
- // I changed the query below; I replaced a 'WHERE NOT EXISTS' construct
- // by an outer join, in the hope that it will be faster (#188).
- $dao = CRM_Core_DAO::executeQuery( "SELECT c.contact_id
- FROM tmp_mailchimp_push_c c
- LEFT OUTER JOIN tmp_mailchimp_push_m m ON m.email = c.email
- WHERE m.email IS NULL;");
- // Loop the $dao object creating/finding contacts in CiviCRM.
- while ($dao->fetch()) {
- $groupContactRemoves[$membership_group_id][] =$dao->contact_id;
- $stats[$listID]['removed']++;
- }
- // Log group contacts which are going to be added to CiviCRM
- CRM_Core_Error::debug_var( 'Mailchimp $groupContact= ', $groupContact);
-
- // FIXME: dirty hack setting a variable in session to skip post hook
- require_once 'CRM/Core/Session.php';
- $session = CRM_Core_Session::singleton();
- $session->set('skipPostHook', 'yes');
-
- if ($groupContact) {
- // We have some contacts to add into groups...
- foreach($groupContact as $groupID => $contactIDs ) {
- CRM_Contact_BAO_GroupContact::addContactsToGroup($contactIDs, $groupID, 'Admin', 'Added');
- }
- }
+ return CRM_Queue_Task::TASK_SUCCESS;
+ }
- // Log group contacts which are going to be removed from CiviCRM
- CRM_Core_Error::debug_var( 'Mailchimp $groupContactRemoves= ', $groupContactRemoves);
-
- if ($groupContactRemoves) {
- // We have some contacts to add into groups...
- foreach($groupContactRemoves as $groupID => $contactIDs ) {
- CRM_Contact_BAO_GroupContact::removeContactsFromGroup($contactIDs, $groupID, 'Admin', 'Removed');
- }
- }
-
- // FIXME: unset variable in session
- $session->set('skipPostHook', '');
+ /**
+ * New contacts and profile changes need bringing into CiviCRM.
+ */
+ public static function syncPullFromMailchimp(CRM_Queue_TaskContext $ctx, $listID, $dry_run) {
- static::updatePullStats($stats);
+ // Do the batch update. Might take a while :-O
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $sync->dry_run = $dry_run;
+ // this generates updates and group changes.
+ $stats[$listID] = $sync->updateCiviFromMailchimp();
// Finally, finish up by removing the two temporary tables
- CRM_Core_DAO::executeQuery("DROP TABLE tmp_mailchimp_push_m;");
- CRM_Core_DAO::executeQuery("DROP TABLE tmp_mailchimp_push_c;");
+ // @todo re-enable this: CRM_Mailchimp_Sync::dropTemporaryTables();
+ static::updatePullStats($stats);
return CRM_Queue_Task::TASK_SUCCESS;
}
@@ -307,11 +282,19 @@ static function syncPullUpdates(CRM_Queue_TaskContext $ctx, $listID) {
/**
* Update the pull stats setting.
*/
- static function updatePullStats($updates) {
+ public static function updatePullStats($updates) {
$stats = CRM_Core_BAO_Setting::getItem(CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'pull_stats');
- foreach ($updates as $listId=>$settings) {
+ foreach ($updates as $list_id=>$settings) {
+ if ($list_id == 'dry_run') {
+ continue;
+ }
foreach ($settings as $key=>$val) {
- $stats[$listId][$key] = $val;
+ if (!empty($stats[$list_id][$key])) {
+ $stats[$list_id][$key] += $val;
+ }
+ else {
+ $stats[$list_id][$key] = $val;
+ }
}
}
CRM_Core_BAO_Setting::setItem($stats, CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'pull_stats');
diff --git a/CRM/Mailchimp/Form/Setting.php b/CRM/Mailchimp/Form/Setting.php
index 31cc682..1985e96 100644
--- a/CRM/Mailchimp/Form/Setting.php
+++ b/CRM/Mailchimp/Form/Setting.php
@@ -4,7 +4,7 @@ class CRM_Mailchimp_Form_Setting extends CRM_Core_Form {
const
MC_SETTING_GROUP = 'MailChimp Preferences';
-
+
/**
* Function to pre processing
*
@@ -18,7 +18,7 @@ function preProcess() {
CRM_Core_Session::setStatus("You need to upgrade to version 4.4 or above to work with extension Mailchimp","Version:");
}
}
-
+
public static function formRule($params){
$currentVer = CRM_Core_BAO_Domain::version(TRUE);
$errors = array();
@@ -27,7 +27,7 @@ public static function formRule($params){
}
return empty($errors) ? TRUE : $errors;
}
-
+
/**
* Function to actually build the form
*
@@ -36,29 +36,25 @@ public static function formRule($params){
*/
public function buildQuickForm() {
$this->addFormRule(array('CRM_Mailchimp_Form_Setting', 'formRule'), $this);
-
+
CRM_Core_Resources::singleton()->addStyleFile('uk.co.vedaconsulting.mailchimp', 'css/mailchimp.css');
-
+
$webhook_url = CRM_Utils_System::url('civicrm/mailchimp/webhook', 'reset=1', TRUE, NULL, FALSE, TRUE);
$this->assign( 'webhook_url', 'Webhook URL - '.$webhook_url);
-
+
// Add the API Key Element
$this->addElement('text', 'api_key', ts('API Key'), array(
'size' => 48,
));
-
+
// Add the User Security Key Element
$this->addElement('text', 'security_key', ts('Security Key'), array(
'size' => 24,
));
-
+
// Add Enable or Disable Debugging
$enableOptions = array(1 => ts('Yes'), 0 => ts('No'));
$this->addRadio('enable_debugging', ts('Enable Debugging'), $enableOptions, NULL);
-
- // Remove or Unsubscribe Preference
- $removeOptions = array(1 => ts('Delete MailChimp Subscriber'), 0 => ts('Unsubscribe MailChimp Subscriber'));
- $this->addRadio('list_removal', ts('List Removal'), $removeOptions, NULL);
// Create the Submit Button.
$buttons = array(
@@ -67,19 +63,23 @@ public function buildQuickForm() {
'name' => ts('Save & Test'),
),
);
- $groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), null, $membership_only = TRUE);
- foreach ($groups as $group_id => $details) {
- $list = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
- $webhookoutput = $list->webhooks($details['list_id']);
- if($webhookoutput[0]['sources']['api'] == 1) {
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Setting - API is set in Webhook setting for listID', $details['list_id']);
- $listID = $details['list_id'];
- CRM_Core_Session::setStatus(ts('API is set in Webhook setting for listID %1', array(1 => $listID)), ts('Error'), 'error');
- break;
- }
- }
+
// Add the Buttons.
$this->addButtons($buttons);
+
+ try {
+ // Initially we won't be able to do this as we don't have an API key.
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+
+ // Check for warnings and output them as status messages.
+ $warnings = CRM_Mailchimp_Utils::checkGroupsConfig();
+ foreach ($warnings as $message) {
+ CRM_Core_Session::setStatus($message);
+ }
+ }
+ catch (Exception $e){
+ CRM_Core_Session::setStatus('Could not use the Mailchimp API - ' . $e->getMessage() . ' You will see this message If you have not yet configured your Mailchimp acccount.');
+ }
}
public function setDefaultValues() {
@@ -88,24 +88,18 @@ public function setDefaultValues() {
$apiKey = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP,
'api_key', NULL, FALSE
);
-
+
$securityKey = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP,
'security_key', NULL, FALSE
);
-
+
$enableDebugging = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP,
'enable_debugging', NULL, FALSE
);
-
- $listRemoval = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP,
- 'list_removal', NULL, FALSE
- );
-
$defaults['api_key'] = $apiKey;
$defaults['security_key'] = $securityKey;
$defaults['enable_debugging'] = $enableDebugging;
- $defaults['list_removal'] = $listRemoval;
-
+
return $defaults;
}
@@ -118,48 +112,43 @@ public function setDefaultValues() {
*/
public function postProcess() {
// Store the submitted values in an array.
- $params = $this->controller->exportValues($this->_name);
-
+ $params = $this->controller->exportValues($this->_name);
+
// Save the API Key & Save the Security Key
if (CRM_Utils_Array::value('api_key', $params) || CRM_Utils_Array::value('security_key', $params)) {
CRM_Core_BAO_Setting::setItem($params['api_key'],
self::MC_SETTING_GROUP,
'api_key'
);
-
+
CRM_Core_BAO_Setting::setItem($params['security_key'],
self::MC_SETTING_GROUP,
'security_key'
);
- CRM_Core_BAO_Setting::setItem($params['enable_debugging'], self::MC_SETTING_GROUP, 'enable_debugging'
- );
-
- CRM_Core_BAO_Setting::setItem($params['list_removal'], self::MC_SETTING_GROUP, 'list_removal'
- );
+ CRM_Core_BAO_Setting::setItem($params['enable_debugging'], self::MC_SETTING_GROUP, 'enable_debugging');
try {
- $mcClient = new Mailchimp($params['api_key']);
- $mcHelper = new Mailchimp_Helper($mcClient);
- $details = $mcHelper->accountDetails();
- } catch (Mailchimp_Invalid_ApiKey $e) {
- CRM_Core_Session::setStatus($e->getMessage());
- return FALSE;
- } catch (Mailchimp_HttpError $e) {
+ $mcClient = CRM_Mailchimp_Utils::getMailchimpApi(TRUE);
+ $response = $mcClient->get('/');
+ if (empty($response->data->account_name)) {
+ throw new Exception("Could not retrieve account details, although a response was received. Somthing's not right.");
+ }
+
+ } catch (Exception $e) {
CRM_Core_Session::setStatus($e->getMessage());
return FALSE;
}
-
-
- $message = "Following is the account information received from API callback:
-
-
Company:
{$details['contact']['company']}
-
First Name:
{$details['contact']['fname']}
-
Last Name:
{$details['contact']['lname']}
-
";
- CRM_Core_Session::setStatus($message);
+
+ $message = "Following is the account information received from API callback:
+
" . ts('The following lists will be synchronised') . "
$will
";
+
+ $this->addElement('checkbox', 'mc_dry_run',
+ ts('Dry Run? (if ticked no changes will be made to CiviCRM or Mailchimp.)'));
+
+ // Create the Submit Button.
+ $buttons = array(
+ array(
+ 'type' => 'submit',
+ 'name' => ts('Sync Contacts'),
+ ),
+ );
+ $this->addButtons($buttons);
+ }
+ if ($wont) {
+ $msg .= "
" . ts('The following lists will be NOT synchronised') . "
The following list(s) no longer exist at Mailchimp.
$wont
";
+ }
+ $this->assign('summary', $msg);
}
/**
@@ -63,7 +108,10 @@ public function buildQuickForm() {
* @return None
*/
public function postProcess() {
- $runner = self::getRunner();
+ $vals = $this->_submitValues;
+ $runner = self::getRunner(FALSE, !empty($vals['mc_dry_run']));
+ // Clear out log table.
+ CRM_Mailchimp_Sync::dropLogTable();
if ($runner) {
// Run Everything in the Queue via the Web.
$runner->runAllViaWeb();
@@ -72,7 +120,10 @@ public function postProcess() {
}
}
- static function getRunner($skipEndUrl = FALSE) {
+ /**
+ * Set up the queue.
+ */
+ public static function getRunner($skipEndUrl = FALSE, $dry_run = FALSE) {
// Setup the Queue
$queue = CRM_Queue_Service::singleton()->create(array(
'name' => self::QUEUE_NAME,
@@ -81,46 +132,50 @@ static function getRunner($skipEndUrl = FALSE) {
));
// reset push stats
- CRM_Core_BAO_Setting::setItem(Array(), CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'push_stats');
- $stats = array();
-
+ $stats = ['dry_run' => $dry_run];
+ CRM_Core_BAO_Setting::setItem($stats, CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'push_stats');
+
// We need to process one list at a time.
$groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), null, $membership_only = TRUE);
CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Sync getRunner $groups= ', $groups);
-
- if (!$groups) {
- // Nothing to do.
- return FALSE;
- }
// Each list is a task.
- $listCount = 1;
+ $listCount = 0;
foreach ($groups as $group_id => $details) {
- $stats[$details['list_id']] = array(
- 'mc_count' => 0,
- 'c_count' => 0,
- 'in_sync' => 0,
- 'added' => 0,
- 'removed' => 0,
- 'group_id' => 0,
- 'error_count' => 0
- );
+ if (empty($details['list_name'])) {
+ // This list has been deleted at Mailchimp, or for some other reason we
+ // could not access its name. Best not to sync it.
+ continue;
+ }
+
+ $stats[$details['list_id']] = [
+ 'c_count' => 0,
+ 'mc_count' => 0,
+ 'in_sync' => 0,
+ 'updates' => 0,
+ 'additions' => 0,
+ 'unsubscribes' => 0,
+ ];
$identifier = "List " . $listCount++ . " " . $details['civigroup_title'];
$task = new CRM_Queue_Task(
- array ('CRM_Mailchimp_Form_Sync', 'syncPushList'),
- array($details['list_id'], $identifier),
- "Preparing queue for $identifier"
+ ['CRM_Mailchimp_Form_Sync', 'syncPushList'],
+ [$details['list_id'], $identifier, $dry_run],
+ "$identifier: collecting data from CiviCRM."
);
// Add the Task to the Queue
$queue->createItem($task);
}
+ if (count($stats)==1) {
+ // Nothing to do. (only key is 'dry_run')
+ return FALSE;
+ }
// Setup the Runner
$runnerParams = array(
- 'title' => ts('Mailchimp Sync: CiviCRM to Mailchimp'),
+ 'title' => ($dry_run ? ts('Dry Run: ') : '') . ts('Mailchimp Push Sync: update Mailchimp from CiviCRM'),
'queue' => $queue,
'errorMode'=> CRM_Queue_Runner::ERROR_ABORT,
'onEndUrl' => CRM_Utils_System::url(self::END_URL, self::END_PARAMS, TRUE, NULL, FALSE),
@@ -133,7 +188,6 @@ static function getRunner($skipEndUrl = FALSE) {
$runner = new CRM_Queue_Runner($runnerParams);
static::updatePushStats($stats);
-
CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Form_Sync getRunner $identifier= ', $identifier);
return $runner;
@@ -142,51 +196,46 @@ static function getRunner($skipEndUrl = FALSE) {
/**
* Set up (sub)queue for syncing a Mailchimp List.
*/
- static function syncPushList(CRM_Queue_TaskContext $ctx, $listID, $identifier) {
+ public static function syncPushList(CRM_Queue_TaskContext $ctx, $listID, $identifier, $dry_run) {
CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushList $listID= ', $listID);
// Split the work into parts:
- // @todo 'force' method not implemented here.
- // Add the Mailchimp collect data task to the queue
+ // Add the CiviCRM collect data task to the queue
+ // It's important that this comes before the Mailchimp one, as some
+ // fast contact matching SQL can run if it's done this way.
$ctx->queue->createItem( new CRM_Queue_Task(
- array('CRM_Mailchimp_Form_Sync', 'syncPushCollectMailchimp'),
+ array('CRM_Mailchimp_Form_Sync', 'syncPushCollectCiviCRM'),
array($listID),
- "$identifier: Fetched data from Mailchimp"
+ "$identifier: Fetched data from CiviCRM, fetching from Mailchimp..."
));
- // Add the CiviCRM collect data task to the queue
+ // Add the Mailchimp collect data task to the queue
$ctx->queue->createItem( new CRM_Queue_Task(
- array('CRM_Mailchimp_Form_Sync', 'syncPushCollectCiviCRM'),
+ array('CRM_Mailchimp_Form_Sync', 'syncPushCollectMailchimp'),
array($listID),
- "$identifier: Fetched data from CiviCRM"
+ "$identifier: Fetched data from Mailchimp. Matching..."
));
- // Add the removals task to the queue
+ // Add the slow match process for difficult contacts.
$ctx->queue->createItem( new CRM_Queue_Task(
- array('CRM_Mailchimp_Form_Sync', 'syncPushRemove'),
+ array('CRM_Mailchimp_Form_Sync', 'syncPushDifficultMatches'),
array($listID),
- "$identifier: Removed those who should no longer be subscribed"
+ "$identifier: Matched up contacts. Comparing..."
));
- // Add the batchUpdate to the queue
+ // Add the Mailchimp collect data task to the queue
$ctx->queue->createItem( new CRM_Queue_Task(
- array('CRM_Mailchimp_Form_Sync', 'syncPushAdd'),
+ array('CRM_Mailchimp_Form_Sync', 'syncPushIgnoreInSync'),
array($listID),
- "$identifier: Added new subscribers and updating existing data changes"
+ "$identifier: Ignored any in-sync already. Updating Mailchimp..."
));
- return CRM_Queue_Task::TASK_SUCCESS;
- }
-
- /**
- * Collect Mailchimp data into temporary working table.
- */
- static function syncPushCollectMailchimp(CRM_Queue_TaskContext $ctx, $listID) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushCollectMailchimp $listID= ', $listID);
-
- $stats[$listID]['mc_count'] = static::syncCollectMailchimp($listID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushCollectMailchimp $stats[$listID][mc_count]', $stats[$listID]['mc_count']);
- static::updatePushStats($stats);
+ // Add the Mailchimp changes
+ $ctx->queue->createItem( new CRM_Queue_Task(
+ array('CRM_Mailchimp_Form_Sync', 'syncPushToMailchimp'),
+ array($listID, $dry_run),
+ "$identifier: Completed additions/updates/unsubscribes."
+ ));
return CRM_Queue_Task::TASK_SUCCESS;
}
@@ -194,485 +243,95 @@ static function syncPushCollectMailchimp(CRM_Queue_TaskContext $ctx, $listID) {
/**
* Collect CiviCRM data into temporary working table.
*/
- static function syncPushCollectCiviCRM(CRM_Queue_TaskContext $ctx, $listID) {
+ public static function syncPushCollectCiviCRM(CRM_Queue_TaskContext $ctx, $listID) {
CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushCollectCiviCRM $listID= ', $listID);
- $stats[$listID]['c_count'] = static::syncCollectCiviCRM($listID);
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $stats[$listID]['c_count'] = $sync->collectCiviCrm('push');
+
CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushCollectCiviCRM $stats[$listID][c_count]= ', $stats[$listID]['c_count']);
-
static::updatePushStats($stats);
return CRM_Queue_Task::TASK_SUCCESS;
}
/**
- * Unsubscribe contacts that are subscribed at Mailchimp but not in our list.
+ * Collect Mailchimp data into temporary working table.
*/
- static function syncPushRemove(CRM_Queue_TaskContext $ctx, $listID) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushRemove $listID= ', $listID);
- // Delete records have the same hash - these do not need an update.
- static::updatePushStats(array($listID=>array('in_sync'=> static::syncIdentical())));
-
- // Now identify those that need removing from Mailchimp.
- // @todo implement the delete option, here just the unsubscribe is implemented.
- $dao = CRM_Core_DAO::executeQuery(
- "SELECT m.email, m.euid, m.leid
- FROM tmp_mailchimp_push_m m
- WHERE NOT EXISTS (
- SELECT email FROM tmp_mailchimp_push_c c WHERE c.email = m.email
- );");
-
- // Loop the $dao object to make a list of emails to unsubscribe|delete from MC
- // http://apidocs.mailchimp.com/api/2.0/lists/batch-unsubscribe.php
- $batch = array();
- $stats[$listID]['removed'] = 0;
- while ($dao->fetch()) {
- $batch[] = array('email' => $dao->email, 'euid' => $dao->euid, 'leid' => $dao->leid);
- $stats[$listID]['removed']++;
- }
- if (!$batch) {
- // Nothing to do
- return CRM_Queue_Task::TASK_SUCCESS;
- }
-
- // Log the batch unsubscribe details
- CRM_Core_Error::debug_var('Mailchimp batchUnsubscribe syncPushRemove $batch= ', $batch);
- $delete = CRM_Core_BAO_Setting::getItem(CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'list_removal', NULL, FALSE);
- // Send Mailchimp Lists API Call: http://apidocs.mailchimp.com/api/2.0/lists/batch-unsubscribe.php
- $list = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
- $result = $list->batchUnsubscribe( $listID, $batch, $delete, $send_bye=FALSE, $send_notify=FALSE);
-
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Sync syncPushRemove $batchUnsubscriberesult= ', $result);
- // @todo check errors? $result['errors'] $result['success_count']
+ public static function syncPushCollectMailchimp(CRM_Queue_TaskContext $ctx, $listID) {
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushCollectMailchimp $listID= ', $listID);
- // Finally we can delete the emails that we just processed from the mailchimp temp table.
- CRM_Core_DAO::executeQuery(
- "DELETE FROM tmp_mailchimp_push_m
- WHERE NOT EXISTS (
- SELECT email FROM tmp_mailchimp_push_c c WHERE c.email = tmp_mailchimp_push_m.email
- );");
+ // Nb. collectCiviCrm must have run before we call this.
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $stats[$listID]['mc_count'] = $sync->collectMailchimp('push', $civi_collect_has_already_run=TRUE);
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushCollectMailchimp $stats[$listID][mc_count]', $stats[$listID]['mc_count']);
static::updatePushStats($stats);
+
return CRM_Queue_Task::TASK_SUCCESS;
}
/**
- * Batch update Mailchimp with new contacts that need to be subscribed, or have changed data.
- *
- * This also does the clean-up tasks of removing the temporary tables.
+ * Do the difficult matches.
*/
- static function syncPushAdd(CRM_Queue_TaskContext $ctx, $listID) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushAdd $listID= ', $listID);
-
- // @todo take the remaining details from tmp_mailchimp_push_c
- // and construct a batchUpdate (do they need to be batched into 1000s? I can't recal).
-
- $dao = CRM_Core_DAO::executeQuery( "SELECT * FROM tmp_mailchimp_push_c;");
- $stats = array();
- // Loop the $dao object to make a list of emails to subscribe/update
- $batch = array();
- while ($dao->fetch()) {
- $merge = array(
- 'FNAME' => $dao->first_name,
- 'LNAME' => $dao->last_name,
- );
- // set the groupings.
- $groupings = unserialize($dao->groupings);
- // this is a array(groupingid=>array(groupid=>bool membership))
- $merge_groups = array();
- foreach ($groupings as $grouping_id => $groups) {
- // CRM_Mailchimp_Utils::checkDebug('get groups $groups= ', $groups);
- $merge_groups[$grouping_id] = array('id' => $grouping_id, 'groups' => array());
-
-
- foreach ($groups as $group_id => $is_member) {
- if ($is_member) {
- $merge_groups[$grouping_id]['groups'][] = CRM_Mailchimp_Utils::getMCGroupName($listID, $grouping_id, $group_id);
-
- }
- }
- }
- // remove the significant array indexes, in case Mailchimp cares.
- $merge['groupings'] = array_values($merge_groups);
-
- $batch[$dao->email] = array('email' => array('email' => $dao->email), 'email_type' => 'html', 'merge_vars' => $merge);
- $stats[$listID]['added']++;
- }
- if (!$batch) {
- // Nothing to do
- return CRM_Queue_Task::TASK_SUCCESS;
- }
-
- // Log the batch subscribe details
- CRM_Core_Error::debug_var('Mailchimp syncPushAdd batchSubscribe $batch= ', $batch);
- // Send Mailchimp Lists API Call.
- // http://apidocs.mailchimp.com/api/2.0/lists/batch-subscribe.php
- $list = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
- $batchs = array_chunk($batch, 50, true);
- $batchResult = array();
- $result = array('errors' => array());
- foreach($batchs as $id => $batch) {
- $batchResult[$id] = $list->batchSubscribe( $listID, $batch, $double_optin=FALSE, $update=TRUE, $replace_interests=TRUE);
- $result['error_count'] += $batchResult[$id]['error_count'];
- // @TODO: updating stats for errors, create sql error "Data too long for column 'value'" (for long array)
- if ($batchResult[$id]['errors']) {
- foreach ($batchResult[$id]['errors'] as $errorDetails){
- // Resubscribe if email address is reported as unsubscribed
- // they want to resubscribe.
- if ($errorDetails['code'] == 212) {
- $unsubscribedEmail = $errorDetails['email']['email'];
- $list->subscribe( $listID, $batch[$unsubscribedEmail]['email'], $batch[$unsubscribedEmail]['merge_vars'], $batch[$unsubscribedEmail]['email_type'], FALSE, TRUE, FALSE, FALSE);
- $result['error_count'] -= 1;
- }
- else {
- $result['errors'][] = $errorDetails;
- }
- }
- }
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Sync syncPushAdd $batchsubscriberesultinloop= ', $batchResult[$id]);
- }
- // debug: file_put_contents(DRUPAL_ROOT . '/logs/' . date('Y-m-d-His') . '-MC-push.log', print_r($result,1));
-
- $get_GroupId = CRM_Mailchimp_Utils::getGroupsToSync(array(), $listID);
-
- CRM_Mailchimp_Utils::checkDebug('$get_GroupId= ', $get_GroupId);
- // @todo check result (keys: error_count, add_count, update_count)
-
- $stats[$listID]['group_id'] = array_keys($get_GroupId);
- $stats[$listID]['error_count'] = $result['error_count'];
- $stats[$listID]['error_details'] = $result['errors'];
-
- static::updatePushStats($stats);
-
- // Finally, finish up by removing the two temporary tables
- CRM_Core_DAO::executeQuery("DROP TABLE tmp_mailchimp_push_m;");
- CRM_Core_DAO::executeQuery("DROP TABLE tmp_mailchimp_push_c;");
-
+ public static function syncPushDifficultMatches(CRM_Queue_TaskContext $ctx, $listID) {
+ // Nb. collectCiviCrm must have run before we call this.
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $c = $sync->matchMailchimpMembersToContacts();
+ CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Sync syncPushDifficultMatches count=', $c);
return CRM_Queue_Task::TASK_SUCCESS;
-
}
-
/**
* Collect Mailchimp data into temporary working table.
*/
- static function syncCollectMailchimp($listID) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncCollectMailchimp $listID= ', $listID);
- // Create a temporary table.
- // Nb. these are temporary tables but we don't use TEMPORARY table because they are
- // needed over multiple sessions because of queue.
-
- CRM_Core_DAO::executeQuery( "DROP TABLE IF EXISTS tmp_mailchimp_push_m;");
- CRM_Core_DAO::executeQuery(
- "CREATE TABLE tmp_mailchimp_push_m (
- email VARCHAR(200),
- first_name VARCHAR(100),
- last_name VARCHAR(100),
- euid VARCHAR(10),
- leid VARCHAR(10),
- hash CHAR(32),
- groupings VARCHAR(4096),
- cid_guess INT(10),
- PRIMARY KEY (email, hash))
- ENGINE=InnoDB DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ;");
- // I'll use the cid_guess column to store the cid when it is
- // immediately clear. This will speed up pulling updates (see #118).
- // Create an index so that this cid_guess can be used for fast
- // searching.
- $dao = CRM_Core_DAO::executeQuery(
- "CREATE INDEX index_cid_guess ON tmp_mailchimp_push_m(cid_guess);");
-
- // Cheekily access the database directly to obtain a prepared statement.
- $db = $dao->getDatabaseConnection();
- $insert = $db->prepare('INSERT INTO tmp_mailchimp_push_m(email, first_name, last_name, euid, leid, hash, groupings) VALUES(?, ?, ?, ?, ?, ?, ?)');
-
- // We need to know what grouping data we care about. The rest we completely ignore.
- // We only care about CiviCRM groups that are mapped to this MC List:
- $mapped_groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), $listID);
-
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Sync syncCollectMailchimp $mapped_groups', $mapped_groups);
-
- // Prepare to access Mailchimp export API
- // See http://apidocs.mailchimp.com/export/1.0/list.func.php
- // Example result (spacing added)
- // ["Email Address" , "First Name" , "Last Name" , "CiviCRM" , "MEMBER_RATING" , "OPTIN_TIME" , "OPTIN_IP" , "CONFIRM_TIME" , "CONFIRM_IP" , "LATITUDE" , "LONGITUDE" , "GMTOFF" , "DSTOFF" , "TIMEZONE" , "CC" , "REGION" , "LAST_CHANGED" , "LEID" , "EUID" , "NOTES"]
- // ["f2@example.com" , "Fred" , "Flintstone" , "general, special" , 2 , "" , null , "2014-09-11 19:57:53" , "212.x.x.x" , null , null , null , null , null , null , null , "2014-09-11 20:02:26" , "180020969" , "884d72639d" , null]
- $apiKey = CRM_Core_BAO_Setting::getItem(CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'api_key');
- // The datacentre is usually appended to the apiKey after a hyphen.
- $dataCentre = 'us1'; // default.
- if (preg_match('/-(.+)$/', $apiKey, $matches)) {
- $dataCentre = $matches[1];
- }
- $url = "https://$dataCentre.api.mailchimp.com/export/1.0/list?apikey=$apiKey&id=$listID";
- $chunk_size = 4096; //in bytes
- $handle = @fopen($url,'r');
- if (!$handle) {
- // @todo not sure a vanilla exception is best?
- throw new \Exception("Failed to access Mailchimp export API");
- }
-
- // Load headers from the export.
- // This is an array of strings. We need to find the array indexes for the columns we're interested in.
- $buffer = fgets($handle, $chunk_size);
- if (trim($buffer)=='') {
- // @todo not sure a vanilla exception is best?
- throw new \Exception("Failed to read from Mailchimp export API");
- }
- $header = json_decode($buffer);
- // We need to know the indexes of our groupings
- foreach ($mapped_groups as $civi_group_id => &$details) {
- if (!$details['grouping_name']) {
- // this will be the membership group.
- continue;
- }
- $details['idx'] = array_search($details['grouping_name'], $header);
- }
- unset($details);
- // ... and LEID and EUID fields.
- $leid_idx = array_search('LEID', $header);
- $euid_idx = array_search('EUID', $header);
-
- //
- // Main loop of all the records.
- //
- while (!feof($handle)) {
- $buffer = trim(fgets($handle, $chunk_size));
- if (!$buffer) {
- continue;
- }
- // fetch array of columns.
- $subscriber = json_decode($buffer);
-
- // Find out which of our mapped groups apply to this subscriber.
- $info = array();
- foreach ($mapped_groups as $civi_group_id => $details) {
- if (!$details['grouping_name']) {
- // this will be the membership group.
- continue;
- }
-
- // Fetch the data for this grouping.
- $mc_groups = explode(', ', $subscriber[ $details['idx'] ]);
- // Is this mc group included?
- $info[ $details['grouping_id'] ][ $details['group_id'] ] = in_array($details['group_name'], $mc_groups);
- }
- // Serialize the grouping array for SQL storage - this is the fastest way.
- $info = serialize($info);
-
- // we're ready to store this but we need a hash that contains all the info
- // for comparison with the hash created from the CiviCRM data (elsewhere).
- // email, first name, last name, groupings
- $hash = md5($subscriber[0] . $subscriber[1] . $subscriber[2] . $info);
- // run insert prepared statement
- $db->execute($insert, array($subscriber[0], $subscriber[1], $subscriber[2], $subscriber[$euid_idx], $subscriber[$leid_idx], $hash, $info));
- }
+ public static function syncPushIgnoreInSync(CRM_Queue_TaskContext $ctx, $listID) {
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushIgnoreInSync $listID= ', $listID);
- // Tidy up.
- fclose($handle);
- $db->freePrepared($insert);
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $stats[$listID]['in_sync'] = $sync->removeInSync('push');
- // Guess the contact ID's, to speed up syncPullUpdates (See issue #188).
- CRM_Mailchimp_Utils::guessCidsMailchimpContacts();
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushIgnoreInSync $stats[$listID][in_sync]', $stats[$listID]['in_sync']);
+ static::updatePushStats($stats);
- $dao = CRM_Core_DAO::executeQuery("SELECT COUNT(*) c FROM tmp_mailchimp_push_m");
- $dao->fetch();
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Form_Sync syncCollectMailchimp $listID= ', $listID);
- return $dao->c;
+ return CRM_Queue_Task::TASK_SUCCESS;
}
/**
- * Collect CiviCRM data into temporary working table.
+ * Batch update Mailchimp with new contacts that need to be subscribed, or
+ * have changed data including unsubscribes.
*/
- static function syncCollectCiviCRM($listID) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncCollectCiviCRM $listID= ', $listID);
- // Nb. these are temporary tables but we don't use TEMPORARY table because they are
- // needed over multiple sessions because of queue.
- CRM_Core_DAO::executeQuery( "DROP TABLE IF EXISTS tmp_mailchimp_push_c;");
- $dao = CRM_Core_DAO::executeQuery("CREATE TABLE tmp_mailchimp_push_c (
- contact_id INT(10) UNSIGNED NOT NULL,
- email_id INT(10) UNSIGNED NOT NULL,
- email VARCHAR(200),
- first_name VARCHAR(100),
- last_name VARCHAR(100),
- hash CHAR(32),
- groupings VARCHAR(4096),
- PRIMARY KEY (email_id, email, hash))
- ENGINE=InnoDB DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ;");
- // Cheekily access the database directly to obtain a prepared statement.
- $db = $dao->getDatabaseConnection();
- $insert = $db->prepare('INSERT INTO tmp_mailchimp_push_c VALUES(?, ?, ?, ?, ?, ?, ?)');
-
- //create table for mailchim civicrm syn errors
- $dao = CRM_Core_DAO::executeQuery("CREATE TABLE IF NOT EXISTS mailchimp_civicrm_syn_errors (
- id int(11) NOT NULL AUTO_INCREMENT,
- email VARCHAR(200),
- error VARCHAR(200),
- error_count int(10),
- group_id int(20),
- list_id VARCHAR(20),
- PRIMARY KEY (id)
- );");
-
- // We need to know what groupings we have maps to.
- // We only care about CiviCRM groups that are mapped to this MC List:
- $mapped_groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), $listID);
-
- // First, get all subscribers from the membership group for this list.
- // ... Find CiviCRM group id for the membership group.
- // ... And while we're at it, build an SQL-safe array of groupIds for groups mapped to groupings.
- // (we use that later)
- $membership_group_id = FALSE;
- // There used to be a distinction between the handling of 'normal' groups
- // and smart groups. But now the API will take care of this.
- $grouping_group_ids = array();
- $default_info = array();
-
- // The CiviCRM Contact API returns group titles instead of group ID's.
- // Nobody knows why. So let's build this array to convert titles to ID's.
- $title2gid = array();
-
- foreach ($mapped_groups as $group_id => $details) {
- $title2gid[$details['civigroup_title']] = $group_id;
- CRM_Contact_BAO_GroupContactCache::loadAll($group_id);
- if (!$details['grouping_id']) {
- $membership_group_id = $group_id;
- }
- else {
- $grouping_group_ids[] = (int)$group_id;
- $default_info[ $details['grouping_id'] ][ $details['group_id'] ] = FALSE;
- }
- }
- if (!$membership_group_id) {
- throw new Exception("No CiviCRM group is mapped to determine membership of Mailchimp list $listID");
- }
- // Use a nice API call to get the information for tmp_mailchimp_push_c.
- // The API will take care of smart groups.
- $result = civicrm_api3('Contact', 'get', array(
- 'is_deleted' => 0,
- // The email filter below does not work (CRM-18147)
- // 'email' => array('IS NOT NULL' => 1),
- // Now I think that on_hold is NULL when there is no e-mail, so if
- // we are lucky, the filter below implies that an e-mail address
- // exists ;-)
- 'on_hold' => 0,
- 'is_opt_out' => 0,
- 'do_not_email' => 0,
- 'group' => $membership_group_id,
- 'return' => array('first_name', 'last_name', 'email_id', 'email', 'group'),
- 'options' => array('limit' => 0),
- ));
-
- foreach ($result['values'] as $contact) {
- // Find out the ID's of the groups the $contact belongs to, and
- // save in $info.
- $info = $default_info;
-
- $contact_group_titles = explode(',', $contact['groups'] );
- foreach ($contact_group_titles as $title) {
- $group_id = $title2gid[$title];
- if (in_array($group_id, $grouping_group_ids)) {
- $details = $mapped_groups[$group_id];
- $info[$details['grouping_id']][$details['group_id']] = TRUE;
- }
- }
-
- // OK we should now have all the info we need.
- // Serialize the grouping array for SQL storage - this is the fastest way.
- $info = serialize($info);
+ public static function syncPushToMailchimp(CRM_Queue_TaskContext $ctx, $listID, $dry_run) {
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncPushAdd $listID= ', $listID);
- // we're ready to store this but we need a hash that contains all the info
- // for comparison with the hash created from the CiviCRM data (elsewhere).
- // email, first name, last name, groupings
- $hash = md5($contact['email'] . $contact['first_name'] . $contact['last_name'] . $info);
- // run insert prepared statement
- $db->execute($insert, array($contact['id'], $contact['email_id'], $contact['email'], $contact['first_name'], $contact['last_name'], $hash, $info));
- }
+ // Do the batch update. Might take a while :-O
+ $sync = new CRM_Mailchimp_Sync($listID);
+ $sync->dry_run = $dry_run;
+ // this generates updates and unsubscribes
+ $stats[$listID] = $sync->updateMailchimpFromCivi();
+ // Finally, finish up by removing the two temporary tables
+ //CRM_Mailchimp_Sync::dropTemporaryTables();
+ static::updatePushStats($stats);
- // Tidy up.
- $db->freePrepared($insert);
- // count
- $dao = CRM_Core_DAO::executeQuery("SELECT COUNT(*) c FROM tmp_mailchimp_push_c");
- $dao->fetch();
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Form_Sync syncCollectCiviCRM $listID= ', $listID);
- return $dao->c;
+ return CRM_Queue_Task::TASK_SUCCESS;
}
/**
* Update the push stats setting.
*/
- static function updatePushStats($updates) {
+ public static function updatePushStats($updates) {
CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync updatePushStats $updates= ', $updates);
+
$stats = CRM_Core_BAO_Setting::getItem(CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'push_stats');
-
foreach ($updates as $listId=>$settings) {
+ if ($listId == 'dry_run') {
+ continue;
+ }
foreach ($settings as $key=>$val) {
- // avoid error details to store in civicrm_settings table
- // create sql error "Data too long for column 'value'" (for long array)
- if ($key == 'error_details') {
- continue;
- }
$stats[$listId][$key] = $val;
}
}
CRM_Core_BAO_Setting::setItem($stats, CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'push_stats');
-
- //$email = $error_count = $error = $list_id = array();
-
- foreach ($updates as $list => $listdetails) {
- if (isset($updates[$list]['error_count']) && !empty($updates[$list]['error_count'])) {
- $error_count = $updates[$list]['error_count'];
- }
- $list_id = $list;
-
- if (isset($updates[$list]['group_id']) && !empty($updates[$list]['group_id'])) {
- foreach ($updates[$list]['group_id'] as $keys => $values) {
- $group_id = $values;
- $deleteQuery = "DELETE FROM `mailchimp_civicrm_syn_errors` WHERE group_id =$group_id";
- CRM_Core_DAO::executeQuery($deleteQuery);
- }
- }
-
- if (isset($updates[$list]['error_details']) && !empty($updates[$list]['error_details'])) {
- foreach ($updates[$list]['error_details'] as $key => $value) {
- $error = $value['error'];
- $email = $value['email']['email'];
-
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Sync updatePushStats $group_id=', $group_id);
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Sync updatePushStats $error_count=', $error_count);
-
- $insertQuery = "INSERT INTO `mailchimp_civicrm_syn_errors` (`email`, `error`, `error_count`, `list_id`, `group_id`) VALUES (%1,%2, %3, %4, %5)";
- $queryParams = array(
- 1 => array($email, 'String'),
- 2 => array($error, 'String'),
- 3 => array($error_count, 'Integer'),
- 4 => array($list_id, 'String'),
- 5 => array($group_id, 'Integer')
- );
- CRM_Core_DAO::executeQuery($insertQuery, $queryParams);
- }
- }
- }
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Form_Sync updatePushStats $updates= ', $updates);
- }
-
- /**
- * Removes from the temporary tables those records that do not need processing.
- */
- static function syncIdentical() {
- //CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncIdentical $count= ', $count);
- // Delete records have the same hash - these do not need an update.
- // count
- $dao = CRM_Core_DAO::executeQuery("SELECT COUNT(c.email) co FROM tmp_mailchimp_push_m m
- INNER JOIN tmp_mailchimp_push_c c ON m.email = c.email AND m.hash = c.hash;");
- $dao->fetch();
- $count = $dao->co;
- CRM_Core_DAO::executeQuery(
- "DELETE m, c
- FROM tmp_mailchimp_push_m m
- INNER JOIN tmp_mailchimp_push_c c ON m.email = c.email AND m.hash = c.hash;");
-
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Form_Sync syncIdentical $count= ', $count);
- return $count;
}
}
diff --git a/CRM/Mailchimp/NetworkErrorException.php b/CRM/Mailchimp/NetworkErrorException.php
new file mode 100644
index 0000000..b9fba81
--- /dev/null
+++ b/CRM/Mailchimp/NetworkErrorException.php
@@ -0,0 +1,7 @@
+userPermissionClass->isModulePermissionSupported() && !CRM_Mailchimp_Permission::check('allow webhook posts')) {
- CRM_Core_Error::fatal();
- }
-
- // Check the key
- // @todo is this a DOS attack vector? seems a lot of work for saying 403, go away, to a robot!
- if(!isset($_GET['key']) || $_GET['key'] != $my_key ) {
- CRM_Core_Error::fatal();
- }
-
- if (!empty($_POST['data']['list_id']) && !empty($_POST['type'])) {
- $requestType = $_POST['type'];
- $requestData = $_POST['data'];
- // Return if API is set in webhook setting for lists
- $list = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
- $webhookoutput = $list->webhooks($requestData['list_id']);
- if($webhookoutput[0]['sources']['api'] == 1) {
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Page_WebHook run API is set in Webhook setting for listID', $requestData['list_id'] );
- return;
- }
+ /**
+ * CiviCRM contact id.
+ */
+ public $contact_id;
+ /**
+ * Process a webhook request from Mailchimp.
+ *
+ * The only documentation for this *sigh* is (May 2016) at
+ * https://apidocs.mailchimp.com/webhooks/
+ */
+ public function run() {
- switch ($requestType) {
- case 'subscribe':
- case 'unsubscribe':
- case 'profile':
- // Create/Update contact details in CiviCRM
- $delay = ( $requestType == 'profile' );
- $contactID = CRM_Mailchimp_Utils::updateContactDetails($requestData['merges'], $delay);
- $contactArray = array($contactID);
-
- // Subscribe/Unsubscribe to related CiviCRM groups
- self::manageCiviCRMGroupSubcription($contactID, $requestData, $requestType);
-
- CRM_Mailchimp_Utils::checkDebug('Start - CRM_Mailchimp_Page_WebHook run $_POST= ', $_POST);
- CRM_Mailchimp_Utils::checkDebug('Start - CRM_Mailchimp_Page_WebHook run $contactID= ', $contactID);
- CRM_Mailchimp_Utils::checkDebug('Start - CRM_Mailchimp_Page_WebHook run $requestData= ', $requestData);
- CRM_Mailchimp_Utils::checkDebug('Start - CRM_Mailchimp_Page_WebHook run $requestType= ', $requestType);
- break;
-
- case 'upemail':
- // Mailchimp Email Update event
- // Try to find the email address
- $email = new CRM_Core_BAO_Email();
- $email->get('email', $requestData['old_email']);
-
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Page_WebHook run- case upemail $requestData[old_email]= ', $requestData['old_email']);
-
- // If the Email was found.
- if (!empty($email->contact_id)) {
- $email->email = $requestData['new_email'];
- $email->save();
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Page_WebHook run- case upemail inside condition $requestData[new_email]= ', $requestData['new_email']);
- }
- break;
- case 'cleaned':
- // Try to find the email address
- $email = new CRM_Core_BAO_Email();
- $email->get('email', $requestData['email']);
-
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Page_WebHook run - case cleaned $requestData[new_email]= ', $requestData['email']);
- // If the Email was found.
- if (!empty($email->contact_id)) {
- $email->on_hold = 1;
- $email->holdEmail($email);
- $email->save();
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Page_WebHook run - case cleaned inside condition $email= ', $email);
- CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Page_WebHook run - case cleaned inside condition $requestData[new_email]= ', $requestData['email']);
- }
- break;
- default:
- // unhandled webhook
- CRM_Mailchimp_Utils::checkDebug('End- CRM_Mailchimp_Page_WebHook run $contactID= ', $contactID);
- CRM_Mailchimp_Utils::checkDebug('End- CRM_Mailchimp_Page_WebHook run $requestData= ', $requestData);
- CRM_Mailchimp_Utils::checkDebug('End- CRM_Mailchimp_Page_WebHook run $requestType= ', $requestType);
- CRM_Mailchimp_Utils::checkDebug('End - CRM_Mailchimp_Page_WebHook run $email= ', $email);
- }
+ CRM_Mailchimp_Utils::checkDebug("Webhook POST: " . serialize($_POST));
+ // Empty response object, default response code.
+ try {
+ $expected_key = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP, 'security_key', NULL, FALSE);
+ $given_key = isset($_GET['key']) ? $_GET['key'] : null;
+ list($response_code, $response_object) = $this->processRequest($expected_key, $given_key, $_POST);
+ CRM_Mailchimp_Utils::checkDebug("Webhook response code $response_code (200 = ok)");
+ }
+ catch (RuntimeException $e) {
+ $response_code = $e->getCode();
+ $response_object = NULL;
+ CRM_Mailchimp_Utils::checkDebug("Webhook RuntimeException code $response_code (200 means OK): " . $e->getMessage());
+ }
+ catch (Exception $e) {
+ // Broad catch.
+ $response_code = 500;
+ $response_object = NULL;
+ CRM_Mailchimp_Utils::checkDebug("Webhook " . get_class($e) . ": " . $e->getMessage());
}
- // Return the JSON output
- header('Content-type: application/json');
- $data = NULL;// We should ideally throw some status
- print json_encode($data);
+ // Serve HTTP response.
+ if ($response_code != 200) {
+ // Some fault.
+ header("HTTP/1.1 $response_code");
+ }
+ else {
+ // Return the JSON output
+ header('Content-type: application/json');
+ print json_encode($response_object);
+ }
CRM_Utils_System::civiExit();
}
- /*
- * Add/Remove contact from CiviCRM Groups mapped with Mailchimp List & Groups
+ /**
+ * Validate and process the request.
+ *
+ * This is separated from the run() method for testing purposes.
+ *
+ * This method serves as a router to other methods named after the type of
+ * webhook we're called with.
+ *
+ * Methods may return data for mailchimp, or may throw RuntimeException
+ * objects, the error code of which will be used for the response.
+ * So you can throw a `RuntimeException("Invalid webhook configuration", 500);`
+ * to tell mailchimp the webhook failed, but you can equally throw a
+ * `RuntimeException("soft fail", 200)` which will not tell Mailchimp there
+ * was any problem. Mailchimp retries if there was a problem.
+ *
+ * If an exception is thrown, it is logged. @todo where?
+ *
+ * @return array with two values: $response_code, $response_object.
*/
- static function manageCiviCRMGroupSubcription($contactID = array(), $requestData , $action) {
- CRM_Mailchimp_Utils::checkDebug('Start- CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $contactID= ', $contactID);
- CRM_Mailchimp_Utils::checkDebug('Start- CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $requestData= ', $requestData);
- CRM_Mailchimp_Utils::checkDebug('Start- CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $requestType= ', $action);
-
- if (empty($contactID) || empty($requestData['list_id']) || empty($action)) {
- return NULL;
- }
- $listID = $requestData['list_id'];
- $groupContactRemoves = $groupContactAdditions = array();
-
- // Deal with subscribe/unsubscribe.
- // We need the CiviCRM membership group for this list.
- $groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), $listID, $membership_only=TRUE);
- $allGroups = CRM_Mailchimp_Utils::getGroupsToSync(array(), $listID, $membership_only = FALSE);
- if (!$groups) {
- // This list is not mapped to a group in CiviCRM.
- return NULL;
- }
- $_ = array_keys($groups);
- $membershipGroupID = $_[0];
- if ($action == 'subscribe') {
- $groupContactAdditions[$membershipGroupID][] = $contactID;
- }
- elseif ($action == 'unsubscribe') {
- $groupContactRemoves[$membershipGroupID][] = $contactID;
-
- $mcGroupings = array();
- foreach (empty($requestData['merges']['GROUPINGS']) ? array() : $requestData['merges']['GROUPINGS'] as $grouping) {
- foreach (explode(', ', $grouping['groups']) as $group) {
- $mcGroupings[$grouping['id']][$group] = 1;
+ public function processRequest($expected_key, $key, $request_data) {
+
+ // Check CMS's permission for (presumably) anonymous users.
+ if (CRM_Core_Config::singleton()->userPermissionClass->isModulePermissionSupported() && !CRM_Mailchimp_Permission::check('allow webhook posts')) {
+ throw new RuntimeException("Missing allow webhook posts permission.", 500);
+ }
+
+ // Check the 2 keys exist and match.
+ if (!$key || !$expected_key || $key != $expected_key ) {
+ throw new RuntimeException("Invalid security key.", 500);
+ }
+
+ if (empty($request_data['data']['list_id']) || empty($request_data['type'])
+ || !in_array($request_data['type'], ['subscribe', 'unsubscribe', 'profile', 'upemail', 'cleaned'])
+ ) {
+ // We are not programmed to respond to this type of request.
+ // But maybe Mailchimp introduced something new, so we'll just say OK.
+ throw new RuntimeException("Missing or invalid data in request: " . json_encode($request_data), 200);
+ }
+
+ $method = $request_data['type'];
+
+ // Check list config at Mailchimp.
+ $list_id = $request_data['data']['list_id'];
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+ $result = $api->get("/lists/$list_id/webhooks")->data->webhooks;
+ $url = CRM_Mailchimp_Utils::getWebhookUrl();
+ // Find our webhook and check for a particularly silly configuration.
+ foreach ($result as $webhook) {
+ if ($webhook->url == $url) {
+ if ($webhook->sources->api) {
+ // To continue could cause a nasty loop.
+ throw new RuntimeException("The list '$list_id' is not configured correctly at Mailchimp. It has the 'API' source set so processing this using the API could cause a loop.", 500);
}
}
- foreach ($allGroups as $groupID => $details) {
- if ($groupID != $membershipGroupID && $details['is_mc_update_grouping']) {
- if (!empty($mcGroupings[$details['grouping_id']][$details['group_name']])) {
- $groupContactRemoves[$groupID][] = $contactID;
- }
- }
+ }
+
+ // Disable post hooks. We're updating *from* Mailchimp so we don't want
+ // to fire anything *at* Mailchimp.
+ CRM_Mailchimp_Utils::$post_hook_enabled = FALSE;
+
+ // Pretty much all the request methods use these:
+ $this->sync = new CRM_Mailchimp_Sync($request_data['data']['list_id']);
+ $this->request_data = $request_data['data'];
+ // Call the appropriate handler method.
+ CRM_Mailchimp_Utils::checkDebug("Webhook: $method with request data: " . json_encode($request_data));
+ $this->$method();
+
+ // re-set the post hooks.
+ CRM_Mailchimp_Utils::$post_hook_enabled = TRUE;
+ // Return OK response.
+ return [200, NULL];
+ }
+
+ /**
+ * Handle subscribe requests.
+ *
+ * For subscribes we rely on the following in request_data:
+ *
+ * - "[list_id]": "a6b5da1054",
+ * - "[email]": "api@mailchimp.com",
+ * - "[merges][FNAME]": "MailChimp",
+ * - "[merges][LNAME]": "API",
+ * - "[merges][INTERESTS]": "Group1,Group2",
+ *
+ */
+ public function subscribe() {
+ // This work is shared with 'profile', so kept in a separate method for
+ // clarity.
+ $this->findOrCreateSubscribeAndUpdate();
+ }
+ /**
+ * Handle unsubscribe requests.
+ *
+ * For unsubscribes we rely on the following in request_data:
+ *
+ * - "data[list_id]": "a6b5da1054",
+ * - "data[email]": "api@mailchimp.com",
+ * - "data[merges][FNAME]": "MailChimp",
+ * - "data[merges][LNAME]": "API",
+ *
+ */
+ public function unsubscribe() {
+
+ try {
+ $this->contact_id = $this->sync->guessContactIdSingle(
+ $this->request_data['email'],
+ $this->request_data['merges']['FNAME'],
+ $this->request_data['merges']['LNAME'],
+ $must_be_in_group=TRUE
+ );
+ if (!$this->contact_id) {
+ // Hmm. We don't think they *are* subscribed.
+ // Nothing for us to do.
+ return;
}
}
+ catch (CRM_Mailchimp_DuplicateContactsException $e) {
+ // We cannot process this webhook.
+ throw new RuntimeException("Duplicate contact: " . $e->getMessage(), 500);
+ }
+
+ // Contact has just unsubscribed, we'll need to remove them from the group.
+ civicrm_api3('GroupContact', 'create', [
+ 'contact_id' => $this->contact_id,
+ 'group_id' => $this->sync->membership_group_id,
+ 'status' => 'Removed',
+ ]);
+ }
+ /**
+ * Handle profile update requests.
+ *
+ * Works as subscribe does.
+ */
+ public function profile() {
+
+ // Profile changes trigger two webhooks simultaneously. This is
+ // upsetting as we can end up creating a new contact twice. So we delay
+ // the profile one a bit so that if a contact needs creating, this will
+ // be done before the profile update one.
+ // Mailchimp expects a response to webhooks within 15s, so we have to
+ // keep the delay short enough.
+ sleep(10);
+
+ // Do same work as subscribe. While with subscribe it's more typical to find
+ // a contact that is not in CiviCRM, it's still a possible situation for a
+ // profile update, e.g. if the subscribe webhook failed or was not fired.
+ $this->findOrCreateSubscribeAndUpdate();
+ }
+
+ /**
+ * Subscriber updated their email.
+ *
+ * Relies on the following keys in $this->request_data:
+ *
+ * - list_id
+ * - new_email
+ * - old_email
+ *
+ */
+ public function upemail() {
+ if (empty($this->request_data['new_email'])
+ || empty($this->request_data['old_email'])) {
+ // Weird.
+ throw new RuntimeException("Attempt to change an email address without specifying both email addresses.", 400);
+ }
+
+ // Identify contact.
+ try {
+ $contact_id = $this->sync->guessContactIdSingle(
+ $this->request_data['old_email'], NULL, NULL, $must_be_in_group=TRUE);
+ }
+ catch (CRM_Mailchimp_DuplicateContactsException $e) {
+ throw new RuntimeException("Duplicate contact: " . $e->getMessage(), 500);
+ }
+ if (!$contact_id) {
+ // We don't know this person. Log an error for us, but no need for
+ // Mailchimp to retry the webhook call.
+ throw new RuntimeException("Contact unknown", 200);
+ }
+
+ // Now find the old email.
+
+ // Find bulk email address for this contact.
+ $result = civicrm_api3('Email', 'get', [
+ 'sequential' => 1,
+ 'contact_id' => $contact_id,
+ 'is_bulkmail' => 1,
+ ]);
+ if ($result['count'] == 1) {
+ // They do have a dedicated bulk email, change it.
+ $result = civicrm_api3('Email', 'create', [
+ 'id' => $result['values'][0]['id'],
+ 'email' => $this->request_data['new_email'],
+ ]);
+ return;
+ }
+
+ // They don't yet have a bulk email, give them one set to this new email.
+ $result = civicrm_api3('Email', 'create', [
+ 'sequential' => 1,
+ 'contact_id' => $contact_id,
+ 'email' => $this->request_data['new_email'],
+ 'is_bulkmail' => 1,
+ ]);
+
+ }
+ /**
+ * Email removed by Mailchimp.
+ *
+ * The request data we rely on is:
+ *
+ * - data[list_id]
+ * - data[campaign_id]
+ * - data[reason] This will be hard|abuse
+ * - data[email]
+ *
+ * Put the email on hold.
+ */
+ public function cleaned() {
+ if (empty($this->request_data['email'])) {
+ // Weird.
+ throw new RuntimeException("Attempt to clean an email address without an email address.", 400);
+ }
+
+ // Find the email address and whether the contact is in this list's
+ // membership group.
+ $result = civicrm_api3('Email', 'get', [
+ 'email' => $this->request_data['email'],
+ 'api.Contact.get' => [
+ 'group' => $this->sync->membership_group_id,
+ 'return' => "contact_id",
+ ],
+ ]);
+
+ if ($result['count'] == 0) {
+ throw new RuntimeException("Email unknown", 200);
+ }
- // Now deal with all the groupings that are mapped to CiviCRM groups for this list
- // and that have the allow MC updates flag set.
- /* Sample groupings from MC:
- *
- * [GROUPINGS] => Array(
- * [0] => Array(
- * [id] => 11365
- * [name] => CiviCRM
- * [groups] => special
- * ))
- * Re-map to mcGroupings[grouping_id][group_name] = 1;
- */
- $mcGroupings = array();
- foreach (empty($requestData['merges']['GROUPINGS']) ? array() : $requestData['merges']['GROUPINGS'] as $grouping) {
- foreach (explode(', ', $grouping['groups']) as $group){
- $mcGroupings[$grouping['id']][$group] = 1;
+ // Loop found emails.
+ $found = 0;
+ foreach ($result['values'] as $email) {
+ // hard: always set on hold.
+ // abuse: set on hold only if contact is in the list.
+ if ($this->request_data['reason'] == 'hard'
+ || (
+ $this->request_data['reason'] == 'abuse'
+ && $email['api.Contact.get']['count'] == 1)
+ ) {
+ // Set it on hold.
+ civicrm_api3('Email', 'create', ['on_hold' => 1] + $email);
+ $found++;
}
}
- $groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), $listID, $membership_only = FALSE);
- CRM_Mailchimp_Utils::checkDebug('Middle- CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $groups ', $groups);
- CRM_Mailchimp_Utils::checkDebug('Middle- CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $mcGroupings ', $mcGroupings);
+ if ($this->request_data['reason'] == 'abuse' && $found == 0) {
+ // We got an abuse request but we could not find a contact that was
+ // subscribed; we have not put any emails on hold.
+ throw new RuntimeException("Email unknown", 200);
+ }
+ }
+
+ // Helper functions.
+ /**
+ * Find/create, and update.
+ *
+ * - "[list_id]": "a6b5da1054",
+ * - "[email]": "api@mailchimp.com",
+ * - "[merges][FNAME]": "MailChimp",
+ * - "[merges][LNAME]": "API",
+ * - "[merges][INTERESTS]": "Group1,Group2",
+ *
+ */
+ public function findOrCreateSubscribeAndUpdate() {
+
+ $this->findOrCreateContact();
+
+ // Check whether names have changed.
+ $contact = civicrm_api3('Contact', 'getsingle', ['contact_id' => $this->contact_id]);
+ $edits = CRM_Mailchimp_Sync::updateCiviFromMailchimpContactLogic(
+ [
+ 'first_name' => empty($this->request_data['merges']['FNAME']) ? '' : $this->request_data['merges']['FNAME'],
+ 'last_name' => empty($this->request_data['merges']['LNAME']) ? '' : $this->request_data['merges']['LNAME'],
+ ],
+ $contact);
+ if ($edits) {
+ // We do need to make some changes.
+ civicrm_api3('Contact', 'create', ['contact_id' => $this->contact_id] + $edits);
+ }
+
+ // Contact has just subscribed, we'll need to add them to the list.
+ civicrm_api3('GroupContact', 'create', [
+ 'contact_id' => $this->contact_id,
+ 'group_id' => $this->sync->membership_group_id,
+ 'status' => 'Added',
+ ]);
- foreach ($groups as $groupID=>$details) {
- if ($groupID != $membershipGroupID && $details['is_mc_update_grouping']) {
- // This is a group we allow updates for.
-
- if (empty($mcGroupings[$details['grouping_id']][$details['group_name']])) {
- $groupContactRemoves[$groupID][] = $contactID;
- }
- else {
- $groupContactAdditions[$groupID][] = $contactID;
+ $this->updateInterestsFromMerges();
+ }
+ /**
+ * Finds or creates the contact from email, first and last name.
+ *
+ * Sets $this->contact_id if successful.
+ *
+ * @throw RuntimeException if a duplicate contact in CiviCRM means we cannot
+ * identify a contact.
+ */
+ public function findOrCreateContact() {
+ // Find contact.
+ try {
+ // Check for missing merges fields.
+ $this->request_data['merges'] += ['FNAME' => '', 'LNAME' => ''];
+ if ( empty($this->request_data['merges']['FNAME'])
+ && empty($this->request_data['merges']['LNAME'])
+ && !empty($this->request_data['merges']['NAME'])) {
+ // No first or last names received, but we have a NAME merge field so
+ // try splitting that.
+ $names = explode(' ', $this->request_data['merges']['NAME']);
+ $this->request_data['merges']['FNAME'] = trim(array_shift($names));
+ if ($names) {
+ // Rest of names go as last name.
+ $this->request_data['merges']['LNAME'] = implode(' ', $names);
}
}
- }
- // Add contacts to groups, if anything to do.
- foreach($groupContactAdditions as $groupID => $contactIDs ) {
- CRM_Contact_BAO_GroupContact::addContactsToGroup($contactIDs, $groupID, 'Admin', 'Added');
+ // Nb. the following will throw an exception if duplication prevents us
+ // adding a contact, so execution will only continue if we were able
+ // either to identify an existing contact, or to identify that the
+ // incomming contact is a new one that we're OK to create.
+ $this->contact_id = $this->sync->guessContactIdSingle(
+ $this->request_data['email'],
+ $this->request_data['merges']['FNAME'],
+ $this->request_data['merges']['LNAME']
+ );
+ if (!$this->contact_id) {
+ // New contact, create now.
+ $result = civicrm_api3('Contact', 'create', [
+ 'contact_type' => 'Individual',
+ 'first_name' => $this->request_data['merges']['FNAME'],
+ 'last_name' => $this->request_data['merges']['LNAME'],
+ ]);
+ if (!$result['id']) {
+ throw new RuntimeException("Failed to create contact", 500);
+ }
+ $this->contact_id = $result['id'];
+ // Create bulk email.
+ $result = civicrm_api3('Email', 'create', [
+ 'contact_id' => $this->contact_id,
+ 'email' => $this->request_data['email'],
+ 'is_bulkmail' => 1,
+ ]);
+ if (!$result['id']) {
+ throw new RuntimeException("Failed to create contact's email", 500);
+ }
+ }
+ }
+ catch (CRM_Mailchimp_DuplicateContactsException $e) {
+ // We cannot process this webhook.
+ throw new RuntimeException("Duplicate contact: " . $e->getMessage(), 500);
}
+ }
+ /**
+ * Mailchimp still sends interests to webhooks in an old school way.
+ *
+ * So it's left to us to identify the interests and groups that they refer to.
+ */
+ public function updateInterestsFromMerges() {
+
+ // Get a list of CiviCRM group Ids that this contact should be in.
+ $should_be_in = $this->sync->splitMailchimpWebhookGroupsToCiviGroupIds($this->request_data['merges']['INTERESTS']);
- // Remove contacts from groups, if anything to do.
- foreach($groupContactRemoves as $groupID => $contactIDs ) {
- CRM_Contact_BAO_GroupContact::removeContactsFromGroup($contactIDs, $groupID, 'Admin', 'Removed');
+ // Now get a list of all the groups they *are* in.
+ $result = civicrm_api3('Contact', 'getsingle', ['return' => 'group', 'contact_id' => $this->contact_id]);
+ $is_in = CRM_Mailchimp_Utils::splitGroupTitles($result['groups'], $this->sync->interest_group_details);
+
+ // Finally loop all the mapped interest groups and process any differences.
+ foreach ($this->sync->interest_group_details as $group_id => $details) {
+ if ($details['is_mc_update_grouping'] == 1) {
+ // We're allowed to update Civi from Mailchimp for this one.
+ if (in_array($group_id, $should_be_in) && !in_array($group_id, $is_in)) {
+ // Not in this group, but should be.
+ civicrm_api3('GroupContact', 'create', [
+ 'contact_id' => $this->contact_id,
+ 'group_id' => $group_id,
+ 'status' => 'Added',
+ ]);
+ }
+ elseif (!in_array($group_id, $should_be_in) && in_array($group_id, $is_in)) {
+ // Is in this group, but should not be.
+ civicrm_api3('GroupContact', 'create', [
+ 'contact_id' => $this->contact_id,
+ 'group_id' => $group_id,
+ 'status' => 'Removed',
+ ]);
+ }
+ }
}
-
- CRM_Mailchimp_Utils::checkDebug('End - CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $groupContactRemoves ', $groupContactRemoves);
- CRM_Mailchimp_Utils::checkDebug('End - CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $groupContactAdditions ', $groupContactAdditions);
- CRM_Mailchimp_Utils::checkDebug('End - CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $contactID= ', $contactID);
- CRM_Mailchimp_Utils::checkDebug('End - CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $requestData= ', $requestData);
- CRM_Mailchimp_Utils::checkDebug('End - CRM_Mailchimp_Page_WebHook manageCiviCRMGroupSubcription $requestType= ', $action);
}
+
}
diff --git a/CRM/Mailchimp/RequestErrorException.php b/CRM/Mailchimp/RequestErrorException.php
new file mode 100644
index 0000000..0455335
--- /dev/null
+++ b/CRM/Mailchimp/RequestErrorException.php
@@ -0,0 +1,7 @@
+group_details['61'] = (array [12])
+ ⬦ $this->group_details['61']['list_id'] = (string [10]) `4882f4fdb8`
+ ⬦ $this->group_details['61']['category_id'] = (null)
+ ⬦ $this->group_details['61']['category_name'] = (null)
+ ⬦ $this->group_details['61']['interest_id'] = (null)
+ ⬦ $this->group_details['61']['interest_name'] = (null)
+ ⬦ $this->group_details['61']['is_mc_update_grouping'] = (string [1]) `0`
+ ⬦ $this->group_details['61']['civigroup_title'] = (string [28]) `mailchimp_integration_test_1`
+ ⬦ $this->group_details['61']['civigroup_uses_cache'] = (bool) 0
+ ⬦ $this->group_details['61']['grouping_id'] = (null)
+ ⬦ $this->group_details['61']['grouping_name'] = (null)
+ ⬦ $this->group_details['61']['group_id'] = (null)
+ ⬦ $this->group_details['61']['group_name'] = (null)
+ */
+ protected $group_details;
+ /**
+ * As above but without membership group.
+ */
+ protected $interest_group_details;
+ /**
+ * The CiviCRM group id responsible for membership at Mailchimp.
+ */
+ protected $membership_group_id;
+
+ /** If true no changes will be made to Mailchimp or CiviCRM. */
+ protected $dry_run = FALSE;
+ public function __construct($list_id) {
+ $this->list_id = $list_id;
+ $this->group_details = CRM_Mailchimp_Utils::getGroupsToSync($groupIDs=[], $list_id, $membership_only=FALSE);
+ foreach ($this->group_details as $group_id => $group_details) {
+ if (empty($group_details['category_id'])) {
+ $this->membership_group_id = $group_id;
+ }
+ }
+ if (empty($this->membership_group_id)) {
+ throw new InvalidArgumentException("Failed to find mapped membership group for list '$list_id'");
+ }
+ // Also cache without the membership group, i.e. interest groups only.
+ $this->interest_group_details = $this->group_details;
+ unset($this->interest_group_details[$this->membership_group_id]);
+ }
+ /**
+ * Getter.
+ */
+ public function __get($property) {
+ switch ($property) {
+ case 'list_id':
+ case 'membership_group_id':
+ case 'group_details':
+ case 'interest_group_details':
+ case 'dry_run':
+ return $this->$property;
+ }
+ throw new InvalidArgumentException("'$property' property inaccessible or unknown");
+ }
+ /**
+ * Setter.
+ */
+ public function __set($property, $value) {
+ switch ($property) {
+ case 'dry_run':
+ return $this->$property = (bool) $value;
+ }
+ throw new InvalidArgumentException("'$property' property inaccessible or unknown");
+ }
+ // The following methods are the key steps of the pull and push syncs.
+ /**
+ * Collect Mailchimp data into temporary working table.
+ *
+ * There are two modes of operation:
+ *
+ * In **pull** mode we only collect data that comes from Mailchimp that we are
+ * allowed to update in CiviCRM.
+ *
+ * In **push** mode we collect data that we would update in Mailchimp from
+ * CiviCRM.
+ *
+ * Crucially the difference is for CiviCRM groups mapped to a Mailchimp
+ * interest: these can either allow updates *from* Mailchimp or not. Typical
+ * use case is a hidden-from-subscriber 'interest' called 'donor type' which
+ * might include 'major donor' and 'minor donor' based on some valuation by
+ * the organisation recorded in CiviCRM groups.
+ *
+ * @param string $mode pull|push.
+ * @return int number of contacts collected.
+ */
+ public function collectMailchimp($mode) {
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncCollectMailchimp $this->list_id= ', $this->list_id);
+ if (!in_array($mode, ['pull', 'push'])) {
+ throw new InvalidArgumentException(__FUNCTION__ . " expects push/pull but called with '$mode'.");
+ }
+ $dao = static::createTemporaryTableForMailchimp();
+
+ // Cheekily access the database directly to obtain a prepared statement.
+ $db = $dao->getDatabaseConnection();
+ $insert = $db->prepare('INSERT INTO tmp_mailchimp_push_m
+ (email, first_name, last_name, hash, interests)
+ VALUES (?, ?, ?, ?, ?)');
+
+ CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Form_Sync syncCollectMailchimp: ', $this->interest_group_details);
+
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+ $offset = 0;
+ $batch_size = 1000;
+ $total = null;
+ $list_id = $this->list_id;
+ $fetch_batch = function() use($api, &$offset, &$total, $batch_size, $list_id) {
+ if ($total !== null && $offset >= $total) {
+ // End of results.
+ return [];
+ }
+ $response = $api->get("/lists/$this->list_id/members", [
+ 'offset' => $offset, 'count' => $batch_size,
+ 'status' => 'subscribed',
+ 'fields' => 'total_items,members.email_address,members.merge_fields,members.interests',
+ ]);
+ $total = (int) $response->data->total_items;
+ $offset += $batch_size;
+ return $response->data->members;
+ };
+
+ //
+ // Main loop of all the records.
+ $collected = 0;
+ while ($members = $fetch_batch()) {
+ $start = microtime(TRUE);
+ foreach ($members as $member) {
+ $first_name = isset($member->merge_fields->FNAME) ? $member->merge_fields->FNAME : '';
+ $last_name = isset($member->merge_fields->LNAME) ? $member->merge_fields->LNAME : '';
+
+ if (!$first_name && !$last_name && !empty($member->merge_fields->NAME)) {
+ // No first or last names received, but we have a NAME merge field so
+ // try splitting that.
+ $names = explode(' ', $member->merge_fields->NAME);
+ $first_name = trim(array_shift($names));
+ if ($names) {
+ // Rest of names go as last name.
+ $last_name = implode(' ', $names);
+ }
+ }
+ // Find out which of our mapped groups apply to this subscriber.
+ // Serialize the grouping array for SQL storage - this is the fastest way.
+ $interests = serialize($this->getComparableInterestsFromMailchimp($member->interests, $mode));
+
+ // we're ready to store this but we need a hash that contains all the info
+ // for comparison with the hash created from the CiviCRM data (elsewhere).
+ //
+ // Previous algorithms included email here, but we actually allow
+ // mailchimp to have any email that belongs to the contact in the
+ // membership group, even though for new additions we'd use the bulk
+ // email. So we don't count an email mismatch as a problem.
+ // $hash = md5($member->email_address . $first_name . $last_name . $interests);
+ $hash = md5($first_name . $last_name . $interests);
+ // run insert prepared statement
+ $result = $db->execute($insert, [
+ $member->email_address,
+ $first_name,
+ $last_name,
+ $hash,
+ $interests,
+ ]);
+ if ($result instanceof DB_Error) {
+ throw new Exception ($result->message . "\n" . $result->userinfo);
+ }
+ $collected++;
+ }
+ CRM_Mailchimp_Utils::checkDebug('collectMailchimp took ' . round(microtime(TRUE) - $start,2) . 's to copy ' . count($members) . ' mailchimp Members to tmp table.');
+ }
+
+ // Tidy up.
+ fclose($handle);
+ $db->freePrepared($insert);
+ return $collected;
+ }
+ /**
+ * Collect CiviCRM data into temporary working table.
+ *
+ * Speed notes.
+ *
+ * Various strategies have been tried here to speed things up. Originally we
+ * used the API with a chained API call, but this was very slow (~10s for
+ * ~5k contacts), so now we load all the contacts, then all the emails in a
+ * 2nd API call. This is about 10x faster, taking less than 1s for ~5k
+ * contacts. Likewise the structuring of the emails on the contact array has
+ * been tried various ways, and this structure-by-type way has reduced the
+ * origninal loop time from 7s down to just under 4s.
+ *
+ *
+ * @param string $mode pull|push.
+ * @return int number of contacts collected.
+ */
+ public function collectCiviCrm($mode) {
+ CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Form_Sync syncCollectCiviCRM $this->list_id= ', $this->list_id);
+ if (!in_array($mode, ['pull', 'push'])) {
+ throw new InvalidArgumentException(__FUNCTION__ . " expects push/pull but called with '$mode'.");
+ }
+ // Cheekily access the database directly to obtain a prepared statement.
+ $dao = static::createTemporaryTableForCiviCRM();
+ $db = $dao->getDatabaseConnection();
+
+ // There used to be a distinction between the handling of 'normal' groups
+ // and smart groups. But now the API will take care of this but this
+ // requires the following function to have run.
+ foreach ($this->interest_group_details as $group_id => $details) {
+ if ($mode == 'push' || $details['is_mc_update_grouping'] == 1) {
+ // Either we are collecting for a push from C->M,
+ // or we're pulling and this group is configured to allow updates.
+ // Therefore we need to make sure the cache is filled.
+ CRM_Contact_BAO_GroupContactCache::loadAll($group_id);
+ }
+ }
+
+ // Use a nice API call to get the information for tmp_mailchimp_push_c.
+ // The API will take care of smart groups.
+ $start = microtime(TRUE);
+ $result = civicrm_api3('Contact', 'get', [
+ 'is_deleted' => 0,
+ // The email filter in comment below does not work (CRM-18147)
+ // 'email' => array('IS NOT NULL' => 1),
+ // Now I think that on_hold is NULL when there is no e-mail, so if
+ // we are lucky, the filter below implies that an e-mail address
+ // exists ;-)
+ 'is_opt_out' => 0,
+ 'do_not_email' => 0,
+ 'on_hold' => 0,
+ 'is_deceased' => 0,
+ 'group' => $this->membership_group_id,
+ 'return' => ['first_name', 'last_name', 'group'],
+ 'options' => ['limit' => 0],
+ //'api.Email.get' => ['on_hold'=>0, 'return'=>'email,is_bulkmail'],
+ ]);
+
+ if ($result['count'] == 0) {
+ // No-one is in the group according to CiviCRM.
+ return 0;
+ }
+
+ // Load emails for these contacts.
+ $emails = civicrm_api3('Email', 'get', [
+ 'on_hold' => 0,
+ 'return' => 'contact_id,email,is_bulkmail,is_primary',
+ 'contact_id' => ['IN' => array_keys($result['values'])],
+ 'options' => ['limit' => 0],
+ ]);
+ // Index emails by contact_id.
+ foreach ($emails['values'] as $email) {
+ if ($email['is_bulkmail']) {
+ $result['values'][$email['contact_id']]['bulk_email'] = $email['email'];
+ }
+ elseif ($email['is_primary']) {
+ $result['values'][$email['contact_id']]['primary_email'] = $email['email'];
+ }
+ else {
+ $result['values'][$email['contact_id']]['other_email'] = $email['email'];
+ }
+ }
+ /**
+ * We have a contact that has no other deets.
+ */
+
+ $start = microtime(TRUE);
+
+ $collected = 0;
+ $insert = $db->prepare('INSERT INTO tmp_mailchimp_push_c VALUES(?, ?, ?, ?, ?, ?)');
+ // Loop contacts:
+ foreach ($result['values'] as $id=>$contact) {
+ // Which email to use?
+ $email = isset($contact['bulk_email'])
+ ? $contact['bulk_email']
+ : (isset($contact['primary_email'])
+ ? $contact['primary_email']
+ : (isset($contact['other_email'])
+ ? $contact['other_email']
+ : NULL));
+ if (!$email) {
+ // Hmmm.
+ continue;
+ }
+
+ // Find out the ID's of the groups the $contact belongs to, and
+ // save in $info.
+ $info = $this->getComparableInterestsFromCiviCrmGroups($contact['groups'], $mode);
+
+ // OK we should now have all the info we need.
+ // Serialize the grouping array for SQL storage - this is the fastest way.
+ $info = serialize($info);
+
+ // we're ready to store this but we need a hash that contains all the info
+ // for comparison with the hash created from the CiviCRM data (elsewhere).
+ // email, first name, last name, groupings
+ // See note above about why we don't include email in the hash.
+ // $hash = md5($email . $contact['first_name'] . $contact['last_name'] . $info);
+ $hash = md5($contact['first_name'] . $contact['last_name'] . $info);
+ // run insert prepared statement
+ $db->execute($insert, array($contact['id'], $email, $contact['first_name'], $contact['last_name'], $hash, $info));
+ $collected++;
+ }
+
+ // Tidy up.
+ $db->freePrepared($insert);
+
+ return $collected;
+ }
+ /**
+ * Match mailchimp records to particular contacts in CiviCRM.
+ *
+ * This requires that both collect functions have been run in the same mode
+ * (push/pull).
+ *
+ * First we attempt a number of SQL based strategies as these are the fastest.
+ *
+ * If the fast SQL matches have failed, we need to do it the slow way.
+ *
+ * @return array of counts - for tests really.
+ * - bySubscribers
+ * - byUniqueEmail
+ * - byNameEmail
+ * - bySingle
+ * - totalMatched
+ * - newContacts (contacts that should be created in CiviCRM)
+ * - failures (duplicate contacts in CiviCRM)
+ */
+ public function matchMailchimpMembersToContacts() {
+
+ // Ensure we have the mailchimp_log table.
+ $dao = CRM_Core_DAO::executeQuery(
+ "CREATE TABLE IF NOT EXISTS mailchimp_log (
+ id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
+ group_id int(20),
+ email VARCHAR(200),
+ name VARCHAR(200),
+ message VARCHAR(512),
+ KEY (group_id)
+ );");
+ // Clear out any old errors to do with this list.
+ CRM_Core_DAO::executeQuery(
+ "DELETE FROM mailchimp_log WHERE group_id = %1;",
+ [1 => [$this->membership_group_id, 'Integer' ]]);
+
+ $stats = [
+ 'bySubscribers' => 0,
+ 'byUniqueEmail' => 0,
+ 'byNameEmail' => 0,
+ 'bySingle' => 0,
+ 'totalMatched' => 0,
+ 'newContacts' => 0,
+ 'failures' => 0,
+ ];
+ // Do the fast SQL identification against CiviCRM contacts.
+ $start = microtime(TRUE);
+ $stats['bySubscribers'] = static::guessContactIdsBySubscribers();
+ CRM_Mailchimp_Utils::checkDebug('guessContactIdsBySubscribers took ' . round(microtime(TRUE) - $start, 2) . 's');
+ $start = microtime(TRUE);
+ $stats['byUniqueEmail'] = static::guessContactIdsByUniqueEmail();
+ CRM_Mailchimp_Utils::checkDebug('guessContactIdsByUniqueEmail took ' . round(microtime(TRUE) - $start, 2) . 's');
+ $start = microtime(TRUE);
+ $stats['byNameEmail'] = static::guessContactIdsByNameAndEmail();
+ CRM_Mailchimp_Utils::checkDebug('guessContactIdsByNameAndEmail took ' . round(microtime(TRUE) - $start, 2) . 's');
+ $start = microtime(TRUE);
+
+ // Now slow match the rest.
+ $dao = CRM_Core_DAO::executeQuery( "SELECT * FROM tmp_mailchimp_push_m m WHERE cid_guess IS NULL;");
+ $db = $dao->getDatabaseConnection();
+ $update = $db->prepare('UPDATE tmp_mailchimp_push_m
+ SET cid_guess = ? WHERE email = ? AND hash = ?');
+ $failures = $new = 0;
+ while ($dao->fetch()) {
+ try {
+ $contact_id = $this->guessContactIdSingle($dao->email, $dao->first_name, $dao->last_name);
+ if (!$contact_id) {
+ // We use zero to mean create a contact.
+ $contact_id = 0;
+ $new++;
+ }
+ else {
+ // Successful match.
+ $stats['bySingle']++;
+ }
+ }
+ catch (CRM_Mailchimp_DuplicateContactsException $e) {
+ $contact_id = NULL;
+ $failures++;
+ }
+ if ($contact_id !== NULL) {
+ // Contact found, or a zero (create needed).
+ $result = $db->execute($update, [
+ $contact_id,
+ $dao->email,
+ $dao->hash,
+ ]);
+ if ($result instanceof DB_Error) {
+ throw new Exception ($result->message . "\n" . $result->userinfo);
+ }
+ }
+ }
+ $db->freePrepared($update);
+ $took = microtime(TRUE) - $start;
+ CRM_Mailchimp_Utils::checkDebug('guessContactIdSingle took ' . round($took,2)
+ . "s for $stats[bySingle] records (" . round($took/$stats['bySingle'],2) . "s/record");
+ $stats['totalMatched'] = array_sum($stats);
+ $stats['newContacts'] = $new;
+ $stats['failures'] = $failures;
+
+ if ($stats['failures']) {
+ // Copy errors into the mailchimp_log table.
+ CRM_Core_DAO::executeQuery(
+ "INSERT INTO mailchimp_log (group_id, message)
+ SELECT %1 group_id,
+ email,
+ CONCAT_WS(' ', first_name, last_name) name,
+ 'titanic' message
+ FROM tmp_mailchimp_push_m
+ WHERE cid_guess IS NULL;",
+ [1 => [$this->membership_group_id, 'Integer']]);
+ }
+
+ return $stats;
+ }
+ /**
+ * Removes from the temporary tables those records that do not need processing
+ * because they are identical.
+ *
+ * In *push* mode this will also remove any rows in the CiviCRM temp table
+ * where there's an email match in the mailchimp table but the cid_guess is
+ * different. This is to cover the case when two contacts in CiviCRM have the
+ * same email and both are added to the membership group. Without this the
+ * Push operation would attempt to craeate a 2nd Mailchimp member but with the
+ * email address that's already on the list. This would mean the names kept
+ * getting flipped around since it would be updating the same member twice -
+ * very confusing.
+ *
+ * So for deleting the contacts from the CiviCRM table on *push* we avoid
+ * this. However on *pull* we leave the contact in the table - they will then
+ * get removed from the group, leaving just the single contact/member with
+ * that particular email address.
+ *
+ * @param string $mode pull|push.
+ * @return int
+ */
+ public function removeInSync($mode) {
+
+ // In push mode, delete duplicate CiviCRM contacts.
+ $doubles = 0;
+ if ($mode == 'push') {
+ $doubles = CRM_Mailchimp_Sync::runSqlReturnAffectedRows(
+ 'DELETE c
+ FROM tmp_mailchimp_push_c c
+ INNER JOIN tmp_mailchimp_push_m m ON c.email=m.email AND m.cid_guess != c.contact_id;
+ ');
+ if ($doubles) {
+ CRM_Mailchimp_Utils::checkDebug("removeInSync removed $doubles contacts who are in the membership group but have the same email address as another contact that is also in the membership group.");
+ }
+ }
+
+ // Delete records have the same hash - these do not need an update.
+ // count for testing purposes.
+ $dao = CRM_Core_DAO::executeQuery("SELECT COUNT(c.email) co FROM tmp_mailchimp_push_m m
+ INNER JOIN tmp_mailchimp_push_c c ON m.cid_guess = c.contact_id AND m.hash = c.hash;");
+ $dao->fetch();
+ $count = $dao->co;
+ if ($count > 0) {
+ CRM_Core_DAO::executeQuery(
+ "DELETE m, c
+ FROM tmp_mailchimp_push_m m
+ INNER JOIN tmp_mailchimp_push_c c ON m.cid_guess = c.contact_id AND m.hash = c.hash;");
+ }
+ CRM_Mailchimp_Utils::checkDebug("removeInSync removed $count in-sync contacts.");
+
+
+ return $count + $doubles;
+ }
+ /**
+ * "Push" sync.
+ *
+ * Sends additions, edits (compared to tmp_mailchimp_push_m), deletions.
+ *
+ * Note that an 'update' counted in the return stats could be a change or an
+ * addition.
+ *
+ * @return array ['updates' => INT, 'unsubscribes' => INT]
+ */
+ public function updateMailchimpFromCivi() {
+ CRM_Mailchimp_Utils::checkDebug("updateMailchimpFromCivi for group #$this->membership_group_id");
+ $operations = [];
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+ $dao = CRM_Core_DAO::executeQuery(
+ "SELECT
+ c.interests c_interests, c.first_name c_first_name, c.last_name c_last_name,
+ c.email c_email,
+ m.interests m_interests, m.first_name m_first_name, m.last_name m_last_name,
+ m.email m_email
+ FROM tmp_mailchimp_push_c c
+ LEFT JOIN tmp_mailchimp_push_m m ON c.contact_id = m.cid_guess;");
+
+ $url_prefix = "/lists/$this->list_id/members/";
+ $changes = $additions = 0;
+ // We need to know that the mailchimp list has certain merge fields.
+ $result = $api->get("/lists/$this->list_id/merge-fields", ['fields' => 'merge_fields.tag'])->data->merge_fields;
+ $merge_fields = [];
+ foreach ($result as $field) {
+ $merge_fields[$field->tag] = TRUE;
+ }
+
+ while ($dao->fetch()) {
+
+ $params = static::updateMailchimpFromCiviLogic(
+ $merge_fields,
+ ['email' => $dao->c_email, 'first_name' => $dao->c_first_name, 'last_name' => $dao->c_last_name, 'interests' => $dao->c_interests],
+ ['email' => $dao->m_email, 'first_name' => $dao->m_first_name, 'last_name' => $dao->m_last_name, 'interests' => $dao->m_interests]);
+
+ if (!$params) {
+ // This is the case if the changes could not be made due to policy
+ // reasons, e.g. a missing name in CiviCRM should not overwrite a
+ // provided name in Mailchimp; this is a difference but it's not one we
+ // will correct.
+ continue;
+ }
+
+ if ($this->dry_run) {
+ // Log the operation description.
+ $_ = "Would " . ($dao->m_email ? 'update' : 'create')
+ . " mailchimp member: $dao->m_email";
+ if (key_exists('email_address', $params)) {
+ $_ .= " change email to '$params[email_address]'";
+ }
+ if (key_exists('merge_fields', $params)) {
+ foreach ($params['merge_fields'] as $field=>$value) {
+ $_ .= " set $field = $value";
+ }
+ }
+ CRM_Mailchimp_Utils::checkDebug($_);
+ }
+ else {
+ // Add the operation to the batch.
+ $params['status'] = 'subscribed';
+ $operations []= ['PUT', $url_prefix . md5(strtolower($dao->c_email)), $params];
+ }
+
+ if ($dao->m_email) {
+ $changes++;
+ } else {
+ $additions++;
+ }
+ }
+
+ // Now consider deletions of those not in membership group at CiviCRM but
+ // there at Mailchimp.
+ $removals = $this->getEmailsNotInCiviButInMailchimp();
+ $unsubscribes = count($removals);
+ if ($this->dry_run) {
+ // Just log.
+ if ($unsubscribes) {
+ CRM_Mailchimp_Utils::checkDebug("Would unsubscribe " . count($unsubscribes) . " Mailchimp members: " . implode(', ', $removals));
+ }
+ else {
+ CRM_Mailchimp_Utils::checkDebug("No Mailchimp members would be unsubscribed.");
+ }
+ }
+ else {
+ // For real, not dry run.
+ foreach ($removals as $email) {
+ $operations []= ['PATCH', $url_prefix . md5(strtolower($email)), ['status' => 'unsubscribed']];
+ }
+ if ($operations) {
+ $result = $api->batchAndWait($operations);
+ }
+ }
+
+ return ['additions' => $additions, 'updates' => $changes, 'unsubscribes' => $unsubscribes];
+ }
+
+ /**
+ * "Pull" sync.
+ *
+ * Updates CiviCRM from Mailchimp using the tmp_mailchimp_push_[cm] tables.
+ *
+ * It is assumed that collections (in 'pull' mode) and `removeInSync` have
+ * already run.
+ *
+ * 1. Loop the full tmp_mailchimp_push_m table:
+ *
+ * 1. Contact identified by collectMailchimp()?
+ * - Yes: update name if different.
+ * - No: Create or find-and-update the contact.
+ *
+ * 2. Check for changes in groups; record what needs to be changed for a
+ * batch update.
+ *
+ * 2. Batch add/remove contacts from groups.
+ *
+ * @return array With the following keys:
+ *
+ * - created: was in MC not CiviCRM so a new contact was created
+ * - joined : email matched existing contact that was joined to the membership
+ * group.
+ * - in_sync: was in MC and on membership group already.
+ * - removed: was not in MC but was on membership group, so removed from
+ * membership group.
+ * - updated: No. in_sync or joined contacts that were updated.
+ *
+ * The initials of these categories c, j, i, r correspond to this diagram:
+ *
+ * From Mailchimp: ************
+ * From CiviCRM : ********
+ * Result : ccccjjjjiiiirrrr
+ *
+ * Of the contacts known in both systems (j, i) we also record how many were
+ * updated (e.g. name, interests).
+ *
+ * Work in pass 1:
+ *
+ * - create|find
+ * - join
+ * - update names
+ * - update interests
+ *
+ * Work in pass 2:
+ *
+ * - remove
+ */
+ public function updateCiviFromMailchimp() {
+
+ // Ensure posthooks don't trigger while we make GroupContact changes.
+ CRM_Mailchimp_Utils::$post_hook_enabled = FALSE;
+
+ // This is a functional variable, not a stats. one
+ $changes = ['removals' => [], 'additions' => []];
+
+ CRM_Mailchimp_Utils::checkDebug("updateCiviFromMailchimp for group #$this->membership_group_id");
+
+ // Stats.
+ $stats = [
+ 'created' => 0,
+ 'joined' => 0,
+ 'in_sync' => 0,
+ 'removed' => 0,
+ 'updated' => 0,
+ ];
+
+ // all Mailchimp table *except* titanics: where the contact matches multiple
+ // contacts in CiviCRM.
+ $dao = CRM_Core_DAO::executeQuery( "SELECT m.*,
+ c.contact_id c_contact_id,
+ c.interests c_interests, c.first_name c_first_name, c.last_name c_last_name
+ FROM tmp_mailchimp_push_m m
+ LEFT JOIN tmp_mailchimp_push_c c ON m.cid_guess = c.contact_id
+ WHERE m.cid_guess IS NOT NULL
+ ;");
+
+ // Create lookup hash to map Mailchimp Interest Ids to CiviCRM Groups.
+ $interest_to_group_id = [];
+ foreach ($this->interest_group_details as $group_id=>$details) {
+ $interest_to_group_id[$details['interest_id']] = $group_id;
+ }
+
+ // Loop records found at Mailchimp, creating/finding contacts in CiviCRM.
+ while ($dao->fetch()) {
+ $existing_contact_changed = FALSE;
+
+ if (!empty($dao->cid_guess)) {
+ // Matched existing contact: result: joined or in_sync
+ $contact_id = $dao->cid_guess;
+
+ if ($dao->c_contact_id) {
+ // Contact is already in the membership group.
+ $stats['in_sync']++;
+ }
+ else {
+ // Contact needs joining to the membership group.
+ $stats['joined']++;
+ if (!$this->dry_run) {
+ // Live.
+ $changes['additions'][$this->membership_group_id][] = $contact_id;
+ }
+ else {
+ // Dry Run.
+ CRM_Mailchimp_Utils::checkDebug("Would add existing contact to membership group. Email: $dao->email Contact Id: $dao->cid_guess");
+ }
+ }
+
+ // Update the first name and last name of the contacts we know
+ // if needed and making sure we don't overwrite
+ // something with nothing. See issue #188.
+ $edits = static::updateCiviFromMailchimpContactLogic(
+ ['first_name' => $dao->first_name, 'last_name' => $dao->last_name],
+ ['first_name' => $dao->c_first_name, 'last_name' => $dao->c_last_name]
+ );
+ if ($edits) {
+ if (!$this->dry_run) {
+ // There are changes to be made so make them now.
+ civicrm_api3('Contact', 'create', ['id' => $contact_id] + $edits);
+ }
+ else {
+ // Dry run.
+ CRM_Mailchimp_Utils::checkDebug("Would update CiviCRM contact $dao->cid_guess "
+ . (empty($edits['first_name']) ? '' : "First name from $dao->c_first_name to $dao->first_name ")
+ . (empty($edits['last_name']) ? '' : "Last name from $dao->c_last_name to $dao->last_name "));
+ }
+ $existing_contact_changed = TRUE;
+ }
+ }
+ else {
+ // Contact does not exist, create a new one.
+ if (!$this->dry_run) {
+ // Live:
+ $result = civicrm_api3('Contact', 'create', [
+ 'contact_type' => 'Individual',
+ 'first_name' => $dao->first_name,
+ 'last_name' => $dao->last_name,
+ 'email' => $dao->email,
+ 'sequential' => 1,
+ ]);
+ $contact_id = $result['values'][0]['id'];
+ $changes['additions'][$this->membership_group_id][] = $contact_id;
+ }
+ else {
+ // Dry Run:
+ CRM_Mailchimp_Utils::checkDebug("Would create new contact with email: $dao->email, name: $dao->first_name $dao->last_name");
+ $contact_id = 'dry-run';
+ }
+ $stats['created']++;
+ }
+
+ // Do interests need updating?
+ if ($dao->c_interests && $dao->c_interests == $dao->interests) {
+ // Nothing to change.
+ }
+ else {
+ // Unpack the interests reported by MC
+ $mc_interests = unserialize($dao->interests);
+ if ($dao->c_interests) {
+ // Existing contact.
+ $existing_contact_changed = TRUE;
+ $civi_interests = unserialize($dao->c_interests);
+ }
+ else {
+ // Newly created contact is not in any interest groups.
+ $civi_interests = [];
+ }
+
+ // Discover what needs changing to bring CiviCRM inline with Mailchimp.
+ foreach ($mc_interests as $interest=>$member_has_interest) {
+ if ($member_has_interest && empty($civi_interests[$interest])) {
+ // Member is interested in something, but CiviCRM does not know yet.
+ if (!$this->dry_run) {
+ $changes['additions'][$interest_to_group_id[$interest]][] = $contact_id;
+ }
+ else {
+ CRM_Mailchimp_Utils::checkDebug("Would add CiviCRM contact $dao->cid_guess to interest group "
+ . $interest_to_group_id[$interest]);
+ }
+ }
+ elseif (!$member_has_interest && !empty($civi_interests[$interest])) {
+ // Member is not interested in something, but CiviCRM thinks it is.
+ if (!$this->dry_run) {
+ $changes['removals'][$interest_to_group_id[$interest]][] = $contact_id;
+ }
+ else {
+ CRM_Mailchimp_Utils::checkDebug("Would remove CiviCRM contact $dao->cid_guess from interest group "
+ . $interest_to_group_id[$interest]);
+ }
+ }
+ }
+ }
+
+ if ($existing_contact_changed) {
+ $stats['updated']++;
+ }
+ }
+
+ // And now, what if a contact is not in the Mailchimp list?
+ // We must remove them from the membership group.
+ // Accademic interest (#188): what's faster, this or a 'WHERE NOT EXISTS'
+ // construct?
+ $dao = CRM_Core_DAO::executeQuery( "
+ SELECT c.contact_id
+ FROM tmp_mailchimp_push_c c
+ LEFT OUTER JOIN tmp_mailchimp_push_m m ON m.cid_guess = c.contact_id
+ WHERE m.email IS NULL;
+ ");
+ // Collect the contact_ids that need removing from the membership group.
+ while ($dao->fetch()) {
+ if (!$this->dry_run) {
+ $changes['removals'][$this->membership_group_id][] =$dao->contact_id;
+ }
+ else {
+ CRM_Mailchimp_Utils::checkDebug("Would remove CiviCRM contact $dao->contact_id from membership group - no longer subscribed at Mailchimp.");
+ }
+ $stats['removed']++;
+ }
+
+ if (!$this->dry_run) {
+ // Log group contacts which are going to be added/removed to/from CiviCRM
+ CRM_Mailchimp_Utils::checkDebug('Mailchimp $changes', $changes);
+
+ // Make the changes.
+ if ($changes['additions']) {
+ // We have some contacts to add into groups...
+ foreach($changes['additions'] as $groupID => $contactIDs) {
+ CRM_Contact_BAO_GroupContact::addContactsToGroup($contactIDs, $groupID, 'Admin', 'Added');
+ }
+ }
+
+ if ($changes['removals']) {
+ // We have some contacts to add into groups...
+ foreach($changes['removals'] as $groupID => $contactIDs) {
+ CRM_Contact_BAO_GroupContact::removeContactsFromGroup($contactIDs, $groupID, 'Admin', 'Removed');
+ }
+ }
+ }
+
+ // Re-enable the post hooks.
+ CRM_Mailchimp_Utils::$post_hook_enabled = TRUE;
+
+ return $stats;
+ }
+
+ // Other methods follow.
+ /**
+ * Convert a 'groups' string as provided by CiviCRM's API to a structured
+ * array of arrays whose keys are Mailchimp interest ids and whos value is
+ * boolean.
+ *
+ * Nb. this is then key-sorted, which results in a standardised array for
+ * comparison.
+ *
+ * @param string $groups as returned by CiviCRM's API.
+ * @param string $mode pull|push.
+ * @return array of interest_ids to booleans.
+ */
+ public function getComparableInterestsFromCiviCrmGroups($groups, $mode) {
+ $civi_groups = $groups
+ ? array_flip(CRM_Mailchimp_Utils::splitGroupTitles($groups, $this->interest_group_details))
+ : [];
+ $info = [];
+ foreach ($this->interest_group_details as $civi_group_id => $details) {
+ if ($mode == 'pull' && $details['is_mc_update_grouping'] != 1) {
+ // This group is configured to disallow updates from Mailchimp to
+ // CiviCRM.
+ continue;
+ }
+ $info[$details['interest_id']] = key_exists($civi_group_id, $civi_groups);
+ }
+ ksort($info);
+ return $info;
+ }
+
+ /**
+ * Convert interests object received from the Mailchimp API into
+ * a structure identical to that produced by
+ * getComparableInterestsFromCiviCrmGroups.
+ *
+ * Note this will only return information about interests mapped in CiviCRM.
+ * Any other interests that may have been created on Mailchimp are not
+ * included here.
+ *
+ * @param object $interests 'interests' as returned by GET
+ * /list/.../members/...?fields=interests
+ * @param string $mode pull|push.
+ */
+ public function getComparableInterestsFromMailchimp($interests, $mode) {
+ $info = [];
+ // If pulling data from Mailchimp to CiviCRM we ignore any changes to
+ // interests where such changes are disallowed by configuration.
+ $ignore_non_updatables = $mode == 'pull';
+ foreach ($this->interest_group_details as $details) {
+ if ($ignore_non_updatables && $details['is_mc_update_grouping'] != 1) {
+ // This group is configured to disallow updates from Mailchimp to
+ // CiviCRM.
+ continue;
+ }
+ $info[$details['interest_id']] = !empty($interests->{$details['interest_id']});
+ }
+ ksort($info);
+ return $info;
+ }
+
+ /**
+ * Convert a 'groups' string as provided by Mailchimp's Webhook request API to
+ * an array of CiviCRM group ids.
+ *
+ * Nb. a Mailchimp webhook is the equivalent of a 'pull' operation so we
+ * ignore any groups that Mailchimp is not allowed to update.
+ *
+ * @param string $groups as returned by Mailchimp's merges.INTERESTS request
+ * data.
+ * @return array of interest_ids to booleans.
+ */
+ public function splitMailchimpWebhookGroupsToCiviGroupIds($group_input) {
+
+ // Create a map of Mailchimp interest names to Civi Groups.
+ $map = [];
+ foreach ($this->interest_group_details as $group_id => $details) {
+ if ($details['is_mc_update_grouping'] == 1) {
+ // This group is configured to allow updates from Mailchimp to CiviCRM.
+ $map[$details['interest_name']] = $group_id;
+ }
+ }
+ // Sort longest strings first.
+ uksort($map, function($a, $b) { return strlen($a) - strlen($b); });
+
+ // Remove the found titles longest first.
+ $groups = [];
+ $group_input = ",$group_input,";
+ foreach ($map as $interest_name => $civi_group_id) {
+ $i = strpos($group_input, ",$interest_name,");
+ if ($i !== FALSE) {
+ $groups[] = $civi_group_id;
+ // Remove this from the string.
+ $group_input = substr($group_input, 0, $i+1) . substr($group_input, $i + strlen(",$interest_group_details,"));
+ }
+ }
+
+ return $groups;
+ }
+ /**
+ * Get list of emails to unsubscribe.
+ *
+ * We *exclude* any emails in Mailchimp that matched multiple contacts in
+ * CiviCRM - these have their cid_guess field set to NULL.
+ *
+ * @return array
+ */
+ public function getEmailsNotInCiviButInMailchimp() {
+ $dao = CRM_Core_DAO::executeQuery(
+ "SELECT m.email
+ FROM tmp_mailchimp_push_m m
+ WHERE cid_guess IS NOT NULL
+ AND NOT EXISTS (
+ SELECT c.contact_id FROM tmp_mailchimp_push_c c WHERE c.contact_id = m.cid_guess
+ );");
+
+ $emails = [];
+ while ($dao->fetch()) {
+ $emails[] = $dao->email;
+ }
+ return $emails;
+ }
+ /**
+ * Return a count of the members on Mailchimp from the tmp_mailchimp_push_m
+ * table.
+ */
+ public function countMailchimpMembers() {
+ $dao = CRM_Core_DAO::executeQuery("SELECT COUNT(*) c FROM tmp_mailchimp_push_m");
+ $dao->fetch();
+ return $dao->c;
+ }
+
+ /**
+ * Return a count of the members on CiviCRM from the tmp_mailchimp_push_c
+ * table.
+ */
+ public function countCiviCrmMembers() {
+ $dao = CRM_Core_DAO::executeQuery("SELECT COUNT(*) c FROM tmp_mailchimp_push_c");
+ $dao->fetch();
+ return $dao->c;
+ }
+
+ /**
+ * Sync a single contact's membership and interests for this list from their
+ * details in CiviCRM.
+ *
+ * @todo rename as push
+ */
+ public function syncSingleContact($contact_id) {
+
+ // Get all the groups related to this list that the contact is currently in.
+ // We have to use this dodgy API that concatenates the titles of the groups
+ // with a comma (making it unsplittable if a group title has a comma in it).
+ $contact = civicrm_api3('Contact', 'getsingle', [
+ 'contact_id' => $contact_id,
+ 'return' => ['first_name', 'last_name', 'email_id', 'email', 'group'],
+ 'sequential' => 1
+ ]);
+
+ $in_groups = CRM_Mailchimp_Utils::splitGroupTitles($contact['groups'], $this->group_details);
+ $currently_a_member = in_array($this->membership_group_id, $in_groups);
+
+ if (empty($contact['email'])) {
+ // Without an email we can't do anything.
+ return;
+ }
+ $subscriber_hash = md5(strtolower($contact['email']));
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+
+ if (!$currently_a_member) {
+ // They are not currently a member.
+ //
+ // We should ensure they are unsubscribed from Mailchimp. They might
+ // already be, but as we have no way of telling exactly what just changed
+ // at our end, we have to make sure.
+ //
+ // Nb. we don't bother updating their interests for unsubscribes.
+ try {
+ $result = $api->patch("/lists/$this->list_id/members/$subscriber_hash",
+ ['status' => 'unsubscribed']);
+ }
+ catch (CRM_Mailchimp_RequestErrorException $e) {
+ if ($e->response->http_code == 404) {
+ // OK. Mailchimp didn't know about them anyway. Fine.
+ }
+ else {
+ CRM_Core_Session::setStatus(ts('There was a problem trying to unsubscribe this contact at Mailchimp; any differences will remain until a CiviCRM to Mailchimp Sync is done.'));
+ }
+ }
+ catch (CRM_Mailchimp_NetworkErrorException $e) {
+ CRM_Core_Session::setStatus(ts('There was a network problem trying to unsubscribe this contact at Mailchimp; any differences will remain until a CiviCRM to Mailchimp Sync is done.'));
+ }
+ return;
+ }
+
+ // Now left with 'subscribe' case.
+ //
+ // Do this with a PUT as this allows for both updating existing and
+ // creating new members.
+ $data = [
+ 'status' => 'subscribed',
+ 'email_address' => $contact['email'],
+ 'merge_fields' => [
+ 'FNAME' => $contact['first_name'],
+ 'LNAME' => $contact['last_name'],
+ ],
+ ];
+ // Do interest groups.
+ $data['interests'] = $this->getComparableInterestsFromCiviCrmGroups($contact['groups']);
+ if (empty($data['interests'])) {
+ unset($data['interests']);
+ }
+ try {
+ $result = $api->put("/lists/$this->list_id/members/$subscriber_hash", $data);
+ }
+ catch (CRM_Mailchimp_RequestErrorException $e) {
+ CRM_Core_Session::setStatus(ts('There was a problem trying to subscribe this contact at Mailchimp:') . $e->getMessage());
+ }
+ catch (CRM_Mailchimp_NetworkErrorException $e) {
+ CRM_Core_Session::setStatus(ts('There was a network problem trying to unsubscribe this contact at Mailchimp; any differences will remain until a CiviCRM to Mailchimp Sync is done.'));
+ }
+ }
+ /**
+ * Identify a contact who is expected to be subscribed to this list.
+ *
+ * This is used in a couple of cases, for finding a contact from incomming
+ * data for:
+ * - a possibly new contact,
+ * - a contact that is expected to be in this membership group.
+ *
+ * Here's how we match a contact:
+ *
+ * - Only non-deleted contacts are returned.
+ *
+ * - Email is unique in CiviCRM
+ * Contact identified, unless limited to in-group only and not in group.
+ *
+ * - Email is entered 2+ times, but always on the same contact.
+ * Contact identified, unless limited to in-group only and not in group.
+ *
+ * - Email belongs to 2+ different contacts. In this situation, if there are
+ * some contacts that are in the membership group, we ignore the other match
+ * candidates. If limited to in-group contacts and there aren't any, we give
+ * up now.
+ *
+ * - Email identified if it belongs to only one contact that is in the
+ * membership list.
+ *
+ * - Look to the candidates whose last name matches.
+ * - Email identified if there's only one last name match.
+ * - If there are any contacts that also match first name, return one of
+ * these. We say it doesn't matter if there's duplicates - just pick
+ * one since everything matches.
+ *
+ * - Email identified if there's a single contact that matches on first
+ * name.
+ *
+ * We fail with a CRM_Mailchimp_DuplicateContactsException if the email
+ * belonged to several contacts and we could not narrow it down by name.
+ *
+ * @param string $email
+ * @param string|null $first_name
+ * @param string|null $last_name
+ * @param bool $must_be_on_list If TRUE, only return an ID if this contact
+ * is known to be on the list. defaults to
+ * FALSE.
+ * @throw CRM_Mailchimp_DuplicateContactsException if the email is known bit
+ * it fails to identify one contact.
+ * @return int|null Contact Id if found.
+ */
+ public function guessContactIdSingle($email, $first_name=NULL, $last_name=NULL, $must_be_on_list=FALSE) {
+
+ // API call returns all matching emails, and all contacts attached to those
+ // emails IF the contact is in our group.
+ $result = civicrm_api3('Email', 'get', [
+ 'sequential' => 1,
+ 'email' => $email,
+ 'api.Contact.get' => [
+ 'is_deleted' => 0,
+ 'return' => "first_name,last_name"],
+ ]);
+
+ // Candidates are any emails that belong to a not-deleted contact.
+ $email_candidates = array_filter($result['values'], function($_) {
+ return ($_['api.Contact.get']['count'] == 1);
+ });
+ if (count($email_candidates) == 0) {
+ // Never seen that email, mate.
+ return NULL;
+ }
+
+ // $email_candidates is currently a sequential list of emails. Instead map it to
+ // be indexed by contact_id.
+ $candidates = [];
+ foreach ($email_candidates as $_) {
+ $candidates[$_['contact_id']] = $_['api.Contact.get']['values'][0];
+ }
+
+ // Now we need to know which, if any of these contacts is in the group.
+ // Build list of contact_ids.
+ $result = civicrm_api3('Contact', 'get', [
+ 'group' => $this->membership_group_id,
+ 'contact_id' => ['IN' => array_keys($candidates)],
+ 'return' => 'contact_id',
+ ]);
+ $in_group = $result['values'];
+
+ // If must be on the membership list, then reduce the candidates to just
+ // those on the list.
+ if ($must_be_on_list) {
+ $candidates = array_intersect_key($candidates, $in_group);
+ if (count($candidates) == 0) {
+ // This email belongs to a contact *not* in the group.
+ return NULL;
+ }
+ }
+
+ if (count($candidates) == 1) {
+ // If there's only one one contact match on this email anyway, then we can
+ // assume that's the person. (we make this assumption in
+ // guessContactIdsByUniqueEmail too.)
+ return key($candidates);
+ }
+
+ // Now we're left with the case that the email matched more than one
+ // different contact.
+
+ if (count($in_group) == 1) {
+ // There's only one contact that is in the membership group with this
+ // email, use that.
+ return key($in_group);
+ }
+
+ // The email belongs to multiple contacts.
+ if ($in_group) {
+ // There are multiple contacts that share the same email and several are
+ // in this group. Narrow our serach to just those in the group.
+ $candidates = array_intersect_key($candidates, $in_group);
+ }
+
+ // Make indexes on names.
+ $last_name_matches = $first_name_matches = [];
+ foreach ($candidates as $candidate) {
+ if (!empty($candidate['first_name']) && ($first_name == $candidate['first_name'])) {
+ $first_name_matches[$candidate['contact_id']] = $candidate;
+ }
+ if (!empty($candidate['last_name']) && ($last_name == $candidate['last_name'])) {
+ $last_name_matches[$candidate['contact_id']] = $candidate;
+ }
+ }
+
+ // Now see if we can find them by name match.
+ if ($last_name_matches) {
+ // Some of the contacts have the same last name.
+ if (count($last_name_matches) == 1) {
+ // Only one contact with this email has the same last name, let's say
+ // it's them.
+ return key($last_name_matches);
+ }
+ // Multiple contacts with same last name. Reduce by same first name.
+ $last_name_matches = array_intersect_key($last_name_matches, $first_name_matches);
+ if (count($last_name_matches) > 0) {
+ // Either there was only one with same last and first name.
+ // Or, there were multiple contacts, but they have the same email and
+ // name so let's say that we're safe enough to pick the first one of
+ // them.
+ return key($last_name_matches);
+ }
+ }
+ // Last name didn't get there. Final chance. If the email and first name
+ // match a single contact, we'll grudgingly(!) say that's OK.
+ if (count($first_name_matches) == 1) {
+ // Only one contact with this email has the same first name, let's say
+ // it's them.
+ return key($first_name_matches);
+ }
+
+ // The email given belonged to several contacts and we were unable to narrow
+ // it down by the names, either. There's nothing we can do here, it's going
+ // to get messy.
+ throw new CRM_Mailchimp_DuplicateContactsException($candidates);
+ }
+ /**
+ * Guess the contact id for contacts whose email is found in the temporary
+ * table made by collectCiviCrm.
+ *
+ * If collectCiviCrm has been run, then we can identify matching contacts very
+ * easily. This avoids problems with multiple contacts in CiviCRM having the
+ * same email address but only one of them is subscribed. :-)
+ *
+ * **WARNING** it would be dangerous to run this if collectCiviCrm() had been run
+ * on a different list(!). For this reason, these conditions are checked by
+ * collectMailchimp().
+ *
+ * This is in a separate method so it can be tested.
+ *
+ * @return int affected rows.
+ */
+ public static function guessContactIdsBySubscribers() {
+ return static::runSqlReturnAffectedRows(
+ "UPDATE tmp_mailchimp_push_m m
+ INNER JOIN tmp_mailchimp_push_c c ON m.email = c.email
+ SET m.cid_guess = c.contact_id
+ WHERE m.cid_guess IS NULL");
+ }
+
+ /**
+ * Guess the contact id by there only being one email in CiviCRM that matches.
+ *
+ * Change in v2.0: it now checks uniqueness by contact id, so if the same
+ * email belongs multiple times to one contact, we can still conclude we've
+ * got the right contact.
+ *
+ * This is in a separate method so it can be tested.
+ * @return int affected rows.
+ */
+ public static function guessContactIdsByUniqueEmail() {
+ // If an address is unique, that's the one we need.
+ return static::runSqlReturnAffectedRows(
+ "UPDATE tmp_mailchimp_push_m m
+ INNER JOIN (
+ SELECT email, c.id AS contact_id
+ FROM civicrm_email e
+ JOIN civicrm_contact c ON e.contact_id = c.id AND c.is_deleted = 0
+ GROUP BY email
+ HAVING COUNT(DISTINCT c.id)=1
+ ) uniques ON m.email = uniques.email
+ SET m.cid_guess = uniques.contact_id
+ ");
+ }
+ /**
+ * Guess the contact id for contacts whose only email matches.
+ *
+ * This is in a separate method so it can be tested.
+ * See issue #188
+ *
+ * v2 includes rewritten SQL because of a bug that caused the test to fail.
+ * @return int affected rows.
+ */
+ public static function guessContactIdsByNameAndEmail() {
+
+ // In the other case, if we find a unique contact with matching
+ // first name, last name and e-mail address, it is probably the one we
+ // are looking for as well.
+
+ // look for email and names that match where there's only one match.
+ return static::runSqlReturnAffectedRows(
+ "UPDATE tmp_mailchimp_push_m m
+ INNER JOIN (
+ SELECT email, first_name, last_name, c.id AS contact_id
+ FROM civicrm_email e
+ JOIN civicrm_contact c ON e.contact_id = c.id AND c.is_deleted = 0
+ GROUP BY email, first_name, last_name
+ HAVING COUNT(DISTINCT c.id)=1
+ ) uniques ON m.email = uniques.email AND m.first_name = uniques.first_name AND m.last_name = uniques.last_name
+ SET m.cid_guess = uniques.contact_id
+ WHERE m.first_name != '' AND m.last_name != ''
+ ");
+ }
+ /**
+ * Drop tmp_mailchimp_push_m and tmp_mailchimp_push_c, if they exist.
+ *
+ * Those tables are created by collectMailchimp() and collectCiviCrm()
+ * for the purposes of syncing to/from Mailchimp/CiviCRM and are not needed
+ * outside of those operations.
+ */
+ public static function dropTemporaryTables() {
+ CRM_Core_DAO::executeQuery("DROP TABLE IF EXISTS tmp_mailchimp_push_m;");
+ CRM_Core_DAO::executeQuery("DROP TABLE IF EXISTS tmp_mailchimp_push_c;");
+ }
+ /**
+ * Drop mailchimp_log table if it exists.
+ *
+ * This table holds errors from multiple lists in Mailchimp where the contact
+ * could not be identified in CiviCRM; typically these contacts are
+ * un-sync-able ("Titanics").
+ */
+ public static function dropLogTable() {
+ CRM_Core_DAO::executeQuery("DROP TABLE IF EXISTS mailchimp_log;");
+ }
+ /**
+ * Create new tmp_mailchimp_push_m.
+ *
+ * Nb. these are temporary tables but we don't use TEMPORARY table because
+ * they are needed over multiple sessions because of queue.
+ *
+ *
+ * cid_guess column is the contact id that this record will be sync-ed to.
+ * It after both collections and a matchMailchimpMembersToContacts call it
+ * will be
+ *
+ * - A contact id
+ * - Zero meaning we can create a new contact
+ * - NULL meaning we must ignore this because otherwise we might end up
+ * making endless duplicates.
+ *
+ * Because a lot of matching is done on this, it has an index. Nb. a test was
+ * done trying the idea of adding the non-unique key at the end of the
+ * collection; heavily-keyed tables can slow down mass-inserts, so sometimes's
+ * it's quicker to add an index after an update. However this only saved 0.1s
+ * over 5,000 records import, so this code was removed for the sake of KISS.
+ *
+ * The speed of collecting from Mailchimp, is, as you might expect, determined
+ * by Mailchimp's API which seems to take about 3s for 1,000 records.
+ * Inserting them into the tmp table takes about 1s per 1,000 records on my
+ * server, so about 4s/1000 members.
+ */
+ public static function createTemporaryTableForMailchimp() {
+ CRM_Core_DAO::executeQuery( "DROP TABLE IF EXISTS tmp_mailchimp_push_m;");
+ $dao = CRM_Core_DAO::executeQuery(
+ "CREATE TABLE tmp_mailchimp_push_m (
+ email VARCHAR(200) NOT NULL,
+ first_name VARCHAR(100) NOT NULL,
+ last_name VARCHAR(100) NOT NULL,
+ hash CHAR(32) NOT NULL,
+ interests VARCHAR(4096) NOT NULL,
+ cid_guess INT(10) DEFAULT NULL,
+ PRIMARY KEY (email, hash),
+ KEY (cid_guess))
+ ENGINE=InnoDB DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ;");
+
+ // Convenience in collectMailchimp.
+ return $dao;
+ }
+ /**
+ * Create new tmp_mailchimp_push_c.
+ *
+ * Nb. these are temporary tables but we don't use TEMPORARY table because
+ * they are needed over multiple sessions because of queue.
+ */
+ public static function createTemporaryTableForCiviCRM() {
+ CRM_Core_DAO::executeQuery( "DROP TABLE IF EXISTS tmp_mailchimp_push_c;");
+ $dao = CRM_Core_DAO::executeQuery("CREATE TABLE tmp_mailchimp_push_c (
+ contact_id INT(10) UNSIGNED NOT NULL,
+ email VARCHAR(200) NOT NULL,
+ first_name VARCHAR(100) NOT NULL,
+ last_name VARCHAR(100) NOT NULL,
+ hash CHAR(32) NOT NULL,
+ interests VARCHAR(4096) NOT NULL,
+ PRIMARY KEY (email, hash),
+ KEY (contact_id)
+ )
+ ENGINE=InnoDB DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ;");
+ return $dao;
+ }
+ /**
+ * Logic to determine update needed.
+ *
+ * This is separate from the method that collects a batch update so that it
+ * can be tested more easily.
+ *
+ * @param array $merge_fields an array where the *keys* are 'tag' names from
+ * Mailchimp's merge_fields resource. e.g. FNAME, LNAME.
+ * @param array $civi_details Array of civicrm details from
+ * tmp_mailchimp_push_c
+ * @param array $mailchimp_details Array of mailchimp details from
+ * tmp_mailchimp_push_m
+ * @return array changes in format required by Mailchimp API.
+ */
+ public static function updateMailchimpFromCiviLogic($merge_fields, $civi_details, $mailchimp_details) {
+
+ $params = [];
+ // I think possibly some installations don't have Multibyte String Functions
+ // installed?
+ $lower = function_exists('mb_strtolower') ? 'mb_strtolower' : 'strtolower';
+
+ if ($civi_details['email'] && $lower($civi_details['email']) != $lower($mailchimp_details['email'])) {
+ // This is the case for additions; when we're adding someone new.
+ $params['email_address'] = $civi_details['email'];
+ }
+
+ if ($civi_details['interests'] && $civi_details['interests'] != $mailchimp_details['interests']) {
+ // Civi's Interest field will unpack to an empty array if we don't have
+ // any mapped interest groups. In this case we don't need to send the
+ // interests to Mailchimp at all, so we check for that.
+ // In the case of adding a new person from CiviCRM to Mailchimp, the
+ // Mailchimp interests passed in will be empty, but the CiviCRM one will
+ // be 'a:0:{}' since that is the serialized version of [].
+ $interests = unserialize($civi_details['interests']);
+ if (!empty($interests)) {
+ $params['interests'] = $interests;
+ }
+ }
+
+ $name_changed = FALSE;
+ if ($civi_details['first_name'] && $civi_details['first_name'] != $mailchimp_details['first_name']) {
+ $name_changed = TRUE;
+ // First name mismatch.
+ if (isset($merge_fields['FNAME'])) {
+ // FNAME field exists, so set it.
+ $params['merge_fields']['FNAME'] = $civi_details['first_name'];
+ }
+ }
+ if ($civi_details['last_name'] && $civi_details['last_name'] != $mailchimp_details['last_name']) {
+ $name_changed = TRUE;
+ if (isset($merge_fields['LNAME'])) {
+ // LNAME field exists, so set it.
+ $params['merge_fields']['LNAME'] = $civi_details['last_name'];
+ }
+ }
+ if ($name_changed && key_exists('NAME', $merge_fields)) {
+ // The name was changed and this list has a NAME field. Supply first last
+ // names to this field.
+ $params['merge_fields']['NAME'] = trim("$civi_details[first_name] $civi_details[last_name]");
+ }
+
+ return $params;
+ }
+
+ /**
+ * Logic to determine update needed for pull.
+ *
+ * This is separate from the method that collects a batch update so that it
+ * can be tested more easily.
+ *
+ * @param array $mailchimp_details Array of mailchimp details from
+ * tmp_mailchimp_push_m, with keys first_name, last_name
+ * @param array $civi_details Array of civicrm details from
+ * tmp_mailchimp_push_c, with keys first_name, last_name
+ * @return array changes in format required by Mailchimp API.
+ */
+ public static function updateCiviFromMailchimpContactLogic($mailchimp_details, $civi_details) {
+
+ $edits = [];
+
+ foreach (['first_name', 'last_name'] as $field) {
+ if ($mailchimp_details[$field] && $mailchimp_details[$field] != $civi_details[$field]) {
+ $edits[$field] = $mailchimp_details[$field];
+ }
+ }
+
+ return $edits;
+ }
+
+ /**
+ * There's probably a better way to do this.
+ */
+ public static function runSqlReturnAffectedRows($sql, $params) {
+ $dao = new CRM_Core_DAO();
+ $q = CRM_Core_DAO::composeQuery($sql, $params);
+ $result = $dao->query($q);
+ if (is_a($result, 'DB_Error')) {
+ throw new Exception ($result->message . "\n" . $result->userinfo);
+ }
+ $dao->free();
+ return $result;
+ }
+}
+
diff --git a/CRM/Mailchimp/Upgrader.php b/CRM/Mailchimp/Upgrader.php
index 05961de..2eb6da9 100644
--- a/CRM/Mailchimp/Upgrader.php
+++ b/CRM/Mailchimp/Upgrader.php
@@ -72,6 +72,133 @@ public function upgrade_17() {
return TRUE;
}
+ /**
+ * Mailchimp in their wisdom changed all the Ids for interests.
+ *
+ * So we have to map on names and then update our stored Ids.
+ *
+ * Also change cronjobs.
+ */
+ public function upgrade_20() {
+ $this->ctx->log->info('Applying update to v2.0 Updating Mailchimp Interest Ids to fit their new API');
+ // New
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+ // Old
+ $mcLists = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
+
+ // Use new API to get lists. Allow for 10,000 lists so we don't bother
+ // batching.
+ $lists = [];
+ foreach ($api->get("/lists", ['fields'=>'lists.id,lists.name','count'=>10000])->data->lists
+ as $list) {
+ $lists[$list->id] = ['name' => $list->name];
+ }
+
+ $queries = [];
+ // Loop lists.
+ foreach (array_keys($lists) as $list_id) {
+ // Fetch Interest categories.
+ $categories = $api->get("/lists/$list_id/interest-categories", ['count' => 10000, 'fields' => 'categories.id,categories.title'])->data->categories;
+ if (!$categories) {
+ continue;
+ }
+
+ // Old: fetch all categories (groupings) and interests (groups) in one go:
+ $old = $mcLists->interestGroupings($list_id);
+ // New: fetch interests for each category.
+ foreach ($categories as $category) {
+ // $lists[$list_id]['categories'][$category->id] = ['name' => $category->title];
+
+ // Match this category by name with the old 'groupings'
+ $matched_old_grouping = FALSE;
+ foreach($old as $old_grouping) {
+ if ($old_grouping['name'] == $category->title) {
+ $matched_old_grouping = $old_grouping;
+ break;
+ }
+ }
+ if ($matched_old_grouping) {
+ // Found a match.
+ $cat_queries []= ['list_id' => $list_id, 'old' => $matched_old_grouping['id'], 'new' => $category->id];
+
+ // Now do interests (old: groups)
+ $interests = $api->get("/lists/$list_id/interest-categories/$category->id/interests", ['fields'=>'interests.id,interests.name','count'=>10000])->data->interests;
+ foreach ($interests as $interest) {
+ // Can we find this interest by name?
+ $matched_old_group = FALSE;
+ foreach($matched_old_grouping['groups'] as $old_group) {
+ if ($old_group['name'] == $interest->name) {
+ $int_queries []= ['list_id' => $list_id, 'old' => $old_group['id'], 'new' => $interest->id];
+ break;
+ }
+ }
+ }
+ }
+ }
+ }
+
+ foreach ($cat_queries as $params) {
+ CRM_Core_DAO::executeQuery('UPDATE civicrm_value_mailchimp_settings '
+ . 'SET mc_grouping_id = %1 '
+ . 'WHERE mc_list_id = %2 AND mc_grouping_id = %3;'
+ , [
+ 1 => [$params['new'], 'String'],
+ 2 => [$params['list_id'], 'String'],
+ 3 => [$params['old'], 'String'],
+ ]);
+ }
+
+ foreach ($int_queries as $params) {
+ CRM_Core_DAO::executeQuery('UPDATE civicrm_value_mailchimp_settings '
+ . 'SET mc_group_id = %1 '
+ . 'WHERE mc_list_id = %2 AND mc_group_id = %3;'
+ , [
+ 1 => [$params['new'], 'String'],
+ 2 => [$params['list_id'], 'String'],
+ 3 => [$params['old'], 'String'],
+ ]);
+ }
+
+ // Now cron jobs. Delete all mailchimp ones.
+ $result = civicrm_api3('Job', 'get', array(
+ 'sequential' => 1,
+ 'api_entity' => "mailchimp",
+ ));
+ if ($result['count']) {
+ // Should only be one, but just in case...
+ foreach ($result['values'] as $old) {
+ // Double check id exists!
+ if (!empty($old['id'])) {
+ civicrm_api3('Job', 'delete', ['id' => $old['id']]);
+ }
+ }
+ }
+
+ // Create Push Sync job.
+ $params = array(
+ 'sequential' => 1,
+ 'name' => 'Mailchimp Push Sync',
+ 'description' => 'Sync contacts between CiviCRM and MailChimp, assuming CiviCRM to be correct. Please understand the implications before using this.',
+ 'run_frequency' => 'Daily',
+ 'api_entity' => 'Mailchimp',
+ 'api_action' => 'pushsync',
+ 'is_active' => 0,
+ );
+ $result = civicrm_api3('job', 'create', $params);
+ // Create Pull Sync job.
+ $params = array(
+ 'sequential' => 1,
+ 'name' => 'Mailchimp Pull Sync',
+ 'description' => 'Sync contacts between CiviCRM and MailChimp, assuming Mailchimp to be correct. Please understand the implications before using this.',
+ 'run_frequency' => 'Daily',
+ 'api_entity' => 'Mailchimp',
+ 'api_action' => 'pullsync',
+ 'is_active' => 0,
+ );
+ $result = civicrm_api3('job', 'create', $params);
+
+ return TRUE;
+ }
/**
* Example: Run an external SQL script
*
diff --git a/CRM/Mailchimp/Upgrader/Base.php b/CRM/Mailchimp/Upgrader/Base.php
index 512fda2..17da06b 100644
--- a/CRM/Mailchimp/Upgrader/Base.php
+++ b/CRM/Mailchimp/Upgrader/Base.php
@@ -8,7 +8,7 @@
class CRM_Mailchimp_Upgrader_Base {
/**
- * @var varies, subclass of htis
+ * @var varies, subclass of ttis
*/
static $instance;
@@ -33,7 +33,7 @@ class CRM_Mailchimp_Upgrader_Base {
private $revisions;
/**
- * Obtain a refernece to the active upgrade handler
+ * Obtain a reference to the active upgrade handler.
*/
static public function instance() {
if (! self::$instance) {
@@ -73,7 +73,7 @@ public function __construct($extensionName, $extensionDir) {
// ******** Task helpers ********
/**
- * Run a CustomData file
+ * Run a CustomData file.
*
* @param string $relativePath the CustomData XML file path (relative to this extension's dir)
* @return bool
@@ -87,6 +87,7 @@ public function executeCustomDataFile($relativePath) {
* Run a CustomData file
*
* @param string $xml_file the CustomData XML file path (absolute path)
+ *
* @return bool
*/
protected static function executeCustomDataFileByAbsPath($xml_file) {
@@ -97,9 +98,10 @@ protected static function executeCustomDataFileByAbsPath($xml_file) {
}
/**
- * Run a SQL file
+ * Run a SQL file.
*
* @param string $relativePath the SQL file path (relative to this extension's dir)
+ *
* @return bool
*/
public function executeSqlFile($relativePath) {
@@ -111,7 +113,7 @@ public function executeSqlFile($relativePath) {
}
/**
- * Run one SQL query
+ * Run one SQL query.
*
* This is just a wrapper for CRM_Core_DAO::executeSql, but it
* provides syntatic sugar for queueing several tasks that
@@ -119,13 +121,14 @@ public function executeSqlFile($relativePath) {
*/
public function executeSql($query, $params = array()) {
// FIXME verify that we raise an exception on error
- CRM_Core_DAO::executeSql($query, $params);
+ CRM_Core_DAO::executeQuery($query, $params);
return TRUE;
}
/**
- * Syntatic sugar for enqueuing a task which calls a function
- * in this class. The task is weighted so that it is processed
+ * Syntatic sugar for enqueuing a task which calls a function in this class.
+ *
+ * The task is weighted so that it is processed
* as part of the currently-pending revision.
*
* After passing the $funcName, you can also pass parameters that will go to
@@ -145,7 +148,7 @@ public function addTask($title) {
// ******** Revision-tracking helpers ********
/**
- * Determine if there are any pending revisions
+ * Determine if there are any pending revisions.
*
* @return bool
*/
@@ -164,7 +167,7 @@ public function hasPendingRevisions() {
}
/**
- * Add any pending revisions to the queue
+ * Add any pending revisions to the queue.
*/
public function enqueuePendingRevisions(CRM_Queue_Queue $queue) {
$this->queue = $queue;
@@ -197,7 +200,7 @@ public function enqueuePendingRevisions(CRM_Queue_Queue $queue) {
}
/**
- * Get a list of revisions
+ * Get a list of revisions.
*
* @return array(revisionNumbers) sorted numerically
*/
@@ -287,11 +290,13 @@ public function onDisable() {
}
public function onUpgrade($op, CRM_Queue_Queue $queue = NULL) {
- switch($op) {
+ switch ($op) {
case 'check':
return array($this->hasPendingRevisions());
+
case 'enqueue':
return $this->enqueuePendingRevisions($queue);
+
default:
}
}
diff --git a/CRM/Mailchimp/Utils.php b/CRM/Mailchimp/Utils.php
index 6cfc1ac..fddb2ad 100644
--- a/CRM/Mailchimp/Utils.php
+++ b/CRM/Mailchimp/Utils.php
@@ -3,43 +3,331 @@
class CRM_Mailchimp_Utils {
const MC_SETTING_GROUP = 'MailChimp Preferences';
- static function mailchimp() {
- $apiKey = CRM_Core_BAO_Setting::getItem(CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'api_key');
- $mcClient = new Mailchimp($apiKey);
- //CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils mailchimp $mcClient', $mcClient);
- return $mcClient;
+
+ /** Mailchimp API object to use. */
+ static protected $mailchimp_api;
+
+ /** Holds runtime cache of group details */
+ static protected $mailchimp_interest_details = [];
+
+ /** Holds a cache of list names from Mailchimp */
+ static protected $mailchimp_lists;
+
+ /**
+ * Checked by mailchimp_civicrm_post before it acts on anything.
+ *
+ * That post hook might send requests to Mailchimp's API, but in the cases
+ * where we're responding to data from Mailchimp, this could possibly result
+ * in a loop, so we have a central on/off switch here.
+ *
+ * In previous versions it was a session variable, but this is not necessary.
+ */
+ public static $post_hook_enabled = TRUE;
+
+ /**
+ * Split a string of group titles into an array of groupIds.
+ *
+ * The Contact:get API is the only place you can get a list of all the groups
+ * (smart and normal) that a contact has membership of. But it returns them as
+ * a comma separated string. You can't split on a comma because there is no
+ * restriction on commas in group titles. So instead we take a list of
+ * candidate titles and look for those.
+ *
+ * This function solves the problem of:
+ * Group name: "Sponsored walk, 2015"
+ * Group name: "Sponsored walk"
+ *
+ * Contact 1's groups: "Sponsored walk,Sponsored walk, 2015"
+ * This contact is in both groups.
+ *
+ * Contact 2's groups: "Sponsored walk"
+ * This contact is only in the one group.
+ *
+ * If we just split on comma then the contacts would only be in the "sponsored
+ * walk" group and never the one with the comma in.
+ *
+ * @param string $group_titles As output by the CiviCRM api for a contact when
+ * you request the 'group' output (which comes in a key called 'groups').
+ * @param array $group_details As from CRM_Mailchimp_Utils::getGroupsToSync
+ * but only including groups you're interested in.
+ * @return array CiviCRM groupIds.
+ */
+ public static function splitGroupTitles($group_titles, $group_details) {
+ $groups = [];
+
+ // Sort the group titles by length, longest first.
+ uasort($group_details, function($a, $b) {
+ return (strlen($b['civigroup_title']) - strlen($a['civigroup_title']));
+ });
+ // Remove the found titles longest first.
+ $group_titles = ",$group_titles,";
+
+ foreach ($group_details as $civi_group_id => $detail) {
+ $i = strpos($group_titles, ",$detail[civigroup_title],");
+ if ($i !== FALSE) {
+ $groups[] = $civi_group_id;
+ // Remove this from the string.
+ $group_titles = substr($group_titles, 0, $i+1) . substr($group_titles, $i + strlen(",$detail[civigroup_title],"));
+ }
+ }
+ return $groups;
+ }
+ /**
+ * Returns the webhook URL.
+ */
+ public static function getWebhookUrl() {
+ $security_key = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP, 'security_key', NULL, FALSE);
+ if (empty($security_key)) {
+ // @Todo what exception should this throw?
+ throw new InvalidArgumentException("You have not set a security key for your Mailchimp integration. Please do this on the settings page at civicrm/mailchimp/settings");
+ }
+ $webhook_url = CRM_Utils_System::url('civicrm/mailchimp/webhook',
+ $query = 'reset=1&key=' . urlencode($security_key),
+ $absolute = TRUE,
+ $fragment = NULL,
+ $htmlize = FALSE,
+ $fronteend = TRUE);
+
+ return $webhook_url;
+ }
+ /**
+ * Returns an API class for talking to Mailchimp.
+ *
+ * This is a singleton pattern with a factory method to create an object of
+ * the normal API class. You can set the Api object with
+ * CRM_Mailchimp_Utils::setMailchimpApi() which is essential for being able to
+ * passin mocks for testing.
+ *
+ * @param bool $reset If set it will replace the API object with a default.
+ * Only useful after changing stored credentials.
+ */
+ public static function getMailchimpApi($reset=FALSE) {
+ if ($reset) {
+ static::$mailchimp_api = NULL;
+ }
+
+ // Singleton pattern.
+ if (!isset(static::$mailchimp_api)) {
+ $params = ['api_key' => CRM_Core_BAO_Setting::getItem(CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'api_key')];
+ $debugging = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP, 'enable_debugging', NULL, FALSE);
+ if ($debugging == 1) {
+ // We want debugging. Inject a logging callback.
+ $params['log_facility'] = function($message) {
+ CRM_Core_Error::debug_log_message($message, FALSE, 'mailchimp');
+ };
+ }
+ $api = new CRM_Mailchimp_Api3($params);
+ static::setMailchimpApi($api);
+ }
+
+ return static::$mailchimp_api;
}
+ /**
+ * Set the API object.
+ *
+ * This is for testing purposes only.
+ */
+ public static function setMailchimpApi(CRM_Mailchimp_Api3 $api) {
+ static::$mailchimp_api = $api;
+ }
+
+ /**
+ * Reset caches.
+ */
+ public static function resetAllCaches() {
+ static::$mailchimp_api = NULL;
+ static::$mailchimp_lists = NULL;
+ static::$mailchimp_interest_details = [];
+ }
+ /**
+ * Check all mapped groups' lists.
+ *
+ * Nb. this does not output anything itself so we can test it works. It is
+ * used by the settings page.
+ *
+ * @param null|Array $groups array of membership groups to check, or NULL to
+ * check all.
+ *
+ * @return Array of message strings that should be output with CRM_Core_Error
+ * or such.
+ *
+ */
+ public static function checkGroupsConfig($groups=NULL) {
+ if ($groups === NULL) {
+ $groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), null, $membership_only = TRUE);
+ }
+ if (!is_array($groups)) {
+ throw new InvalidArgumentException("expected array argument, if provided");
+ }
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+
+ $warnings = [];
+ // Check all our groups do not have the sources:API set in the webhook, and
+ // that they do have the webhook set.
+ foreach ($groups as $group_id => $details) {
+
+ $group_settings_link = ""
+ . htmlspecialchars($details['civigroup_title']) . "";
+
+ $message_prefix = ts('CiviCRM group "%1" (Mailchimp list %2): ',
+ [1 => $group_settings_link, 2 => $details['list_id']]);
+
+ try {
+ $test_warnings = CRM_Mailchimp_Utils::configureList($details['list_id'], $dry_run=TRUE);
+ foreach ($test_warnings as $_) {
+ $warnings []= $message_prefix . $_;
+ }
+ }
+ catch (CRM_Mailchimp_NetworkErrorException $e) {
+ $warnings []= $message_prefix . ts("Problems (possibly temporary) fetching details from Mailchimp. ") . $e->getMessage();
+ }
+ catch (CRM_Mailchimp_RequestErrorException $e) {
+ $message = $e->getMessage();
+ if ($e->response->http_code == 404) {
+ // A little more helpful than "resource not found".
+ $warnings []= $message_prefix . ts("The Mailchimp list that this once worked with has "
+ ."been deleted on Mailchimp. Please edit the CiviCRM group settings to "
+ ."either specify a different Mailchimp list that exists, or to remove "
+ ."the Mailchimp integration for this group.");
+ }
+ else {
+ $warnings []= $message_prefix . ts("Problems fetching details from Mailchimp. ") . $e->getMessage();
+ }
+ }
+ }
+
+ if ($warnings) {
+ CRM_Core_Error::debug_log_message('Mailchimp list check warnings' . var_export($warnings,1));
+ }
+ return $warnings;
+ }
+ /**
+ * Configure webhook with Mailchimp.
+ *
+ * Returns a list of messages to display to the user.
+ *
+ * @param string $list_id Mailchimp List Id.
+ * @param bool $dry_run If set no changes are made.
+ * @return array
+ */
+ public static function configureList($list_id, $dry_run = FALSE) {
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+ $expected = [
+ 'url' => CRM_Mailchimp_Utils::getWebhookUrl(),
+ 'events' => [
+ 'subscribe' => TRUE,
+ 'unsubscribe' => TRUE,
+ 'profile' => TRUE,
+ 'cleaned' => TRUE,
+ 'upemail' => TRUE,
+ 'campaign' => FALSE,
+ ],
+ 'sources' => [
+ 'user' => TRUE,
+ 'admin' => TRUE,
+ 'api' => FALSE,
+ ],
+ ];
+ $verb = $dry_run ? 'Need to change ' : 'Changed ';
+ try {
+ $result = $api->get("/lists/$list_id/webhooks");
+ $webhooks = $result->data->webhooks;
+ //$webhooks = $api->get("/lists/$list_id/webhooks")->data->webhooks;
+
+ if (empty($webhooks)) {
+ $messages []= ts(($dry_run ? 'Need to create' : 'Created') .' a webhook at Mailchimp');
+ }
+ else {
+ // Existing webhook(s) - check thoroughly.
+ if (count($webhooks) > 1) {
+ // Unusual case, leave it alone.
+ $messages [] = "Mailchimp list $list_id has more than one webhook configured. This is unusual, and so CiviCRM has not made any changes. Please ensure the webhook is set up correctly.";
+ return $messages;
+ }
+
+ // Got a single webhook, check it looks right.
+ $messages = [];
+ // Correct URL?
+ if ($webhooks[0]->url != $expected['url']) {
+ $messages []= ts($verb . 'webhook URL from %1 to %2', [1 => $webhooks[0]->url, 2 => $expected['url']]);
+ }
+ // Correct sources?
+ foreach ($expected['sources'] as $source => $expected_value) {
+ if ($webhooks[0]->sources->$source != $expected_value) {
+ $messages []= ts($verb . 'webhook source %1 from %2 to %3', [1 => $source, 2 => (int) $webhooks[0]->sources->$source, 3 => (int)$expected_value]);
+ }
+ }
+ // Correct events?
+ foreach ($expected['events'] as $event => $expected_value) {
+ if ($webhooks[0]->events->$event != $expected_value) {
+ $messages []= ts($verb . 'webhook event %1 from %2 to %3', [1 => $event, 2 => (int) $webhooks[0]->events->$event, 3 => (int) $expected_value]);
+ }
+ }
+
+ if (empty($messages)) {
+ // All fine.
+ return;
+ }
+
+ if (!$dry_run) {
+ // As of May 2016, there doesn't seem to be an update method for
+ // webhooks, so we just delete this and add another.
+ $api->delete("/lists/$list_id/webhooks/" . $webhooks[0]->id);
+ }
+ }
+ if (!$dry_run) {
+ // Now create the proper one.
+ $result = $api->post("/lists/$list_id/webhooks", $expected);
+ }
+
+ }
+ catch (CRM_Mailchimp_RequestErrorException $e) {
+ if ($e->request->method == 'GET' && $e->response->http_code == 404) {
+ $messages [] = ts("The Mailchimp list that this once worked with has been deleted");
+ }
+ else {
+ $messages []= ts("Problems updating or fetching from Mailchimp. Please manually check the configuration. ") . $e->getMessage();
+ }
+ }
+ catch (CRM_Mailchimp_NetworkErrorException $e) {
+ $messages []= ts("Problems (possibly temporary) talking to Mailchimp. ") . $e->getMessage();
+ }
+
+ return $messages;
+ }
/**
* Look up an array of CiviCRM groups linked to Maichimp groupings.
*
- * Indexed by CiviCRM groupId, including:
*
- * - list_id (MC)
- * - grouping_id(MC)
- * - group_id (MC)
- * - is_mc_update_grouping (bool) - is the subscriber allowed to update this via MC interface?
- * - group_name (MC)
- * - grouping_name (MC)
- * - civigroup_title
- * - civigroup_uses_cache boolean
*
* @param $groupIDs mixed array of CiviCRM group Ids to fetch data for; or empty to return ALL mapped groups.
* @param $mc_list_id mixed Fetch for a specific Mailchimp list only, or null.
* @param $membership_only bool. Only fetch mapped membership groups (i.e. NOT linked to a MC grouping).
- *
+ * @return array keyed by CiviCRM group id whose values are arrays of details
+ * including:
+ * // Details about Mailchimp
+ * 'list_id'
+ * 'list_name'
+ * 'category_id'
+ * 'category_name'
+ * 'interest_id'
+ * 'interest_name'
+ * // Details from CiviCRM
+ * 'civigroup_title'
+ * 'civigroup_uses_cache'
+ * 'is_mc_update_grouping' bool: is the subscriber allowed to update this
+ * via MC interface?
+ * // Deprecated DO NOT USE from Mailchimp.
+ * 'grouping_id'
+ * 'grouping_name'
+ * 'group_id'
+ * 'group_name'
*/
- static function getGroupsToSync($groupIDs = array(), $mc_list_id = null, $membership_only = FALSE) {
+ public static function getGroupsToSync($groupIDs = array(), $mc_list_id = null, $membership_only = FALSE) {
$params = $groups = $temp = array();
-
- foreach ($groupIDs as $value) {
- if($value){
- $temp[] = $value;
- }
- }
-
- $groupIDs = $temp;
+ $groupIDs = array_filter(array_map('intval',$groupIDs));
if (!empty($groupIDs)) {
$groupIDs = implode(',', $groupIDs);
@@ -67,16 +355,28 @@ static function getGroupsToSync($groupIDs = array(), $mc_list_id = null, $member
WHERE $whereClause";
$dao = CRM_Core_DAO::executeQuery($query, $params);
while ($dao->fetch()) {
+ $list_name = CRM_Mailchimp_Utils::getMCListName($dao->mc_list_id);
+ $interest_name = CRM_Mailchimp_Utils::getMCInterestName($dao->mc_list_id, $dao->mc_grouping_id, $dao->mc_group_id);
+ $category_name = CRM_Mailchimp_Utils::getMCCategoryName($dao->mc_list_id, $dao->mc_grouping_id);
$groups[$dao->entity_id] =
array(
+ // Details about Mailchimp
'list_id' => $dao->mc_list_id,
- 'grouping_id' => $dao->mc_grouping_id,
- 'group_id' => $dao->mc_group_id,
+ 'list_name' => $list_name,
+ 'category_id' => $dao->mc_grouping_id,
+ 'category_name' => $category_name,
+ 'interest_id' => $dao->mc_group_id,
+ 'interest_name' => $interest_name,
+ // Details from CiviCRM
'is_mc_update_grouping' => $dao->is_mc_update_grouping,
- 'group_name' => CRM_Mailchimp_Utils::getMCGroupName($dao->mc_list_id, $dao->mc_grouping_id, $dao->mc_group_id),
- 'grouping_name' => CRM_Mailchimp_Utils::getMCGroupingName($dao->mc_list_id, $dao->mc_grouping_id),
'civigroup_title' => $dao->civigroup_title,
'civigroup_uses_cache' => (bool) (($dao->saved_search_id > 0) || (bool) $dao->children),
+
+ // Deprecated from Mailchimp.
+ 'grouping_id' => $dao->mc_grouping_id,
+ 'grouping_name' => $category_name,
+ 'group_id' => $dao->mc_group_id,
+ 'group_name' => $interest_name,
);
}
@@ -85,83 +385,26 @@ static function getGroupsToSync($groupIDs = array(), $mc_list_id = null, $member
return $groups;
}
- static function getGroupIDsToSync() {
- $groupIDs = self::getGroupsToSync();
- return array_keys($groupIDs);
- }
-
- static function getMemberCountForGroupsToSync($groupIDs = array()) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupsToSync $groupIDs', $groupIDs);
- $group = new CRM_Contact_DAO_Group();
- foreach ($groupIDs as $key => $value) {
- $group->id = $value;
- }
- $group->find(TRUE);
-
- if (empty($groupIDs)) {
- $groupIDs = self::getGroupIDsToSync();
- }
- if(!empty($groupIDs) && $group->saved_search_id){
- $groupIDs = implode(',', $groupIDs);
- $smartGroupQuery = "
- SELECT count(*)
- FROM civicrm_group_contact_cache smartgroup_contact
- WHERE smartgroup_contact.group_id IN ($groupIDs)";
- $count = CRM_Core_DAO::singleValueQuery($smartGroupQuery);
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getMemberCountForGroupsToSync $count', $count);
- return $count;
-
-
- }
- else if (!empty($groupIDs)) {
- $groupIDs = implode(',', $groupIDs);
- $query = "
- SELECT count(*)
- FROM civicrm_group_contact
- WHERE status = 'Added' AND group_id IN ($groupIDs)";
- $count = CRM_Core_DAO::singleValueQuery($query);
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getMemberCountForGroupsToSync $count', $count);
- return $count;
- }
- return 0;
- }
-
/**
- * return the group name for given list, grouping and group
+ * Return the name at mailchimp for the given Mailchimp list id.
*
+ * @return string.
*/
- static function getMCGroupName($listID, $groupingID, $groupID) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getMCGroupName $listID', $listID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getMCGroupName $groupingID', $groupingID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getMCGroupName $groupID', $groupID);
-
- $info = static::getMCInterestGroupings($listID);
-
- // Check list, grouping, and group exist
- if (empty($info[$groupingID]['groups'][$groupID])) {
- return NULL;
+ public static function getMCListName($list_id) {
+ if (!isset(static::$mailchimp_lists)) {
+ static::$mailchimp_lists[$list_id] = [];
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+ $lists = $api->get('/lists', ['fields' => 'lists.id,lists.name','count'=>10000])->data->lists;
+ foreach ($lists as $list) {
+ static::$mailchimp_lists[$list->id] = $list->name;
+ }
}
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getMCGroupName $info', $info[$groupingID]['groups'][$groupID]['name']);
-
- return $info[$groupingID]['groups'][$groupID]['name'];
- }
-
- /**
- * Return the grouping name for given list, grouping MC Ids.
- */
- static function getMCGroupingName($listID, $groupingID) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getMCGroupingName $listID', $listID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getMCGroupingName $groupingID', $groupingID);
-
- $info = static::getMCInterestGroupings($listID);
- // Check list, grouping, and group exist
- if (empty($info[$groupingID])) {
- return NULL;
+ if (!isset(static::$mailchimp_lists[$list_id])) {
+ // Return ZLS if not found.
+ return '';
}
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getMCGroupingName $info ', $info[$groupingID]['name']);
-
- return $info[$groupingID]['name'];
+ return static::$mailchimp_lists[$list_id];
}
/**
@@ -170,79 +413,117 @@ static function getMCGroupingName($listID, $groupingID) {
* Nb. general API function used by several other helper functions.
*
* Returns an array like {
- * [groupingId] => array(
- * 'id' => [groupingId],
- * 'name' => ...,
- * 'form_field' => ..., (not v interesting)
- * 'display_order' => ..., (not v interesting)
- * 'groups' => array(
- * [MC groupId] => array(
- * 'id' => [MC groupId],
- * 'bit' => ..., ?
- * 'name' => ...,
- * 'display_order' => ...,
- * 'subscribers' => ..., ?
+ * [category_id] => array(
+ * 'id' => category_id,
+ * 'name' => Category name
+ * 'interests' => array(
+ * [interest_id] => array(
+ * 'id' => interest_id,
+ * 'name' => interest name
* ),
* ...
* ),
* ...
- * )
+ * )
*
*/
- static function getMCInterestGroupings($listID) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getMCInterestGroupings $listID', $listID);
+ public static function getMCInterestGroupings($listID) {
if (empty($listID)) {
+ CRM_Mailchimp_Utils::checkDebug('CRM_Mailchimp_Utils::getMCInterestGroupings called without list id');
return NULL;
}
- static $mapper = array();
+ $mapper = &static::$mailchimp_interest_details;
if (!array_key_exists($listID, $mapper)) {
$mapper[$listID] = array();
- $mcLists = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
try {
- $results = $mcLists->interestGroupings($listID);
-
+ // Get list name.
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
+ $categories = $api->get("/lists/$listID/interest-categories",
+ ['fields' => 'categories.id,categories.title','count'=>10000])
+ ->data->categories;
+ }
+ catch (CRM_Mailchimp_RequestErrorException $e) {
+ if ($e->response->http_code == 404) {
+ // Controlled response
+ CRM_Core_Error::debug_log_message("Mailchimp error: List $listID is not found.");
+ return NULL;
+ }
+ else {
+ CRM_Core_Error::debug_log_message('Unhandled Mailchimp error: ' . $e->getMessage());
+ throw $e;
+ }
}
- catch (Exception $e) {
+ catch (CRM_Mailchimp_NetworkErrorException $e) {
+ CRM_Core_Error::debug_log_message('Unhandled Mailchimp network error: ' . $e->getMessage());
+ throw $e;
return NULL;
}
- /* re-map $result for quick access via grouping_id and groupId
- *
- * Nb. keys for grouping:
- * - id
- * - name
- * - form_field (not v interesting)
- * - display_order (not v interesting)
- * - groups: array as follows, keyed by GroupId
- *
- * Keys for each group
- * - id
- * - bit ?
- * - name
- * - display_order
- * - subscribers ?
- *
- */
- foreach ($results as $grouping) {
-
-
- $mapper[$listID][$grouping['id']] = $grouping;
- unset($mapper[$listID][$grouping['id']]['groups']);
- foreach ($grouping['groups'] as $group) {
- $mapper[$listID][$grouping['id']]['groups'][$group['id']] = $group;
+ // Re-map $categories from this:
+ // id = (string [10]) `f192c59e0d`
+ // title = (string [7]) `CiviCRM`
+
+ foreach ($categories as $category) {
+ // Need to look up interests for this category.
+ $interests = CRM_Mailchimp_Utils::getMailchimpApi()
+ ->get("/lists/$listID/interest-categories/$category->id/interests",
+ ['fields' => 'interests.id,interests.name','count'=>10000])
+ ->data->interests;
+
+ $mapper[$listID][$category->id] = [
+ 'id' => $category->id,
+ 'name' => $category->title,
+ 'interests' => [],
+ ];
+ foreach ($interests as $interest) {
+ $mapper[$listID][$category->id]['interests'][$interest->id] =
+ ['id' => $interest->id, 'name' => $interest->name];
}
}
}
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getMCInterestGroupings $mapper', $mapper[$listID]);
+ CRM_Mailchimp_Utils::checkDebug("CRM_Mailchimp_Utils::getMCInterestGroupings for list '$listID' returning ", $mapper[$listID]);
return $mapper[$listID];
}
- /*
+ /**
+ * return the group name for given list, grouping and group
+ *
+ */
+ public static function getMCInterestName($listID, $category_id, $interest_id) {
+ $info = static::getMCInterestGroupings($listID);
+
+ // Check list, grouping, and group exist
+ if (empty($info[$category_id]['interests'][$interest_id])) {
+ $name = null;
+ }
+ else {
+ $name = $info[$category_id]['interests'][$interest_id]['name'];
+ }
+ CRM_Mailchimp_Utils::checkDebug(__FUNCTION__ . " called for list '$listID', category '$category_id', interest '$interest_id', returning '$name'");
+ return $name;
+ }
+
+ /**
+ * Return the grouping name for given list, grouping MC Ids.
+ */
+ public static function getMCCategoryName($listID, $category_id) {
+ $info = static::getMCInterestGroupings($listID);
+
+ // Check list, grouping, and group exist
+ $name = NULL;
+ if (!empty($info[$category_id])) {
+ $name = $info[$category_id]['name'];
+ }
+ CRM_Mailchimp_Utils::checkDebug("CRM_Mailchimp_Utils::getMCCategoryName for list $listID cat $category_id returning $name");
+ return $name;
+ }
+
+ /**
* Get Mailchimp group ID group name
*/
- static function getMailchimpGroupIdFromName($listID, $groupName) {
+ public static function getMailchimpGroupIdFromName($listID, $groupName) {
CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getMailchimpGroupIdFromName $listID', $listID);
CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getMailchimpGroupIdFromName $groupName', $groupName);
@@ -268,533 +549,36 @@ static function getMailchimpGroupIdFromName($listID, $groupName) {
}
}
- static function getGroupIdForMailchimp($listID, $groupingID, $groupID) {
-
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupIdForMailchimp $listID', $listID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupIdForMailchimp $groupingID', $groupingID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupIdForMailchimp $groupID', $groupID);
-
- if (empty($listID)) {
- return NULL;
- }
-
- if (!empty($groupingID) && !empty($groupID)) {
- $whereClause = "mc_list_id = %1 AND mc_grouping_id = %2 AND mc_group_id = %3";
- } else {
- $whereClause = "mc_list_id = %1";
- }
-
- $query = "
- SELECT entity_id
- FROM civicrm_value_mailchimp_settings mcs
- WHERE $whereClause";
- $params =
- array(
- '1' => array($listID , 'String'),
- '2' => array($groupingID , 'String'),
- '3' => array($groupID , 'String'),
- );
- $dao = CRM_Core_DAO::executeQuery($query, $params);
- if ($dao->fetch()) {
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getGroupIdForMailchimp $dao->entity_id', $dao->entity_id);
- return $dao->entity_id;
- }
-
- }
-
/**
- * Try to find out already if we can find a unique contact for this e-mail
- * address.
- */
- static function guessCidsMailchimpContacts() {
- // If an address is unique, that's the one we need.
- CRM_Core_DAO::executeQuery(
- "UPDATE tmp_mailchimp_push_m m
- JOIN civicrm_email e1 ON m.email = e1.email
- LEFT OUTER JOIN civicrm_email e2 ON m.email = e2.email AND e1.id <> e2.id
- SET m.cid_guess = e1.contact_id
- WHERE e2.id IS NULL")->free();
- // In the other case, if we find a unique contact with matching
- // first name, last name and e-mail address, it is probably the one we
- // are looking for as well.
- CRM_Core_DAO::executeQuery(
- "UPDATE tmp_mailchimp_push_m m
- JOIN civicrm_email e1 ON m.email = e1.email
- JOIN civicrm_contact c1 ON e1.contact_id = c1.id AND c1.first_name = m.first_name AND c1.last_name = m.last_name
- LEFT OUTER JOIN civicrm_email e2 ON m.email = e2.email
- LEFT OUTER JOIN civicrm_contact c2 on e2.contact_id = c2.id AND c2.first_name = m.first_name AND c2.last_name = m.last_name AND c2.id <> c1.id
- SET m.cid_guess = e1.contact_id
- WHERE m.cid_guess IS NULL AND c2.id IS NULL")->free();
- }
-
- /**
- * Update first name and last name of the contacts of which we already
- * know the contact id.
- */
- static function updateGuessedContactDetails() {
- // In theory I could do this with one SQL join statement, but this way
- // we would bypass user defined hooks. So I will use the API, but only
- // in the case that the names are really different. This will save
- // some expensive API calls. See issue #188.
-
- $dao = CRM_Core_DAO::executeQuery(
- "SELECT c.id, m.first_name, m.last_name
- FROM tmp_mailchimp_push_m m
- JOIN civicrm_contact c ON m.cid_guess = c.id
- WHERE m.first_name NOT IN ('', COALESCE(c.first_name, ''))
- OR m.last_name NOT IN ('', COALESCE(c.last_name, ''))");
-
- while ($dao->fetch()) {
- $params = array('id' => $dao->id);
- if ($dao->first_name) {
- $params['first_name'] = $dao->first_name;
- }
- if ($dao->last_name) {
- $params['last_name'] = $dao->last_name;
- }
- civicrm_api3('Contact', 'create', $params);
- }
- $dao->free();
- }
-
- /*
- * Create/Update contact details in CiviCRM, based on the data from Mailchimp webhook
- */
- static function updateContactDetails(&$params, $delay = FALSE) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils updateContactDetails $params', $params);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils updateContactDetails $delay', $delay);
-
- if (empty($params)) {
- return NULL;
- }
- $params['status'] = array('Added' => 0, 'Updated' => 0);
- $contactParams =
- array(
- 'version' => 3,
- 'contact_type' => 'Individual',
- 'first_name' => $params['FNAME'],
- 'last_name' => $params['LNAME'],
- 'email' => $params['EMAIL'],
- );
-
- if($delay){
- //To avoid a new duplicate contact to be created as both profile and upemail events are happening at the same time
- sleep(20);
- }
- $contactids = CRM_Mailchimp_Utils::getContactFromEmail($params['EMAIL']);
-
- if(count($contactids) > 1) {
- CRM_Core_Error::debug_log_message( 'Mailchimp Pull/Webhook: Multiple contacts found for the email address '. print_r($params['EMAIL'], true), $out = false );
- return NULL;
- }
- if(count($contactids) == 1) {
- $contactParams = CRM_Mailchimp_Utils::updateParamsExactMatch($contactids, $params);
- $params['status']['Updated'] = 1;
- }
- if(empty($contactids)) {
- //check for contacts with no primary email address
- $id = CRM_Mailchimp_Utils::getContactFromEmail($params['EMAIL'], FALSE);
-
- if(count($id) > 1) {
- CRM_Core_Error::debug_log_message( 'Mailchimp Pull/Webhook: Multiple contacts found for the email address which is not primary '. print_r($params['EMAIL'], true), $out = false );
- return NULL;
- }
- if(count($id) == 1) {
- $contactParams = CRM_Mailchimp_Utils::updateParamsExactMatch($id, $params);
- $params['status']['Updated'] = 1;
- }
- // Else create new contact
- if(empty($id)) {
- $params['status']['Added'] = 1;
- }
-
- }
- // Create/Update Contact details
- $contactResult = civicrm_api('Contact' , 'create' , $contactParams);
-
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils updateContactDetails $contactID', $contactResult['id']);
-
- return $contactResult['id'];
- }
-
- static function getContactFromEmail($email, $primary = TRUE) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getContactFromEmail $email', $email);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getContactFromEmail $primary', $primary);
-
- $primaryEmail = 1;
- if(!$primary) {
- $primaryEmail = 0;
- }
- $contactids = array();
- $query = "
- SELECT `contact_id` FROM civicrm_email ce
- INNER JOIN civicrm_contact cc ON ce.`contact_id` = cc.id
- WHERE ce.email = %1 AND ce.is_primary = {$primaryEmail} AND cc.is_deleted = 0 ";
- $dao = CRM_Core_DAO::executeQuery($query, array( '1' => array($email, 'String')));
- while($dao->fetch()) {
- $contactids[] = $dao->contact_id;
- }
-
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getContactFromEmail $contactids', $contactids);
-
- return $contactids;
- }
-
- static function updateParamsExactMatch($contactids = array(), $params) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils updateParamsExactMatch $params', $params);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils updateParamsExactMatch $contactids', $contactids);
-
-
- $contactParams =
- array(
- 'version' => 3,
- 'contact_type' => 'Individual',
- 'first_name' => $params['FNAME'],
- 'last_name' => $params['LNAME'],
- 'email' => $params['EMAIL'],
- );
- if(count($contactids) == 1) {
- $contactParams['id'] = $contactids[0];
- unset($contactParams['contact_type']);
- // Don't update firstname/lastname if it was empty
- if(empty($params['FNAME']))
- unset($contactParams['first_name']);
- if(empty($params['LNAME']))
- unset ($contactParams['last_name']);
- }
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils updateParamsExactMatch $contactParams', $contactParams);
-
- return $contactParams;
- }
- /*
- * Function to get the associated CiviCRM Groups IDs for the Grouping array
- * sent from Mialchimp Webhook.
- *
- * Note: any groupings from Mailchimp that do not map to CiviCRM groups are
- * silently ignored. Also, if a subscriber has no groupings, this function
- * will not return any CiviCRM groups (because all groups must be mapped to
- * both a list and a grouping).
- */
- static function getCiviGroupIdsforMcGroupings($listID, $mcGroupings) {
-
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getCiviGroupIdsforMcGroupings $listID', $listID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getCiviGroupIdsforMcGroupings $mcGroupings', $mcGroupings);
-
- if (empty($listID) || empty($mcGroupings)) {
- return array();
- }
- $civiGroups = array();
- foreach ($mcGroupings as $key => $mcGrouping) {
- if(!empty($mcGrouping['groups'])) {
- $mcGroups = @explode(',', $mcGrouping['groups']);
- foreach ($mcGroups as $mcGroupKey => $mcGroupName) {
- // Get Mailchimp group ID from group name. Only group name is passed in by Webhooks
- $mcGroupID = self::getMailchimpGroupIdFromName($listID, trim($mcGroupName));
- // Mailchimp group ID is unavailable
- if (empty($mcGroupID)) {
- // Try the next one.
- continue;
- }
-
- // Find the CiviCRM group mapped with the Mailchimp List and Group
- $civiGroupID = self::getGroupIdForMailchimp($listID, $mcGrouping['id'] , $mcGroupID);
- if (!empty($civiGroupID)) {
- $civiGroups[] = $civiGroupID;
- }
- }
- }
- }
-
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getCiviGroupIdsforMcGroupings $civiGroups', $civiGroups);
- return $civiGroups;
- }
-
- /*
- * Function to get CiviCRM Groups for the specific Mailchimp list in which the Contact is Added to
+ * Log a message and optionally a variable, if debugging is enabled.
*/
- static function getGroupSubscriptionforMailchimpList($listID, $contactID) {
-
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupSubscriptionforMailchimpList $listID', $listID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupSubscriptionforMailchimpList $contactID', $contactID);
-
- if (empty($listID) || empty($contactID)) {
- return NULL;
- }
-
- $civiMcGroups = array();
- $query = "
- SELECT entity_id
- FROM civicrm_value_mailchimp_settings mcs
- WHERE mc_list_id = %1";
- $params = array('1' => array($listID, 'String'));
-
- $dao = CRM_Core_DAO::executeQuery($query ,$params);
- while ($dao->fetch()) {
- $groupContact = new CRM_Contact_BAO_GroupContact();
- $groupContact->group_id = $dao->entity_id;
- $groupContact->contact_id = $contactID;
- $groupContact->whereAdd("status = 'Added'");
- $groupContact->find();
- if ($groupContact->fetch()) {
- $civiMcGroups[] = $dao->entity_id;
- }
- }
-
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getGroupSubscriptionforMailchimpList $civiGroups', $civiGroups);
+ public static function checkDebug($description, $variable='VARIABLE_NOT_PROVIDED') {
+ $debugging = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP, 'enable_debugging', NULL, FALSE);
- return $civiMcGroups;
- }
-
- /*
- * Function to delete Mailchimp contact for given CiviCRM email ID
- */
- static function deleteMCEmail($emailId = array() ) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils deleteMCEmail $emailId', $emailId);
- /*
- modified by mathavan@vedaconsulting.co.uk
- table name civicrm_mc_sync has no longer exist
- and dont have leid, euid, list_id informations
- so returning null to avoid the script
- */
-
- return NULL;
-
- //end
-
- if (empty($emailId)) {
- return NULL;
- }
- $toDelete = array();
- $listID = array();
- $email = NULL;
- $query = NULL;
-
- if (!empty($emailId)) {
- $emailIds = implode(',', $emailId);
- // @todo I think this code meant to include AND is_latest.
- // Looks very inefficient otherwise?
- #Mathavan@vedaconsulting.co.uk, commmenting the query, table no longer exist
- //$query = "SELECT * FROM civicrm_mc_sync WHERE email_id IN ($emailIds) ORDER BY id DESC";
- }
- $dao = CRM_Core_DAO::executeQuery($query);
-
- while ($dao->fetch()) {
- $leidun = $dao->mc_leid;
- $euidun = $dao->mc_euid;
- $listID = $dao->mc_list_id;
- $mc_group = $dao->mc_group;
- $email_id = $dao->email_id;
- $email = CRM_Core_DAO::getFieldValue('CRM_Core_DAO_Email', $dao->email_id, 'email', 'id');
-
- $toDelete[$listID]['batch'][] = array(
- 'email' => $email,
- 'euid' => $euidun,
- 'leid' => $leidun,
- );
-
- $params = array(
- 'email_id' => $dao->email_id,
- 'mc_list_id' => $listID,
- 'mc_group' => $mc_group,
- 'mc_euid' => $euidun,
- 'mc_leid' => $leidun,
- 'sync_status' => 'Removed'
- );
-
- CRM_Mailchimp_BAO_MCSync::create($params);
- }
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils deleteMCEmail $toDelete', $toDelete);
- foreach ($toDelete as $listID => $vals) {
- // sync contacts using batchunsubscribe
- $mailchimp = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
- $results = $mailchimp->batchUnsubscribe(
- $listID,
- $vals['batch'],
- TRUE,
- TRUE,
- TRUE
- );
- }
-
- return $toDelete;
- }
-
- /**
- * Function to call syncontacts with smart groups and static groups
- *
- * Returns object that can iterate over a slice of the live contacts in given group.
- */
- static function getGroupContactObject($groupID, $start = null) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupContactObject $groupID', $groupID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupContactObject $start', $start);
-
- $group = new CRM_Contact_DAO_Group();
- $group->id = $groupID;
- $group->find();
-
- if($group->fetch()){
- //Check smart groups (including parent groups, which function as smart groups).
- if($group->saved_search_id || $group->children){
- $groupContactCache = new CRM_Contact_BAO_GroupContactCache();
- $groupContactCache->group_id = $groupID;
- if ($start !== null) {
- $groupContactCache->limit($start, CRM_Mailchimp_Form_Sync::BATCH_COUNT);
- }
- $groupContactCache->find();
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getGroupContactObject $groupContactCache', $groupContactCache);
- return $groupContactCache;
+ if ($debugging == 1) {
+ if ($variable === 'VARIABLE_NOT_PROVIDED') {
+ // Simple log message.
+ CRM_Core_Error::debug_log_message($description, FALSE, 'mailchimp');
}
else {
- $groupContact = new CRM_Contact_BAO_GroupContact();
- $groupContact->group_id = $groupID;
- $groupContact->whereAdd("status = 'Added'");
- if ($start !== null) {
- $groupContact->limit($start, CRM_Mailchimp_Form_Sync::BATCH_COUNT);
- }
- $groupContact->find();
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getGroupContactObject $groupContact', $groupContact);
- return $groupContact;
+ // Log a variable.
+ CRM_Core_Error::debug_log_message(
+ $description . "\n" . var_export($variable,1)
+ , FALSE, 'mailchimp');
}
}
- return FALSE;
}
- /**
- * Function to call syncontacts with smart groups and static groups xxx delete
- *
- * Returns object that can iterate over a slice of the live contacts in given group.
- */
- static function getGroupMemberships($groupIDs) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils getGroupMemberships $groupIDs', $groupIDs);
-
- $group = new CRM_Contact_DAO_Group();
- $group->id = $groupID;
- $group->find();
-
- if($group->fetch()){
- //Check smart groups
- if($group->saved_search_id){
- $groupContactCache = new CRM_Contact_BAO_GroupContactCache();
- $groupContactCache->group_id = $groupID;
- if ($start !== null) {
- $groupContactCache->limit($start, CRM_Mailchimp_Form_Sync::BATCH_COUNT);
- }
- $groupContactCache->find();
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getGroupMemberships $groupContactCache', $groupContactCache);
- return $groupContactCache;
- }
- else {
- $groupContact = new CRM_Contact_BAO_GroupContact();
- $groupContact->group_id = $groupID;
- $groupContact->whereAdd("status = 'Added'");
- if ($start !== null) {
- $groupContact->limit($start, CRM_Mailchimp_Form_Sync::BATCH_COUNT);
- }
- $groupContact->find();
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils getGroupMemberships $groupContact', $groupContact);
- return $groupContact;
- }
- }
-
- return FALSE;
- }
-
- /*
- * Function to subscribe/unsubscribe civicrm contact in Mailchimp list
- *
- * $groupDetails - Array
- * (
- * [list_id] => ec641f8988
- * [grouping_id] => 14397
- * [group_id] => 35609
- * [is_mc_update_grouping] =>
- * [group_name] =>
- * [grouping_name] =>
- * [civigroup_title] => Easter Newsletter
- * [civigroup_uses_cache] =>
- * )
- *
- * $action - subscribe/unsubscribe
+ // Deprecated - remove these once Mailchimp turn off APIv2, scheduled end of
+ // 2016.
+ /**
+ * deprecated (soon!) v1, v2 API
*/
- static function subscribeOrUnsubsribeToMailchimpList($groupDetails, $contactID, $action) {
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils subscribeOrUnsubsribeToMailchimpList $groupDetails', $groupDetails);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils subscribeOrUnsubsribeToMailchimpList $contactID', $contactID);
- CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils subscribeOrUnsubsribeToMailchimpList $action', $action);
-
- if (empty($groupDetails) || empty($contactID) || empty($action)) {
- return NULL;
- }
-
- // We need to get contact's email before subscribing in Mailchimp
- $contactParams = array(
- 'version' => 3,
- 'id' => $contactID,
- );
- $contactResult = civicrm_api('Contact' , 'get' , $contactParams);
- // This is the primary email address of the contact
- $email = $contactResult['values'][$contactID]['email'];
-
- if (empty($email)) {
- // Its possible to have contacts in CiviCRM without email address
- // and add to group offline
- return;
- }
-
- // Optional merges for the email (FNAME, LNAME)
- $merge = array(
- 'FNAME' => $contactResult['values'][$contactID]['first_name'],
- 'LNAME' => $contactResult['values'][$contactID]['last_name'],
- );
-
- $listID = $groupDetails['list_id'];
- $grouping_id = $groupDetails['grouping_id'];
- $group_id = $groupDetails['group_id'];
- if (!empty($grouping_id) AND !empty($group_id)) {
- $merge_groups[$grouping_id] = array('id'=> $groupDetails['grouping_id'], 'groups'=>array());
- $merge_groups[$grouping_id]['groups'][] = CRM_Mailchimp_Utils::getMCGroupName($listID, $grouping_id, $group_id);
-
- // remove the significant array indexes, in case Mailchimp cares.
- $merge['groupings'] = array_values($merge_groups);
- }
-
- // Send Mailchimp Lists API Call.
- $list = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
- switch ($action) {
- case "subscribe":
- // http://apidocs.mailchimp.com/api/2.0/lists/subscribe.php
- try {
- $result = $list->subscribe($listID, array('email' => $email), $merge, $email_type='html', $double_optin=FALSE, $update_existing=FALSE, $replace_interests=TRUE, $send_welcome=FALSE);
- }
- catch (Exception $e) {
- // Don't display if the error is that we're already subscribed.
- $message = $e->getMessage();
- if ($message !== $email . ' is already subscribed to the list.') {
- CRM_Core_Session::setStatus($message);
- }
- }
- break;
- case "unsubscribe":
- // https://apidocs.mailchimp.com/api/2.0/lists/unsubscribe.php
- $delete = CRM_Core_BAO_Setting::getItem(CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'list_removal', NULL, FALSE);
- try {
- $result = $list->unsubscribe($listID, array('email' => $email), $delete, $send_goodbye=false, $send_notify=false);
- }
- catch (Exception $e) {
- CRM_Core_Session::setStatus($e->getMessage());
- }
- break;
- }
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils subscribeOrUnsubsribeToMailchimpList $groupDetails', $groupDetails);
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils subscribeOrUnsubsribeToMailchimpList $contactID', $contactID);
- CRM_Mailchimp_Utils::checkDebug('End-CRM_Mailchimp_Utils subscribeOrUnsubsribeToMailchimpList $action', $action);
+ public static function mailchimp() {
+ $apiKey = CRM_Core_BAO_Setting::getItem(CRM_Mailchimp_Form_Setting::MC_SETTING_GROUP, 'api_key');
+ $mcClient = new Mailchimp($apiKey);
+ //CRM_Mailchimp_Utils::checkDebug('Start-CRM_Mailchimp_Utils mailchimp $mcClient', $mcClient);
+ return $mcClient;
}
- static function checkDebug($class_function = 'classname', $debug) {
- $debugging = CRM_Core_BAO_Setting::getItem(self::MC_SETTING_GROUP, 'enable_debugging', NULL, FALSE
- );
-
- if ($debugging == 1) {
- CRM_Core_Error::debug_var($class_function, $debug);
- }
- }
}
diff --git a/README-tech.md b/README-tech.md
new file mode 100644
index 0000000..814ab4c
--- /dev/null
+++ b/README-tech.md
@@ -0,0 +1,274 @@
+# Mailchimp sync operations, including tests.
+
+Sync efforts fall into four categories:
+
+1. Push from CiviCRM to Mailchimp.
+2. Pull from Mailchimp to CiviCRM.
+3. CiviCRM-fired hooks.
+4. Mailchimp-fired Webhooks
+
+Note that a *key difference between push and pull*, other than the direction of
+authority, is that mapped interest groups can be declared as being allowed to be
+updated on a pull or not. This is useful when Mailchimp has no right to change
+that interest group, e.g. a group that you identify with a smart group in
+CiviCRM. Typically such groups should be considered internal and therefore
+hidden from subscribers at all times.
+
+One of the challenges is to *identify the CiviCRM* contact that a mailchimp
+member matches. The code for this is centralised in
+`CRM_Mailchimp_Sync::guessContactIdSingle()`, which has tests at
+`MailchimpApiIntegrationMockTest::testGuessContactIdSingle()`.
+
+Look at the comment block for that test and for the `guessContactIdSingle`
+method for details of how contacts are identified. However, this is slow and so
+for the bulk operations there's some SQL shortcuts for efficiency which are in the methods:
+
+ - `guessContactIdsBySubscribers`
+ - `guessContactIdsByNameAndEmail`
+ - `guessContactIdsByUniqueEmail`
+
+## About Names
+
+Mailchimp lists default to having `FNAME` and `LNAME` merge fields to store
+first and last names. Some people change/delete these merge fields which makes
+things difficult. A common reason is that people wanted a single name field on a
+Mailchimp-provided sign-up form. This extension allows for the existance of a
+`NAME` merge field. Names found here are split automatically (on spaces) with
+the first word becomming the first name and any names following being used as
+last names. See unit tests for conditions and handling of blanks.
+
+A 'pull' sync will split the names and then work as if those names
+were in FNAME, LNAME merge fields, but only if the FNAME/LNAME fields don't
+exist or are both empty.
+
+A 'push' sync will combine the first and last names into a single string and
+submit that to the `NAME` merge field, if it exists.
+
+
+## About email selection.
+
+In order to be subscribed, the contact must:
+
+- have an email available
+- not be deceased
+- not have `is_opt_out` set
+- not have `do_not_email` set
+
+In terms of subscribing people from CiviCRM to Mailchimp, it will use the first
+available (i.e. not "on hold") email in this order:
+
+1. Specified bulk email address
+2. Primary email address
+3. Any other email address
+
+
+## Tests are provided at different levels.
+
+- Unit tests check the logic of certain bits of the system. These can be run
+ without CiviCRM or Mailchimp services.
+
+- Integration tests require a CiviCRM install but mock the Mailchimp service.
+ This enables testing such as checking that CiviCRM is making the expected
+ calls to the API
+
+- Integration tests that run with live Mailchimp. These test that the Mailchimp
+ API is behaving as expected, and/or that the use of it is achieving what we
+ think it is achieving.
+
+# Push CiviCRM to Mailchimp Sync for a list.
+
+The Push Sync is done by the `CRM_Mailchimp_Sync` class. The steps are:
+
+1. Fetch required data from Mailchimp for all the list's members.
+2. Fetch required data from CiviCRM for all the list's CiviCRM membership group.
+3. Add those who are not on Mailchimp, and update those whose details are
+ different on CiviCRM compared to Mailchimp.
+4. Remove from mailchimp those not on CiviCRM.
+
+The test cases are as follows:
+
+## A subscribed contact not on Mailchimp is added.
+
+`testPushAddsNewPerson()` checks this.
+
+## Name changes to subscribed contacts are pushed except deletions.
+
+ CiviCRM Mailchimp Result (at Mailchimp)
+ --------+----------+---------------------
+ Fred Fred (added)
+ Fred Fred Fred (no change)
+ Fred Barney Fred (corrected)
+ Fred Fred (no change)
+ --------+----------+---------------------
+
+This logic is tested by `tests/unit/SyncTest.php`
+
+The collection, comparison and API calls are tested in
+`tests/integration/MailchimpApiIntegrationTest.php1`
+
+## Interest changes to subscribed contacts are pushed.
+
+This logic is tested by `tests/unit/SyncTest.php`
+
+The collection, comparison and API calls are tested in
+`tests/integration/MailchimpApiIntegrationTest.php`
+
+## Changes to unsubscribed contacts are not pushed.
+
+This is tested in `tests/integration/MailchimpApiIntegrationTest.php`
+in `testPushUnsubscribes()`
+
+## A contact no longer subscribed at CiviCRM should be unsubscribed at Mailchimp.
+
+This is tested in `tests/integration/MailchimpApiIntegrationTest.php`
+in `testPushUnsubscribes()`
+
+
+# Pull Mailchimp to CiviCRM Sync for a list.
+
+The Pull Sync is done by the `CRM_Mailchimp_Sync` class. The steps are:
+
+1. Fetch required data from Mailchimp for all the list's members.
+2. Fetch required data from CiviCRM for all the list's CiviCRM membership group.
+3. Identify a single contact in CiviCRM that corresponds to the Mailchimp member,
+ create a contact if needed.
+4. Update the contact with name and interest group changes (only for interests
+ that are configured to allow Mailchimp to CiviCRM updates)
+5. Remove contacts from the membership group if they are not subscribed at Mailchimp.
+
+The test cases are as follows:
+
+## Test identification of contact by known membership group.
+
+An email from Mailchimp can be used to identify the CiviCRM contact if if
+matches among a list of CiviCRM contacts that are in the membership group.
+
+This is done with `SyncIntegrationTest::testGuessContactIdsBySubscribers`
+
+## Test identification of contact by the email only matching one contact.
+
+An email can be matched if it's unique to a particular contact in CiviCRM.
+
+This is done with `SyncIntegrationTest::testGuessContactIdsByUniqueEmail`
+
+## Test identification of contact by email and name match.
+
+An email can be matched along with a first and last name if they all match only
+one contact in CiviCRM.
+
+This is done with `SyncIntegrationTest::testGuessContactIdsByNameAndEmail`
+
+
+## Test that name changes from Mailchimp are properly pulled.
+
+See integration test `testPullChangesName()` and for the name logic see unit test
+`testUpdateCiviFromMailchimpContactLogic`.
+
+## Test that interest group changes from Mailchimp are properly pulled.
+
+See integration tests:
+- `testPullChangesInterests()` For when the group is configured with update
+ permission from Mailchimp to Civi.
+- `testPullChangesNonPullInterests()` For when the group is NOT configured with
+ update permission.
+
+## Test that contacts unknown to CiviCRM when pulled get added.
+
+See integration test `testPullAddsContact()`.
+
+## Test that contacts not received from Mailchimp but in membership group get removed from membership group.
+
+See integration test `testPullRemovesContacts()`.
+
+# Mailchimp Webhooks
+
+Mailchimp's webhooks are an important part of the system. If they are
+functioning correctly then the Pull sync should never need to make any changes.
+
+But they're a nightmare for non-techy users to configure, so now this extension
+takes care of them. When you visit the settings page all groups' webhooks are
+checked, with errors shown to the user. You can correct a list's webhooks by
+editing the CiviCRM group settings. There's a tickbox for doing the webhook
+changes which defaults to ticked, and when you save it will ensure everything is
+correct.
+
+Tests
+- `MailchimpApiIntegrationMockTest::testCheckGroupsConfig`
+- `MailchimpApiIntegrationMockTest::testConfigureList`
+
+
+
+# Posthook used to immediately add/remove a single person.
+
+If you *add/remove/delete a single contact* from a group that is associated with a
+Mailchimp list then the posthook is used to detect this and make the change at
+Mailchimp.
+
+There are several cases that this does not cover (and it's therefore of questionable use):
+
+- Smart groups. If you have a smart group of all with last name Flintstone and
+ you change someone's name to Flintstone, thus giving them membership of that
+ group, this hook will *not* be triggered (@todo test).
+
+- Block additions. If you add more than one contact to a group, the immediate
+ Mailchimp updates are not triggered. This is because each contact requires a
+ separate API call. Add thousands and this will cause big problems.
+
+If the group you added someone to was synced to an interest at Mailchimp then
+the person's membership is checked. If they are, according to CiviCRM in the
+group mapped to that lists's membership, then their interests are updated at
+Mailchimp. If they are not currently in the membership CiviCRM group then the
+interest change is not attempted to be registered with Mailchimp.
+
+See Tests:
+
+- `MailchimpApiIntegrationMockTest::testPostHookForMembershipListChanges()`
+- `MailchimpApiIntegrationMockTest::testPostHookForInterestGroupChanges()`
+
+Because of these limitations, you cannot rely on this hook to keep your list
+up-to-date and will always need to do a CiviCRM to Mailchimp Push sync before
+sending a mailing.
+
+# Settings page
+
+The settings page stores details like the API key etc.
+
+However it also serves to check the mapped groups and lists are properly set up. Specifically it:
+
+- Checks that the list still exists on Mailchimp
+- Checks that the list's webhook is set and configured exactly.
+
+Warnings are displayed on screen when these settings are wrong and these include
+a link to the group's settings page, from which you can auto-configure the list
+to the correct settings on Save.
+
+These warnings are tested in `MailchimpApiIntegrationMockTest::testCheckGroupsConfig()`.
+
+
+# "Titanics": Duplicate contacts that can't be sunk!
+
+One thing we can't cope with is duplicate contacts. This is now fairly rare
+because of the more liberal matching of CiviCRM contacts in version 2.0.
+
+Specifically: an email coming from Mailchimp belonged to several contacts and we
+were unable to narrow it down by the names (perhaps there was no name in
+Mailchimp).
+
+On *push*, the temporary mailchimp table has NULL in it for these contacts.
+Normally we would unsubscribe emails from Mailchimp that are not matched in the
+CiviCRM table, but we'll avoid unsubscribing the ones that are NULL.
+
+On *pull*, we will *not* create a contact for NULL `cid_guess` records.
+
+**This means that the contact will stay on Mailchimp unaffected and un-synced by
+any sync operations.** They are therefore un-sync-able.
+
+The alternatives?
+
+1. create new contact. Can't do this; it could result in creating a new contact
+ on every sync, since every creation would cause the duplication to increase.
+
+2. pick one of the matching contacts at random but they could be different
+ people sharing an email so we wouldn't want to merge in any names or
+ interests based on the wrong contact.
+
diff --git a/README.md b/README.md
index 79fa525..ab01d90 100644
--- a/README.md
+++ b/README.md
@@ -1,36 +1,53 @@
uk.co.vedaconsulting.mailchimp
==============================
-The new extension builds on the existing work done by the science gallery, adding the ability to pick the Mailchimp List the CiviCRM group should be integrated to as well as making the entire process a much simpler one to setup.
-
-For each Mailchimp list that you want to integrate, you set up a CiviCRM group.
-This will control the subscribers in that list. Add contacts to the group in
-CiviCRM and after pushing the sync button, those will be subscribed to your Mailchimp
-list. If you remove contacts from the group in CiviCRM, the sync will unsubscribe
-them at Mailchimp. If anyone clicks an unsubscribe link in a Mailchimp email,
-they are automatically removed from your CiviCRM group.
-
-Additionally, if you use Mailchimp's "Interest Groupings", you can map particular
-Mailchimp groups to a CiviCRM group. You can choose whether this is a group that the
-subscriber can edit (using Mailchimp's forms), or not.
-
-So if you have a list of fundraisers you might use an interest grouping called
-"Interests" and give subscribers options like "Shaking tins", "Door knocking",
-"Climbing mountains". Each of these can be mapped to a CiviCRM group (if you
-so choose) and membership of these groups will be updated.
-
-Alternatively, what if you have groups in CiviCRM like "major donor" or
-"miserly meanie" that you want to use to segment your mailings but you don't
-want subscribers seeing or being able to edit these? This is accommodated, too.
-It's up to you to set up your Mailchimp Interest Groupings so that this fieldset
-is hidden from subscribers, but then you can just link a CiviCRM group to
-one of those. These groups will never update from Mailchimp to CiviCRM.
-
-NB Mailchimp sometimes calls Interest Groupings just "Groups", which gets
-very confusing because you have Groups of Groups, and of course CiviCRM uses
-the word Group, too! Here I will stick to calling the Mailchimp fields
-"Interest Groupings" which each contain a number of "Mailchimp Groups", to
-differentiate them from CiviCRM groups.
+## Introduction
+
+This extension helps you keep a CiviCRM group in sync with subscribers of a
+Mailchimp list. It can sync different CiviCRM groups to different Mailchimp lists.
+
+Additionally, if you use Mailchimp's *Interests* feature, you can map particular
+Mailchimp interests to a CiviCRM group. You can choose whether this is a group
+that the subscriber can edit (using Mailchimp's forms), or not.
+
+Some updates happen in near-real time. So once set-up, if someone new subscribes
+using a Mailchimp embedded sign-up form, they will be added to CiviCRM (if not
+already found) and joined to the appropriate group. Likewise, if you use the
+"Add to Group" widget on a contact's Groups tab to add someone to a group that
+is sync-ed with a Mailchimp List, they will be immediately added. Likewise with
+individual unsubscribes/removals.
+
+However not all updates are possible (or desireable) this way, and to cope with
+these there's two mechanisms offered:
+
+1. **Pull Sync: updates CiviCRM from Mailchimp**, assuming Mailchimp is correct.
+ You'd do this if you had just made a bulk change at Mailchimp, e.g. you'd
+ just imported a new list of contacts to a list and you wanted to make sure
+ that these contacts were in CiviCRM.
+
+2. **Push Sync: updates Mailchimp from CiviCRM**, assuming CiviCRM is correct.
+ You'd do this if you'd just made a bulk change at CiviCRM, e.g. added/removed
+ a load of contacts to one or more sync-ed groups, or changed records such
+ that now they qualify (or cease to qualify) for membership of a Smart Group.
+
+Typically day-to-day changes made at Mailchimp (poeple clicking unsubscribe, or
+individuals subscribing or updating their preferences) are all done right away,
+so except for bulk changes that you do to your list deliberately, you usually do
+not need to use the **Pull**.
+
+You can set up the **Push** to run at scheduled intervals, if you find that's
+useful, otherwise do it after a change, or at least before you send out an email.
+
+**Note: syncing works best when done regularly**. If changes are made at both
+ends there's no way to figure out which way is correct and you'll be forced to
+choose: pull or push? So it's important you have an awareness of this in your
+day-to-day workflows.
+
+## Take care. This can make large-scale bulk updates to your data.
+
+Until you're confident in the way this works and your own workflows, **make sure
+to backup both your mailchimp and civicrm contacts**. Regular periodic backups
+are also sensible practise.
## How to Install
@@ -39,55 +56,51 @@ differentiate them from CiviCRM groups.
3. When you reload the Manage Extensions page the new “Mailchimp” extension should be listed with an Install link.
4. Proceed with install.
-Before the extension can be used you must set up your API keys...
-
-To get your accounts API you should follow these instructions http://kb.mailchimp.com/accounts/management/about-api-keys
+Before the extension can be used you must set up your API keys. To get your
+Mailchimp account's API you should follow [Mailchimp's
+instructions](http://kb.mailchimp.com/accounts/management/about-api-keys).
-Once you’ve setup your Mailchimp API key it can be added to CiviCRM through "Mailings >> Mailchimp Settings" screen, with url https://<>/civicrm/mailchimp/settings?reset=1. Using “Save & Test” button will test that a connection can be made to your Mailchimp account, and if your API settings are correct.
+Once you’ve setup your Mailchimp API key it can be added to CiviCRM through
+"Mailings >> Mailchimp Settings" screen, with url
+`https://<>/civicrm/mailchimp/settings?reset=1`. Using “Save & Test”
+button will test that a connection can be made to your Mailchimp account, and if
+your API settings are correct.
-## Basic Use
+## Basic Use Example
-In Mailchimp: Set up an empty list, lets call it Newsletter. You'll also need
-to set up this list's **Webhooks**.
-
-Steps to configure Mailchimp Webhook settings with the relevant CiviCRM Url:
-
-1. To know the relevant CiviCRM url visit https://<>/civicrm/mailchimp/settings?reset=1.
-2. In the “Security Key” field entering a key shows the complete webhook url. Note down the complete url.
-3. Make sure webhook url is accessible to public. If not, just make sure anonymous / public user has “allow webhook posts” permission.
- Note: For Wordpress, the following page is mandatory for right behaviour of webhook url.
- (images/wordpress_civi_default_page.png)
-4. Log in to your MailChimp account.
-5. Navigate to your Lists.
-6. Click Webhooks under Settings menu and Click ‘Add a New Webhook’ button.
-7. Enter the CiviCRM Webhook URL, noted in #2 ( https://<>/civicrm/mailchimp/webhook?reset=1&key=ABCD ) in Callback URL field.
-8. Tick the relevant options for type of updates and when to send an update.
-9. Click Save.
+In Mailchimp: Set up an empty list, lets call it Newsletter.
In CiviCRM: you need a group to track subscribers to your Mailchimp Newsletter
List. You can create a new blank group, or choose an existing group (or smart
group). The CiviCRM Group's settings page has an additional fieldset called
Mailchimp.
-Choose the integration option, called "Sync membership of this group with membership of a Mailchimp List" then choose your list name.
+Choose the integration option, called "*Membership Sync: Contacts in this group
+should be subscribed to a Mailchimp List*" then choose your list name.
-
+Ensure the tickbox is ticked that says "*Ensure lists's webhook settings are
+correct*".
Save your group's settings.
The next step is to get CiviCRM and Mailchimp in sync. **Which way you do this
-is important**. In our example we have assumed a new, blank Mailchimp list and
-a populated CiviCRM Group. So we want to do a **CiviCRM to Mailchimp** Sync.
-However, if we had set up an empty group in CiviCRM for a pre-existing
-Mailchimp list, we would want to do a **Mailchimp to CiviCRM** sync. If you get
-it wrong you'll end up removing/unsubscribing everyone!
+is important**. In our example we have assumed a new, blank Mailchimp list and a
+populated CiviCRM Group. So we want to do a **Push CiviCRM to Mailchimp** Sync.
+However, if we had set up an empty group in CiviCRM for a pre-existing Mailchimp
+list, we would want to do a **Pull Mailchimp to CiviCRM** sync. If you get it
+wrong you'll end up removing/unsubscribing everyone!
So for our example, with an empty Mailchimp list and a CiviCRM newsletter group
with contacts in, you'll find the **CiviCRM to Mailchimp Sync** function in the
**Mailings** menu.
-Push the Sync button and after a while (for a large
-list/group) you should see a summary screen.
+You'll notice a tick-box for **Dry-Run**. If this is ticked then all the work is
+done except for the actual updates. Use this if you're at all unclear on what's
+about to happen and check the results make sense.
+
+Push the Sync button and after a while (for a large list/group) you should see a
+summary screen.
+
### From here on...
@@ -105,29 +118,27 @@ We have an upcoming feature that will give you the option to force
a CiviCRM to Mailchimp sync which will automatically do the necessary deletions,
but this is not included in the current version. Watch this space.
-## Interest groupings
+## Interests Example
For this example we'll set up two interest groupings in Mailchimp, one called
-Interests that publically viewable, and one called Private that is hidden from
-subscribers. Within "Interests" add Mailchimp Groups such as "bananas",
+*Things I like* that is publically viewable, and one called *Private* that is hidden from
+subscribers. Within "Things I like" add Mailchimp Groups such as "bananas",
"organic farming", "climate change activism". Within the "Private" Mailchimp
Interest Grouping, you might add Mailchimp Groups called "major donor", "VIPs"
etc.
-Please take care and follow Mailchimp's help pages for how to restrict the
+Please **take care** and follow Mailchimp's help pages for how to restrict the
visibility of the Private interest grouping.
Now back in CiviCRM, setup groups to map to these Mailchimp Groups. When you
-look at the CiviCRM group's settings page, choose "Sync membership with a
-Mailchimp interest grouping" you'll then see something like:
-
-
+look at the CiviCRM group's settings page, choose "*Interest Sync: Contacts in
+this group should have an "interest" set at Mailchimp*".
Here you can see the two options about whether Mailchimp subscribers are supposed
to be able to edit their membership of this interest grouping.
So for the Private interest grouping, choose the first, No option, for the
-public "Interests" one, choose the second option.
+public "Things I like" one, choose the second option.
**Please note** that while it's possible to configure one Mailchimp Group to be
updatable and another to be non-updatable within the same mailchimp interest
@@ -135,18 +146,78 @@ grouping, this will lead to unpredictable results. Stick to the rule: if it's
public, it should be updateable, if it's hidden/private, it should be not
updatable.
-When you run the sync, these grouping will be updated accordingly. Nb. a webhook
-immediately processes changes made from the Mailchimp end.
+When you run the sync, these grouping will be updated accordingly. So again,
+when you first set it up, which source has the data: Mailchimp or CiviCRM?
+Choose Pull or Push accordingly.
+
+
+
+## How contacts are matched and about "Titanics" - un-sync-able contacts.
+
+The extension tries hard to match contact details from Mailchimp with existing
+contacts in CiviCRM. This is explained in the [README-tech](README-tech.md) file
+and documented in more detail in the code comments including tests.
+
+The basics are:
+
+1. Emails are the primary thing to match (obviously!). While CiviCRM will always
+ choose a "bulk mail" email address for giving to Mailchimp, it will check
+ every email, bulk mail or other when trying to find a contact.
+
+2. The default Mailchimp `FNAME` and `LNAME` "*merge fields*" are assumed and
+ are an important part of a successful sync workflow. There is the option to
+ use a `NAME` merge field at Mailchimp, but this is far less helpful in many
+ ways, so avoid this if you can.
+
+3. Precidence will be given to contacts that are in the sync-ed membership
+ group. So if the same email and name belongs to two contacts but one is in
+ the membership group, it will assume that's the one to work with.
+
+4. If there are multiple contacts with the same name and email, and none are in
+ the group, a random one will be picked. Then after this will be preferred
+ (see point 3).
+
+5. "*Titanics*" There are still cases of messy data in CiviCRM that cannot be
+ resolved. e.g. 2 contacts have the same email, neither is in the group, and
+ neither has a name that matches the incoming data from Mailchimp. These are
+ considered un-sync-able. It would be wrong to create another contact
+ (possibiliy of adding to the duplicates) and we can't choose between them.
+ Such contacts are excluded from the Sync and will remain as they are on
+ Mailchimp, until such time that the duplication or incorrect names in CiviCRM
+ is sorted out. These contacts are listed on the summary page after a sync
+ operation.
+
+## 'Cleaned' emails
+
+When Mailchimp determines an email must be 'cleaned' CiviCRM will put that email
+"on hold". Cleaned in Mailchimp parlance means the email is duff, e.g. hard
+bounces.
+
+## Difficult Mailchimp policies
-## Sync - which way?
+Mailchimp have certain policies in place to protect their own interests and
+sender reputation. These can cause problems with sync.
-If you are integrating Mailchimp account that's been there for a while (lots of contacts), with a relatively new CiviCRM setup (less or no contacts), might want to do mailchimp to civicrm sync first (aka pull). Any updates to Mailchimp after that would immediately be handled via webhooks. And updates to CiviCRM could be passed onto Mailchimp via Civi to Mailchimp Sync.
+For instance if someone unsubscribes from a list using the link at the bottom of
+their email, **you are not allowed to re-subscribe them**. Previously there have
+been work-arounds to this (e.g. delete the member and re-add them) and while
+there are still loopholes in Mailchimp's API that could be exploited to this
+end, as it is against their policy we should not design a system that makes use
+of these since the loopholes (bugs) could be fixed at any time, breaking our
+system without notice.
-In other words, most Mailchimp changes are handled immediately via webhook. So it is now rare to need the Mailchimp to CiviCRM sync operation, whereas the CiviCRM to Mailchimp sync should become part of your pre-campaign sending preparation.
+Note that this rule does not apply if it was us (CiviCRM) who unsubscribed the
+member.
-In any case sync process between CiviCRM and Mailchimp can be automated and run on regular basis using the “Mailchimp Sync” scheduled job shipped by extension. The job can be enabled and configured from the Scheduled Job screen: Administer > System Settings > Scheduled Jobs.
+Likewise you cannot (or are not *supposed* to be able to) re-subscribe a
+'cleaned' email address.
-Version 1.5 and above also ships a scheduled job for "Mailchimp Sync".
+Mailchimp's policy is that these emails can only be updated to "pending" by the
+API, which means Mailchimp sends a "do you want to subscribe to..." email. The
+extension does not currently handle this case.
+## Thanks and Authors.
-**Note: Before you do any sync make sure to backup your mailchimp and civicrm contacts.**
+Originally work was done by the science gallery, then Veda Consulting and
+Artful Robot. Thanks also to Sumatran Orangutan Society for funding some of the work to
+implement Mailchimp API v3.
diff --git a/api/v3/Mailchimp.php b/api/v3/Mailchimp.php
index dd0bce0..be47c71 100644
--- a/api/v3/Mailchimp.php
+++ b/api/v3/Mailchimp.php
@@ -19,31 +19,121 @@
* @throws API_Exception
*/
function civicrm_api3_mailchimp_getlists($params) {
- $mcLists = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
-
- $lists = array();
+ $api = CRM_Mailchimp_Utils::getMailchimpApi();
- /**
- * Fix for #155 - Sync limited to 25 MailChimp Lists
- **/
- $results = $mcLists->getList(NULL, 0, 100); //get max number of mailing lists i.e. 100
+ $query = ['offset' => 0, 'count' => 100, 'fields'=>'lists.id,lists.name,total_items'];
- foreach($results['data'] as $list) {
- $lists[$list['id']] = $list['name'];
- }
+ $lists = [];
+ do {
+ $data = $api->get('/lists', $query)->data;
+ foreach ($data->lists as $list) {
+ $lists[$list->id] = $list->name;
+ }
+ $query['offset'] += 100;
+ } while ($query['offset'] * 100 < $data->total_items);
- $pages = ceil($results['total']/100); //calculate the number of page requests to be made to fetch all the mailing lists
+ return civicrm_api3_create_success($lists);
+}
- for( $i=1; $i< $pages; $i++ ) {
- $results = $mcLists->getList(NULL, $i, 100); //get 100 results for each page
- foreach($results['data'] as $list) {
- $lists[$list['id']] = $list['name'];
+/**
+ * Get Mailchimp Interests.
+ *
+ * Returns an array whose keys are interest hashes and whose values are
+ * arrays. Nb. Mailchimp now (2016) talks "Interest Categories" which each
+ * contain "Interests". It used to talk of "groupings and groups" which was much
+ * more confusing!
+ *
+ * @param array $params
+ * @return array API result descriptor
+ * @see civicrm_api3_create_success
+ * @see civicrm_api3_create_error
+ * @throws API_Exception
+ */
+function civicrm_api3_mailchimp_getinterests($params) {
+ try {
+ $list_id = $params['id'];
+ $results = CRM_Mailchimp_Utils::getMCInterestGroupings($list_id);
+ }
+ catch (Exception $e) {
+ return array();
+ }
+
+ $interests = [];
+ foreach ($results as $category_id => $category_details) {
+ $interests[$category_id]['id'] = $category_id;
+ $interests[$category_id]['name'] = $category_details['name'];
+ foreach ($category_details['interests'] as $interest_id => $interest_details) {
+ $interests[$category_id]['interests'][$interest_id] = "$category_details[name]::$interest_details[name]";
}
}
- return civicrm_api3_create_success($lists);
+ return civicrm_api3_create_success($interests);
+}
+
+/**
+ * CiviCRM to Mailchimp Push Sync.
+ *
+ * This is a schedulable job.
+ *
+ * Note this was previously named 'sync' and did a pull, then a push request.
+ * However this is problematic because each of these syncs brings the membership
+ * exactly in-line, so there's nothing for the 'push' to do anyway. The pull
+ * will remove any contacts from the synced membership group that are not in the
+ * Mailchimp list. This means any contacts added to the membership group that
+ * have not been sent up to Mailchimp (there are several scenarios when this
+ * happens: bulk additions, smart groups, ...) will be removed from the group
+ * before they've ever been subscribed.
+ *
+ * @param array $params
+ * @return array API result descriptor
+ * @see civicrm_api3_create_success
+ * @see civicrm_api3_create_error
+ * @throws API_Exception
+ */
+function civicrm_api3_mailchimp_pushsync($params) {
+
+ // Do push from CiviCRM to mailchimp
+ $runner = CRM_Mailchimp_Form_Sync::getRunner($skipEndUrl = TRUE);
+ if ($runner) {
+ $result = $runner->runAll();
+ }
+
+ if ($result['is_error'] == 0) {
+ return civicrm_api3_create_success();
+ }
+ else {
+ return civicrm_api3_create_error();
+ }
+}
+/**
+ * Pull sync from Mailchimp to CiviCRM.
+ *
+ * This is a schedulable job.
+ *
+ * @param array $params
+ * @return array API result descriptor
+ * @see civicrm_api3_create_success
+ * @see civicrm_api3_create_error
+ * @throws API_Exception
+ */
+function civicrm_api3_mailchimp_pullsync($params) {
+
+ // Do push from CiviCRM to mailchimp
+ $runner = CRM_Mailchimp_Form_Pull::getRunner($skipEndUrl = TRUE);
+ if ($runner) {
+ $result = $runner->runAll();
+ }
+
+ if ($result['is_error'] == 0) {
+ return civicrm_api3_create_success();
+ }
+ else {
+ return civicrm_api3_create_error();
+ }
}
+// Deprecated below here. No code in this extension uses these, so if your 3rd
+// party code does use them time to take action.
/**
* Mailchimp Get Mailchimp Membercount API
*
@@ -65,7 +155,12 @@ function civicrm_api3_mailchimp_getmembercount($params) {
return civicrm_api3_create_success($listmembercount);
}
/**
- * Mailchimp Get Mailchimp Groups API
+ * Mailchimp Get Mailchimp Groups API.
+ *
+ * Returns an array whose keys are interest grouping Ids and whose values are
+ * arrays. Nb. Mailchimp now (2016) talks "Interest Categories" which each
+ * contain "Interests". It used to talk of "groupings and groups" which was much
+ * more confusing!
*
* @param array $params
* @return array API result descriptor
@@ -168,57 +263,3 @@ function civicrm_api3_mailchimp_getcivicrmgroupmailchimpsettings($params) {
return civicrm_api3_create_success($groups);
}
-/**
- * CiviCRM to Mailchimp Sync
- *
- * @param array $params
- * @return array API result descriptor
- * @see civicrm_api3_create_success
- * @see civicrm_api3_create_error
- * @throws API_Exception
- */
-function civicrm_api3_mailchimp_sync($params) {
- $groups = CRM_Mailchimp_Utils::getGroupsToSync(array(), null, $membership_only = TRUE);
- foreach ($groups as $group_id => $details) {
- $list = new Mailchimp_Lists(CRM_Mailchimp_Utils::mailchimp());
- $webhookoutput = $list->webhooks($details['list_id']);
- if($webhookoutput[0]['sources']['api'] == 1) {
- return civicrm_api3_create_error('civicrm_api3_mailchimp_sync - API is set in Webhook setting for listID '.$details['list_id'].' Please uncheck API' );
- }
- }
- $result = $pullResult = array();
-
- // Do pull first from mailchimp to CiviCRM
- $pullRunner = CRM_Mailchimp_Form_Pull::getRunner($skipEndUrl = TRUE);
- if ($pullRunner) {
- $pullResult = $pullRunner->runAll();
- }
-
- // Do push from CiviCRM to mailchimp
- $runner = CRM_Mailchimp_Form_Sync::getRunner($skipEndUrl = TRUE);
- if ($runner) {
- $result = $runner->runAll();
- }
-
- if ($pullResult['is_error'] == 0 && $result['is_error'] == 0) {
- return civicrm_api3_create_success();
- }
- else {
- return civicrm_api3_create_error();
- }
-}
-
-/*function civicrm_api3_mailchimp_pull($params) {
- $result = array();
- $runner = CRM_Mailchimp_Form_Pull::getRunner($params);
- if ($runner) {
- $result = $runner->runAll();
- }
-
- if ($result['is_error'] == 0) {
- return civicrm_api3_create_success();
- }
- else {
- return civicrm_api3_create_error();
- }
-}*/
diff --git a/info.xml b/info.xml
index d3f7feb..fe757f1 100644
--- a/info.xml
+++ b/info.xml
@@ -8,8 +8,8 @@
Deepak Srivastava, Kajan, Parvez Saleh and Rich Lott (Artful Robot)support@vedaconsulting.co.uk
- 2016-06-01
- 1.8.7
+ 2016-05-08
+ 2.0stable4.4
diff --git a/mailchimp.civix.php b/mailchimp.civix.php
index 1119352..e1b0d69 100644
--- a/mailchimp.civix.php
+++ b/mailchimp.civix.php
@@ -3,34 +3,38 @@
// AUTO-GENERATED FILE -- Civix may overwrite any changes made to this file
/**
- * (Delegated) Implementation of hook_civicrm_config
+ * (Delegated) Implements hook_civicrm_config().
*
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_config
*/
function _mailchimp_civix_civicrm_config(&$config = NULL) {
static $configured = FALSE;
- if ($configured) return;
+ if ($configured) {
+ return;
+ }
$configured = TRUE;
$template =& CRM_Core_Smarty::singleton();
- $extRoot = dirname( __FILE__ ) . DIRECTORY_SEPARATOR;
+ $extRoot = dirname(__FILE__) . DIRECTORY_SEPARATOR;
$extDir = $extRoot . 'templates';
if ( is_array( $template->template_dir ) ) {
array_unshift( $template->template_dir, $extDir );
- } else {
+ }
+ else {
$template->template_dir = array( $extDir, $template->template_dir );
}
$include_path = $extRoot . PATH_SEPARATOR . get_include_path( );
- set_include_path( $include_path );
+ set_include_path($include_path);
}
/**
- * (Delegated) Implementation of hook_civicrm_xmlMenu
+ * (Delegated) Implements hook_civicrm_xmlMenu().
*
* @param $files array(string)
+ *
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_xmlMenu
*/
function _mailchimp_civix_civicrm_xmlMenu(&$files) {
@@ -40,7 +44,7 @@ function _mailchimp_civix_civicrm_xmlMenu(&$files) {
}
/**
- * Implementation of hook_civicrm_install
+ * Implements hook_civicrm_install().
*
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_install
*/
@@ -52,7 +56,7 @@ function _mailchimp_civix_civicrm_install() {
}
/**
- * Implementation of hook_civicrm_uninstall
+ * Implements hook_civicrm_uninstall().
*
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_uninstall
*/
@@ -64,7 +68,7 @@ function _mailchimp_civix_civicrm_uninstall() {
}
/**
- * (Delegated) Implementation of hook_civicrm_enable
+ * (Delegated) Implements hook_civicrm_enable().
*
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_enable
*/
@@ -78,7 +82,7 @@ function _mailchimp_civix_civicrm_enable() {
}
/**
- * (Delegated) Implementation of hook_civicrm_disable
+ * (Delegated) Implements hook_civicrm_disable().
*
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_disable
* @return mixed
@@ -93,7 +97,7 @@ function _mailchimp_civix_civicrm_disable() {
}
/**
- * (Delegated) Implementation of hook_civicrm_upgrade
+ * (Delegated) Implements hook_civicrm_upgrade().
*
* @param $op string, the type of operation being performed; 'check' or 'enqueue'
* @param $queue CRM_Queue_Queue, (for 'enqueue') the modifiable list of pending up upgrade tasks
@@ -115,7 +119,8 @@ function _mailchimp_civix_civicrm_upgrade($op, CRM_Queue_Queue $queue = NULL) {
function _mailchimp_civix_upgrader() {
if (!file_exists(__DIR__.'/CRM/Mailchimp/Upgrader.php')) {
return NULL;
- } else {
+ }
+ else {
return CRM_Mailchimp_Upgrader_Base::instance();
}
}
@@ -158,7 +163,7 @@ function _mailchimp_civix_find_files($dir, $pattern) {
return $result;
}
/**
- * (Delegated) Implementation of hook_civicrm_managed
+ * (Delegated) Implements hook_civicrm_managed().
*
* Find any *.mgd.php files, merge their content, and return.
*
@@ -178,7 +183,7 @@ function _mailchimp_civix_civicrm_managed(&$entities) {
}
/**
- * (Delegated) Implementation of hook_civicrm_caseTypes
+ * (Delegated) Implements hook_civicrm_caseTypes().
*
* Find any and return any files matching "xml/case/*.xml"
*
@@ -206,6 +211,31 @@ function _mailchimp_civix_civicrm_caseTypes(&$caseTypes) {
}
}
+/**
+ * (Delegated) Implements hook_civicrm_angularModules().
+ *
+ * Find any and return any files matching "ang/*.ang.php"
+ *
+ * Note: This hook only runs in CiviCRM 4.5+.
+ *
+ * @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_angularModules
+ */
+function _mailchimp_civix_civicrm_angularModules(&$angularModules) {
+ if (!is_dir(__DIR__ . '/ang')) {
+ return;
+ }
+
+ $files = _mailchimp_civix_glob(__DIR__ . '/ang/*.ang.php');
+ foreach ($files as $file) {
+ $name = preg_replace(':\.ang\.php$:', '', basename($file));
+ $module = include $file;
+ if (empty($module['ext'])) {
+ $module['ext'] = 'uk.co.vedaconsulting.mailchimp';
+ }
+ $angularModules[$name] = $module;
+ }
+}
+
/**
* Glob wrapper which is guaranteed to return an array.
*
@@ -224,30 +254,24 @@ function _mailchimp_civix_glob($pattern) {
}
/**
- * Inserts a navigation menu item at a given place in the hierarchy
+ * Inserts a navigation menu item at a given place in the hierarchy.
*
- * $menu - menu hierarchy
- * $path - path where insertion should happen (ie. Administer/System Settings)
- * $item - menu you need to insert (parent/child attributes will be filled for you)
- * $parentId - used internally to recurse in the menu structure
+ * @param array $menu - menu hierarchy
+ * @param string $path - path where insertion should happen (ie. Administer/System Settings)
+ * @param array $item - menu you need to insert (parent/child attributes will be filled for you)
*/
-function _mailchimp_civix_insert_navigation_menu(&$menu, $path, $item, $parentId = NULL) {
- static $navId;
-
+function _mailchimp_civix_insert_navigation_menu(&$menu, $path, $item) {
// If we are done going down the path, insert menu
if (empty($path)) {
- if (!$navId) $navId = CRM_Core_DAO::singleValueQuery("SELECT max(id) FROM civicrm_navigation");
- $navId ++;
- $menu[$navId] = array (
- 'attributes' => array_merge($item, array(
+ $menu[] = array(
+ 'attributes' => array_merge(array(
'label' => CRM_Utils_Array::value('name', $item),
'active' => 1,
- 'parentID' => $parentId,
- 'navID' => $navId,
- ))
+ ), $item),
);
- return true;
- } else {
+ return TRUE;
+ }
+ else {
// Find an recurse into the next level down
$found = false;
$path = explode('/', $path);
@@ -263,13 +287,58 @@ function _mailchimp_civix_insert_navigation_menu(&$menu, $path, $item, $parentId
}
/**
- * (Delegated) Implementation of hook_civicrm_alterSettingsFolders
+ * (Delegated) Implements hook_civicrm_navigationMenu().
+ */
+function _mailchimp_civix_navigationMenu(&$nodes) {
+ if (!is_callable(array('CRM_Core_BAO_Navigation', 'fixNavigationMenu'))) {
+ _mailchimp_civix_fixNavigationMenu($nodes);
+ }
+}
+
+/**
+ * Given a navigation menu, generate navIDs for any items which are
+ * missing them.
+ */
+function _mailchimp_civix_fixNavigationMenu(&$nodes) {
+ $maxNavID = 1;
+ array_walk_recursive($nodes, function($item, $key) use (&$maxNavID) {
+ if ($key === 'navID') {
+ $maxNavID = max($maxNavID, $item);
+ }
+ });
+ _mailchimp_civix_fixNavigationMenuItems($nodes, $maxNavID, NULL);
+}
+
+function _mailchimp_civix_fixNavigationMenuItems(&$nodes, &$maxNavID, $parentID) {
+ $origKeys = array_keys($nodes);
+ foreach ($origKeys as $origKey) {
+ if (!isset($nodes[$origKey]['attributes']['parentID']) && $parentID !== NULL) {
+ $nodes[$origKey]['attributes']['parentID'] = $parentID;
+ }
+ // If no navID, then assign navID and fix key.
+ if (!isset($nodes[$origKey]['attributes']['navID'])) {
+ $newKey = ++$maxNavID;
+ $nodes[$origKey]['attributes']['navID'] = $newKey;
+ $nodes[$newKey] = $nodes[$origKey];
+ unset($nodes[$origKey]);
+ $origKey = $newKey;
+ }
+ if (isset($nodes[$origKey]['child']) && is_array($nodes[$origKey]['child'])) {
+ _mailchimp_civix_fixNavigationMenuItems($nodes[$origKey]['child'], $maxNavID, $nodes[$origKey]['attributes']['navID']);
+ }
+ }
+}
+
+/**
+ * (Delegated) Implements hook_civicrm_alterSettingsFolders().
*
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_alterSettingsFolders
*/
function _mailchimp_civix_civicrm_alterSettingsFolders(&$metaDataFolders = NULL) {
static $configured = FALSE;
- if ($configured) return;
+ if ($configured) {
+ return;
+ }
$configured = TRUE;
$settingsDir = __DIR__ . DIRECTORY_SEPARATOR . 'settings';
diff --git a/mailchimp.php b/mailchimp.php
index 049fe55..7869a95 100644
--- a/mailchimp.php
+++ b/mailchimp.php
@@ -4,6 +4,7 @@
require_once 'vendor/mailchimp/Mailchimp.php';
require_once 'vendor/mailchimp/Mailchimp/Lists.php';
+
/**
* Implementation of hook_civicrm_config
*
@@ -34,26 +35,27 @@ function mailchimp_civicrm_install() {
// Create a cron job to do sync data between CiviCRM and MailChimp.
$params = array(
'sequential' => 1,
- 'name' => 'Mailchimp Sync',
- 'description' => 'Sync contacts between CiviCRM and MailChimp. Pull from mailchimp is performed before push.',
+ 'name' => 'Mailchimp Push Sync',
+ 'description' => 'Sync contacts between CiviCRM and MailChimp, assuming CiviCRM to be correct. Please understand the implications before using this.',
'run_frequency' => 'Daily',
'api_entity' => 'Mailchimp',
- 'api_action' => 'sync',
+ 'api_action' => 'pushsync',
'is_active' => 0,
);
$result = civicrm_api3('job', 'create', $params);
- // create a pull job
- /*$params = array(
+
+ // Create Pull Sync job.
+ $params = array(
'sequential' => 1,
- 'name' => 'Mailchimp Pull',
- 'description' => 'Pull contacts from mailchimp to civi.',
+ 'name' => 'Mailchimp Pull Sync',
+ 'description' => 'Sync contacts between CiviCRM and MailChimp, assuming Mailchimp to be correct. Please understand the implications before using this.',
'run_frequency' => 'Daily',
'api_entity' => 'Mailchimp',
- 'api_action' => 'pull',
+ 'api_action' => 'pullsync',
'is_active' => 0,
);
- $result = civicrm_api3('job', 'create', $params);*/
+ $result = civicrm_api3('job', 'create', $params);
return _mailchimp_civix_civicrm_install();
}
@@ -135,7 +137,9 @@ function mailchimp_civicrm_alterSettingsFolders(&$metaDataFolders = NULL) {
}
/**
- * Implementation of hook_civicrm_buildForm
+ * Implementation of hook_civicrm_buildForm.
+ *
+ * Alter the group settings form to add in our offer of Mailchimp integration.
*
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_buildForm
*/
@@ -161,17 +165,21 @@ function mailchimp_civicrm_buildForm($formName, &$form) {
$options = array(
ts('No integration'),
- ts('Sync membership of this group with membership of a Mailchimp List'),
- ts('Sync membership of with a Mailchimp interest grouping')
+ ts('Membership Sync: Contacts in this group should be subscribed to a Mailchimp List'),
+ ts('Interest Sync: Contacts in this group should have an "interest" set at Mailchimp')
);
$form->addRadio('mc_integration_option', '', $options, NULL, ' ');
+ $form->addElement('checkbox', 'mc_fixup',
+ ts('Ensure list\'s webhook settings are correct at Mailchimp when saved.'));
+
// Prepopulate details if 'edit' action
$groupId = $form->getVar('_id');
if ($form->getAction() == CRM_Core_Action::UPDATE AND !empty($groupId)) {
$mcDetails = CRM_Mailchimp_Utils::getGroupsToSync(array($groupId));
+ $defaults['mc_fixup'] = 1;
if (!empty($mcDetails)) {
$defaults['mailchimp_list'] = $mcDetails[$groupId]['list_id'];
$defaults['is_mc_update_grouping'] = $mcDetails[$groupId]['is_mc_update_grouping'];
@@ -266,23 +274,70 @@ function mailchimp_civicrm_validateForm( $formName, &$fields, &$files, &$form, &
}
}
/**
- * Implementation of hook_civicrm_pageRun
+ * When the group settings form is saved, configure the mailchimp list if
+ * appropriate.
+ *
+ * Implements hook_civicrm_postProcess($formName, &$form)
+ *
+ * @link https://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_postProcess
+ */
+function mailchimp_civicrm_postProcess($formName, &$form) {
+ if ($formName == 'CRM_Group_Form_Edit') {
+ $vals = $form->_submitValues;
+ if (!empty($vals['mc_fixup']) && !empty($vals['mailchimp_list'])
+ && !empty($vals['mc_integration_option']) && $vals['mc_integration_option'] == 1) {
+ // This group is supposed to have Mailchimp integration and the user wants
+ // us to check the Mailchimp list is properly configured.
+ $messages = CRM_Mailchimp_Utils::configureList($vals['mailchimp_list']);
+ foreach ($messages as $message) {
+ CRM_Core_Session::setStatus($message);
+ }
+ }
+ }
+}
+/**
+ * Implementation of hook_civicrm_pageRun.
+ *
*
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_pageRun
*/
function mailchimp_civicrm_pageRun( &$page ) {
if ($page->getVar('_name') == 'CRM_Group_Page_Group') {
- $params = array(
- 'version' => 3,
- 'sequential' => 1,
- );
- // Get all the mailchimp lists/groups and pass it to template as JS array
- // To reduce the no. of AJAX calls to get the list/group name in Group Listing Page
- $result = civicrm_api('Mailchimp', 'getlistsandgroups', $params);
- if(!$result['is_error']){
- $list_and_groups = json_encode($result['values']);
- $page->assign('lists_and_groups', $list_and_groups);
+ // Manage Groups page at /civicrm/group?reset=1
+
+ // Some implementations of javascript don't like using integers for object
+ // keys. Prefix with 'id'.
+ // This combined with templates/CRM/Group/Page/Group.extra.tpl provides js
+ // with a mailchimp_lists variable like: {
+ // 'id12345': 'Membership sync to list Foo',
+ // 'id98765': 'Interest sync to Bar on list Foo',
+ // }
+ $js_safe_object = [];
+ foreach (CRM_Mailchimp_Utils::getGroupsToSync() as $group_id => $group) {
+ if ($group['interest_id']) {
+ if ($group['interest_name']) {
+ $val = strtr(ts("Interest sync to %interest_name on list %list_name"),
+ [
+ '%interest_name' => htmlspecialchars($group['interest_name']),
+ '%list_name' => htmlspecialchars($group['list_name']),
+ ]);
+ }
+ else {
+ $val = ts("BROKEN interest sync. (perhaps list was deleted?)");
+ }
+ }
+ else {
+ if ($group['list_name']) {
+ $val = strtr(ts("Membership sync to list %list_name"),
+ [ '%list_name' => htmlspecialchars($group['list_name']), ]);
+ }
+ else {
+ $val = ts("BROKEN membership sync. (perhaps list was deleted?)");
+ }
+ }
+ $js_safe_object['id' . $group_id] = $val;
}
+ $page->assign('mailchimp_groups', json_encode($js_safe_object));
}
}
@@ -300,6 +355,7 @@ function mailchimp_civicrm_pre( $op, $objectName, $id, &$params ) {
);
if($objectName == 'Email') {
+ return; // @todo
// If about to delete an email in CiviCRM, we must delete it from Mailchimp
// because we won't get chance to delete it once it's gone.
//
@@ -325,6 +381,7 @@ function mailchimp_civicrm_pre( $op, $objectName, $id, &$params ) {
// If deleting an individual, delete their (bulk) email address from Mailchimp.
if ($op == 'delete' && $objectName == 'Individual') {
+ return; // @todo
$result = civicrm_api('Contact', 'get', $params1);
foreach ($result['values'] as $key => $value) {
$emailId = $value['email_id'];
@@ -408,69 +465,107 @@ function mailchimp_civicrm_navigationMenu(&$params){
* @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_post
*/
function mailchimp_civicrm_post( $op, $objectName, $objectId, &$objectRef ) {
+
+ if (!CRM_Mailchimp_Utils::$post_hook_enabled) {
+ // Post hook is disabled at this point in the running.
+ return;
+ }
/***** NO BULK EMAILS (User Opt Out) *****/
if ($objectName == 'Individual' || $objectName == 'Organization' || $objectName == 'Household') {
// Contact Edited
- if ($op == 'edit' || $op == 'create') {
- if($objectRef->is_opt_out == 1) {
- $action = 'unsubscribe';
- } else {
- $action = 'subscribe';
- }
-
- // Get all groups, the contact is subscribed to
- $civiGroups = CRM_Contact_BAO_GroupContact::getGroupList($objectId);
- $civiGroups = array_keys($civiGroups);
-
- if (empty($civiGroups)) {
- return;
- }
-
- // Get mailchimp details
- $groups = CRM_Mailchimp_Utils::getGroupsToSync($civiGroups);
-
- if (!empty($groups)) {
- // Loop through all groups and unsubscribe the email address from mailchimp
- foreach ($groups as $groupId => $groupDetails) {
- CRM_Mailchimp_Utils::subscribeOrUnsubsribeToMailchimpList($groupDetails, $objectId, $action);
- }
- }
- }
+ // @todo artfulrobot: I don't understand the cases this is dealing with.
+ // Perhaps it was trying to check that if someone's been
+ // marked as 'opt out' then they're unsubscribed from all
+ // mailings. I could not follow the logic though -
+ // without tests in place I thought it was better
+ // disabled.
+ if (FALSE) {
+ if ($op == 'edit' || $op == 'create') {
+ if($objectRef->is_opt_out == 1) {
+ $action = 'unsubscribe';
+ } else {
+ $action = 'subscribe';
+ }
+
+ // Get all groups, the contact is subscribed to
+ $civiGroups = CRM_Contact_BAO_GroupContact::getGroupList($objectId);
+ $civiGroups = array_keys($civiGroups);
+
+ if (empty($civiGroups)) {
+ return;
+ }
+
+ // Get mailchimp details
+ $groups = CRM_Mailchimp_Utils::getGroupsToSync($civiGroups);
+
+ if (!empty($groups)) {
+ // Loop through all groups and unsubscribe the email address from mailchimp
+ foreach ($groups as $groupId => $groupDetails) {
+ // method removed. CRM_Mailchimp_Utils::subscribeOrUnsubsribeToMailchimpList($groupDetails, $objectId, $action);
+ }
+ }
+ }
+
+ }
}
/***** Contacts added/removed/deleted from CiviCRM group *****/
if ($objectName == 'GroupContact') {
+ // Determine if the action being taken needs to affect Mailchimp at all.
- // FIXME: Dirty hack to skip hook
- require_once 'CRM/Core/Session.php';
- $session = CRM_Core_Session::singleton();
- $skipPostHook = $session->get('skipPostHook');
-
- // Added/Removed/Deleted - This works for both bulk action and individual add/remove/delete
- if (($op == 'create' || $op == 'edit' || $op == 'delete') && empty($skipPostHook)) {
- // Decide mailchimp action based on $op
- // Add / Rejoin Group
- if ($op == 'create' || $op == 'edit') {
- $action = 'subscribe';
- }
- // Remove / Delete
- elseif ($op == 'delete') {
- $action = 'unsubscribe';
- }
-
- // Get mailchimp details for the group
- $groups = CRM_Mailchimp_Utils::getGroupsToSync(array($objectId));
-
- // Proceed only if the group is configured with mailing list/groups
- if (!empty($groups[$objectId])) {
-
- // Loop through all contacts added/removed from the group
- foreach ($objectRef as $contactId) {
- // Subscribe/Unsubscribe in Mailchimp
- CRM_Mailchimp_Utils::subscribeOrUnsubsribeToMailchimpList($groups[$objectId], $contactId, $action);
- }
- }
- }
+ if ($op == 'view') {
+ // Nothing changed; nothing to do.
+ return;
+ }
+
+ // Get mailchimp details for the group.
+ // $objectId here means CiviCRM group Id.
+ $groups = CRM_Mailchimp_Utils::getGroupsToSync(array($objectId));
+ if (empty($groups[$objectId])) {
+ // This group has nothing to do with Mailchimp.
+ return;
+ }
+
+ // The updates we need to make can be complex.
+ // If someone left/joined a group synced as the membership group for a
+ // Mailchimp list, then that's a subscribe/unsubscribe option.
+ // If however it was a group synced to an interest in Mailchimp, then
+ // the join/leave on the CiviCRM side only means updating interests on the
+ // Mailchimp side, not a subscribe/unsubscribe.
+ // There is also the case that somone's been put into an interest group, but
+ // is not in the membership group, which should not result in them being
+ // subscribed at MC.
+ //
+ // Finally this hook is useful for small changes only; if you just added
+ // thousands of people to a group then this is NOT the way to tell Mailchimp
+ // about it as it would require thousands of separate API calls. This would
+ // probably cause big problems (like hitting the API rate limits, or
+ // crashing CiviCRM due to PHP max execution times etc.). Such updates must
+ // happen in the more controlled bulk update (push).
+
+ if (count($objectRef) > 1) {
+ // Limit application to one contact only.
+ return;
+ }
+
+ if ($groups[$objectId]['interest_id']) {
+ // This is a change to an interest grouping.
+ // We only need update Mailchimp about this if the contact is in the
+ // membership group.
+ $list_id = $groups[$objectId]['list_id'];
+ // find membership group, then find out if the contact is in that group.
+ $membership_group_details = CRM_Mailchimp_Utils::getGroupsToSync(array(), $list_id, TRUE);
+ $result = civicrm_api3('Contact', 'getsingle', ['return'=>'group','contact_id'=>$objectRef[0]]);
+ if (!CRM_Mailchimp_Utils::splitGroupTitles($result['groups'], $membership_group_details)) {
+ // This contact is not in the membership group, so don't bother telling
+ // Mailchimp about a change in their interests.
+ return;
+ }
+ }
+
+ // Trigger mini sync for this person and this list.
+ $sync = new CRM_Mailchimp_Sync($groups[$objectId]['list_id']);
+ $sync->syncSingleContact($objectRef[0]);
}
-}
\ No newline at end of file
+}
diff --git a/templates/CRM/Group/MailchimpSettings.tpl b/templates/CRM/Group/MailchimpSettings.tpl
index 766b6c0..3ae1df0 100644
--- a/templates/CRM/Group/MailchimpSettings.tpl
+++ b/templates/CRM/Group/MailchimpSettings.tpl
@@ -26,6 +26,17 @@
{$form.is_mc_update_grouping.label}
{$form.is_mc_update_grouping.html}
+
+
{$form.mc_fixup.html}{$form.mc_fixup.label}
+ {ts}If this is ticked when you press Save,
+ CiviCRM will edit the webhook settings of this list at Mailchimp to make
+ sure they're configured correctly. The only time you would want to
+ untick this box is if you are doing some development on a local
+ server because that would result in supplying an invalid webhook URL to a
+ possibly production list at mailchimp. So basically leave this ticked,
+ unless you know what you're doing :-){/ts}
+
+ {if $dry_run}
+ {ts}Dry Run: no contacts/members actually changed.{/ts}
+ {/if}
{ts}Import completed with result counts as:{/ts}
{foreach from=$stats item=group}
{$group.name}
-
+
+
{ts}Contacts on CiviCRM and in membership group (originally){/ts}:
{$group.stats.c_count}
+
{ts}Of these, kept because subscribed at Mailchimp:{/ts}:
{$group.stats.in_sync}
+
{ts}Of these, removed because not subscribed at Mailchimp:{/ts}:
{$group.stats.removed}
{ts}Contacts on Mailchimp{/ts}:
{$group.stats.mc_count}
-
{ts}Contacts on CiviCRM (originally){/ts}:
{$group.stats.c_count}
-
{ts}Contacts that were in sync already{/ts}:
{$group.stats.in_sync}
-
{ts}Contacts Added to the CiviCRM group{/ts}:
{$group.stats.added}
-
{ts}Contacts Removed from the CiviCRM group{/ts}:
{$group.stats.removed}
+
{ts}Of these, already in membership group{/ts}:
{$group.stats.in_sync}
+
{ts}Of these, existing contacts added to membership group{/ts}:
{$group.stats.joined}
+
{ts}Of these, new contacts created{/ts}:
{$group.stats.created}
+
{ts}Existing contacts updated{/ts}:
{$group.stats.updated}
{/foreach}
- {/if}
-
-
+ {if $error_messages}
+
Error messages
+
These errors have come from the last sync operation (whether that was a 'pull' or a 'push').
+
+
Group Id
Name and Email
Error
+
+
+ {foreach from=$error_messages item=msg}
+
{$msg.group}
+
{$msg.name} {$msg.email}
+
{$msg.message}
+
+ {/foreach}
+
+ {/if}
+ {else}
+
{ts}Running this will assume that the information in Mailchimp about who is
+ supposed to be a in the CiviCRM membership group is correct.{/ts}
+
{ts}Points to know:{/ts}
+
+
{ts}If a contact is not subscribed at Mailchimp, they will be removed from the CiviCRM membership group (if they were in it).{/ts}
+
{ts}If a contact is subscribed at Mailchimp, they will be added to the CiviCRM membership group (if they were in it). If the contact cannot be found in CiviCRM, a new contact will be created. {/ts}
+
{ts}Any and all CiviCRM groups set up to sync to Mailchimp Interests and configured to allow updates from Mailchimp will be consulted and changes made as needed, adding/removing contacts from these CiviCRM groups.{/ts}
+
{ts}If somone's name is different, the CiviCRM name is replaced by the Mailchimp name (unless there is a name at CiviCRM but no name at Mailchimp).{/ts}
+
{ts}This is a "pull" from Mailchimp operation. You may want the "push" to Mailchimp instead.{/ts}
{$form.security_key.html}
- {ts} Define a security key to be used with webhooks{/ts}
+ {ts}Define a security key to be used with
+ webhooks. e.g. a 12+ character random string of upper- and
+ lower-case letters and numbers. Note if you change this once lists
+ are set up you'll need to update all the groups that serve as
+ memberships for Mailchimp lists.{/ts}
{ts}{$webhook_url}{/ts}
@@ -28,13 +32,6 @@
{$form.enable_debugging.html}
-
-
{$form.list_removal.label}
-
{$form.list_removal.html}
- {ts} Delete or Unsubscribe at MailChimp when corresponding CiviCRM contact is removed?{/ts}
-
-
These errors have come from the last sync operation (whether that was a 'pull' or a 'push').
+
+
Group Id
Name and Email
Error
+
+
+ {foreach from=$error_messages item=msg}
+
{$msg.group}
+
{$msg.name} {$msg.email}
+
{$msg.message}
+
+ {/foreach}
+
+ {/if}
+ {else}
+
{ts}Push contacts from CiviCRM to Mailchimp{/ts}
+
{ts}Running this will assume that the information in CiviCRM about who is
+ supposed to be subscribed to the Mailchimp list is correct.{/ts}
+
{ts}Points to know:{/ts}
+
+
{ts}If a contact is not in the membership group at CiviCRM, they will be unsubscribed from Mailchimp (assuming they are currently subscribed at Mailchimp).{/ts}
+
{ts}If a contact is in the membership group, they will be subscribed at Mailchimp. This could cost you money if adding subscribers exceeds your current tariff.
+ Check the numbers of contacts in each group and/or do a Dry Run first.{/ts}
+
{ts}Any and all CiviCRM groups set up to sync to Mailchimp Interests will be consulted and changes made to members' interests at Mailchimp, as needed.{/ts}
+
{ts}If somone's name is different, the Mailchimp name is replaced by the CiviCRM name (unless there is a name at Mailchimp but no name at CiviCRM).{/ts}
+
{ts}This is a "push" to Mailchimp operation. You may want the "pull" from Mailchimp instead.{/ts}