Module breaks on Silverstripe 4.12

[SanderHamaka]

Situation:

Clean Silverstripe 4.12 install. ModelAdmin with GridField Queued Export button.

Problem:
Job queues fine but always breaks immediately when the task starts to process the queue.
When the project is downgraded tot Silverstripe 4.11 all is fine.

Cause:
LeftAndMain now has afterHandleRequest that checks for $this->response->isError()
If an error is found, the current response (a GridFieldQueuedExportButtonResponse) is replaced by a new HTTPResponse representing an error state.
This is problematic because in the constructor of GridFieldQueuedExportButtonResponse a status of 500 is assigned to the response. As far as I can see this status is never altered after that and triggers the new afterHandleRequest functionality.

Fix:
This could be a fix. Perhaps someone can suggest someting more elaborate to detect the correct setting of the body on the response:

class GridFieldQueuedExportButtonResponse extends HTTPResponse
{
    /**
     * @var GridField
     */
    protected $gridField;

    public function __construct(GridField $gridField)
    {
        $this->gridField = $gridField;

        if($this->gridField) {
            parent::__construct('', 200);
        }
        else {
            parent::__construct('', 500);
        }
    }
}

Edit:

The specific error I'm seeing when I try to run the job via the CLI is:

Call to a member function getManipulatedList() on null at /var/www/vendor/silverstripe/gridfieldqueuedexport/src/Jobs/GenerateCSVJob.php:395)

PR

• FIX Don't mark mocked response as an error #82

[GuySartorelli]

I can't reproduce this failure with a fresh installation of Silverstripe CMS 4.12. Can you please provide the following:

1. results of composer show in your terminal
2. any errors related to this that appear in your error logs, or in the messages tab of the failed job
3. if you have not yet done so, please try to reproduce this error from a fresh installation, so that you can provide steps for reproducing it.

[wilr]

I can confirm the above fix works on 4.12 so it would be good to PR this @SanderHamaka. Noticed it with User Defined Form exports failing in production after 4.12 upgrade.

Thanks @SanderHamaka

[GuySartorelli]

@wilr Given you're able to confirm the fix works, I assume that means you were able to reproduce the issue - can you please provide steps to reproduce the issue?

[wilr]

@GuySartorelli I don't have 'steps' per-se. I can give you access to the affected project if you need to see if in action yourself.

• Open up a certain 4.13 project (PHP 8.1) which has userforms installed and queuedexport (apart from that no other major modules)
• Try to export results from 'Submissions' tab
• Export failed.

After applying the patch it now works again

I have yet to try on Silverstripe 5.

Result of composer info for this project in question.

apimatic/jsonmapper                        v1.3.1             Map nested JSON structures onto PHP classes
asyncphp/doorman                           3.2.0              Child process management
aws/aws-crt-php                            v1.2.1             AWS Common Runtime for PHP
aws/aws-sdk-php                            3.271.6            AWS SDK for PHP - Use Amazon Web Services in your PHP project
axllent/silverstripe-trailing-slash        2.2.6              Ensure that a single trailing slash is always added to the URL
bramus/ansi-php                            3.1                ANSI Control Functions and ANSI Control Sequences (Colors, Erasing, etc.) for PHP CLI Apps
bramus/monolog-colored-line-formatter      2.0.3              Colored Line Formatter for Monolog
bummzack/sortablefile                      2.2.0              An extension for SilverStripe that adds sorting to UploadField.
clue/stream-filter                         v1.6.0             A simple and modern approach to stream filtering in PHP
colymba/gridfield-bulk-editing-tools       3.1.0              SilverStripe GridField component to upload images/files and edit records in bulk
composer/ca-bundle                         1.3.5              Lets you find a path to the system CA bundle, and includes a fallback to the Mozilla CA bundle.
composer/installers                        v1.12.0            A multi-framework Composer library installer
composer/semver                            3.3.2              Semver library that offers utilities, version constraint parsing and validation.
dnadesign/silverstripe-elemental           4.11.1             Elemental pagetype and collection of Elements
dnadesign/silverstripe-elemental-userforms 3.3.1              Adds a new element for usersforms
doctrine/cache                             1.13.0             PHP Doctrine Cache library is a popular cache implementation that supports many different drivers such ...
doctrine/deprecations                      v1.1.0             A small layer on top of trigger_error(E_USER_DEPRECATED) or PSR-3 logging with options to disable all d...
doctrine/lexer                             2.1.0              PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.
dynamic/flexslider                         4.2.0              Display a FlexSlider on pages of your website
dynamic/silverstripe-elemental-flexslider  2.1.1              Flexslider slideshow content block for Silverstripe Elemental
dynamic/silverstripe-linkable              1.1.0              A couple of handy form fields and objects for managing external and internal links on DataObjects
egulias/email-validator                    3.2.6              A library for validating emails against several RFCs
embed/embed                                v4.4.8             PHP library to retrieve page info using oembed, opengraph, etc
fullscreeninteractive/sapphire             dev-main 4de14a6
fullscreeninteractive/source-cli           dev-main 1bab49f   A collection of server scripts to integrate with Source.
graham-campbell/result-type                v1.1.1             An Implementation Of The Result Type
guzzlehttp/guzzle                          7.7.0              Guzzle is a PHP HTTP client library
guzzlehttp/promises                        1.5.3              Guzzle promises library
guzzlehttp/psr7                            2.5.0              PSR-7 message implementation that also provides common utility methods
heyday/silverstripe-cacheinclude           dev-main a3de5d9   Fast Caching for SilverStripe
heyday/silverstripe-hashpath               3.0.4              Hash path provides a function in SilverStripe templates which given a path to an asset returns a path i...
http-interop/http-factory-guzzle           1.2.0              An HTTP Factory using Guzzle PSR7
intervention/image                         2.7.2              Image handling and manipulation library with support for Laravel integration
jean85/pretty-package-versions             2.0.5              A library to get pretty versions strings of installed dependencies
jonom/focuspoint                           4.0.5              Smarter automatic image cropping for SilverStripe
jonom/silverstripe-betternavigator         5.4.1              Front-end utility menu for Silverstripe websites featuring administration and development tools
jonom/silverstripe-text-target-length      2.0.2              Set character length recommendations on SilverStripe text form fields
kub-at/php-simple-html-dom-parser          1.9.1              PHP Simple HTML DOM Parser with namespace and PHP 7.3 compatible
larapack/dd                                1.1                `dd` is a helper method in Laravel. This package will add the `dd` to your application.
league/csv                                 9.8.0              CSV data manipulation made easy in PHP
league/flysystem                           1.1.10             Filesystem abstraction: Many filesystems, one API.
league/mime-type-detection                 1.11.0             Mime-type detection for Flysystem
m1/env                                     2.2.0              Env is a lightweight library bringing .env file parser compatibility to PHP. In short - it enables you ...
marcj/topsort                              2.0.0              High-Performance TopSort/Dependency resolving algorithm
mashape/unirest-php                        v3.0.4             Unirest PHP
ml/iri                                     1.1.4              IRI handling for PHP
ml/json-ld                                 1.2.1              JSON-LD Processor for PHP
monolog/monolog                            1.27.1             Sends your logs to files, sockets, inboxes, databases and various web services
mtdowling/jmespath.php                     2.6.1              Declaratively specify how to extract elements from a JSON document
nikic/php-parser                           v4.15.5            A PHP parser written in PHP
opis/closure                               3.6.3              A library that can be used to serialize closures (anonymous functions) and arbitrary objects.
oscarotero/html-parser                     v0.1.7             Parse html strings to DOMDocument
php-http/client-common                     2.7.0              Common HTTP Client implementations and tools for HTTPlug
php-http/discovery                         1.18.1             Finds and installs PSR-7, PSR-17, PSR-18 and HTTPlug implementations
php-http/httplug                           2.4.0              HTTPlug, the HTTP client abstraction for PHP
php-http/message                           1.16.0             HTTP Message related tools
php-http/message-factory                   1.1.0              Factory interfaces for PSR-7 HTTP Message
php-http/promise                           1.1.0              Promise used for asynchronous HTTP requests
phpoption/phpoption                        1.9.1              Option Type for PHP
pipedrive/pipedrive                        3.2.0              Pipedrive REST client for PHP
psr/cache                                  2.0.0              Common interface for caching libraries
psr/container                              1.1.2              Common Container Interface (PHP FIG PSR-11)
psr/event-dispatcher                       1.0.0              Standard interfaces for event handling.
psr/http-client                            1.0.2              Common interface for HTTP clients
psr/http-factory                           1.0.2              Common interfaces for PSR-7 HTTP message factories
psr/http-message                           1.1                Common interface for HTTP messages
psr/log                                    1.1.4              Common interface for logging libraries
psr/simple-cache                           1.0.1              Common interfaces for simple caching
quinninteractive/silverstripe-seo          1.1.2              An all-in-one SEO module for SilverStripe
ralouphie/getallheaders                    3.0.3              A polyfill for getallheaders.
sentry/sdk                                 3.4.0              This is a metapackage shipping sentry/sentry with a recommended HTTP client.
sentry/sentry                              3.19.1             A PHP SDK for Sentry (http://sentry.io)
silverstripe-terraformers/embargo-expiry   1.2.3              Simple embargo & expiry for Site Tree
silverstripe/admin                         1.13.2             SilverStripe admin interface
silverstripe/asset-admin                   1.13.1             Asset management for the SilverStripe CMS
silverstripe/assets                        1.13.1             SilverStripe Assets component
silverstripe/blog                          3.12.1             A fresh take on blogging in Silverstripe set out to tackle the issue of a cluttered Site Tree.
silverstripe/campaign-admin                1.13.0             SilverStripe campaign admin interface
silverstripe/cms                           4.13.2             The SilverStripe Content Management System
silverstripe/config                        1.6.0              SilverStripe configuration based on YAML and class statics
silverstripe/contentreview                 4.7.0              Flags pages for periodical author review (incl. reporting)
silverstripe/dynamodb                      4.1.0              SilverStripe DynamoDB integration.
silverstripe/errorpage                     1.13.0             ErrorPage component for SilverStripe CMS
silverstripe/event-dispatcher              0.1.3              Publish and subscribe to events in Silverstripe CMS or your Silverstripe application
silverstripe/framework                     4.13.6             The SilverStripe framework
silverstripe/graphql                       4.3.0              GraphQL server for SilverStripe models and other data
silverstripe/gridfieldqueuedexport         2.x-dev 2c15cd5    Export large data sets from your GridField in the SilverStripe CMS interface through async jobs
silverstripe/iframe                        2.4.0              Add an iFrame pagetype to SilverStripe, to show content from another website
silverstripe/lumberjack                    2.3.1              A module to make managing pages in a GridField easy without losing any of the functionality that you're...
silverstripe/mimevalidator                 2.5.0              Checks uploaded file content roughly matches a known MIME type for the file extension.
silverstripe/recipe-cms                    4.13.0             SilverStripe recipe for fully featured page and asset content editing
silverstripe/recipe-core                   4.13.0             SilverStripe framework-only core recipe
silverstripe/recipe-plugin                 1.7.0              Helper plugin to install SilverStripe recipes
silverstripe/redirectedurls                2.2.0              Provides a system for users to configure arbitrary redirections in the CMS
silverstripe/reports                       4.13.0             Reports module for SilverStripe CMS
silverstripe/segment-field                 2.8.0              A reusable approach to segment-generating fields
silverstripe/serve                         2.4.0              Connects the PHP development server to SilverStripe
silverstripe/session-manager               1.5.0              Allow users to manage and revoke access to multiple login sessions across devices.
silverstripe/sharedraftcontent             2.9.1              Share draft page content with non-CMS users
silverstripe/siteconfig                    4.13.0             Site wide settings administration.
silverstripe/spamprotection                3.4.0              Spam protection module for SilverStripe.
silverstripe/spellcheck                    2.5.0              Improves spellcheck support for SilverStripe CMS, including an implementation for HunSpell.
silverstripe/sqlite3                       2.2.1              Adds SQLite3 support to SilverStripe
silverstripe/tagfield                      2.11.1             Tag field for SilverStripe
silverstripe/taxonomy                      2.5.0              Provide extra taxonomy for cataloguing the data.
silverstripe/userforms                     5.15.1             UserForms enables CMS users to create dynamic forms via a drag and drop interface and without getting i...
silverstripe/vendor-plugin                 1.7.0              Allows vendor modules to expose directories to the webroot
silverstripe/versioned                     1.13.0             SilverStripe Versioned component
silverstripe/versioned-admin               1.13.0             SilverStripe versioned admin interface
sminnee/callbacklist                       0.1.1              PHP class that manages a list of callbacks
stevie-mayhew/silverstripe-svg             dev-master 031d78d Basic SVG support for SilverStripe
swiftmailer/swiftmailer                    v6.3.0             Swiftmailer, free feature-rich PHP mailer
symbiote/silverstripe-gridfieldextensions  3.6.2              A collection of useful grid field components
symbiote/silverstripe-queuedjobs           4.12.0             A framework for defining and running background jobs in a queued manner
symfony/cache                              v4.4.48            Provides extended PSR-6, PSR-16 (and tags) implementations
symfony/cache-contracts                    v2.5.2             Generic abstractions related to caching
symfony/config                             v4.4.44            Helps you find, load, combine, autofill and validate configuration values of any kind
symfony/console                            v5.4.24            Eases the creation of beautiful and testable command line interfaces
symfony/deprecation-contracts              v3.0.2             A generic function and convention to trigger deprecation notices
symfony/event-dispatcher                   v5.4.22            Provides tools that allow your application components to communicate with each other by dispatching eve...
symfony/event-dispatcher-contracts         v3.0.2             Generic abstractions related to dispatching event
symfony/expression-language                v3.4.47            Symfony ExpressionLanguage Component
symfony/filesystem                         v5.4.23            Provides basic utilities for the filesystem
symfony/finder                             v4.4.44            Finds files and directories via an intuitive fluent interface
symfony/http-client                        v6.0.20            Provides powerful methods to fetch HTTP resources synchronously or asynchronously
symfony/http-client-contracts              v3.0.2             Generic abstractions related to HTTP clients
symfony/options-resolver                   v6.0.19            Provides an improved replacement for the array_replace PHP function
symfony/polyfill-ctype                     v1.27.0            Symfony polyfill for ctype functions
symfony/polyfill-iconv                     v1.27.0            Symfony polyfill for the Iconv extension
symfony/polyfill-intl-grapheme             v1.27.0            Symfony polyfill for intl's grapheme_* functions
symfony/polyfill-intl-idn                  v1.27.0            Symfony polyfill for intl's idn_to_ascii and idn_to_utf8 functions
symfony/polyfill-intl-normalizer           v1.27.0            Symfony polyfill for intl's Normalizer class and related functions
symfony/polyfill-mbstring                  v1.27.0            Symfony polyfill for the Mbstring extension
symfony/polyfill-php70                     v1.20.0            Symfony polyfill backporting some PHP 7.0+ features to lower PHP versions
symfony/polyfill-php72                     v1.27.0            Symfony polyfill backporting some PHP 7.2+ features to lower PHP versions
symfony/polyfill-php73                     v1.27.0            Symfony polyfill backporting some PHP 7.3+ features to lower PHP versions
symfony/polyfill-php80                     v1.27.0            Symfony polyfill backporting some PHP 8.0+ features to lower PHP versions
symfony/polyfill-php81                     v1.27.0            Symfony polyfill backporting some PHP 8.1+ features to lower PHP versions
symfony/process                            v4.4.44            Executes commands in sub-processes
symfony/service-contracts                  v2.5.2             Generic abstractions related to writing services
symfony/string                             v6.0.19            Provides an object-oriented API to strings and deals with bytes, UTF-8 code points and grapheme cluster...
symfony/translation                        v4.4.47            Provides tools to internationalize your application
symfony/translation-contracts              v2.5.2             Generic abstractions related to translation
symfony/var-dumper                         v6.0.19            Provides mechanisms for walking through any arbitrary PHP variable
symfony/var-exporter                       v5.4.21            Allows exporting any serializable PHP data structure to plain PHP code
symfony/yaml                               v4.4.45            Loads and dumps YAML files
tractorcow/silverstripe-robots             4.0.2              Simple robots generation module for Silverstripe
ua-parser/uap-php                          v3.9.14            A multi-language port of Browserscope's user agent parser.
unclecheese/display-logic                  2.0.6              Allows assignment of conditions for display and hide of specific form fields based on client side behav...
undefinedoffset/silverstripe-nocaptcha     2.4.0              A spam protector and form field using Google's reCAPTCHA v2 or optionally a foundation v3 implementation
vlucas/phpdotenv                           v5.5.0             Loads environment variables from `.env` to `getenv()`, `$_ENV` and `$_SERVER` automagically.
webonyx/graphql-php                        v14.11.9           A PHP port of GraphQL reference implementation
wildbit/swiftmailer-postmark               3.4.0              A Swiftmailer Transport for Postmark.
wilr/silverstripe-googlesitemaps           2.2.1

[GuySartorelli]

Ahh, okay, I've reproduced this now - it doesn't fail if you run the job through the jobs admin section in your browser - but it does fail if you process jobs via the CLI.
Update: Processing the queue in the browser via /dev/tasks/ProcessJobQueueTask also results in the broken job.

The problem does ultimately stem from the new LeftAndMain::afterHandleRequest() handling... I'm tracking down why processing queued jobs via a BuildTask goes through a LeftAndMain. That seems off to me.

[GuySartorelli]

So, as a way to allow serialising a GridField after calling GenerateCSVJob::setGridField(), that method doesn't just set the GridField instance on a private property - it sets a bunch of other properties related to the gridfield including the gridfield's URL.
Then GenerateCSVJob::getGridField() makes a mock get request to that URL, which ultimately is to some admin controller, which is why LeftAndMain gets involved.

If it was only breaking in CLI I'd recommend that the fix is to add a && !Director::isCli() into LeftAndMain::afterHandleRequest() so it doesn't rewrite CLI error responses. That may still be worth doing as its own exercise, but doesn't resolve this problem.

I'm a little hesitant to change the response code for this special response without knowing why 500 was used in the first place - but that was part of the initial commit 7 years ago without any supporting context... so maybe it's not actually important that it's 500 and they just needed some code to use. It's not used as a real response anyway - the whole request that this response is used in is a mock request just to reconstitute the gridfield based on its URL, as a bizarre form of deserialisation... so the fix in the description is probably okay.

I'm gonna do a little more thinking about this but if I can't find an obvious reason why its problematic I'll open a PR that implements something similar to the suggestion in the description.

[GuySartorelli]

I've raised a PR for this.

[emteknetnz]

Linked PR has been merged and released as 2.8.1 and 3.0.1