Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(stepfunctions-tasks): EMR createCluster command, add Instances > InstanceFleets > LaunchSpecifications > OnDemandSpecification #12501

Closed
1 of 2 tasks
mohanrajendran opened this issue Jan 14, 2021 · 6 comments
Closed
1 of 2 tasks

(stepfunctions-tasks): EMR createCluster command, add Instances > InstanceFleets > LaunchSpecifications > OnDemandSpecification #12501

mohanrajendran opened this issue Jan 14, 2021 · 6 comments
Labels
@aws-cdk/aws-stepfunctions-tasks closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1

Comments

@mohanrajendran
Copy link

The current InstanceFleetProvisioningSpecificationsProperty (https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions-tasks.EmrCreateCluster.InstanceFleetProvisioningSpecificationsProperty.html) only supports spot specification whereas the EMR InstanceFleetProvisioningSpecifications supports OnDemandSpecification (https://docs.aws.amazon.com/emr/latest/APIReference/API_InstanceFleetProvisioningSpecifications.html#EMR-Type-InstanceFleetProvisioningSpecifications-OnDemandSpecification). I would like the functionality to add OnDemandSpecifications for our usecase.

Use Case

Functional parity with options available in EMR and for our internal use where we need to specify AllocationStrategy under OnDemandSpecifications.

Proposed Solution

Add OnDemandProvisioningSpecificationProperty type with allocationStrategyProperty. Add a field called onDemandSpecification to InstanceFleetProvisioningSpecificationsProperty here. https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts#L598

Finally modify this function (https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/emr/private/cluster-utils.ts#L123) to emit the relevant parameters for ondemand specification.

Other

  • 👋 I may be able to implement this feature request
  • ⚠️ This feature might incur a breaking change

This is a 🚀 Feature Request
@mohanrajendran mohanrajendran added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Jan 14, 2021
mohanrajendran pushed a commit to mohanrajendran/aws-cdk that referenced this issue Jan 14, 2021
feat(aws-stepfunctions-tasks): adding on demand option + allocation s…
e5a8b5e 
…trategy for emrcreatecluster step

The current InstanceFleetProvisioningSpecificationsProperty (https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions-tasks.EmrCreateCluster.InstanceFleetProvisioningSpecificationsProperty.html) only supports spot specification whereas the EMR InstanceFleetProvisioningSpecifications supports OnDemandSpecification (https://docs.aws.amazon.com/emr/latest/APIReference/API_InstanceFleetProvisioningSpecifications.html#EMR-Type-InstanceFleetProvisioningSpecifications-OnDemandSpecification). This commit adds the missing options.

fixes aws#12501
@mohanrajendran mohanrajendran mentioned this issue Jan 14, 2021
Closed
mohanrajendran pushed a commit to mohanrajendran/aws-cdk that referenced this issue Jan 24, 2021
feat(aws-stepfunctions-tasks): adding on demand option + allocation s…
58d395c 
…trategy for emrcreatecluster step

The current InstanceFleetProvisioningSpecificationsProperty (https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions-tasks.EmrCreateCluster.InstanceFleetProvisioningSpecificationsProperty.html) only supports spot specification whereas the EMR InstanceFleetProvisioningSpecifications supports OnDemandSpecification (https://docs.aws.amazon.com/emr/latest/APIReference/API_InstanceFleetProvisioningSpecifications.html#EMR-Type-InstanceFleetProvisioningSpecifications-OnDemandSpecification). This commit adds the missing options.

fixes aws#12501
@NGL321 NGL321 changed the title (aws stepfunction-tasks): EMR createCluster command, add Instances > InstanceFleets > LaunchSpecifications > OnDemandSpecification (stepfunction-tasks): EMR createCluster command, add Instances > InstanceFleets > LaunchSpecifications > OnDemandSpecification Jan 25, 2021
@NGL321 NGL321 changed the title (stepfunction-tasks): EMR createCluster command, add Instances > InstanceFleets > LaunchSpecifications > OnDemandSpecification (stepfunctions-tasks): EMR createCluster command, add Instances > InstanceFleets > LaunchSpecifications > OnDemandSpecification Feb 1, 2021
@shivlaks shivlaks added p1 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Jun 7, 2021
@zanussbaum
Copy link

Is this being actively developed? This a feature that would be a big win for our current Step Function and make migrating to flexible fleets incredibly easy for our team. Otherwise, we will need to re-haul our step function.

@github-actions
Copy link

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Jul 28, 2022
@github-actions github-actions bot closed this as completed Aug 2, 2022
@JiahuiTao
Copy link

JiahuiTao commented Aug 25, 2022 •
edited
Loading

This feature will benefit our team a lot. We have many StepFunctions state machines with most steps running on EMR. Creating an EMR cluster with OnDemandSpecification in Sfn with CDK will make our job a lot more efficient.

@itsrts
Copy link

itsrts commented Aug 25, 2022

This becomes really crucial where required. It would be really helpful if this can be picked up.

@rsmayda
Copy link

rsmayda commented Oct 16, 2023 •
edited
Loading

FYI this cannot be used with CallAwsService either

@rsmayda rsmayda mentioned this issue Oct 30, 2023
Closed
2 tasks
@rsmayda
Copy link

rsmayda commented Oct 30, 2023

Created duplicate here #27761 -- feel free to follow progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-stepfunctions-tasks closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(stepfunctions-tasks): adding on demand option + allocation s… mohanrajendran/aws-cdk
6 participants
@mohanrajendran @rsmayda @itsrts @JiahuiTao @shivlaks @zanussbaum