Merge pull request #31 from civitaspo/tailrec

Set `@tailrec` to runInternal in operation
civitaspo · Feb 17, 2023 · ba8a190 · ba8a190
2 parents 0d50949 + fa5f1c5
commit ba8a190
Show file tree

Hide file tree

Showing 3 changed files with 54 additions and 41 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,20 +1,25 @@
+1.0.3 (2023-02-17)
+==================
+
+- [feat] [#31](https://github.com/civitaspo/embulk-input-dynamodb/pull/31) Add `@tailrec` annotation to `runInternal` in operations.
+
 1.0.2 (2023-02-15)
 ==================
 
-- [chore] [#30](https://github.com/lulichn/embulk-input-dynamodb/pull/30) Fix scan log message.
+- [chore] [#30](https://github.com/civitaspo/embulk-input-dynamodb/pull/30) Fix scan log message.
 
 1.0.1 (2023-02-15)
 ==================
 
-- [chore] [#29](https://github.com/lulichn/embulk-input-dynamodb/pull/29) Upgrade dependencies.
+- [chore] [#29](https://github.com/civitaspo/embulk-input-dynamodb/pull/29) Upgrade dependencies.
     - Use `Exec.getPageBuilder` instead of `new PageBuilder`.
     - Upgrade scala 2.13.1 -> 2.13.10
     - Upgrade aws-sdk 1.11.171 -> 1.12.406
 
 1.0.0 (2023-02-15)
 ==================
 
-- [Breaking Change] [#27](https://github.com/lulichn/embulk-input-dynamodb/pull/27) Upgrade embulk 0.9.23 -> 0.10.41 with removing deprecated features.
+- [Breaking Change] [#27](https://github.com/civitaspo/embulk-input-dynamodb/pull/27) Upgrade embulk 0.9.23 -> 0.10.41 with removing deprecated features.
     - Upgrade Gradle 6.1 -> 7.6
     - Apply gradle-embulk-plugins
     - Remove deprecated features
@@ -23,29 +28,29 @@
 0.3.1 (2020-03-29)
 ==================
 
-- [BugFix] [#23](https://github.com/lulichn/embulk-input-dynamodb/pull/23) Throw scala.MatchError when using "limit" option. (Fix [#21](https://github.com/lulichn/embulk-input-dynamodb/issues/21))
-- [Enhancement] [#23](https://github.com/lulichn/embulk-input-dynamodb/pull/23) Use embulk-core tests library instead of embulk-test.
+- [BugFix] [#23](https://github.com/civitaspo/embulk-input-dynamodb/pull/23) Throw scala.MatchError when using "limit" option. (Fix [#21](https://github.com/civitaspo/embulk-input-dynamodb/issues/21))
+- [Enhancement] [#23](https://github.com/civitaspo/embulk-input-dynamodb/pull/23) Use embulk-core tests library instead of embulk-test.
 
 0.3.0 (2020-03-09)
 ==================
 
 - [Enhancement] Update dependencies
-    - [#5](https://github.com/lulichn/embulk-input-dynamodb/pull/5) JRuby Gradle Plugin (0.1.5 => 1.5.0)
-    - [#6](https://github.com/lulichn/embulk-input-dynamodb/pull/6) Scala (2.11.8 => 2.13.1)
-    - [#7](https://github.com/lulichn/embulk-input-dynamodb/pull/7) AWS DynamoDB SDK (1.10.43 => 1.11.711)
-    - [#8](https://github.com/lulichn/embulk-input-dynamodb/pull/8) Embulk (0.8.13 => 0.9.23)
-- [Enhancement] [#9](https://github.com/lulichn/embulk-input-dynamodb/pull/9) Use TestingEmbulk instead of EmbulkEmbed when testing
-- [Enhancement] [#10](https://github.com/lulichn/embulk-input-dynamodb/pull/10) Reduce test dependencies
-- [Enhancement] [#13](https://github.com/lulichn/embulk-input-dynamodb/pull/13) Use Github Actions instead of CircleCI.
-- [Enhancement] [#15](https://github.com/lulichn/embulk-input-dynamodb/pull/15) Improve development environments
+    - [#5](https://github.com/civitaspo/embulk-input-dynamodb/pull/5) JRuby Gradle Plugin (0.1.5 => 1.5.0)
+    - [#6](https://github.com/civitaspo/embulk-input-dynamodb/pull/6) Scala (2.11.8 => 2.13.1)
+    - [#7](https://github.com/civitaspo/embulk-input-dynamodb/pull/7) AWS DynamoDB SDK (1.10.43 => 1.11.711)
+    - [#8](https://github.com/civitaspo/embulk-input-dynamodb/pull/8) Embulk (0.8.13 => 0.9.23)
+- [Enhancement] [#9](https://github.com/civitaspo/embulk-input-dynamodb/pull/9) Use TestingEmbulk instead of EmbulkEmbed when testing
+- [Enhancement] [#10](https://github.com/civitaspo/embulk-input-dynamodb/pull/10) Reduce test dependencies
+- [Enhancement] [#13](https://github.com/civitaspo/embulk-input-dynamodb/pull/13) Use Github Actions instead of CircleCI.
+- [Enhancement] [#15](https://github.com/civitaspo/embulk-input-dynamodb/pull/15) Improve development environments
     - Introduce [scalafmt](https://scalameta.org/scalafmt/) with [spotless](https://github.com/diffplug/spotless)
     - Fix the format violations
     - Add [scalafmt](https://scalameta.org/scalafmt/) to CI
     - Add [CHANGELOG](./CHANGELOG.md)
     - Add [an example](./example)
     - Update README about development
-- [Enhancement] [#16](https://github.com/lulichn/embulk-input-dynamodb/pull/16) Cleanup gradle settings
-- [New Feature] [#18](https://github.com/lulichn/embulk-input-dynamodb/pull/18) Introduce new `auth_method`: `"session"`, `"anonymous"`, `"web_identity_token"`, `"default"`.
+- [Enhancement] [#16](https://github.com/civitaspo/embulk-input-dynamodb/pull/16) Cleanup gradle settings
+- [New Feature] [#18](https://github.com/civitaspo/embulk-input-dynamodb/pull/18) Introduce new `auth_method`: `"session"`, `"anonymous"`, `"web_identity_token"`, `"default"`.
     - `"anonymous"`: uses anonymous access. This auth method can access only public files.
     - `"session"`: uses temporary-generated **access_key_id**, **secret_access_key** and **session_token**.
     - `"assume_role"`: uses temporary-generated credentials by assuming **role_arn** role.
@@ -56,22 +61,22 @@
         1. `"web_identity_token"`
         1. `"profile"`
         1. `"instance"`
-- [New Feature] [#18](https://github.com/lulichn/embulk-input-dynamodb/pull/18) Support `http_proxy` option when generating aws credentials.
-- [Enhancement] [#18](https://github.com/lulichn/embulk-input-dynamodb/pull/18) The default value of `auth_method` option become `"default"`. When `access_key_id` and `secret_access_key` options are set, use `"basic"` as `auth_method` for backward compatibility.
-- [Deprecated] [#18](https://github.com/lulichn/embulk-input-dynamodb/pull/18) Make `access_key` and `secret_key` options deprecated. Use `access_key_id` and `secret_access_key` options instead.
-- [Deprecated] [#18](https://github.com/lulichn/embulk-input-dynamodb/pull/18) Make `end_point` option deprecated. Use `endpoint` option instead.
-- [Deprecated] [#19](https://github.com/lulichn/embulk-input-dynamodb/pull/19) The original operation implementation is deprecated, so the below options become deprecated.
+- [New Feature] [#18](https://github.com/civitaspo/embulk-input-dynamodb/pull/18) Support `http_proxy` option when generating aws credentials.
+- [Enhancement] [#18](https://github.com/civitaspo/embulk-input-dynamodb/pull/18) The default value of `auth_method` option become `"default"`. When `access_key_id` and `secret_access_key` options are set, use `"basic"` as `auth_method` for backward compatibility.
+- [Deprecated] [#18](https://github.com/civitaspo/embulk-input-dynamodb/pull/18) Make `access_key` and `secret_key` options deprecated. Use `access_key_id` and `secret_access_key` options instead.
+- [Deprecated] [#18](https://github.com/civitaspo/embulk-input-dynamodb/pull/18) Make `end_point` option deprecated. Use `endpoint` option instead.
+- [Deprecated] [#19](https://github.com/civitaspo/embulk-input-dynamodb/pull/19) The original operation implementation is deprecated, so the below options become deprecated.
     - **operation**: Use **query** option or **scan** option instead.
     - **limit**: Use **query.batch_size** option or **query.batch_size** instead.
     - **scan_limit**: Use **query.batch_size** option or **query.batch_size** instead.
     - **record_limit**: Use **query.limit** option or **query.limit** instead.
     - **filters**: Use **query.filter_expression** option or **query.filter_expression** instead.
-- [New Feature] [#19](https://github.com/lulichn/embulk-input-dynamodb/pull/19) Introduce new options **scan**, **query** to support all configurations for Dynamodb Scan/Query Operation API except legacy configurations.
+- [New Feature] [#19](https://github.com/civitaspo/embulk-input-dynamodb/pull/19) Introduce new options **scan**, **query** to support all configurations for Dynamodb Scan/Query Operation API except legacy configurations.
     - NOTE: This operation stores `null` AttributeValue as `null`, though, in the deprecated operation, `null` is converted arbitrarily. (`string` -> empty string, `long` -> `0`, `double` -> `0.0`, `boolean` -> `false`)
     - NOTE: This operation stores timestamp values by parsing user-defined format, though the deprecated operation skips storing values when the column type is defined as `timestamp` without any errors.
     - NOTE: This operation can convert the specific type of the attribute that you specify in **column.attribute_type** to Embulk types, though the deprecated operation can only convert Embulk types that match a particular Dynamodb Attribute type.
-- [Enhancement] [#19](https://github.com/lulichn/embulk-input-dynamodb/pull/19) You can store each dynamodb item as JSON, so **columns** option becomes optional.
-- [Enhancement] [#19](https://github.com/lulichn/embulk-input-dynamodb/pull/19) You can specify the `AttributeValue` type (like `"S"`, `"N"`, `"SS"` and so on) used when converting AttributeValue to Embulk type.
-- [BugFix] [#19](https://github.com/lulichn/embulk-input-dynamodb/pull/19) Avoid `NullPointerException` when Type `N` AttributeValue has `null` in the deprecated operation.
-- [Enhancement] [#19](https://github.com/lulichn/embulk-input-dynamodb/pull/19) Examples work without real Dynamodb.
-- [Enhancement] [#19](https://github.com/lulichn/embulk-input-dynamodb/pull/19) Add more examples.
+- [Enhancement] [#19](https://github.com/civitaspo/embulk-input-dynamodb/pull/19) You can store each dynamodb item as JSON, so **columns** option becomes optional.
+- [Enhancement] [#19](https://github.com/civitaspo/embulk-input-dynamodb/pull/19) You can specify the `AttributeValue` type (like `"S"`, `"N"`, `"SS"` and so on) used when converting AttributeValue to Embulk type.
+- [BugFix] [#19](https://github.com/civitaspo/embulk-input-dynamodb/pull/19) Avoid `NullPointerException` when Type `N` AttributeValue has `null` in the deprecated operation.
+- [Enhancement] [#19](https://github.com/civitaspo/embulk-input-dynamodb/pull/19) Examples work without real Dynamodb.
+- [Enhancement] [#19](https://github.com/civitaspo/embulk-input-dynamodb/pull/19) Add more examples.
diff --git a/src/main/scala/org/embulk/input/dynamodb/operation/DynamodbQueryOperation.scala b/src/main/scala/org/embulk/input/dynamodb/operation/DynamodbQueryOperation.scala
@@ -7,6 +7,7 @@ import org.embulk.input.dynamodb.logger
 
 import scala.jdk.CollectionConverters._
 import scala.util.chaining._
+import scala.annotation.tailrec
 
 object DynamodbQueryOperation {
 
@@ -37,6 +38,7 @@ case class DynamodbQueryOperation(task: DynamodbQueryOperation.Task)
       .tap(r => r.setScanIndexForward(task.getScanIndexForward))
   }
 
+  @tailrec
   private def runInternal(
       dynamodb: AmazonDynamoDB,
       f: Seq[Map[String, AttributeValue]] => Unit,
@@ -53,13 +55,15 @@ case class DynamodbQueryOperation(task: DynamodbQueryOperation.Task)
         f(result.getItems.asScala.take(v.toInt).map(_.asScala.toMap).toSeq)
       case _ =>
         f(result.getItems.asScala.map(_.asScala.toMap).toSeq)
-        Option(result.getLastEvaluatedKey).foreach { lastEvaluatedKey =>
-          runInternal(
-            dynamodb,
-            f,
-            lastEvaluatedKey = Option(lastEvaluatedKey.asScala.toMap),
-            loadedRecords = loadedRecords + result.getCount
-          )
+        Option(result.getLastEvaluatedKey) match {
+          case Some(v) =>
+            runInternal(
+              dynamodb,
+              f,
+              lastEvaluatedKey = Option(v.asScala.toMap),
+              loadedRecords = loadedRecords + result.getCount
+            )
+          case None => // do nothing
         }
     }
   }

diff --git a/src/main/scala/org/embulk/input/dynamodb/operation/DynamodbScanOperation.scala b/src/main/scala/org/embulk/input/dynamodb/operation/DynamodbScanOperation.scala
@@ -10,6 +10,7 @@ import org.embulk.input.dynamodb.logger
 
 import scala.jdk.CollectionConverters._
 import scala.util.chaining._
+import scala.annotation.tailrec
 
 object DynamodbScanOperation {
 
@@ -56,6 +57,7 @@ case class DynamodbScanOperation(task: DynamodbScanOperation.Task)
       }
   }
 
+  @tailrec
   private def runInternal(
       dynamodb: AmazonDynamoDB,
       embulkTaskIndex: Int,
@@ -74,14 +76,16 @@ case class DynamodbScanOperation(task: DynamodbScanOperation.Task)
         f(result.getItems.asScala.take(v.toInt).map(_.asScala.toMap).toSeq)
       case _ =>
         f(result.getItems.asScala.map(_.asScala.toMap).toSeq)
-        Option(result.getLastEvaluatedKey).foreach { lastEvaluatedKey =>
-          runInternal(
-            dynamodb,
-            embulkTaskIndex,
-            f,
-            lastEvaluatedKey = Option(lastEvaluatedKey.asScala.toMap),
-            loadedRecords = loadedRecords + result.getCount
-          )
+        Option(result.getLastEvaluatedKey) match {
+          case Some(v) if (loadableRecords.isDefined) =>
+            runInternal(
+              dynamodb,
+              embulkTaskIndex,
+              f,
+              lastEvaluatedKey = Option(v.asScala.toMap),
+              loadedRecords = loadedRecords + result.getCount
+            )
+          case _ => // do nothing
         }
     }
   }