Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update sales receipt discount logic to provide negative amounts #130

Merged
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ dbt_modules/
logs/
.DS_Store
integration_tests/.DS_Store
dbt_packages/
dbt_packages/
env/
16 changes: 15 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,16 @@
# dbt_quickbooks v0.X.X RELEASE TO BE ANNOUNCED

This pull request includes the following updates:
## Bug Fix
- Adjusted logic for discount sales receipt lines in `int_quickbooks__sales_receipt_double_entry` model to bring in these values properly as negative adjusted amounts in the `quickbooks__general_ledger`.
[PR #130](https://github.com/fivetran/dbt_quickbooks/pull/130)

## Under the Hood
- Added consistency tests within integration tests for the `quickbooks__general_ledger` models.
fivetran-avinash marked this conversation as resolved.
Show resolved Hide resolved

## Contributors
- [@mikerenderco](https://github.com/mikerenderco ) ([PR #131](https://github.com/fivetran/dbt_quickbooks/pull/131))
fivetran-avinash marked this conversation as resolved.
Show resolved Hide resolved

# dbt_quickbooks v0.13.1
[PR #125](https://github.com/fivetran/dbt_quickbooks/pull/125) includes the following updates:

Expand Down Expand Up @@ -42,7 +55,8 @@
- The default end date will be the current date

## Under the Hood
- Updated the maintainer PR template to resemble the most up to date format.
- Updated the maintainer PR template to resemble the most
up to date format.
- Removed the check docs GitHub Action as it is no longer necessary.

# dbt_quickbooks v0.12.2
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

37 changes: 5 additions & 32 deletions docs/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/run_results.json

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions integration_tests/ci/sample.profiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ integration_tests:
pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
port: 5439
schema: quickbooks_integration_tests_05
schema: quickbooks_integration_tests_06
threads: 8
bigquery:
type: bigquery
method: service-account-json
project: 'dbt-package-testing'
schema: quickbooks_integration_tests_05
schema: quickbooks_integration_tests_06
threads: 8
keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
snowflake:
Expand All @@ -33,7 +33,7 @@ integration_tests:
role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
schema: quickbooks_integration_tests_05
schema: quickbooks_integration_tests_06
threads: 8
postgres:
type: postgres
Expand All @@ -42,13 +42,13 @@ integration_tests:
pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
port: 5432
schema: quickbooks_integration_tests_05
schema: quickbooks_integration_tests_06
threads: 8
databricks:
catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
schema: quickbooks_integration_tests_05
schema: quickbooks_integration_tests_06
threads: 8
token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
type: databricks
8 changes: 7 additions & 1 deletion integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ version: '0.13.1'
profile: 'integration_tests'
config-version: 2
models:
+schema: "quickbooks_{{ var('directed_schema','dev') }}"
quickbooks:
+materialized: table
double_entry_transactions:
Expand All @@ -17,9 +18,10 @@ models:
+materialized: table
tmp:
+materialized: view

vars:
quickbooks_source:
quickbooks_schema: quickbooks_integration_tests_05
quickbooks_schema: quickbooks_integration_tests_06
quickbooks_account_identifier: "account_data"
quickbooks_address_identifier: "address_data"
quickbooks_bill_line_identifier: "bill_line_data"
Expand Down Expand Up @@ -61,6 +63,10 @@ vars:
quickbooks_vendor_credit_identifier: "vendor_credit_data"
quickbooks_vendor_identifier: "vendor_data"


# For validation testing. To be commented out before release.
# account_type_exclusions: ['Discount']
fivetran-avinash marked this conversation as resolved.
Show resolved Hide resolved

seeds:
+quote_columns: "{{ true if target.type == 'redshift' else false }}"
quickbooks_integration_tests:
Expand Down
2 changes: 1 addition & 1 deletion integration_tests/packages.yml
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
packages:
- local: ../
- local: ../
3 changes: 2 additions & 1 deletion integration_tests/seeds/account_data.csv
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,5 @@ id,_fivetran_synced,account_number,account_sub_type,account_type,active,balance,
33,2020-06-23 3:58:54,,4d604dd8f008145471dc845683399189,Other Current Asset,TRUE,80922,80922,Asset,2020-04-30 17:29:35,USD,,07bd344f6acd4e8ed41ae67f8d8b4aee,07bd344f6acd4e8ed41ae67f8d8b4aee,,FALSE,0,,2020-04-30 17:29:35
34,2020-12-17 19:31:57,6674,263b6c4c47a488aab804c8b8c12b4f76,Bank,TRUE,83287,83287,Asset,2020-04-30 17:36:45,USD,,939ee17c7427610c55083b8623cde590,939ee17c7427610c55083b8623cde590,,FALSE,0,,2020-12-17 19:21:21
57,2020-12-02 15:31:58,,100fb8196b919896e083af2939bbffb2,Other Current Asset,TRUE,84550,84550,Asset,2020-07-13 19:15:58,USD,,5270c9c9f9382af307c28a838e89c57d,5270c9c9f9382af307c28a838e89c57d,,FALSE,0,,2020-12-02 15:19:27
60,2020-11-09 22:47:02,,9337c3fde59c076b0d3ba902fa18e4e8,Other Current Asset,TRUE,65627,65627,Asset,2020-11-09 22:33:35,USD,,4d41392cafa229d9dffd274c9bf6d278,4d41392cafa229d9dffd274c9bf6d278,,FALSE,0,,2020-11-09 22:33:35
60,2020-11-09 22:47:02,,9337c3fde59c076b0d3ba902fa18e4e8,Other Current Asset,TRUE,65627,65627,Asset,2020-11-09 22:33:35,USD,,4d41392cafa229d9dffd274c9bf6d278,4d41392cafa229d9dffd274c9bf6d278,,FALSE,0,,2020-11-09 22:33:35
75,2020-11-09 22:47:02,,DiscountsRefundsGiven,Discount,TRUE,6,6,,2020-11-09 22:33:35,USD,,4d41392cafa229d9dffd274c9bf6d278,4d41392cafa229d9dffd274c9bf6d278,,FALSE,0,,2020-11-09 22:33:35
2 changes: 1 addition & 1 deletion integration_tests/seeds/item_data.csv
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
id,_fivetran_synced,active,asset_account_id,created_at,description,expense_account_id,fully_qualified_name,income_account_id,inventory_start_date,level,name,parent_item_id,purchase_cost,purchase_description,purchase_tax_code_id,purchase_tax_included,quantity_on_hand,sales_tax_code_id,sales_tax_included,stock_keeping_unit,sub_item,sync_token,taxable,track_quantity_on_hand,type,unit_price,updated_at
2,2020-06-23 3:58:55,TRUE,,2020-03-25 19:24:41,,,6a7e73161603d87b26a8eac49dab0a9c,22,,,6a7e73161603d87b26a8eac49dab0a9c,,0,,,,,,,,,0,FALSE,FALSE,92581ed3a1ac6dec57db359c3de94bb0,0,2020-03-25 19:24:41
1,2020-06-23 3:58:55,TRUE,,2020-03-25 19:24:41,,,11ff9f68afb6b8b5b8eda218d7c83a65,22,,,11ff9f68afb6b8b5b8eda218d7c83a65,,0,,,,,,,,,0,FALSE,FALSE,92581ed3a1ac6dec57db359c3de94bb0,0,2020-03-25 19:24:41
3,2020-07-08 18:17:14,TRUE,,2020-07-08 18:08:45,,,1c24051547edf65b7d7658ddf8e97e38,25,,,1c24051547edf65b7d7658ddf8e97e38,,0,,,,,,,,,0,FALSE,FALSE,92581ed3a1ac6dec57db359c3de94bb0,0,2020-07-08 18:08:45
3,2020-07-08 18:17:14,TRUE,,2020-07-08 18:08:45,,,1c24051547edf65b7d7658ddf8e97e38,25,,,1c24051547edf65b7d7658ddf8e97e38,3,0,,,,,,,,,0,FALSE,FALSE,92581ed3a1ac6dec57db359c3de94bb0,0,2020-07-08 18:08:45
2 changes: 1 addition & 1 deletion integration_tests/seeds/sales_receipt_data.csv
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ id,_fivetran_deleted,_fivetran_synced,apply_tax_after_discount,balance,bill_emai
167,TRUE,2020-12-30 15:17:11,,,a73560940b49f37aca1449905871ee08,,,2020-11-06 1:19:55,,,,,,,,,,,,,USD,,a60b25a0dcb8d84901c6123af8d6d8ed,,,,,,,,,,,,,,,,,,,,,,,,2020-11-06 1:19:55
221,TRUE,2020-12-30 15:17:11,,,7330190becfee9792546addabd1accf0,,,2020-11-06 1:23:33,,,,,,,,,,,,,USD,,a60b25a0dcb8d84901c6123af8d6d8ed,,,,,,,,,,,,,,,,,,,,,,,,2020-11-06 1:23:33
215,TRUE,2020-12-30 15:17:11,,,940bb81cef33bcf145cef7dfd59e4f88,,,2020-11-06 1:23:07,,,,,,,,,,,,,USD,,a60b25a0dcb8d84901c6123af8d6d8ed,,,,,,,,,,,,,,,,,,,,,,,,2020-11-06 1:23:07
203,TRUE,2020-12-30 15:17:11,,,f0af68c786e75aa93741f3a61afa069e,,,2020-11-06 1:22:14,,,,,,,,,,,,,USD,,a60b25a0dcb8d84901c6123af8d6d8ed,,,,,,,,,,,,,,,,,,,,,,,,2020-11-06 1:22:14
291,FALSE,2020-12-30 15:17:11,,,f0af68c786e75aa93741f3a61afa069e,,,2020-11-06 1:22:14,,,,,,,,,,,,,USD,,a60b25a0dcb8d84901c6123af8d6d8ed,,,,,,,,,,,,,,,,,,,,,,,,2020-11-06 1:22:14
7 changes: 4 additions & 3 deletions integration_tests/seeds/sales_receipt_line_data.csv
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
index,sales_receipt_id,_fivetran_synced,amount,bundle_id,bundle_quantity,description,description_service_date,description_tax_code_id,discount_account_id,discount_class_id,discount_discount_percent,discount_percent_based,discount_tax_code_id,sales_item_account_id,sales_item_class_id,sales_item_discount_amount,sales_item_discount_rate,sales_item_item_id,sales_item_quantity,sales_item_service_date,sales_item_tax_code_id,sales_item_unit_price,sub_total_item_id
4,291,2020-12-05 2:47:02,132,,,234805c3f942ccac17aad79eae0a844c,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,132,
3,291,2020-12-05 2:47:02,132,,,234805c3f942ccac17aad79eae0a844c,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,132,
1,143,2020-11-06 1:32:08,323,,,3cf2292340ea48c039f874a276262c13,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,323,
0,291,2020-12-05 2:47:02,549,,,3cf2292340ea48c039f874a276262c13,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,549,
3,341,2020-12-05 2:47:02,136,,,3cf2292340ea48c039f874a276262c13,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,136,
2,291,2020-12-05 2:47:02,423,,,3cf2292340ea48c039f874a276262c13,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,423,
1,291,2020-12-05 2:47:02,423,,,3cf2292340ea48c039f874a276262c13,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,423,
0,213,2020-11-06 1:32:08,413,,,a92c0de6bd6879a5510ec56823945c2d,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,413,
0,219,2020-11-06 1:32:08,768,,,a92c0de6bd6879a5510ec56823945c2d,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,768,
0,221,2020-11-06 1:32:08,438,,,a92c0de6bd6879a5510ec56823945c2d,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,438,
6,361,2020-12-05 3:02:17,283,,,4081cf3f04424df0f1b1bc5205a68d6c,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,283,
4,319,2020-12-05 2:47:02,11,,,a1816b6eab016f3272fbff9c292abcfe,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,11,
4,319,2020-12-05 2:47:02,11,,,a1816b6eab016f3272fbff9c292abcfe,,,,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,11,
2,291,2020-12-05 2:47:02,6,,,234805c3f942ccac17aad79eae0a844c,,,75,,,,,25,,,,3,1,,7624ad70a400ab16bed0602ac396f02b,132,
39 changes: 39 additions & 0 deletions integration_tests/tests/consistency_general_ledger_amounts.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{{ config(
tags="fivetran_validations",
enabled=var('fivetran_validation_tests_enabled', false)
) }}

with prod as (

select
transaction_id,
sum(adjusted_amount) as adjusted_amount_cumulative
from {{ target.schema }}_quickbooks_prod.quickbooks__general_ledger
{{ "where account_type not in " ~ var('account_type_exclusions', []) ~ "" if var('account_type_exclusions', []) }}
group by 1
),

dev as (

select
transaction_id,
sum(adjusted_amount) as adjusted_amount_cumulative
from {{ target.schema }}_quickbooks_dev.quickbooks__general_ledger
{{ "where account_type not in " ~ var('account_type_exclusions', []) ~ "" if var('account_type_exclusions', []) }}
group by 1
),

final as (

select
prod.transaction_id,
prod.adjusted_amount_cumulative as prod_adjusted_amount_cumulative,
dev.adjusted_amount_cumulative as dev_adjusted_amount_cumulative
from prod
full outer join dev
on dev.transaction_id = prod.transaction_id
)

select *
from final
where abs(prod_adjusted_amount_cumulative - dev_adjusted_amount_cumulative) >= 0.01
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{{ config(
tags="fivetran_validations",
enabled=var('fivetran_validation_tests_enabled', false)
) }}

with prod as (
select
1 as join_key,
count(*) as total_transactions
from {{ target.schema }}_quickbooks_prod.quickbooks__general_ledger
group by 1
),

dev as (

select
1 as join_key,
count(*) as total_transactions
from {{ target.schema }}_quickbooks_dev.quickbooks__general_ledger
group by 1
),

final as (

select
prod.join_key,
prod.total_transactions as prod_total_transactions,
dev.total_transactions as dev_total_transactions
from prod
full outer join dev
on dev.join_key = prod.join_key
)

select *
from final
where prod_total_transactions != dev_total_transactions
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@ sales_receipt_lines as (
from {{ ref('stg_quickbooks__sales_receipt_line') }}
),

accounts as (

select *
from {{ ref('stg_quickbooks__account') }}
),

fivetran-joemarkiewicz marked this conversation as resolved.
Show resolved Hide resolved
items as (

select
Expand All @@ -36,7 +42,11 @@ sales_receipt_join as (
sales_receipts.source_relation,
sales_receipt_lines.index,
sales_receipts.transaction_date,
sales_receipt_lines.amount,
case
when sales_receipt_lines.discount_account_id is not null
then sales_receipt_lines.amount * (-1)
else sales_receipt_lines.amount
end as amount,
sales_receipts.deposit_to_account_id as debit_to_account_id,
coalesce(sales_receipt_lines.discount_account_id, sales_receipt_lines.sales_item_account_id, items.parent_income_account_id, items.income_account_id) as credit_to_account_id,
sales_receipts.customer_id,
Expand Down