Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add VPN P3A usage questions #23892

Closed
DJAndries opened this issue Jul 5, 2022 · 5 comments · Fixed by brave/brave-core#14168
Closed

Add VPN P3A usage questions #23892

DJAndries opened this issue Jul 5, 2022 · 5 comments · Fixed by brave/brave-core#14168
Assignees
@DJAndries
Labels
feature/vpn features/P3A OS/Android Fixes related to Android browser functionality OS/Desktop QA Pass - Android ARM QA Pass-Win64 QA/Yes release-notes/exclude
Milestone
1.44.x - Release

Comments

@DJAndries
Copy link
Collaborator

DJAndries commented Jul 5, 2022 •
edited
Loading

Description

Adding three P3A questions for VPN.

As a Brave VPN user, when was the last time I enabled the Brave VPN?
Brave.VPN.LastUsageTime

  1. 0 - 6 days ago
  2. 7 - 13 days ago
  3. 14 - 20 days ago
  4. 21 - 27 days ago
  5. 28 - 59 days ago
  6. 60 days ago or more

As Brave VPN user, how many different days did I enable the Brave VPN in the last 30 days?
Brave.VPN.DaysInMonthUsed

  1. 0 days
  2. 1 day
  3. 2 days
  4. 3 to 5 days
  5. 6 to 10 days
  6. 11 to 15 days
  7. 16 to 20 days
  8. More than 20 days

As a first time user of the Brave VPN this week, did I enable it again the following day?
Brave.VPN.NewUserReturning

  1. I have never enabled the Brave VPN
  2. I have enabled the Brave VPN, but I'm not a first time Brave VPN user this week
  3. I'm a first time Brave VPN user this week but, no, I did not enable the Brave VPN the rest of the week
  4. I'm a first time Brave VPN user this week and, yes, I did enable it again the following day
  5. I'm a first time Brave VPN user this week and, yes, I enabled it again this week but not the following day

Miscellaneous Information:

Will raise PR soon.
@DJAndries DJAndries added OS/Android Fixes related to Android browser functionality OS/Desktop labels Jul 5, 2022
@DJAndries DJAndries self-assigned this Jul 5, 2022
@DJAndries DJAndries mentioned this issue Jul 12, 2022
Closed
25 tasks
@brave-builds brave-builds added this to the 1.43.x - Nightly milestone Jul 13, 2022
@DJAndries DJAndries mentioned this issue Jul 13, 2022
Merged
25 tasks
@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Sep 2, 2022 •
edited
Loading

Verification PASSED on

Brave | 1.44.70 Chromium: 105.0.5195.68 (Official Build) beta (64-bit)
-- | --
Revision | ad13e82529051bac6a0e65f455e6d7a1e5fd7938-refs/branch-heads/5195@{#903}
OS | Windows 10 Version 21H2 (Build 19044.1889)
Brave.VPN.LastUsageTime_PASSED
Case 1: Brave.VPN.LastUsageTime (0 to 6 days ago) =1_PASSED

Case 1:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Open brave://local-state file and ensured that the metrics Brave.VPN.LastUsageTime response value is 1 as expected
  • Close the browser and set the sys date ahead by 1 day
  • Relaunch the browser and ensured that the metrics Brave.VPN.LastUsageTime response value is 1 as expected
    image

Case 2:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 6 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 1 as expected
    image
Case 2: Brave.VPN.LastUsageTime (7 to 13 days ago) =2_PASSED

Case 1:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 7 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 2 as expected
    image

Case 2:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 10 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 2 as expected
    image

Case 3:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 13 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 2 as expected
    image
Case 3: Brave.VPN.LastUsageTime (14 to 20 days ago) =3_PASSED

Case 1:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 14 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 3 as expected
    image

Case 2:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 18 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 3 as expected
    image

Case 3:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 20 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 3 as expected
    image
Case 4: Brave.VPN.LastUsageTime (21 to 27 days ago) =4_PASSED

Case 1:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 21 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 4 as expected
    image

Case 2:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 27 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 4 as expected

Case 3:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 25 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 4 as expected
Case 5: Brave.VPN.LastUsageTime (28 to 59 days ago) =5_PASSED

Case 1:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 28 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 5 as expected

image

Case 2:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 59 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 5 as expected

image

Case 3:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 50 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 5 as expected

image

Case 6: Brave.VPN.LastUsageTime (60 days ago or more) =6_PASSED

Case 1:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 60 days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 6 as expected
    image

Case 2:

  • Clean profile 1.44.x
  • Enable Brave VPN
  • Disonnect the VPN
  • Close the browser and set the sys date ahead by 60+ days
  • Relaunch the browser and open brave://local-state file
  • Ensured that the metrics Brave.VPN.LastUsageTime response value is 6 as expected
    image
Brave.VPN.DaysInMonthUsed_PASSED
Case 2:Brave.VPN.DaysInMonthUsed = 1_PASSED
  • Ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 1 when VPN is enabled for the 1st time in a month
    image
Case 3:Brave.VPN.DaysInMonthUsed = 2_PASSED
  • Clean profile 1.44.x
  • Enable Brave VPN
  • Close the browser, set the sys time ahead by one day
  • Relaunch the browser
  • Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 2 as expected
    image
Case 4:Brave.VPN.DaysInMonthUsed (3 to 5 days)= 3_PASSED
  1. Clean profile 1.44.x
  2. Enable Brave VPN
  3. Close the browser, set the sys time ahead by one day
  4. Relaunch the browser and enusre VPN is enabled
  5. Close the browser, set the sys time ahead by one week
  6. Relaunch the browser and enusre VPN is enabled
  7. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 3 as expected
    image
  8. Repeat the step 3 and 4 twice
  9. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 3 as expected
Case 5:Brave.VPN.DaysInMonthUsed (6 to 10 days)= 4_PASSED
  1. Clean profile 1.44.x
  2. Enable Brave VPN
  3. Close the browser, set the sys time ahead by one day
  4. Relaunch the browser and enusre VPN is enabled
  5. Repeat the step 3, 4 for 4 times
  6. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 4 as expected
    image
  7. Repeat the step 3, 4 for 6 times
  8. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 4 as expected
Case 6:Brave.VPN.DaysInMonthUsed (11 to 15 days)= 5_PASSED
  1. Clean profile 1.44.x
  2. Enable Brave VPN
  3. Close the browser, set the sys time ahead by one day
  4. Relaunch the browser and enusre VPN is enabled
  5. Repeat the step 3, 4 for 9 times
  6. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 5 as expected
    image
  7. Repeat the step 3, 4 for 4 times
  8. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 5 as expected
Case 7:Brave.VPN.DaysInMonthUsed (16 to 20 days)= 6_PASSED
  1. Clean profile 1.44.x
  2. Enable Brave VPN
  3. Close the browser, set the sys time ahead by one day
  4. Relaunch the browser and enusre VPN is enabled
  5. Repeat the step 3, 4 for 14 times
  6. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 6 as expected
    image
  7. Repeat the step 3, 4 for 4 times
  8. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 6 as expected
Case 8:Brave.VPN.DaysInMonthUsed (21 days and more)= 7_PASSED
  1. Clean profile 1.44.x
  2. Enable Brave VPN
  3. Close the browser, set the sys time ahead by one day
  4. Relaunch the browser and enusre VPN is enabled
  5. Repeat the step 3, 4 for 19 times
  6. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 7 as expected
    image
  7. Repeat the step 3, 4 for 4 times
  8. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 7 as expected
Case 9: VPN Usage days (30 days) distributed within two months_PASSED
  1. Clean profile 1.44.x
  2. Enable Brave VPN today (SEP 9)
  3. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 1 as expected
  4. Close the browser and set the sys date ahead by one day (SEP 10)
  5. Relaunch the browser
  6. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 2 as expected
  7. Close the browser and set the sys date to OCT 1
  8. Relaunch the browser
  9. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 3 as expected
  10. Close the browser and set the sys date to OCT 2
  11. Relaunch the browser
  12. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 3 as expected
  13. Close the browser and set the sys date to OCT 7
  14. Relaunch the browser
  15. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 3 as expected
  16. Close the browser and set the sys date ahead by one day OCT 8
  17. Relaunch the browser
  18. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 4 as expected
  19. Close the browser and set the sys date to OCT 20
  20. Relaunch the browser and ensure VPN is disconnected
  21. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 3 as expected (Sep 20 to Oct 20 is 30days and the VPN usage is 4 days (OCT1, OCT2, OCT7 and OCT8) during this period, 3 to 5 days usage fall under the response bucket 3)
  22. Close the browser and set the sys date to NOV 3
  23. Relaunch the browser and ensure VPN is disconnected
  24. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 2 as expected
    (Oct 3 to Nov 3 is 30days and the VPN usage is 2 days (OCT7 and OCT8) during this period, 2 days usage fall under the response bucket 2)
  25. Close the browser and set the sys date to NOV 8
  26. Relaunch the browser and ensure VPN is disconnected
  27. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 1 as expected
    (Oct 8 to Nov 8 is 30days and the VPN usage is 1 day (OCT8) during this period, 1 day usage fall under the response bucket 1)
  28. Close the browser and set the sys date to NOV 9
  29. Relaunch the browser and ensure VPN is disconnected
  30. Open brave://local-state file and ensured that the metrics Brave.VPN.DaysInMonthUsed response value is 0 as expected
    (Oct 9 to Nov 9 is 30days and the VPN usage is 0 day during this period)
Brave.VPN.NewUserReturning_PASSED
Case 1:Brave.VPN.NewUserReturning = 0_PASSED
  • Ensured that the metrics Brave.VPN.NewUserReturning response value is 0 when VPN is not enabled
    image
Case 2:Brave.VPN.NewUserReturning = 1_PASSED
  • Clean profile 1.44.x
  • Enable Brave VPN
  • Close the browser, set the sys time ahead by one day
  • Relaunch the browser
  • Ensure VPN is enabled
  • Close the browser, set the sys time ahead by one week
  • Relaunch the browser and ensure VPN is enabled
  • Open brave://local-state file and ensured that the metrics Brave.VPN.NewUserReturning response value is 1 as expected
    image
Case 3:Brave.VPN.NewUserReturning = 2_PASSED
  • Ensured that the metrics Brave.VPN.NewUserReturning response value is 1 when user enable the VPN first time this week but, did not enable the Brave VPN the rest of the week
    image
Case 4:Brave.VPN.NewUserReturning = 3_PASSED
  • Clean profile 1.44.x
  • Enable Brave VPN
  • Close the browser, set the sys time ahead by one day
  • Relaunch the browser
  • Ensure VPN is enabled
  • Open brave://local-state file and ensured that the metrics Brave.VPN.NewUserReturning response value is 3 as expected
    image
Case 5:Brave.VPN.NewUserReturning = 4_PASSED
  • Clean profile 1.44.x
  • Enable Brave VPN
  • Close the browser, set the sys time ahead by 3 days
  • Relaunch the browser
  • Ensure VPN is enabled
  • Open brave://local-state file and ensured that the metrics Brave.VPN.NewUserReturning response value is 4 as expected
    image

@GeetaSarvadnya GeetaSarvadnya added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Sep 2, 2022
@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Sep 5, 2022 •
edited
Loading

@DJAndries Few questions related to the test plan brave/brave-core#14168. Could you please review them?

  1. The test plan says turn on VPN in steps 2,6 and 10. Once the VPN is enabled user can access the VPN even if they advance the system clock and VPN doesn't gets disconnected once the browser is closed and system clock is set ahead, I am not sure why we need to turn on the VPN in steps 2,6 and 10. The browser state (Open/close) doesn't determine the VPN connection.
  2. The metrics Brave.VPN.LastUsageTime response value are not working as expected. Steps which I followed are as below:
    1. Clean profile 1.44.x
    2. Enable Brave VPN
    3. Open brave://local-state file and ensured Brave.VPN.LastUsageTime metrics response value is 1 as expected
    4. Set the system clock ahead by one week or more than one week, the metrics reponse value is 1 instead of 2
  3. The Brave.VPN.DaysInMonthUsed metrics response range overlapping in two response buckets in Add VPN P3A usage questions #23892 (comment) hence the response range needs to updated for 4, 5 and 6 as below
    4) 5 to 10 days
    5) 11 to 15 days
    6) 16 to 20 days

@GeetaSarvadnya GeetaSarvadnya mentioned this issue Sep 5, 2022
Closed
@DJAndries
Copy link
Collaborator Author

@GeetaSarvadnya These are good points. I updated the Windows test plan, P3A wiki and original issue to address your concerns. Thanks

@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Sep 9, 2022 •
edited
Loading

@DJAndries Thanks for the updated test plan. Still the test plan needs to be updated, when the user reconnects (step 9) the VPN on the future date the server throws the error and unbale to connect the VPN (#25244 (comment)) In this case we won't get the expected result which you mentioned in step 10

I would need one more clarification on the metrics Brave.VPN.NewUserReturning - I assume a week is considered from the day when VPN is enabled not the calendar week. Is my understanding correct?

@GeetaSarvadnya GeetaSarvadnya mentioned this issue Sep 9, 2022
Closed
@GeetaSarvadnya GeetaSarvadnya added QA Pass-Win64 and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Sep 9, 2022
@Uni-verse Uni-verse added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Sep 22, 2022
@Uni-verse
Copy link
Contributor

Uni-verse commented Sep 22, 2022 •
edited
Loading

Verified on Samsung Galaxy S21 5G running Android version 12 using the following build(s):

Brave	1.44.97 Chromium: 106.0.5249.55 (Official Build) (64-bit) 
Revision	4d5f098fca6ab7f4b6b7c240be3d9593c2357709-refs/branch-heads/5249@{#531}
OS	Android 12; Build/SP1A.210812.016

Test Scenarios based on #23892 (comment)

Brave.VPN.LastUsageTime - Passed

1 = 0 - 6 days ago
2 = 7 - 13 days ago
3 = 14 - 20 days ago
4 = 21 - 27 days ago
5 = 28 - 59 days ago
6 = 60 days ago or more

0 - 6 days ago 7 - 13 days ago 14 - 20 days ago 21 - 27 days ago 28 - 59 days ago 60 days ago or more
screenshot-1663881538490 screenshot-1663881702465 screenshot-1663881799188 screenshot-1663881895628 screenshot-1663881965096 screenshot-1663882049972

Brave.VPN.DaysInMonthUsed - Pass

0 = 0 days
1 = 1 day
2 = 2 days
3 = 3 to 5 days
4 = 6 to 10 days
5 = 11 to 15 days
6 = 16 to 20 days
7 = More than 20 days

1 day 2 days 3 to 5 days 6 to 10 days 11 to 15 days 16 to 20 days More than 20 days
screenshot-1663944706244 screenshot-1663945333974 screenshot-1663948728286 screenshot-1663949519587 screenshot-1663949908711 screenshot-1663950179644 screenshot-1663950448916

Brave.VPN.NewUserReturning - Passed

0 = I have never enabled the Brave VPN
1 = I have enabled the Brave VPN, but I'm not a first time Brave VPN user this week
2 = I'm a first time Brave VPN user this week but, no, I did not enable the Brave VPN the rest of the week
3 = I'm a first time Brave VPN user this week and, yes, I did enable it again the following day
4 = I'm a first time Brave VPN user this week and, yes, I enabled it again this week but not the following day

0 1 2 3 4
screenshot-1663951186127 screenshot-1663950726747 screenshot-1663951090039 screenshot-1663945377067 screenshot-1663948511588

@Uni-verse Uni-verse added QA Pass - Android ARM and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Sep 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DJAndries DJAndries

Labels
feature/vpn features/P3A OS/Android Fixes related to Android browser functionality OS/Desktop QA Pass - Android ARM QA Pass-Win64 QA/Yes release-notes/exclude
Projects
None yet
Milestone
1.44.x - Release
Development

Successfully merging a pull request may close this issue.

Added VPN usage P3A questions brave/brave-core
Added VPN usage P3A brave/brave-core
5 participants
@bsclifton @DJAndries @Uni-verse @GeetaSarvadnya @brave-builds