Skip to content

Commit

Permalink
Databus Throttling Changes (#847)
Browse files Browse the repository at this point in the history 
* feat: execution name sanitization and unit test for stepfn

* master_size api changes

* Admin API changes (#845)

1. /UpdateParameterStore
2. /QueueExecutionAttributes/<queue_type>/<queue_name>

* PD-257044-changes-for-databus (#846)

* added datadog metrics changes

* [PD-257044] databus changes rebase (#848)

* PD-257044-changes-for-databus

* duplicated kafka and ssm util packages in emo sor module

* removed redundant dependencies

* admin api and queue name split

* Pd 257044 databus changes rebase (#849)

* PD-257044-changes-for-databus

* duplicated kafka and ssm util packages in emo sor module

* removed redundant dependencies

* additional changes

* additional changes

* additional changes

* removed unneccessary dependencies

* Revert "added datadog metrics changes"

This reverts commit 62dc31e.

* build fix

* endpoint change and databus changes

* updated updateRefInDatabus method

* address review commnets

* branch admin -prepare release emodb-6.5.205

* branch admin -prepare for next development iteration

---------

Co-authored-by: Rishu Yadav <rishu.yadav@bazaarvoice.com>
Co-authored-by: shanmohammed-bv <ntyshaan@gmail.com>
Co-authored-by: Anurag Dubey <anurag.dubey@bazaarvoice.com>
Co-authored-by: Anandu Jayan <anandu.jayan@bazaarvoice.com>
Co-authored-by: jenkins <bv-jenkins@bazaarvoice.com>
  • Loading branch information
@nabajyotiDash-hub @rishuyadavbv
@shan-1094 @anurag0510 @anandujayan
6 people authored Nov 6, 2024
1 parent a344557 commit 387db36
Show file tree
Hide file tree
Showing 101 changed files with 1,489 additions and 157 deletions.
2 changes: 1 addition & 1 deletion auth/auth-client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-core/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-store/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-util/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>emodb</artifactId>
<groupId>com.bazaarvoice.emodb</groupId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion blob-api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client-common/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client-jersey2/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion cachemgr/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/astyanax/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client-jax-rs-2/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client-jersey2/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/dropwizard/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/jersey-client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/json/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/stash/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/uuid/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/zookeeper/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
4 changes: 4 additions & 0 deletions databus-api/src/main/java/com/bazaarvoice/emodb/databus/api/Databus.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ Subscription getSubscription(String subscription)
*/
long getEventCountUpTo(String subscription, long limit);

default long getMasterCount(){
return 0;
};

/** Counts the number of events with outstanding claims that cause the events to be skipped by {@link #poll}. */
long getClaimCount(String subscription);

Expand Down
2 changes: 1 addition & 1 deletion databus-client-common/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client-jersey2/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
5 changes: 5 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/DatabusFactory.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,11 @@ public long getEventCountUpTo(String subscription, long limit) {
return _ownerAwareDatabus.getEventCountUpTo(ownerId, subscription, limit);
}

@Override
public long getMasterCount() {
return _ownerAwareDatabus.getMasterEventCountUncached(ownerId);
}

@Override
public long getClaimCount(String subscription) {
return _ownerAwareDatabus.getClaimCount(ownerId, subscription);
Expand Down
12 changes: 12 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/DefaultDatabus.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,18 @@ public long getEventCount(String ownerId, String subscription) {
return getEventCountUpTo(ownerId, subscription, Long.MAX_VALUE);
}

@Override
public long getMasterEventCountUncached(String ownerId) {
_log.info("Inside getMasterEventCountUncached with _masterFanoutChannels length {}", _masterFanoutChannels.size());
long size = 0;
for(String channel : _masterFanoutChannels) {
size += _eventStore.getSizeEstimate(channel, Long.MAX_VALUE);
_log.info("From channel size {} {}:", channel, size);
}
return size;
}


@Override
public long getEventCountUpTo(String ownerId, String subscription, long limit) {
checkSubscriptionOwner(ownerId, subscription);
Expand Down
3 changes: 3 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/OwnerAwareDatabus.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ Subscription getSubscription(String ownerId, String subscription)
long getEventCount(String ownerId, String subscription)
throws UnauthorizedSubscriptionException;

public long getMasterEventCountUncached(String ownerId)
throws UnauthorizedSubscriptionException;

long getEventCountUpTo(String ownerId, String subscription, long limit)
throws UnauthorizedSubscriptionException;

Expand Down
2 changes: 1 addition & 1 deletion datacenter/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion event/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion job-api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion job/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion kafka/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion megabus/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
4 changes: 2 additions & 2 deletions parent/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<packaging>pom</packaging>

<name>EmoDB Parent</name>
Expand All @@ -20,7 +20,7 @@
<connection>scm:git:git@github.com:bazaarvoice/emodb.git</connection>
<developerConnection>scm:git:git@github.com:bazaarvoice/emodb.git</developerConnection>
<url>scm:git:git@github.com:bazaarvoice/emodb.git</url>
<tag>emodb-6.5.98</tag>
<tag>emodb-6.5.208-SNAPSHOT</tag>
</scm>

<properties>
Expand Down
2 changes: 1 addition & 1 deletion plugins/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
4 changes: 2 additions & 2 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>parent/pom.xml</relativePath>
</parent>

Expand All @@ -18,7 +18,7 @@
<connection>scm:git:git@github.com:bazaarvoice/emodb.git</connection>
<developerConnection>scm:git:git@github.com:bazaarvoice/emodb.git</developerConnection>
<url>scm:git:git@github.com:bazaarvoice/emodb.git</url>
<tag>emodb-6.5.98</tag>
<tag>emodb-6.5.208-SNAPSHOT</tag>
</scm>

<modules>
Expand Down
2 changes: 1 addition & 1 deletion quality/integration/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
Loading

0 comments on commit 387db36

Please sign in to comment.