[fix][broker] fix MetadataStoreException$NotFoundException while doing topic lookup by codelipenghui · Pull Request #15633 · apache/pulsar

codelipenghui · 2022-05-17T09:45:37Z

Motivation

#14839 fixes the HTTP lookup with multiple advertised listeners, and the added test
is trying to verify the advertised listeners with 2 brokers, but the config of the
the additional broker will not update, so the test gets passed.

The root cause is we are using the (advertised address + WebService port) to register
the broker under the loadbalance path, but while creating the lookup result, we will
use the WebService URL to get the data, so we will get MetadataStoreException$NotFoundException.

2022-05-17T14:18:55,658+0800 [pulsar-load-manager-38-1] INFO  org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl - Acquired resource lock on /loadbalance/brokers/BROKER-0:15004
2022-05-17T14:19:01,126+0800 [pulsar-4-1] WARN  org.apache.pulsar.broker.lookup.TopicLookupBase - Failed to lookup broker for topic persistent://public/default/my-topic: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: /loadbalance/brokers/xxxx:15004
java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: /loadbalance/brokers/xxxx:15004
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
	at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:747) ~[?:?]
	at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:735) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2182) ~[?:?]
	at org.apache.pulsar.broker.namespace.NamespaceService.searchForCandidateBroker(NamespaceService.java:599) ~[classes/:?]
	at org.apache.pulsar.broker.namespace.NamespaceService.lambda$findBrokerServiceUrl$8(NamespaceService.java:405) ~[classes/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.76.Final.jar:4.1.76.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: /loadbalance/brokers/xxxx:15004
	at org.apache.pulsar.broker.namespace.NamespaceService.lambda$createLookupResult$18(NamespaceService.java:642) ~[classes/:?]
	at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:757) ~[?:?]
	at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:735) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2182) ~[?:?]
	at org.apache.pulsar.broker.namespace.NamespaceService.createLookupResult(NamespaceService.java:620) ~[classes/:?]
	at org.apache.pulsar.broker.namespace.NamespaceService$MockitoMock$1465504970.createLookupResult$accessor$rPJehdnB(Unknown Source) ~[classes/:?]
	at org.apache.pulsar.broker.namespace.NamespaceService$MockitoMock$1465504970$auxiliary$608QTNMe.call(Unknown Source) ~[classes/:?]
	at org.mockito.internal.invocation.RealMethod$FromCallable$1.call(RealMethod.java:40) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.invocation.RealMethod$FromBehavior.invoke(RealMethod.java:62) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.invocation.InterceptedInvocation.callRealMethod(InterceptedInvocation.java:142) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:45) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.Answers.answer(Answers.java:99) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:110) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:56) ~[mockito-core-3.12.4.jar:?]
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptSuperCallable(MockMethodInterceptor.java:145) ~[mockito-core-3.12.4.jar:?]
	at org.apache.pulsar.broker.namespace.NamespaceService$MockitoMock$1465504970.createLookupResult(Unknown Source) ~[classes/:?]
	at org.apache.pulsar.broker.namespace.NamespaceService.searchForCandidateBroker(NamespaceService.java:598) ~[classes/:?]
	... 9 more

Modification

Added properties support for ResourceUnit, so that we can carry more information while getting the least loaded broker
For creating the lookup result, use the broker znode name to get the broker data if it presents in the ResourceUnit

Verification

No new tests are needed, just fix the test that was added in #14839

Documentation

Check the box below or label this PR directly.

Need to update docs?

no-need-doc
(bug fix)

…g topic lookup ### Motivation apache#14839 fixes the http lookup with multiple advertised listeners, and the added test is try to verify the advertised listeners with 2 brokers, but the config of the additional broker will not update, so the test get passed. The root cause is we are using the (advertised address + WebService port) to register the broker under the loadbalance path, but while creating the lookup result, we will use the WebService URL to get the data, so we will get `MetadataStoreException$NotFoundException`. ### Modification - Added properties support for ResourceUnit, so that we can carry more information while getting least loaded broker - For creating the lookup result, use the broker znode name to get the broker data if it presents in the ResourceUnit ### Verification No new tests needed, just fix the test that added in apache#14839

github-actions · 2022-05-17T09:46:15Z

@codelipenghui:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

github-actions · 2022-05-17T09:46:15Z

@codelipenghui:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

github-actions · 2022-05-17T09:46:27Z

@codelipenghui:Thanks for providing doc info!

codelipenghui · 2022-05-17T13:03:52Z

/pulsarbot run-failure-checks

…g topic lookup (#15633) (cherry picked from commit 70551a6)

…g topic lookup (apache#15633) (cherry picked from commit 70551a6) (cherry picked from commit 6415b31)

…g topic lookup (#15633) (cherry picked from commit 70551a6)

codelipenghui requested a review from merlimat May 17, 2022 09:45

codelipenghui self-assigned this May 17, 2022

codelipenghui added type/bug The PR fixed a bug or issue reported a bug area/broker labels May 17, 2022

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label May 17, 2022

codelipenghui added release/2.9.0 release/2.10.1 and removed doc-not-needed Your PR changes do not impact docs labels May 17, 2022

github-actions Bot added the doc-label-missing label May 17, 2022

codelipenghui added the doc-not-needed Your PR changes do not impact docs label May 17, 2022

codelipenghui added this to the 2.11.0 milestone May 17, 2022

github-actions Bot removed the doc-label-missing label May 17, 2022

codelipenghui requested review from 315157973, BewareMyPower, RobertIndie, Technoboy-, gaoran10, hangc0276 and mattisonchao May 17, 2022 09:46

315157973 approved these changes May 18, 2022

View reviewed changes

Technoboy- approved these changes May 18, 2022

View reviewed changes

BewareMyPower added release/2.9.3 and removed release/2.9.0 labels May 18, 2022

BewareMyPower reviewed May 18, 2022

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/SimpleResourceUnit.java Outdated

BewareMyPower reviewed May 18, 2022

View reviewed changes

Comment thread pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/AdvertisedListenersTest.java Outdated

BewareMyPower reviewed May 18, 2022

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java

Address comments.

3ea5963

BewareMyPower approved these changes May 19, 2022

View reviewed changes

Demogorgon314 approved these changes May 19, 2022

View reviewed changes

Technoboy- merged commit 70551a6 into apache:master May 19, 2022

codelipenghui deleted the penghui/fix-broker-lookup branch May 20, 2022 07:05

codelipenghui added a commit that referenced this pull request May 20, 2022

[fix][broker] fix MetadataStoreException$NotFoundException while doin…

6415b31

…g topic lookup (#15633) (cherry picked from commit 70551a6)

codelipenghui added the cherry-picked/branch-2.10 label May 20, 2022

nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request May 23, 2022

[fix][broker] fix MetadataStoreException$NotFoundException while doin…

7b37a8f

…g topic lookup (apache#15633) (cherry picked from commit 70551a6) (cherry picked from commit 6415b31)

mattisonchao pushed a commit that referenced this pull request May 25, 2022

[fix][broker] fix MetadataStoreException$NotFoundException while doin…

0813a1b

…g topic lookup (#15633) (cherry picked from commit 70551a6)

mattisonchao added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label May 25, 2022

mattisonchao mentioned this pull request May 25, 2022

[branch-2.9][fix][broker] Fix jdk API compatibility issues. #15779

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix][broker] fix MetadataStoreException$NotFoundException while doing topic lookup#15633

[fix][broker] fix MetadataStoreException$NotFoundException while doing topic lookup#15633
Technoboy- merged 2 commits into
apache:masterfrom
codelipenghui:penghui/fix-broker-lookup

codelipenghui commented May 17, 2022 •

edited

Loading

Uh oh!

github-actions Bot commented May 17, 2022

Uh oh!

github-actions Bot commented May 17, 2022

Uh oh!

github-actions Bot commented May 17, 2022

Uh oh!

codelipenghui commented May 17, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

codelipenghui commented May 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

Verification

Documentation

Uh oh!

github-actions Bot commented May 17, 2022

Uh oh!

github-actions Bot commented May 17, 2022

Uh oh!

github-actions Bot commented May 17, 2022

Uh oh!

codelipenghui commented May 17, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codelipenghui commented May 17, 2022 •

edited

Loading