Skip to content

HIVE-29451: Optimize MapWork to configure JobConf once per table #6317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hemanthumashankar0511 wants to merge 1 commit into
base: master
Choose a base branch
from
+11 −1
Open

HIVE-29451: Optimize MapWork to configure JobConf once per table #6317

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -658,8 +658,18 @@ public String getSamplingTypeString() {
@Override
public void configureJobConf(JobConf job) {
super.configureJobConf(job);
// Configure each table only once, even if we read thousands of its partitions.
// This avoids repeating expensive work (like loading storage drivers) for every single partition.
Set<TableDesc> processedTables = new HashSet<>();

for (PartitionDesc partition : aliasToPartnInfo.values()) {
PlanUtils.configureJobConf(partition.getTableDesc(), job);
TableDesc tableDesc = partition.getTableDesc();

// If we haven't seen this table before, configure it and remember it.
// If we have seen it, skip it.
if (tableDesc != null && processedTables.add(tableDesc)) {
Copy link
Contributor

@soumyakanti3578 soumyakanti3578 Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although add works here, what do you think about using contains instead? It separates the "checking" from "adding" logic and it is easier to read and maintain the code.

Copy link
Author

@hemanthumashankar0511 hemanthumashankar0511 Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I agree readability is important.

I'm happy to change it to contains() + add() if you think that's clearer. Let me know your preference and I'll update the PR accordingly.

if (tableDesc != null && !processedTables.contains(tableDesc)) {
  processedTables.add(tableDesc);
  PlanUtils.configureJobConf(tableDesc, job);
}

PlanUtils.configureJobConf(tableDesc, job);
}
}
Collection<Operator<?>> mappers = aliasToWork.values();
for (IConfigureJobConf icjc : OperatorUtils.findOperators(mappers, IConfigureJobConf.class)) {
Expand Down