Add Melt method to DataFrame

[sevenzees]

Add DataFrame.Melt() method for transforming wide to long format

Description

This PR implements a Melt() method for the DataFrame class that transforms data from wide format to long format, similar to Pandas' pandas.melt() function. This is a fundamental data reshaping operation that "unpivots" multiple value columns into a pair of variable-value columns.

Fixes #7577

What does this change do?

The Melt() method:

• Accepts identifier columns that remain fixed in the output
• Unpivots specified value columns (or auto-detects them if not specified) into two new columns: one containing the original column names and one containing the values
• Supports customizable column names for the variable and value columns
• Handles mixed data types across value columns by converting to string when necessary
• Optionally filters out null or empty values from the result
• Includes comprehensive input validation with clear error messages

Why this approach?

Performance optimizations:

• Pre-calculates the total output size to allocate columns once upfront, eliminating expensive incremental resize operations
• Uses direct iteration instead of creating intermediate index arrays
• Caches column references to reduce repeated dictionary lookups

Design decisions:

• Separated data processing into focused helper methods for maintainability
• Follows the Pandas API design for familiarity to users coming from Python
• Maintains type safety by preserving column types when all value columns share the same type
• Defaults to using all non-ID columns as value columns for convenience (matches Pandas behavior)

API signature:

public DataFrame Melt(
    IEnumerable<string> idColumns, 
    IEnumerable<string> valueColumns = null, 
    string variableName = "variable", 
    string valueName = "value", 
    bool dropNulls = false)

Changes included

• DataFrame.cs: Added Melt() method and supporting helper methods
  • CalculateTotalOutputRows(): Pre-calculates output size for efficient allocation
  • InitializeIdColumns(): Sets up ID columns with correct size
  • CreateValueColumn(): Creates appropriately typed value column
  • FillMeltedData(): Performs the actual unpivoting operation

Example usage

// Transform quarterly sales data from wide to long format
var df = new DataFrame(new[]
{
    new StringDataFrameColumn("Region", new[] { "North", "South" }),
    new Int32DataFrameColumn("Q1", new[] { 1000, 800 }),
    new Int32DataFrameColumn("Q2", new[] { 1200, 900 }),
    new Int32DataFrameColumn("Q3", new[] { 1100, 950 })
});

var melted = df.Melt(
    idColumns: new[] { "Region" },
    valueColumns: new[] { "Q1", "Q2", "Q3" },
    variableName: "Quarter",
    valueName: "Sales"
);

// Result:
// | Region | Quarter | Sales |
// |--------|---------|-------|
// | North  | Q1      | 1000  |
// | North  | Q2      | 1200  |
// | North  | Q3      | 1100  |
// | South  | Q1      | 800   |
// | South  | Q2      | 900   |
// | South  | Q3      | 950   |

Additional notes

This implementation brings the .NET DataFrame API closer to feature parity with Pandas and supports common data transformation workflows needed for analysis and visualization. The method is optimized for performance while maintaining code readability and maintainability.

[codecov]

Codecov Report

❌ Patch coverage is 99.74684% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 69.59%. Comparing base (a564b13) to head (a26437d).

Files with missing lines	Patch %	Lines
src/Microsoft.Data.Analysis/DataFrame.cs	99.28%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7578      +/-   ##
==========================================
+ Coverage   69.55%   69.59%   +0.04%     
==========================================
  Files        1484     1484              
  Lines      273209   273604     +395     
  Branches    27919    27949      +30     
==========================================
+ Hits       190029   190423     +394     
- Misses      75817    75820       +3     
+ Partials     7363     7361       -2     

Flag	Coverage Δ
Debug	69.59% <99.74%> (+0.04%)	⬆️
production	63.85% <99.38%> (+0.02%)	⬆️
test	89.63% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/Microsoft.Data.Analysis/Strings.Designer.cs	51.46% <100.00%> (+7.92%)	⬆️
...st/Microsoft.Data.Analysis.Tests/DataFrameTests.cs	99.92% <100.00%> (+0.01%)	⬆️
src/Microsoft.Data.Analysis/DataFrame.cs	93.75% <99.28%> (+1.89%)	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sevenzees]

Not sure who all would want to look at this, but I have another PR here. @tarekgh @ericstj @jeffhandley

[tarekgh]

CC @rokonec @ManickaP who may help with that.

[Copilot]

Pull request overview

Adds a DataFrame.Melt() API to reshape data from wide to long format (similar to pandas.melt), enabling a common “unpivot” transformation within Microsoft.Data.Analysis.

Changes:

• Introduces DataFrame.Melt(...) plus helper methods for validation, sizing, column initialization, and filling.
• Implements optional null/empty filtering (dropNulls) and mixed-type handling (stringifying values when needed).
• Adds new unit tests covering core melt scenarios and some invalid-input cases.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

File	Description
src/Microsoft.Data.Analysis/DataFrame.cs	Adds the Melt() API and its helper methods to produce a long-format DataFrame.
test/Microsoft.Data.Analysis.Tests/DataFrameTests.cs	Adds theory data + tests validating melt output and a few invalid-input cases.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sevenzees]

I have implemented all the Copilot suggestions!

[tarekgh]

@rokonec @ManickaP will you have a chance to look at this PR?

[sevenzees]

@tarekgh Do you know of anyone else who can help?

[tarekgh]

@sevenzees I am following up offline trying to find who can help. Sorry for the delay. I'll try get back to you soon.

[tarekgh]

@sevenzees I just wanted to let you know that I’m still following up on this, so you don’t think I forgot 🙂 Thank you for your patience.

[rokonec]

I was snowed under, I will look at it, and will replay in a three working days.

[sevenzees]

@rokonec Thank you for your review! I have implemented all your suggestions.