LINQ Mastery

Chunk: Slicing data for batch processing

1 Views Updated 5/4/2026

Batch Processing made Easy

Before .NET 6, splitting a list into smaller batches required messy math with Skip and Take. Chunk does it in one line.

1. Why Chunk?

Many APIs and Databases have limits on payload size (e.g., 'You can only insert 500 rows at a time'). Chunk allows you to process a massive list in bite-sized pieces.


// Split 10,000 IDs into batches of 100
var batches = largeListOfIds.Chunk(100);

foreach (var batch in batches) {
    // Process 100 items...
    await SendBatchToApi(batch);
}
    

2. Memory Efficiency

Chunk is lazy-evaluated. It doesn't create all the batches in memory at once. It yields them one by one as you iterate. This allows you to process millions of items with a fixed, small memory footprint.

3. Architect Insight

Q: "What happens to the last chunk?"

Architect Answer: "If you have 105 items and you chunk by 10, you will get 10 chunks of 10, and a final chunk of 5. The last chunk is never 'padded'β€”it simply contains the remaining elements. This makes it safe to use for any list size without extra logic."

LINQ Mastery
General
Introduction to LINQ Mastery
1. Core Foundations
LINQ Fundamentals: Why LINQ? IQueryable vs IEnumerable: The Architect's choice Expression Trees: The power behind LINQ providers Method Syntax vs Query Syntax: Trade-offs
2. Filtering & Transformation
Where & Select: The bread and butter SelectMany: Flattening complex hierarchies OfType vs Cast: Handling heterogeneous collections Distinct & DistinctBy: Mastering unique sets
3. Aggregation & Quantifiers
Any, All, Contains: The boolean quantifiers Count, LongCount, Sum: Basic aggregations Min, Max, Average: Statistical operations Aggregate: The 'Fold' function of .NET
4. Ordering & Partitioning
OrderBy & OrderByDescending: Sorting data ThenBy: Multi-level sorting Take & Skip: Pagination strategies TakeWhile & SkipWhile: Dynamic partitioning
5. Sets & Lookups
Union, Intersect, Except: Set theory in C# Zip: Combining two streams ToDictionary vs ToLookup: One-to-One vs One-to-Many Chunk: Slicing data for batch processing
6. Join & Grouping
Inner Join: The standard match GroupJoin: Creating hierarchical results GroupBy: The SQL counterpart in LINQ Left Outer Join: The manual workaround in LINQ
7. Advanced Providers & Parallelism
PLINQ (Parallel LINQ): Speeding up CPU-bound queries AsParallel vs AsSequential: When to switch LINQ to XML: Processing documents with ease Custom LINQ Providers: How to build your own 'Queryable'
8. Real-world Performance & Patterns
Memory Leaks in LINQ: Capturing variables and closures Architect Case Study: Optimizing a multi-join dashboard query