Lesson 21/31

Tutorials LINQ Mastery

Chunk: Slicing data for batch processing

On this page

Batch Processing made Easy

Before .NET 6, splitting a list into smaller batches required messy math with Skip and Take. Chunk does it in one line.

1. Why Chunk?

Many APIs and Databases have limits on payload size (e.g., 'You can only insert 500 rows at a time'). Chunk allows you to process a massive list in bite-sized pieces.


// Split 10,000 IDs into batches of 100
var batches = largeListOfIds.Chunk(100);

foreach (var batch in batches) {
    // Process 100 items...
    await SendBatchToApi(batch);
}
    

2. Memory Efficiency

Chunk is lazy-evaluated. It doesn't create all the batches in memory at once. It yields them one by one as you iterate. This allows you to process millions of items with a fixed, small memory footprint.

3. Architect Insight

Q: "What happens to the last chunk?"

Architect Answer: "If you have 105 items and you chunk by 10, you will get 10 chunks of 10, and a final chunk of 5. The last chunk is never 'padded'—it simply contains the remaining elements. This makes it safe to use for any list size without extra logic."

LINQ Mastery
Course syllabus
General Introduction to LINQ Mastery
1. Core Foundations LINQ Fundamentals: Why LINQ? IQueryable vs IEnumerable: The Architect's choice Expression Trees: The power behind LINQ providers Method Syntax vs Query Syntax: Trade-offs
2. Filtering & Transformation Where & Select: The bread and butter SelectMany: Flattening complex hierarchies OfType vs Cast: Handling heterogeneous collections Distinct & DistinctBy: Mastering unique sets
3. Aggregation & Quantifiers Any, All, Contains: The boolean quantifiers Count, LongCount, Sum: Basic aggregations Min, Max, Average: Statistical operations Aggregate: The 'Fold' function of .NET
4. Ordering & Partitioning OrderBy & OrderByDescending: Sorting data ThenBy: Multi-level sorting Take & Skip: Pagination strategies TakeWhile & SkipWhile: Dynamic partitioning
5. Sets & Lookups Union, Intersect, Except: Set theory in C# Zip: Combining two streams ToDictionary vs ToLookup: One-to-One vs One-to-Many Chunk: Slicing data for batch processing
6. Join & Grouping Inner Join: The standard match GroupJoin: Creating hierarchical results GroupBy: The SQL counterpart in LINQ Left Outer Join: The manual workaround in LINQ
7. Advanced Providers & Parallelism PLINQ (Parallel LINQ): Speeding up CPU-bound queries AsParallel vs AsSequential: When to switch LINQ to XML: Processing documents with ease Custom LINQ Providers: How to build your own 'Queryable'
8. Real-world Performance & Patterns Memory Leaks in LINQ: Capturing variables and closures Architect Case Study: Optimizing a multi-join dashboard query
Toolliyo Assistant
Ask about tutorials, ebooks, training, pricing, mentor services, and support. I use public site content only—not admin or internal tools.

care@toolliyo.com

Need callback? Share your details