Practical Generators in Go 1.23 for Database Pagination

|

Introduction

Go, known for its simplicity and powerful concurrency model, continues to evolve with every new release. Go 1.23 brings a fresh set of tools and improvements, and one of the most interesting features is the addition of practical generators. In many applications, especially when working with databases, handling pagination is a common task. Pagination ensures that large datasets are presented in a user-friendly manner by fetching smaller subsets of records at a time.

This article will cover practical use cases of generators in Go 1.23, particularly focusing on database pagination. We will first introduce the concept of generators in Go, explain why they’re useful, and then dive deep into a real-world example of database pagination. The article will be packed with practical code examples, detailed explanations, and a comprehensive conclusion to help you understand how to use Go 1.23 generators effectively.

What are Generators in Go?

In Go, a generator is a function that produces a sequence of values, one at a time, in a memory-efficient manner. This is useful when you need to process large datasets without loading the entire dataset into memory. Generators are particularly effective in scenarios like database pagination, where you need to fetch results incrementally rather than all at once.

Before Go 1.23, implementing generators required custom channels or iterators. However, Go 1.23 introduces new patterns that make it easier to write clean, generator-style code for pagination and other use cases.

Why Use Generators for Database Pagination?

Database pagination refers to the practice of retrieving a subset of data from a larger dataset in response to a user’s request. This is essential in web applications where large datasets cannot be presented to the user in a single page due to performance issues or limitations in user interface design.

Generators are ideal for database pagination because:

  1. Memory Efficiency: Rather than loading the entire dataset into memory, generators allow you to fetch records incrementally.
  2. Seamless Iteration: Generators yield records one by one, making it easy to loop through results and process them as needed.
  3. Asynchronous Capabilities: Go’s concurrency features integrate well with generators, allowing you to handle database pagination asynchronously, improving application performance.

Now that we understand why generators are useful, let’s dive into how you can use them for database pagination in Go 1.23.

Setting Up the Database

To demonstrate how generators work with database pagination, we’ll use an example based on the popular PostgreSQL database. However, the concepts can be applied to any relational database.

First, let’s assume you have a users table with the following schema:

sql
CREATE TABLE users (
id SERIAL PRIMARY KEY,
username VARCHAR(100),
email VARCHAR(100)
);

Next, let’s seed the database with some sample data:

sql
INSERT INTO users (username, email)
VALUES
('user1', 'user1@example.com'),
('user2', 'user2@example.com'),
('user3', 'user3@example.com'),
('user4', 'user4@example.com'),
('user5', 'user5@example.com');

For this example, we’ll use the Go pgx driver to interact with PostgreSQL. You can install it with the following command:

bash
go get github.com/jackc/pgx/v5

Implementing a Simple Generator for Pagination

Let’s create a basic generator for paginating over the users table. First, we’ll need to establish a connection to the database.

go

package main

import (
“context”
“fmt”
“log”
“os”

“github.com/jackc/pgx/v5”
)

func connectDB() (*pgx.Conn, error) {
conn, err := pgx.Connect(context.Background(), “postgres://username:password@localhost:5432/dbname”)
if err != nil {
return nil, err
}
return conn, nil
}

Once connected, we can create a generator function that fetches users in batches from the database using pagination.

go
func fetchUsers(ctx context.Context, conn *pgx.Conn, limit int) <-chan []string {
out := make(chan []string)go func() {
defer close(out)
offset := 0for {
rows, err := conn.Query(ctx, “SELECT username, email FROM users LIMIT $1 OFFSET $2”, limit, offset)
if err != nil {
log.Println(“Error querying the database:”, err)
return
}batch := make([]string, 0)for rows.Next() {
var username, email string
if err := rows.Scan(&username, &email); err != nil {
log.Println(“Error scanning row:”, err)
return
}
batch = append(batch, fmt.Sprintf(“User: %s, Email: %s”, username, email))
}if len(batch) == 0 {
return
}out <- batchoffset += limit
}
}()return out
}

Using the Generator for Pagination

Now that we have our generator, let’s write a function to consume it and print the users in batches.

go
func main() {
conn, err := connectDB()
if err != nil {
log.Fatalf("Failed to connect to the database: %v", err)
}
defer conn.Close(context.Background())userChannel := fetchUsers(context.Background(), conn, 2)for batch := range userChannel {
fmt.Println(“Batch of users:”)
for _, user := range batch {
fmt.Println(user)
}
}
}

In this example, we are fetching users two at a time from the database. The fetchUsers function retrieves records in batches using pagination and sends each batch to a channel. The main function consumes this channel and prints out each user.

Improving the Generator with Error Handling

In the above example, the generator handles errors internally by logging them. However, a more idiomatic approach would be to return errors to the caller so that they can be handled appropriately.

Here’s an improved version of the generator that returns an additional error channel.

go
func fetchUsersWithErrors(ctx context.Context, conn *pgx.Conn, limit int) (<-chan []string, <-chan error) {
out := make(chan []string)
errChan := make(chan error)go func() {
defer close(out)
defer close(errChan)
offset := 0for {
rows, err := conn.Query(ctx, “SELECT username, email FROM users LIMIT $1 OFFSET $2”, limit, offset)
if err != nil {
errChan <- err
return
}batch := make([]string, 0)for rows.Next() {
var username, email string
if err := rows.Scan(&username, &email); err != nil {
errChan <- err
return
}
batch = append(batch, fmt.Sprintf(“User: %s, Email: %s”, username, email))
}if len(batch) == 0 {
return
}out <- batch
offset += limit
}
}()return out, errChan
}

With this improved version, you can handle errors separately from the data processing loop.

go
func main() {
conn, err := connectDB()
if err != nil {
log.Fatalf("Failed to connect to the database: %v", err)
}
defer conn.Close(context.Background())userChannel, errChannel := fetchUsersWithErrors(context.Background(), conn, 2)for {
select {
case batch, ok := <-userChannel:
if !ok {
return // Channel closed
}
fmt.Println(“Batch of users:”)
for _, user := range batch {
fmt.Println(user)
}
case err := <-errChannel:
if err != nil {
log.Fatalf(“Error fetching users: %v”, err)
}
}
}
}

Optimizing Pagination with Cursor-Based Pagination

While offset-based pagination works, it can become inefficient for large datasets because the database has to skip over many rows to find the correct offset. An alternative approach is cursor-based pagination, where you keep track of a unique identifier (like an id) to fetch the next set of records.

Here’s how you can implement cursor-based pagination:

go
func fetchUsersByCursor(ctx context.Context, conn *pgx.Conn, limit int, lastID int) (<-chan []string, <-chan error) {
out := make(chan []string)
errChan := make(chan error)go func() {
defer close(out)
defer close(errChan)query := “SELECT id, username, email FROM users WHERE id > $1 ORDER BY id ASC LIMIT $2”for {
rows, err := conn.Query(ctx, query, lastID, limit)
if err != nil {
errChan <- err
return
}batch := make([]string, 0)
var latestID intfor rows.Next() {
var id int
var username, email string
if err := rows.Scan(&id, &username, &email); err != nil {
errChan <- err
return
}
latestID = id
batch = append(batch, fmt.Sprintf(“User: %s, Email: %s”, username, email))
}if len(batch) == 0 {
return
}out <- batch
lastID = latestID
}
}()return out, errChan
}

This approach fetches users starting from the last known id and eliminates the inefficiencies of offset-based pagination.

Conclusion

Go 1.23 introduces new patterns that make writing generators easier, especially when working with tasks like database pagination. In this article, we’ve covered how generators can be applied to efficiently paginate over large datasets using both offset-based and cursor-based pagination methods.

Generators, when combined with Go’s channels and goroutines, provide an elegant way to handle incremental data fetching without consuming too much memory or sacrificing performance. With the examples provided, you can now integrate practical generators into your own Go applications, particularly for database-driven tasks like pagination.

As Go continues to evolve, the patterns and practices covered here will remain valuable tools in your Go developer toolkit. By leveraging generators, you can ensure your applications remain performant and scalable even when dealing with large datasets.