Did you know there’s a hacky we can do automatic argument batching in Go using the variadic arguments?

In fact, I stumbled on a problem where I had to update tens of thousands of SQL rows. My Go code is usually using the sqlx SQL extension, which allows named queries with the IN operator. However, the queries made with IN cannot contain more then 10’000 elements, so you have to batch the queries into multiple ones. Failing to do that will result in a driver level error saying that you have too many arguments in the IN variable, which may be confusing for someone who has not looked at the underlying code. For this reason I decided that the best way to deal with the constraint would be on the model level (my Go web applications usually consist of three different abstraction levels where the model level is the lowest one, but more on that on a later date). Communicating row limits on function signature was something I thought would be too hard to make clear, so I decided to give the variadic arguments and recursive function logic a try.

The logic of the function would basically be the following:

1. Check if you need to make multiple SQL queries

   i.1 If so, create a batch array

   i.2 iterate through the original arguments and append to the array as many times as necessary

   i.3 iterate through the batch array and call the function again with the smaller slices

2. Do the actual SQL query

func (svc PointService) Add(rewardIDs []int, points int, level ...int) error {
    // if rewardIDs is over 10'000 and level is empty, we assume the function call
    // is initial and that the function should do automatic batching
    if len(rewardIDs) > 10000 && len(level) == 0 {
        var batches = make([][]int, 1)
        for index, id := range rewardIDs {
            var currentBatch = len(batches) - 1
            if index != 0 && index%10000 == 0 {
                batches = append(batches, []int{})
            }
            batches[currentBatch] = append(batches[currentBatch], id)
        }
        for i, batch := range batches {
            log.Println("running batch:", i)
            err := svc.Add(batch, points, 1)
            if err != nil {
                return err
            }
        }
        return nil
    }
    // if the are are less than 10K recipients, run the call the usual way
    query, args, err := sqlx.Named(`
    UPDATE customer SET point_balance = point_balance+:points
    WHERE id IN (:ids);`, map[string]interface{}{
        "points": points,
        "ids": rewardIDs,
    })
    if err != nil {
        return err
    }
    query, args, err = sqlx.In(query, args...)
    if err != nil {
        return err
    }
    query = svc.Connection.Rebind(query)
    _, err = svc.Connection.Exec(query, args...)
    if err != nil {
        return err
    }
    return nil
} 

With the following code nobody has to worry about calling the argument with too many recipients. As long as you don’t supply the function with non-zero third argument, everything will go smoothly.