R Code : (After bunzip2 t.csv.bz2) z<-fread("~/q/m32/t.csv") ulist = unique(z$grp) timer=proc.time() z2 <- z[,list(sumcol=sum(col3)), by='col1,col2,grp'][,as.list((sumcol[match(ulist,grp)])), by=c("col1","col2")] proc.time() - timer setnames(z2[],c("col1","col2",ulist)) user system elapsed 27.215 0.503 27.759 Q Code: (this was used to create the t.csv file for use in R) q)n:5000000 q)s:`a`b`c`d`e`f`g`h`i`j`k`l`m`n`o`p`q`r`s`t q)\t exec s#(grp!col4) by col1:col1,col2:col2 from update col4:sum col3 by col1,col2,grp from t 11543 q)save `:t.csv `:t.csv