Orc Stripe tests - File Merging


        
            
                Markdown 0.70 KB
                                    
                        | None                    
                
                                        |
    0    0                            

            
                
                                    raw
                    download
                    clone
                    embed
                    print
                
                                    report
                
                
                            
        
        
            Test file I made over time using LOTS of small ORC files and ALTER TABLE CONCATENATE 
-rwxr-xr-x   3 worker hdfs 1596985161 2017-06-03 21:53 /apps/hive/warehouse/orc_merge_tests/000000_0
hive --orcfiledump /apps/hive/warehouse/orc_merge_tests/000000_0 
62458 stripes, most very small...

see dump here ->

see configs here -> 
HIVE-fd7ba3db-21a3-41b7-8ff0-a1d0bd25a476   
Defaults, Cold Session
select count(*), line from orc_merge_tests group by line;

(with hot session)
Still running after 40 minutes in Tez, Gave up on waiting.Went to MR
Ran 6 mappers in MR..

Elapsed:    7hrs, 28mins, 2sec [IN MR]
Insert Overwrite with a massive (28gb) single reducer to get a big file