ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > 应用开发 > Java > 外部排序优化

外部排序优化

原创 Java 作者：壹頁書 时间：2014-05-29 23:12:22 0 删除 编辑
优化外部排序
主要步骤重新拆分如下
第一阶段
1.将未排序的大文件切分(spilterReceiver)
2.一个线程池专门接收byte[]，并将其转化为long[] (prepareReceiver)
3.一个线程池专门排序long[] (sortReceiver)
4.一个线程池专门将已经有序的long[]写入文件 (writerReceiver)
这个过程直接写long类型的数字到文件，因为写入文本占用IO资源比直接写入long型数字要多。

第二阶段
1.一个线程用于归并排序,将结果写入缓冲区 (mergerReceiver)
2.如果缓冲区快满了，就将其交给一个线程池用于写入最终的文件。 (mergerReceiver)

优化的重点是平衡CPU，内存和硬盘的资源使用。

每个电脑的配置不同，都需要进行不同的调整，才能达到最优化的效果。
单位的电脑是i5的CPU，8G内存,7200转硬盘。
我使用的虚拟机分配了2个核心,4G内存。
跑这个程序大致需要200s左右。
因为第二阶段的第一个步骤是非常消耗CPU资源的，单位电脑CPU主频很高，所以不会成为瓶颈。

而家里的电脑，应该配置更接近吴老师做实验的配置。主频比较低,32位系统,4G内存。
这个配置明显CPU成为瓶颈
在第一个阶段，切分出来的文件，由于CPU不能及时排序并输出，经常导致OOM
只能在切分一个文件出来之后，让切分线程sleep一段时间，大量测试得到的结果是这个线程至少sleep2500毫秒才能不导致OOM。
也就是说,CPU运算的速度跟不上IO的速度。后来想到一个方法，加大分片。原来4G文件使用15个分片，现在使用30个分片。数据快入快出,这样可以避免OOM。至于这个分片的数字，恐怕每个机器都需要实际的测试。
另外在第二阶段，也出现了CPU拖累IO,导致性能下降。
因为归并的时候，需要大量运算各个分片中最小的数据，经常看到计算50M数据,CPU需要2s，而写入一般1s就可以完成。
而这个过程是单线程的，CPU的主频直接决定了外部排序的总体时间。


  1. import java.io.BufferedInputStream;
  2. import java.io.File;
  3. import java.io.FileInputStream;
  4. import java.io.IOException;
  5. import java.io.RandomAccessFile;
  6. import java.nio.ByteBuffer;
  7. import java.nio.MappedByteBuffer;
  8. import java.nio.channels.FileChannel;
  9. import java.nio.channels.FileChannel.MapMode;
  10. import java.util.ArrayList;
  11. import java.util.Arrays;
  12. import java.util.Collections;
  13. import java.util.List;
  14. import java.util.concurrent.BrokenBarrierException;
  15. import java.util.concurrent.CyclicBarrier;
  16. import java.util.concurrent.ExecutorService;
  17. import java.util.concurrent.Executors;

  19. public class Controller {
  20.     public static void main(String[] args) throws IOException {
  21.         new Controller().action(new File("/home/lihuilin/桌面/t.txt"), 30, "/home/lihuilin/桌面/");

  23.     }

  25.     public void action(File file, int pieces, String outDir) throws IOException {
  26.         Invoker invoker = new Invoker(pieces);
  27.         List<Command> commandList = blocking(file, pieces, outDir);
  28.         for (Command command : commandList) {
  29.             command.setInvoker(invoker);
  30.             invoker.executeCommand(command);
  31.         }
  32.     }

  34.     private List<Command> blocking(File file, int pieces, String outDir) throws IOException {
  35.         List<Command> result = new ArrayList<Command>();
  36.         List<Long> list = new ArrayList<Long>();
  37.         list.add(-1L);
  38.         long length = file.length();
  39.         long step = length / pieces;
  40.         long index = 0;
  41.         for (int i = 0; i < pieces; i++) {
  42.             BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
  43.             if (index + step < length) {
  44.                 index = index + step;
  45.                 in.skip(index);
  46.                 while (in.read() != 10) {
  47.                     index = index + 1;
  48.                 }
  49.                 list.add(index);
  50.                 index++;
  51.             }
  52.             in.close();
  53.         }
  54.         list.add(length - 1);
  55.         for (int i = 0; i < list.size() - 1; i++) {
  56.             long skipSize = list.get(i) + 1;
  57.             long l = list.get(i + 1) - list.get(i);
  58.             result.add(new SpilterCommand(file, skipSize, l, outDir));
  59.         }
  60.         return result;
  61.     }

  63. }

  65. interface Command extends Comparable<Command> {
  66.     public void setInvoker(Invoker invoker);

  68.     public void setReceiver(Receiver receiver);

  70.     public void execute() throws IOException;

  72.     public long getSkipSize();

  74.     public int getLength();

  76.     public String getOutDir();

  78.     public File getSourceFile();

  80.     public Invoker getInvoker();

  82.     public void clear();

  84.     public File getOutFile();

  86.     public File getTargetFile();
  87. }

  89. class Invoker {
  90.     private Receiver sortReceiver;
  91.     private Receiver spilterReceiver;
  92.     private Receiver writerReceiver;
  93.     private Receiver prepareReceiver;
  94.     private Receiver mergerReceiver;

  96.     public Invoker(int pieces) {
  97.         writerReceiver = new WriterReceiver(pieces);
  98.         spilterReceiver = new SpilterReceiver();
  99.         sortReceiver = new SorterReceiver();
  100.         prepareReceiver = new PrepareReceiver();
  101.         mergerReceiver = new MergerReceiver();
  102.     }

  104.     public void executeCommand(Command command) {
  105.         try {
  106.             if (command instanceof SpilterCommand) {
  107.                 spilterReceiver.action(command);
  108.             } else if (command instanceof PrepareCommand) {
  109.                 prepareReceiver.action(command);
  110.             } else if (command instanceof SortCommand) {
  111.                 sortReceiver.action(command);
  112.             } else if (command instanceof WriterCommand) {
  113.                 writerReceiver.action(command);
  114.             } else if (command instanceof MergeCommand) {
  115.                 mergerReceiver.action(command);
  116.             } else if (command instanceof KillCommand) {
  117.                 writerReceiver.shutdown();
  118.                 spilterReceiver.shutdown();
  119.                 sortReceiver.shutdown();
  120.                 prepareReceiver.shutdown();
  121.                 mergerReceiver.shutdown();
  122.             }
  123.         } catch (IOException ex) {
  124.             ex.printStackTrace();
  125.         }
  126.     }
  127. }

  129. class SpilterCommand extends AbstractCommand {

  131.     public SpilterCommand(File sourceFile, long skipSize, long length, String outDir) {
  132.         super(sourceFile, skipSize, length, outDir);
  133.     }

  135.     @Override
  136.     public void clear() {

  138.     }

  140. }

  142. abstract class AbstractCommand implements Command {
  143.     private Invoker invoker = null;
  144.     private Receiver receiver = null;
  145.     private long skipSize = 0L;
  146.     private int length = 0;
  147.     private String outDir = null;
  148.     private File sourceFile;
  149.     private File outFile = null;
  150.     public static final long START = System.currentTimeMillis();
  151.     private static int FLAG = 0;
  152.     private File targetFile = null;

  154.     private synchronized int getFlag() {
  155.         return FLAG++;
  156.     }

  158.     public AbstractCommand(Command command) {
  159.         this(command.getSourceFile(), command.getSkipSize(), command.getLength(), command.getOutDir());
  160.         this.invoker = command.getInvoker();
  161.         if (command.getOutFile() != null) {
  162.             this.outFile = command.getOutFile();
  163.         } else {
  164.             this.outFile = new File(this.outDir + getFlag() + ".temp");
  165.         }
  166.     }

  168.     public AbstractCommand(File sourceFile, long skipSize, long length, String outDir) {
  169.         if (length > Integer.MAX_VALUE) {
  170.             throw new RuntimeException("长度溢出");
  171.         }
  172.         this.skipSize = skipSize;
  173.         this.length = (int) length;
  174.         this.outDir = outDir;
  175.         this.sourceFile = sourceFile;
  176.         this.targetFile = new File(outDir + "result.temp");
  177.     }

  179.     public void setInvoker(Invoker invoker) {
  180.         this.invoker = invoker;
  181.     }

  183.     public void setReceiver(Receiver receiver) {
  184.         this.receiver = receiver;
  185.     }

  187.     public long getSkipSize() {
  188.         return skipSize;
  189.     }

  191.     public int getLength() {
  192.         return length;
  193.     }

  195.     public String getOutDir() {
  196.         return outDir;
  197.     }

  199.     public File getSourceFile() {
  200.         return sourceFile;
  201.     }

  203.     @Override
  204.     public String toString() {
  205.         return "Command [skipSize=" + skipSize + ", length=" + length + "]";
  206.     }

  208.     public void execute() throws IOException {
  209.         this.receiver.action(this);
  210.     }

  212.     public Invoker getInvoker() {
  213.         return invoker;
  214.     }

  216.     public File getOutFile() {
  217.         return this.outFile;
  218.     }

  220.     public File getTargetFile() {
  221.         return this.targetFile;
  222.     }

  224.     @Override
  225.     public int compareTo(Command o) {
  226.         if (this.skipSize < o.getSkipSize()) {
  227.             return -1;
  228.         }
  229.         return 1;
  230.     }
  231. }

  233. interface Receiver {
  234.     public void action(Command command) throws IOException;

  236.     public void shutdown();
  237. }

  239. class PrepareReceiver implements Receiver {
  240.     private ExecutorService prepareThreadPool;

  242.     public PrepareReceiver() {
  243.         prepareThreadPool = Executors.newFixedThreadPool(4);
  244.     }

  246.     @Override
  247.     public void action(final Command command) throws IOException {
  248.         prepareThreadPool.submit(new Runnable() {

  250.             @Override
  251.             public void run() {
  252.                 System.out.println("\t整理数据:" + command);
  253.                 long start = System.currentTimeMillis();

  255.                 byte[] data = ((PrepareCommand) command).getData();
  256.                 long[] result = new long[getObjectSize(data)];
  257.                 int resultIndex = 0;
  258.                 int index = 0;
  259.                 int first = 0;
  260.                 while (index < data.length) {
  261.                     if (data[index] == 10) {
  262.                         byte[] tmpData = Arrays.copyOfRange(data, first, index);
  263.                         String str = new String(tmpData);
  264.                         result[resultIndex] = Long.valueOf(str);
  265.                         resultIndex++;
  266.                         first = index + 1;
  267.                     }

  269.                     index++;
  270.                 }
  271.                 command.getInvoker().executeCommand(new SortCommand(command, result));

  273.                 result = null;
  274.                 data = null;
  275.                 command.clear();

  277.                 long end = System.currentTimeMillis();
  278.                 System.out.println("\t结束整理数据:" + command + ",用时:" + (end - start) / 1000);
  279.             }

  281.         });

  283.     }

  285.     private int getObjectSize(byte[] data) {
  286.         int size = 0;
  287.         for (byte b : data) {
  288.             if (b == 10) {
  289.                 size++;
  290.             }
  291.         }
  292.         return size;
  293.     }

  295.     @Override
  296.     public void shutdown() {
  297.         this.prepareThreadPool.shutdown();
  298.     }

  300. }

  302. class PrepareCommand extends AbstractCommand {

  304.     private byte[] data;

  306.     public PrepareCommand(Command command, byte[] data) {
  307.         super(command);
  308.         this.data = data;
  309.     }

  311.     public byte[] getData() {
  312.         return this.data;
  313.     }

  315.     @Override
  316.     public void clear() {
  317.         this.data = null;
  318.     }
  319. }

  321. class SpilterReceiver implements Receiver {
  322.     @Override
  323.     public void action(final Command command) throws IOException {
  324.         try {
  325.             System.out.println("开始读入:" + command);
  326.             long start = System.currentTimeMillis();
  327.             FileChannel in = new RandomAccessFile(command.getSourceFile(), "r").getChannel();
  328.             MappedByteBuffer inBuffer = in.map(MapMode.READ_ONLY, command.getSkipSize(), command.getLength());
  329.             byte[] data = new byte[inBuffer.limit()];
  330.             inBuffer.get(data);
  331.             command.getInvoker().executeCommand(new PrepareCommand(command, data));

  333.             data = null;
  334.             in.close();
  335.             command.clear();

  337.             long end = System.currentTimeMillis();
  338.             System.out.println("结束内存读入:" + (end - start) / 1000 + "s");

  340.         } catch (IOException e) {
  341.             e.printStackTrace();
  342.         }
  343.     }

  345.     @Override
  346.     public void shutdown() {
  347.     }
  348. }

  350. class MergeCommand extends AbstractCommand {
  351.     private List<Command> commandList = null;

  353.     public MergeCommand(List<Command> commandList) {
  354.         super(commandList.get(0));
  355.         this.commandList = commandList;
  356.     }

  358.     @Override
  359.     public void clear() {

  361.     }

  363.     public List<Command> getCommandList() {
  364.         return this.commandList;
  365.     }
  366. }

  368. class KillCommand extends AbstractCommand {

  370.     public KillCommand(Command command) {
  371.         super(command);
  372.     }

  374.     @Override
  375.     public void clear() {

  377.     }

  379. }

  381. class MergerReceiver implements Receiver {
  382.     private ExecutorService mergerThreadPool = Executors.newFixedThreadPool(4);
  383.     private List<Worker> workerList = new ArrayList<Worker>();
  384.     private ByteBuffer bb = null;

  386.     @Override
  387.     public void action(final Command command) throws IOException {
  388.         bb = ByteBuffer.allocate(50 * 1024 * 1024);
  389.         List<Command> list = ((MergeCommand) command).getCommandList();
  390.         for (Command historyCommand : list) {
  391.             Worker worker = new Worker(historyCommand.getOutFile(), workerList);
  392.             workerList.add(worker);
  393.         }

  395.         System.out.println("读取队列,写入目标文件");
  396.         long start = System.currentTimeMillis();
  397.         RandomAccessFile targetFile = new RandomAccessFile(command.getTargetFile(), "rw");
  398.         FileChannel channel = targetFile.getChannel();
  399.         MappedByteBuffer out = null;
  400.         long index = 0L;
  401.         while (workerList.size() != 0) {
  402.             Worker worker = Collections.min(workerList);
  403.             Long data = worker.poll();
  404.             if (data == null) {
  405.                 workerList.remove(worker);
  406.             } else {

  408.                 bb.put((data + "\n").getBytes());

  410.                 if (bb.position() > 49 * 1024 * 1024) {
  411.                     long end = System.currentTimeMillis();
  412.                     System.out.println("归并用时:" + (end - start)/1000 + "s");
  413.                     start = System.currentTimeMillis();
  414.                     bb.flip();
  415.                     long temp = bb.limit();
  416.                     MutiWriter writer = new MutiWriter(channel, bb, index);
  417.                     mergerThreadPool.submit(writer);
  418.                     index = index + temp;
  419.                     bb = ByteBuffer.allocate(50 * 1024 * 1024);
  420.                 }

  422.             }
  423.         }

  425.         bb.flip();

  427.         out = channel.map(MapMode.READ_WRITE, index, bb.limit());
  428.         int i = bb.limit();
  429.         while (i > .0) {
  430.             out.put(bb.get());
  431.             i--;
  432.         }
  433.         out.force();
  434.         channel.close();
  435.         targetFile.close();

  437.         KillCommand killCommand = new KillCommand(command);
  438.         command.getInvoker().executeCommand(killCommand);

  440.         long end = System.currentTimeMillis();
  441.         System.out.println("外部排序总用时:" + (end - AbstractCommand.START) / 1000);

  443.     }

  445.     private class MutiWriter implements Runnable {
  446.         private ByteBuffer buffer;
  447.         private MappedByteBuffer out = null;

  449.         public MutiWriter(FileChannel channel, ByteBuffer buffer, long index) {
  450.             this.buffer = buffer;
  451.             try {
  452.                 out = channel.map(MapMode.READ_WRITE, index, buffer.limit());
  453.             } catch (IOException e) {
  454.                 e.printStackTrace();
  455.             }
  456.         }

  458.         @Override
  459.         public void run() {
  460.             long a1 = System.currentTimeMillis();
  461.             int i = buffer.limit();
  462.             while (i > 0) {
  463.                 out.put(buffer.get());
  464.                 i--;
  465.             }
  466.             out.force();
  467.             out = null;
  468.             long a2 = System.currentTimeMillis();
  469.             System.out.println("内存映射写入:" + (a2 - a1) / 1000 + "s");
  470.         }

  472.     }

  474.     @Override
  475.     public void shutdown() {
  476.         mergerThreadPool.shutdown();
  477.     }

  479.     private class Worker implements Comparable<Worker> {
  480.         private long data;
  481.         private MappedByteBuffer buffer = null;
  482.         private List<Worker> workerList = null;
  483.         private boolean eof = false;

  485.         Worker(File file, List<Worker> workerList) {
  486.             try {
  487.                 RandomAccessFile rFile = new RandomAccessFile(file, "r");
  488.                 FileChannel channel = rFile.getChannel();
  489.                 buffer = channel.map(MapMode.READ_ONLY, 0, channel.size());

  491.                 this.workerList = workerList;
  492.                 data = buffer.getLong();

  494.                 rFile.close();
  495.                 channel.close();

  497.             } catch (IOException e) {
  498.                 // TODO Auto-generated catch block
  499.                 e.printStackTrace();
  500.             }
  501.         }

  503.         public long peek() {
  504.             return data;
  505.         }

  507.         public Long poll() {
  508.             long result = data;
  509.             if (buffer.position() != buffer.limit()) {
  510.                 data = buffer.getLong();
  511.             } else {
  512.                 if (eof == false) {
  513.                     eof = true;
  514.                 } else {
  515.                     return null;
  516.                 }
  517.             }

  519.             return result;
  520.         }

  522.         @Override
  523.         public int compareTo(Worker o) {
  524.             if (this.peek() > o.peek()) {
  525.                 return 1;
  526.             } else if (this.peek() < o.peek()) {
  527.                 return -1;
  528.             } else {
  529.                 return 0;
  530.             }
  531.         }
  532.     }

  534. }

  536. class WriterReceiver implements Receiver {
  537.     private ExecutorService writeThreadPool = null;;
  538.     private CyclicBarrier barrier = null;
  539.     private List<Command> commandList = new ArrayList<Command>();

  541.     public WriterReceiver(int pieces) {
  542.         writeThreadPool = Executors.newFixedThreadPool(pieces + 2);
  543.         barrier = new CyclicBarrier(pieces, new Runnable() {

  545.             @Override
  546.             public void run() {
  547.                 long end = System.currentTimeMillis();
  548.                 System.out.println("合并之前总用时:" + (end - AbstractCommand.START) / 1000);

  550.                 MergeCommand command = new MergeCommand(WriterReceiver.this.commandList);
  551.                 command.getInvoker().executeCommand(command);

  553.             }
  554.         });

  556.     }

  558.     @Override
  559.     public void action(final Command command) {
  560.         writeThreadPool.submit(new Runnable() {

  562.             @Override
  563.             public void run() {
  564.                 long[] data = ((WriterCommand) command).getData();

  566.                 try {
  567.                     System.out.println("\t\t\t开始写入:" + command);
  568.                     long start = System.currentTimeMillis();
  569.                     File outfile = command.getOutFile();
  570.                     FileChannel channel = new RandomAccessFile(outfile, "rw").getChannel();
  571.                     MappedByteBuffer buffer = channel.map(MapMode.READ_WRITE, 0, data.length * 8);
  572.                     for (int i = 0; i < data.length; i++) {
  573.                         buffer.putLong(data[i]);
  574.                     }
  575.                     buffer.force();
  576.                     WriterReceiver.this.commandList.add(command);
  577.                     long end = System.currentTimeMillis();
  578.                     System.out.println("\t\t\t结束写入:" + command + ",用时:" + (end - start) / 1000);

  580.                     command.clear();
  581.                     data = null;
  582.                     channel.close();
  583.                     buffer = null;

  585.                     barrier.await();
  586.                 } catch (IOException ex) {
  587.                     ex.printStackTrace();
  588.                 } catch (InterruptedException e) {
  589.                     e.printStackTrace();
  590.                 } catch (BrokenBarrierException e) {
  591.                     e.printStackTrace();
  592.                 }
  593.             }
  594.         });
  595.     }

  597.     @Override
  598.     public void shutdown() {
  599.         writeThreadPool.shutdown();
  600.     }
  601. }

  603. class SortCommand extends AbstractCommand {
  604.     private long[] data = null;

  606.     public SortCommand(Command command, long[] data) {
  607.         super(command);
  608.         this.data = data;
  609.     }

  611.     public long[] getData() {
  612.         return this.data;
  613.     }

  615.     @Override
  616.     public void clear() {
  617.         this.data = null;
  618.     }

  620. }

  622. class SorterReceiver implements Receiver {
  623.     private ExecutorService sortThreadPool = Executors.newFixedThreadPool(4);

  625.     public void action(final Command command) {
  626.         sortThreadPool.submit(new Runnable() {

  628.             @Override
  629.             public void run() {
  630.                 System.out.println("\t\t开始排序:" + command);
  631.                 long start = System.currentTimeMillis();

  633.                 long[] data = ((SortCommand) command).getData();
  634.                 Arrays.sort(data);

  636.                 command.getInvoker().executeCommand(new WriterCommand(command, data));

  638.                 data = null;
  639.                 command.clear();

  641.                 long end = System.currentTimeMillis();
  642.                 System.out.println("\t\t结束排序:" + command + ",用时:" + (end - start) / 1000);

  644.             }

  646.         });

  648.     }

  650.     @Override
  651.     public void shutdown() {
  652.         sortThreadPool.shutdown();
  653.     }
  654. }

  656. class WriterCommand extends AbstractCommand {
  657.     private long[] data;

  659.     public WriterCommand(Command command, long[] data) {
  660.         super(command);
  661.         this.data = data;
  662.     }

  664.     public long[] getData() {
  665.         return this.data;
  666.     }

  668.     @Override
  669.     public void clear() {
  670.         this.data = null;
  671.     }

  673. }
最近真的很忙,我这一个单线程的CPU居然都跑出了双核的感觉。
我感觉优化就是找到慢的地方，看看是不是有无效的操作,如果都是有效运算,就只能想办法使用多线程，使压力分散在多个核心上。
之前吴老师说他用的是直接缓冲区+通道 IO的方式。自己写程序将long直接转化为byte。
这个优化的方式我试了一下，不能应用在我的场景。
他的本意是减少GC，但是在我的场景，这种方式更加剧了CPU的争用。
优化的本质应该是平衡资源

程序日志,相比最初的版本，性能提升了50s
因为将切分文件，byte数组转为long数组拆分为了两个操作，并且使用了多线程处理。
  1. 开始读入:Command [skipSize=0, length=135863265]
  2. 结束内存读入:3s
  3. 整理数据:Command [skipSize=0, length=135863265]
  4. 开始读入:Command [skipSize=135863265, length=135863254]
  5. 结束整理数据:Command [skipSize=0, length=135863265],用时:4
  6. 开始排序:Command [skipSize=0, length=135863265]
  7. 结束内存读入:4s
  8. 开始读入:Command [skipSize=271726519, length=135863267]
  9. 整理数据:Command [skipSize=135863265, length=135863254]
  10. 结束排序:Command [skipSize=0, length=135863265],用时:2
  11. 开始写入:Command [skipSize=0, length=135863265]
  12. 结束内存读入:3s
  13. 开始读入:Command [skipSize=407589786, length=135863268]
  14. 整理数据:Command [skipSize=271726519, length=135863267]
  15. 结束写入:Command [skipSize=0, length=135863265],用时:2
  16. 结束整理数据:Command [skipSize=135863265, length=135863254],用时:5
  17. 开始排序:Command [skipSize=135863265, length=135863254]
  18. 结束排序:Command [skipSize=135863265, length=135863254],用时:1
  19. 开始写入:Command [skipSize=135863265, length=135863254]
  20. 结束整理数据:Command [skipSize=271726519, length=135863267],用时:4
  21. 开始排序:Command [skipSize=271726519, length=135863267]
  22. 结束排序:Command [skipSize=271726519, length=135863267],用时:1
  23. 开始写入:Command [skipSize=271726519, length=135863267]
  24. 结束写入:Command [skipSize=135863265, length=135863254],用时:6
  25. 结束写入:Command [skipSize=271726519, length=135863267],用时:4
  26. 结束内存读入:11s
  27. 整理数据:Command [skipSize=407589786, length=135863268]
  28. 开始读入:Command [skipSize=543453054, length=135863264]
  29. 结束整理数据:Command [skipSize=407589786, length=135863268],用时:3
  30. 开始排序:Command [skipSize=407589786, length=135863268]
  31. 结束内存读入:4s
  32. 开始读入:Command [skipSize=679316318, length=135863268]
  33. 整理数据:Command [skipSize=543453054, length=135863264]
  34. 结束排序:Command [skipSize=407589786, length=135863268],用时:1
  35. 开始写入:Command [skipSize=407589786, length=135863268]
  36. 结束写入:Command [skipSize=407589786, length=135863268],用时:1
  37. 结束整理数据:Command [skipSize=543453054, length=135863264],用时:3
  38. 开始排序:Command [skipSize=543453054, length=135863264]
  39. 结束排序:Command [skipSize=543453054, length=135863264],用时:1
  40. 开始写入:Command [skipSize=543453054, length=135863264]
  41. 结束写入:Command [skipSize=543453054, length=135863264],用时:1
  42. 结束内存读入:7s
  43. 整理数据:Command [skipSize=679316318, length=135863268]
  44. 开始读入:Command [skipSize=815179586, length=135863264]
  45. 结束内存读入:3s
  46. 开始读入:Command [skipSize=951042850, length=135863270]
  47. 整理数据:Command [skipSize=815179586, length=135863264]
  48. 开始排序:Command [skipSize=679316318, length=135863268]
  49. 结束整理数据:Command [skipSize=679316318, length=135863268],用时:3
  50. 结束排序:Command [skipSize=679316318, length=135863268],用时:2
  51. 开始写入:Command [skipSize=679316318, length=135863268]
  52. 结束整理数据:Command [skipSize=815179586, length=135863264],用时:4
  53. 开始排序:Command [skipSize=815179586, length=135863264]
  54. 结束写入:Command [skipSize=679316318, length=135863268],用时:2
  55. 结束内存读入:5s
  56. 整理数据:Command [skipSize=951042850, length=135863270]
  57. 开始读入:Command [skipSize=1086906120, length=135863262]
  58. 结束排序:Command [skipSize=815179586, length=135863264],用时:1
  59. 开始写入:Command [skipSize=815179586, length=135863264]
  60. 结束写入:Command [skipSize=815179586, length=135863264],用时:1
  61. 整理数据:Command [skipSize=1086906120, length=135863262]
  62. 结束内存读入:3s
  63. 开始读入:Command [skipSize=1222769382, length=135863265]
  64. 开始排序:Command [skipSize=951042850, length=135863270]
  65. 结束整理数据:Command [skipSize=951042850, length=135863270],用时:4
  66. 结束内存读入:3s
  67. 开始读入:Command [skipSize=1358632647, length=135863256]
  68. 整理数据:Command [skipSize=1222769382, length=135863265]
  69. 结束排序:Command [skipSize=951042850, length=135863270],用时:2
  70. 开始写入:Command [skipSize=951042850, length=135863270]
  71. 结束写入:Command [skipSize=951042850, length=135863270],用时:1
  72. 结束整理数据:Command [skipSize=1086906120, length=135863262],用时:5
  73. 开始排序:Command [skipSize=1086906120, length=135863262]
  74. 结束排序:Command [skipSize=1086906120, length=135863262],用时:1
  75. 开始写入:Command [skipSize=1086906120, length=135863262]
  76. 结束内存读入:4s
  77. 整理数据:Command [skipSize=1358632647, length=135863256]
  78. 开始读入:Command [skipSize=1494495903, length=135863268]
  79. 结束写入:Command [skipSize=1086906120, length=135863262],用时:1
  80. 开始排序:Command [skipSize=1222769382, length=135863265]
  81. 结束整理数据:Command [skipSize=1222769382, length=135863265],用时:6
  82. 结束排序:Command [skipSize=1222769382, length=135863265],用时:1
  83. 开始写入:Command [skipSize=1222769382, length=135863265]
  84. 整理数据:Command [skipSize=1494495903, length=135863268]
  85. 结束内存读入:4s
  86. 开始读入:Command [skipSize=1630359171, length=135863255]
  87. 开始排序:Command [skipSize=1358632647, length=135863256]
  88. 结束整理数据:Command [skipSize=1358632647, length=135863256],用时:5
  89. 结束写入:Command [skipSize=1222769382, length=135863265],用时:1
  90. 结束排序:Command [skipSize=1358632647, length=135863256],用时:1
  91. 开始写入:Command [skipSize=1358632647, length=135863256]
  92. 结束整理数据:Command [skipSize=1494495903, length=135863268],用时:3
  93. 结束写入:Command [skipSize=1358632647, length=135863256],用时:2
  94. 开始排序:Command [skipSize=1494495903, length=135863268]
  95. 结束内存读入:4s
  96. 整理数据:Command [skipSize=1630359171, length=135863255]
  97. 开始读入:Command [skipSize=1766222426, length=135863271]
  98. 结束排序:Command [skipSize=1494495903, length=135863268],用时:1
  99. 开始写入:Command [skipSize=1494495903, length=135863268]
  100. 结束写入:Command [skipSize=1494495903, length=135863268],用时:1
  101. 结束整理数据:Command [skipSize=1630359171, length=135863255],用时:3
  102. 开始排序:Command [skipSize=1630359171, length=135863255]
  103. 整理数据:Command [skipSize=1766222426, length=135863271]
  104. 结束内存读入:4s
  105. 开始读入:Command [skipSize=1902085697, length=135863258]
  106. 结束排序:Command [skipSize=1630359171, length=135863255],用时:2
  107. 开始写入:Command [skipSize=1630359171, length=135863255]
  108. 结束写入:Command [skipSize=1630359171, length=135863255],用时:1
  109. 结束整理数据:Command [skipSize=1766222426, length=135863271],用时:3
  110. 开始排序:Command [skipSize=1766222426, length=135863271]
  111. 整理数据:Command [skipSize=1902085697, length=135863258]
  112. 结束内存读入:4s
  113. 开始读入:Command [skipSize=2037948955, length=135863254]
  114. 结束排序:Command [skipSize=1766222426, length=135863271],用时:1
  115. 开始写入:Command [skipSize=1766222426, length=135863271]
  116. 结束写入:Command [skipSize=1766222426, length=135863271],用时:1
  117. 结束整理数据:Command [skipSize=1902085697, length=135863258],用时:3
  118. 开始排序:Command [skipSize=1902085697, length=135863258]
  119. 整理数据:Command [skipSize=2037948955, length=135863254]
  120. 结束内存读入:4s
  121. 开始读入:Command [skipSize=2173812209, length=135863259]
  122. 结束排序:Command [skipSize=1902085697, length=135863258],用时:2
  123. 开始写入:Command [skipSize=1902085697, length=135863258]
  124. 结束写入:Command [skipSize=1902085697, length=135863258],用时:1
  125. 结束整理数据:Command [skipSize=2037948955, length=135863254],用时:3
  126. 开始排序:Command [skipSize=2037948955, length=135863254]
  127. 结束内存读入:4s
  128. 整理数据:Command [skipSize=2173812209, length=135863259]
  129. 开始读入:Command [skipSize=2309675468, length=135863260]
  130. 结束排序:Command [skipSize=2037948955, length=135863254],用时:2
  131. 开始写入:Command [skipSize=2037948955, length=135863254]
  132. 结束整理数据:Command [skipSize=2173812209, length=135863259],用时:3
  133. 开始排序:Command [skipSize=2173812209, length=135863259]
  134. 结束内存读入:4s
  135. 整理数据:Command [skipSize=2309675468, length=135863260]
  136. 开始读入:Command [skipSize=2445538728, length=135863266]
  137. 结束写入:Command [skipSize=2037948955, length=135863254],用时:2
  138. 结束排序:Command [skipSize=2173812209, length=135863259],用时:2
  139. 开始写入:Command [skipSize=2173812209, length=135863259]
  140. 结束写入:Command [skipSize=2173812209, length=135863259],用时:1
  141. 结束整理数据:Command [skipSize=2309675468, length=135863260],用时:3
  142. 开始排序:Command [skipSize=2309675468, length=135863260]
  143. 整理数据:Command [skipSize=2445538728, length=135863266]
  144. 结束内存读入:4s
  145. 开始读入:Command [skipSize=2581401994, length=135863268]
  146. 结束排序:Command [skipSize=2309675468, length=135863260],用时:2
  147. 开始写入:Command [skipSize=2309675468, length=135863260]
  148. 结束写入:Command [skipSize=2309675468, length=135863260],用时:1
  149. 结束整理数据:Command [skipSize=2445538728, length=135863266],用时:4
  150. 开始排序:Command [skipSize=2445538728, length=135863266]
  151. 结束内存读入:4s
  152. 整理数据:Command [skipSize=2581401994, length=135863268]
  153. 开始读入:Command [skipSize=2717265262, length=135863264]
  154. 结束排序:Command [skipSize=2445538728, length=135863266],用时:2
  155. 开始写入:Command [skipSize=2445538728, length=135863266]
  156. 结束写入:Command [skipSize=2445538728, length=135863266],用时:1
  157. 结束整理数据:Command [skipSize=2581401994, length=135863268],用时:3
  158. 开始排序:Command [skipSize=2581401994, length=135863268]
  159. 整理数据:Command [skipSize=2717265262, length=135863264]
  160. 结束内存读入:4s
  161. 开始读入:Command [skipSize=2853128526, length=135863258]
  162. 结束排序:Command [skipSize=2581401994, length=135863268],用时:2
  163. 开始写入:Command [skipSize=2581401994, length=135863268]
  164. 结束写入:Command [skipSize=2581401994, length=135863268],用时:1
  165. 结束整理数据:Command [skipSize=2717265262, length=135863264],用时:3
  166. 开始排序:Command [skipSize=2717265262, length=135863264]
  167. 整理数据:Command [skipSize=2853128526, length=135863258]
  168. 结束内存读入:4s
  169. 开始读入:Command [skipSize=2988991784, length=135863254]
  170. 结束排序:Command [skipSize=2717265262, length=135863264],用时:2
  171. 开始写入:Command [skipSize=2717265262, length=135863264]
  172. 结束写入:Command [skipSize=2717265262, length=135863264],用时:1
  173. 结束整理数据:Command [skipSize=2853128526, length=135863258],用时:3
  174. 开始排序:Command [skipSize=2853128526, length=135863258]
  175. 结束内存读入:4s
  176. 整理数据:Command [skipSize=2988991784, length=135863254]
  177. 开始读入:Command [skipSize=3124855038, length=135863260]
  178. 结束排序:Command [skipSize=2853128526, length=135863258],用时:2
  179. 开始写入:Command [skipSize=2853128526, length=135863258]
  180. 结束写入:Command [skipSize=2853128526, length=135863258],用时:1
  181. 结束整理数据:Command [skipSize=2988991784, length=135863254],用时:3
  182. 开始排序:Command [skipSize=2988991784, length=135863254]
  183. 整理数据:Command [skipSize=3124855038, length=135863260]
  184. 结束内存读入:4s
  185. 开始读入:Command [skipSize=3260718298, length=135863254]
  186. 结束排序:Command [skipSize=2988991784, length=135863254],用时:2
  187. 开始写入:Command [skipSize=2988991784, length=135863254]
  188. 结束写入:Command [skipSize=2988991784, length=135863254],用时:2
  189. 结束整理数据:Command [skipSize=3124855038, length=135863260],用时:4
  190. 开始排序:Command [skipSize=3124855038, length=135863260]
  191. 整理数据:Command [skipSize=3260718298, length=135863254]
  192. 结束内存读入:4s
  193. 开始读入:Command [skipSize=3396581552, length=135863262]
  194. 结束排序:Command [skipSize=3124855038, length=135863260],用时:1
  195. 开始写入:Command [skipSize=3124855038, length=135863260]
  196. 结束写入:Command [skipSize=3124855038, length=135863260],用时:2
  197. 结束整理数据:Command [skipSize=3260718298, length=135863254],用时:3
  198. 开始排序:Command [skipSize=3260718298, length=135863254]
  199. 结束内存读入:4s
  200. 整理数据:Command [skipSize=3396581552, length=135863262]
  201. 开始读入:Command [skipSize=3532444814, length=135863265]
  202. 结束排序:Command [skipSize=3260718298, length=135863254],用时:1
  203. 开始写入:Command [skipSize=3260718298, length=135863254]
  204. 结束写入:Command [skipSize=3260718298, length=135863254],用时:2
  205. 结束整理数据:Command [skipSize=3396581552, length=135863262],用时:3
  206. 开始排序:Command [skipSize=3396581552, length=135863262]
  207. 结束内存读入:4s
  208. 整理数据:Command [skipSize=3532444814, length=135863265]
  209. 开始读入:Command [skipSize=3668308079, length=135863273]
  210. 结束排序:Command [skipSize=3396581552, length=135863262],用时:2
  211. 开始写入:Command [skipSize=3396581552, length=135863262]
  212. 结束写入:Command [skipSize=3396581552, length=135863262],用时:2
  213. 结束整理数据:Command [skipSize=3532444814, length=135863265],用时:3
  214. 开始排序:Command [skipSize=3532444814, length=135863265]
  215. 结束内存读入:4s
  216. 整理数据:Command [skipSize=3668308079, length=135863273]
  217. 开始读入:Command [skipSize=3804171352, length=135863261]
  218. 结束排序:Command [skipSize=3532444814, length=135863265],用时:2
  219. 开始写入:Command [skipSize=3532444814, length=135863265]
  220. 结束写入:Command [skipSize=3532444814, length=135863265],用时:1
  221. 结束整理数据:Command [skipSize=3668308079, length=135863273],用时:3
  222. 开始排序:Command [skipSize=3668308079, length=135863273]
  223. 整理数据:Command [skipSize=3804171352, length=135863261]
  224. 结束内存读入:4s
  225. 开始读入:Command [skipSize=3940034613, length=135862991]
  226. 结束排序:Command [skipSize=3668308079, length=135863273],用时:2
  227. 开始写入:Command [skipSize=3668308079, length=135863273]
  228. 结束写入:Command [skipSize=3668308079, length=135863273],用时:2
  229. 结束整理数据:Command [skipSize=3804171352, length=135863261],用时:3
  230. 开始排序:Command [skipSize=3804171352, length=135863261]
  231. 结束内存读入:4s
  232. 整理数据:Command [skipSize=3940034613, length=135862991]
  233. 结束排序:Command [skipSize=3804171352, length=135863261],用时:1
  234. 开始写入:Command [skipSize=3804171352, length=135863261]
  235. 结束写入:Command [skipSize=3804171352, length=135863261],用时:1
  236. 结束整理数据:Command [skipSize=3940034613, length=135862991],用时:3
  237. 开始排序:Command [skipSize=3940034613, length=135862991]
  238. 结束排序:Command [skipSize=3940034613, length=135862991],用时:1
  239. 开始写入:Command [skipSize=3940034613, length=135862991]
  240. 结束写入:Command [skipSize=3940034613, length=135862991],用时:1
  241. 合并之前总用时:143
  242. 读取队列,写入目标文件
  243. 归并用时:2s
  244. 内存映射写入:1s
  245. 归并用时:2s
  246. 内存映射写入:1s
  247. 归并用时:2s
  248. 内存映射写入:1s
  249. 归并用时:2s
  250. 内存映射写入:1s
  251. 归并用时:2s
  252. 内存映射写入:1s
  253. 归并用时:2s
  254. 内存映射写入:1s
  255. 归并用时:2s
  256. 内存映射写入:1s
  257. 归并用时:2s
  258. 内存映射写入:2s
  259. 归并用时:2s
  260. 内存映射写入:1s
  261. 归并用时:2s
  262. 内存映射写入:1s
  263. 归并用时:2s
  264. 内存映射写入:1s
  265. 归并用时:2s
  266. 内存映射写入:1s
  267. 归并用时:2s
  268. 内存映射写入:1s
  269. 归并用时:2s
  270. 内存映射写入:1s
  271. 归并用时:2s
  272. 归并用时:1s
  273. 内存映射写入:3s
  274. 归并用时:2s
  275. 归并用时:2s
  276. 内存映射写入:5s
  277. 内存映射写入:3s
  278. 归并用时:2s
  279. 内存映射写入:2s
  280. 内存映射写入:2s
  281. 归并用时:2s
  282. 内存映射写入:2s
  283. 归并用时:2s
  284. 归并用时:2s
  285. 内存映射写入:2s
  286. 内存映射写入:1s
  287. 归并用时:2s
  288. 归并用时:2s
  289. 内存映射写入:2s
  290. 内存映射写入:2s
  291. 归并用时:2s
  292. 归并用时:2s
  293. 内存映射写入:2s
  294. 内存映射写入:2s
  295. 归并用时:2s
  296. 内存映射写入:2s
  297. 归并用时:2s
  298. 内存映射写入:2s
  299. 归并用时:2s
  300. 内存映射写入:2s
  301. 归并用时:2s
  302. 内存映射写入:2s
  303. 归并用时:2s
  304. 归并用时:2s
  305. 内存映射写入:2s
  306. 内存映射写入:2s
  307. 归并用时:2s
  308. 归并用时:2s
  309. 内存映射写入:2s
  310. 归并用时:2s
  311. 内存映射写入:2s
  312. 内存映射写入:2s
  313. 归并用时:2s
  314. 内存映射写入:2s
  315. 归并用时:2s
  316. 内存映射写入:2s
  317. 归并用时:2s
  318. 内存映射写入:2s
  319. 归并用时:2s
  320. 内存映射写入:2s
  321. 归并用时:2s
  322. 内存映射写入:2s
  323. 归并用时:2s
  324. 内存映射写入:2s
  325. 归并用时:2s
  326. 内存映射写入:2s
  327. 归并用时:2s
  328. 内存映射写入:2s
  329. 归并用时:2s
  330. 内存映射写入:2s
  331. 归并用时:2s
  332. 内存映射写入:2s
  333. 归并用时:2s
  334. 内存映射写入:2s
  335. 归并用时:2s
  336. 内存映射写入:2s
  337. 归并用时:2s
  338. 内存映射写入:2s
  339. 归并用时:2s
  340. 归并用时:2s
  341. 内存映射写入:4s
  342. 内存映射写入:2s
  343. 归并用时:3s
  344. 内存映射写入:2s
  345. 归并用时:2s
  346. 归并用时:2s
  347. 内存映射写入:2s
  348. 内存映射写入:2s
  349. 归并用时:2s
  350. 内存映射写入:2s
  351. 归并用时:2s
  352. 内存映射写入:2s
  353. 归并用时:2s
  354. 内存映射写入:2s
  355. 归并用时:2s
  356. 内存映射写入:2s
  357. 归并用时:2s
  358. 内存映射写入:2s
  359. 归并用时:2s
  360. 内存映射写入:2s
  361. 归并用时:2s
  362. 内存映射写入:2s
  363. 归并用时:2s
  364. 归并用时:1s
  365. 归并用时:2s
  366. 归并用时:4s
  367. 内存映射写入:6s
  368. 内存映射写入:8s
  369. 内存映射写入:4s
  370. 归并用时:2s
  371. 归并用时:2s
  372. 内存映射写入:3s
  373. 内存映射写入:5s
  374. 内存映射写入:2s
  375. 归并用时:3s
  376. 归并用时:2s
  377. 内存映射写入:2s
  378. 内存映射写入:2s
  379. 归并用时:2s
  380. 内存映射写入:1s
  381. 归并用时:3s
  382. 归并用时:2s
  383. 内存映射写入:2s
  384. 归并用时:2s
  385. 内存映射写入:2s
  386. 内存映射写入:2s
  387. 归并用时:2s
  388. 归并用时:2s
  389. 内存映射写入:2s
  390. 内存映射写入:2s
  391. 归并用时:2s
  392. 内存映射写入:1s
  393. 归并用时:2s
  394. 内存映射写入:2s
  395. 归并用时:2s
  396. 内存映射写入:2s
  397. 归并用时:2s
  398. 内存映射写入:2s
  399. 归并用时:2s
  400. 内存映射写入:1s
  401. 外部排序总用时:351

来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/29254281/viewspace-1172752/ ，如需转载，请注明出处，否则将追究法律责任。

0
0
上一篇： 外部排序
下一篇： JAVA操作MongoDB
请登录后发表评论 登录
全部评论
壹頁書

注册时间：2013-10-19

  • 博文量
    620
  • 访问量
    6339987

最新文章

支持我们 作者招募 用户协议 FAQ Contact Us

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

违法和不良信息举报电话:18101376593