With FastStats Designer we have worked over the last few years on developing multi-threaded versions of our sequential data load algorithms. We have worked a component at a time replacing the sequential processes with new multi-threaded equivalents and in doing so we achieved a 25-30% speed improvement over the course of 2014.
For the 2015 Q2 version of FastStats Designer we are releasing for beta use our new multi-threaded data load and compression component “Apteco.DataBuild.dll”.
This routine still produces our standard cyprus phone number FastStats optimised, compressed binary data file formats but does so with entirely new code that has been written from scratch using the latest parallelisation technology. The technology we decided to base our component on is a Microsoft library called TPL Dataflow that extends the multi-threaded features of the .Net Framework to support data processing pipelines. We re-implemented our compression routines as components of this pipeline to make full use of the multi-threaded capabilities of modern processors.
Previously with the Nov 14 sequential load process we were able to build a TPC-H reference test system with 1.3Bn records (1Bn Line Item records) in 4 hours. The sequential data load portion of this build took nearly 2 hours. The new Apteco.DataBuild component performs the same processing in under 50 minutes dropping the total load time (including auto discovery, sort and compression) to 2 hours 50 minutes.