Morning Joe: Big Data Tools for Review

Big data is a big topic and many tools are starting to Spring up. However, implementations of the old SQL standard are falling far behind these tools. My recent post on using a Fork Join Pool in insertion and pulling data can help if using PostgreSQL but also destroys bandwidth. It cause headaches for everyone whenever I need to do some serious checking of my ETL,parsing, normalization, data work, or other processes outside of our co-location. Although this is nothing new, I’ve compiled a list of tools to go forth and make little rocks from big rocks all day.

I’ve found some companies and a tool for a big review later on, this is my prework post while my IDE starts up. So far, I have found technologies for SQL databases that use Hadoop,Fractal Trees, and Cassandra to speed up the process. They are not focused specifically on speed but can help create faster database access and lower coding time.

What I’ve found so far:

  • Cassandra (open source): promises scalability and availability alongside a plethora of features (maybe for implementing other tools)
  • Oracle Data Integration Adapter for Hadoop: promises the speed of hadoop connected to an Oracle database
  • BigSQL (open source):promises to combine Cassandra,PostgreSQL, and Hadoop into a blazing fast package for analsis.
  • MapR technologies (somewhat open source): offers a wide variety of products to improve speed in querying and analysis from Hive and map reduce, to actual hadoop
  • Fractal Tree Indexing (open source): Tokutech’s fractal tree indexing speeds up insertions using buffers on each tree node
  • Alteryx: a tool for quicker data processing though not quite as fast as the others (good if your budget does not allow clusters but allows something better than Pentaho
  • MongoDB (open source): Combines map reduce and other technologies with large databases. Tokutech tested its fractal tree indexes on MongoDB
  • Pentaho (open source): The open source version of Alteryx

Many of these tools are already implemented in others such as Pentaho. Personally, I would like to see a SQL-like language that uses these tools alongside a query processor. It would make the tasks even faster to write, think Java v. Python. You could have 10 lines of map reduce code, 5 minutes of click and drag, or a 1 line easy-going query that writes as you think.

To be clear, I am not ranking these, only marking them for future review since this is what has piqued my interest today. Cheers!

Python PDF 2: Writing and Manipulating a PDF with PyPDF2 and ReportLab

Note: PdfMiner3K is out and uses a nearly identical API to this one. Fully working code examples are available from my Github account with Python 3 examples at CrawlerAids3 and Python 2 at CrawlerAids (both currently developed)

In my previous post on pdfMiner, I wrote on how to extract information from a pdf. For completeness, I will discuss how PyPDF2 and reportlab can be used to write a pdf and manipulate an existing pdf. I am learning as I go here. This is some low hanging fruit meant to provide a fuller picture. Also, I am quite busy.

PyPDF and reportlab do not offer the completeness in extraction that pdfMiner offers. However, they offer a way of writing to existing pdfs and reportlab allows for document creation. For Java, try PDFBox.

However, PyPdf is becoming extinct and pyPDF2 has broken pages on its website. The packages are still available from pip,easy_install, and from github. The mixture of reportlab and pypdf is a bit bizzare.

PyPDF2 Documentation

PyPdf, unlike pdfMiner, is well documented. The author of the original PyPdf also wrote an in depth review with code samples. If you are looking for an in depth manual for use of the tool, it is best to start there.

Report Lab Documentation

Report lab documentation is available to build from the bitbucket repositories.

Installing PyPdf and ReportLab

Pypdf2 and reportlab are easy to install. Additionally, PyPDF2 can be installed from the python package site and reportlab can be cloned.

   easy_install pypdf2
   pip install pypdf2
   easy_install reportlab
   pip install reportlab

ReportLab Initialization

The necessary part of report lab is the canvas objects. Report lab has several sizes. They are letter,legal, and portrait. The canvas object is instantiated with a string and size.

from reportlab.pdfgen import canvas
from reportlab.lib.pagesizes import portrait

PyPdf2 Initialization

PyPdf2 has a relatively simple setup. A PdfFileWriter is initialized to add to the document, as opposed to the PdfReader which reads from the document. The reader takes a file object as its parameter. The writer takes an output file at write time.

   from pyPDF2 import PdfFileWriter, PdfFileReader
   # a reader
   # a writer

All of this can be found under the review by the writer of the original pyPdf.

Working with PyPdf2 Pages and Objects

Before writing to a pdf, it is useful to know how to create the structure and add objects. With the PdfFileWriter, it is possible to use the following methods (an IDE or the documentation will give more depth).

  • addBlankPage-create a new page
  • addBookmark-add a bookmark to the pdf
  • addLink- add a link in a specific rectangular area
  • addMetaData- add meta data to the pdf
  • insertPage-adds a page at a specific index
  • insertBlankPage-insert a blank page at a specific index
  • addNamedDestination-add a named destination object to the page
  • addNamedDestinationObject-add a created named destination to the page
  • encrypt-encrypt the pdf (setting use_128bit to True creates 128 bit encryption and False creates 40 bit encryption with a default of 128 bits)
  • removeLinks-removes links by object
  • removeText-removes text by text object
  • setPageMode-set the page mode (e.g. /FullScreen,/UseOutlines,/UseThumbs,/UseNone
  • setPageLayout-set the layout(e.g. /NoLayout,/SinglePage,/OneColumn,/TwoColumnLeft)
  • getPage-get a page by index
  • getLayout-get the layout
  • getPageMode-get the page mode
  • getOutlineRoot-get the root outline

ReportLab Objects

Report lab also contains a set of objects. Documentation can be found here. It appears that postscript or something similar is used for writing documents to a page in report lab. Using ghostscript, it is possible to learn postscript. Postscript is like assembler and involves manipulating a stack to create a page. It was developed at least in part by Adobe Systems, Inc. back in the 1980s and before my time on earth began.

Some canvas methods are:

  • addFont-add a font object
  • addOutlineEntry-add an outline type to the pdf
  • addPostscriptCommand-add postscript to the document
  • addPageLabel-add a page label to the document canvas
  • arc-draw an arc in a postscript like manner
  • beginText-create a text element
  • bezier-create a postscript like bezier curve
  • drawString-draw a string
  • drawText-draw a text object
  • drawPath-darw a postscript like path
  • drawAlignedString-draw a string on a pivot character
  • drawImage-draw an image
  • ellipse-draw an elipse on a bounding box
  • circle-draw a circle
  • rect-draw a rectangle

Write a String to a PDF

There are two things that dominate the way of writing pdf files, writing images, and writing strings to the document. This is handled entirely in

Here, I have added some text and a circle to a pdf.

def writeString():
    packet = StringIO.StringIO()
    cv=canvas.Canvas(packet, pagesize=letter)
    #create a string
    cv.drawString(0, 500, "Hello World!")
    #a circle. Do not add another string. This draws on a new page., 250, 20, stroke=1, fill=0)
    #save to string
    #get back to 0
    #write to a file
    with open(fpath,'wb') as fp:

The output of the above code:
Page 1
 photo page1.png

Page 2
 photo page2.png

Unfortunately, adding a new element occurs on a new page after calling the canvas’ save method. Luckily the “closure” of the pdf just creates a new page object. A much larger slice of documentation by reportlab goes over writing a document in more detail. The documentation includes alignment and other factors. Alignments are provided when adding an object to a page.

Manipulating a PDF
Manipulation can occur with ReportLab. ReportLab allows for deletion of pages,insertion of pages, and creation of blank pages. The author of pyPDF goes over this in depth in his review.

This code repeats the previous pages twice in a new pdf. It is also possible to merge (overlay) pdf pages.

    from PyPDF2 import PdfFileWriter,PdfFileReader

    writer = PdfFileWriter()
    # add the page to itself
    for i in range(0,pdf1.getNumPages()):
    for i in range(0,pdf2.getNumPages()):
    # write to file
    with file("destination.pdf", "wb") as outfp:

Overall Feedback
Overall, PyPDF is useful for merging and changing existing documents in terms of the the way they look and reportlab is useful in creating documents from scratch. PyPDF deals mainly with the objects quickly and effectively and reportlab allows for in depth pdf creation. In combination, these tools rival others such as Java’s PdfBox and even exceed it in ways. However, pdfMiner is a better extraction tool.