------Quick Links------

Introduction

Function Breakdown & Code Flow

Comments

---------------------------

TensorFlow TFRecord; Loading In and Popping Out Data (Part 2)

Author: Oluwole Oyetoke (17th December, 2017)

Contact Me Drop A Comment


Introduction

Continuing the previous post on converting a whole dataset into a TFRecord file in TensorFlow, on this post we will examine how we can also pop out the example protos saved into these record files in a seamless manner. For more details on TensorFlow TFRecords and how to create one, see the part 1 of this post

TFRecord


Diagram 1: Streaming Out Data Out of a TF Record

Function Breakdown & Code Flow

Just like I often would say, for any programming challenge, there are usually multiple solutions. Therefore, I believe there are multiple methods/approach of streaming out data out of a TFRecord. This is just one approach out of many. Feel free to post links to other ways of doing this in the comment section. The script contains two major functions which are the _process_multiple_images() function and the _process_one_imag() function. The breakdown below explain how interactions happen between then.

  1. Get path(s) to TF record files
  2. Determine the number of records in the TF Record file
  3. Process & get all images stored in the TFRecord file
    1. connect to TF Record to get raw dataset (line 64)
    2. pass every single content of the TF record through the '_process_one_image' function (line 65) which does the following:
      1. specify features to extract (line 45 to 49)
      2. finesse extracted data to suit your current needs e.g. reshape, cast from unit8 to float32 etc (line 53-59)
    3. create iterator which iterates through the dataset based on the number of batches you specify
    4. evaluate the tensors returned by the iterator for each iteration (line 83)
    5. squeeze the np array returned by the evaluator to ensure they are not out of form
    6. append this newly acquired batch of images (in np array) to the complete_labels and complete_images np arrays
    7. perform step 3d to 3f until 'num_of_epochs' specified is completed
    8. return entire dataset of extracted features e.g. images and labels as 2 big np arrays (line 105)
  4. Print shape of returned array to be sure you got it right

Full Code: Pop_TFRecord.py


HTML Comment Box is loading comments...