I was invited to work with The Works museum on an idea they wanted to execute, but had no idea how to make
it work. My work on the Magic the Gathering card recognition program gave me the tools I needed to apply
object detection and color recognition to not only bring this exhibit to fruition, but also make it awesome.
The key to recognizing the correct colors of lego bricks, regardless of ambient color temperature, was to
create a physical "legend", which contained every brick color that was used in the exhibit, and affixing it
to the submission station under the webcam in a known position. This way, the Java program checking the colors
would have a perfect reference color every time, as long as it could accurately identify the black polygon.
Components of the exhibit that I contributed included:
- Pattern manager website for exhibit administrators
- One-page tablet app that presents exhibitors with a new 8x8 lego pattern for them to assemble
- Webcam & monitor station for kids to check their completed 8x8 tiles for accuracy
In order to account for various lighting conditions and the fluctuations of the automatic webcam gain, I
created an algorithm that constantly modified the threshold value for identifying the boder between the
black lego brick object and the white table surface (the cutoff point at which a given color should be
considered either black or white.) The algorithm would randomly mutate the threshold value, preferring
smaller changes over larger ones, and, after each mutation, check for a black quadrilateral. If the
check failed, the threshold would continue to mutate. Once the black polygon was correctly identified,
the "successful" threshold value would be stored, and then mutations would occur around that value.
Eventually, a collection of successful threshold values would be recorded, and the "ideal" value would be
the mean of those values. The threshold would then continually check itself, and if lighting conditions
changed and the black quadrilateral could not be identified, the random mutations would begin again.
Once the program had "locked on" to the black quadrilateral, perspective correction would be performed
to transform the polygon from an irregular shape to a rectangle. Then, based on known, hard-coded locations
of the "legend" Lego bricks, key colors would be established from five random points within each 1x1 brick.
Then, using a nearest-neighbor by the square of distances, each 1x1 block on the 8x8 tile would be mapped
to one of the colors."
