Brickspace is my University of Houston Summer Undergraduate Research Fellowship 2014 project, developed under the supervision of Ioannis Kakadiaris and the UH Computational Biomedicine Laboratory.

1. Put your Lego bricks on a table.
2. Take a picture of the bricks with Brickspace.
3. Receive custom model instructions featuring your bricks.

Brickspace can detect red, orange, yellow, green, blue, and black 2x4 bricks, as seen in the video below. Download Brickspace from the iOS App Store and try it for yourself!

During the research project, the app included support for the Structure Sensor from Occipital. The Structure is like a mini-Kinect for mobile devices. When the Structure is attached, it provides depth data to the app. Brickspace uses this data to esimate the size of each brick and assign it one of the four supported sizes: 2x1, 2x2, 2x3, or 2x4. The public version does not contain Structure support. If you're interested, I've written an extensive explanation of how this version of Brickspace uses the Structure Sensor.

Download

Use the image above, or type in this link:
http://ansonj.org/brickspaceapp

Code

The source code for the entire application is available on GitHub. The structureEnabled branch contains the code for the frozen research version with Structure Sensor support, and master contains the latest public version.

Press

Turning Cancer to Fat, Drunk Fruit Flies Among Student Projects from Lisa Merkl at UH

Requirements

1. iPad of your choice.
2. Building bricks of various colors. Designed for red, orange, yellow, green, blue, and black Lego 2x4 bricks.
3. A white or lightly-colored surface.

The process

1. Dump out your bricks onto your white surface.
2. Spread the bricks out a bit so that they're not touching each other.
3. Use the app to take a picture of the bricks. The app will detect the bricks in the image.
4. Review the results of the image capture and processing. You may need to change the size or color of the bricks, since the app assumes all bricks are 2x4. Also, the app sometimes counts extra shapes (like shadows) as bricks, which you can delete during this step.
5. The app will then give you custom model instructions using the bricks that it has detected. Build away!

Limitations

• Adding bricks: The app does not yet support tapping the image to add bricks that the image processing missed, but this will be supported in a future release.
• Available models: The app currently knows how to build three different models. More models will be available in future updates.
  1. Basic Tower: All of your 2x4s stacked on top of each other.
  2. Flat Pyramid: All of your 2x4s arranged into a pyramid.
  3. Spiral Tower: All of your 2x4s arranged into a square tower with jagged, spiraling edges and a colorful pattern.
• Scanning / image detection: The image processing stumbles over shadows and any patterns on the scanning surface. Small bricks that are close together are seen as one brick. Tweaking parameters to detect smaller objects often results in the detector registering the studs on top of a brick as bricks.
• Brick size detection: In the research version with support for the Structure Sensor, the app only correctly assigned sizes to bricks about 50% of the time.

Under the hood

Brickspace uses OpenCV for image processing and object detection—specifically, SimpleBlobDetector from features2d.h. This allows the app to detect where bricks are and estimate their color.

In the research version, the Structure SDK handles communication with the Structure Sensor. When Brickspace captures an image, the SDK provides a depth measurement at each pixel. This is used to estimate the volume of each brick. If you're interested, I've written an explanation of how this works.

Stats, just for kicks

• 27 header files
• Lines of code (counting only the header files and their implementations plus main.m):
  • Research version (with Structure support): 4,301
  • Public version: 3,762

Disclaimer

Brickspace is not affiliated with, sponsored by, or related to The Lego Group, Lego Fusion, or Occipital.