Sunday, July 26, 2015

Mobile Barcode Scanning With Zxing.Net


A LOB mobile application I’m working on requires the ability to scan barcodes. A proper hardware solution would be best but sourcing something suitable has proven challenging. As a result we’re trying to scan barcodes using the device’s camera.

When we started the app was a Xamarin project so we tried Zxing.Net.Mobile. Unfortunately performance was abysmal. Many barcodes wouldn’t scan at all and the ones that did required many seconds to get a read. To be fair, we may have configured the settings badly. We found little guidance on the right settings. Even on iOS, enabling Apple's native API’s for decoding barcodes only made marginal difference.

We then looked at commercial solutions. We're not against paying for quality. Most of these SDK's provided a great experience. Yet we found the licenses over the top and difficult to understand. One company wanted tens of thousands of dollars because we will distribute the app via the store. This is despite our app requiring a specialised back end, limiting the audience to a few thousand people. Several others wanted per user per symbology, or per user per platform licenses.  These quickly rack up into the tens of thousands as well. The licenses were all yearly or monthly renewals too. Tracking the exact number of users is problematic. Our devices may not connect to the internet, or connect only periodically.  These devices aren’t personal the same way most mobile phones are. We will have several different users using the same device at different times.  There isn’t an accurate way to track users per device, or total users. No one talked about these issues, or how to report active users for licensing purposes. Another problem is most of these SDK’s only support one or two platforms not all three.

When we changed to a UWP application the situation was even worse. There are few SDK’s for the Windows platform and none that currently target UWP. There are some for WinRT that work in a UWP project, but they have separate binaries for x86/x64 and arm. None have packaged these binaries in a way a UWP project can reference and remain 'universal'. I believe this can be solved by repackaging the binaries ourselves but we never tried.

So we returned to Zxing.Net-, but this time only for decoding the barcodes. We wrote our own camera integration, based on the CameraGetPreviewFrame sample from Microsoft. We also configured Zxing differently. Now we have something with ‘acceptable’ performance for camera based barcode scanning. Unscientific tests show it out performs some implementations our competitors have used on iOS. What follows are tips based on our experience for getting the best out of Zxing. Your mileage may vary. Hardware, environmental conditions and use cases can have an impact. These tips should serve as a good starting point though.

Speed is King

The key take away is this: ensure decoding the barcode is fast. Slow decoding of the barcode is death to the user experience. This might seem counter-intuitive because accuracy would seem just as important. The trick is, with camera based scanning you have a lot of ‘dead’ frames. Frames where the barcode isn't present, isn't in focus or isn't lit correctly. The longer it takes to process each dead frame, the longer it will take to get a read.

  • Use a low to medium resolution. In my testing resolutions between 640x480 and 1024x768 work best. A higher resolution might seem better, as it should improve the barcode definition. In reality it slows down the decoding process and doesn’t help. Most mobile devices pick the highest resolution by default. They assume you’re trying to take video for the sake of video. You need to choose a sensible resolution for barcode scanning.
  • Don’t queue frames. I never actually tried this, but I did think about it. Later I found someone who had tried it, and abandoned it as a bad idea. Most queued frames will be ‘dead’ frames. Processing dead frames means getting a valid read will take longer. Just get a frame, attempt a decode, then get another frame. Don’t worry about the frames that get dropped.
  • Configure Zxing for best performance. Your use cases determine the best settings for you. I recommend these settings as a starting point. 
    • AutoRotate = false. Turning this on means decodes take about three times as long. Note you may need to enable this if you're scanning 2D barcodes. For 1D barcodes it is generally not helpful as orientation doesn't matter anyway. 
    • TryHarder = false. This setting isn’t intended for mobile scanning. It’s for large documents (say A4 document scans) that may contain barcodes anywhere. It makes a big impact to speed. Turn this off. You don’t need it. Trust me.
    • TryInverted = false. I haven't found this ever helped. 
    • Only add barcode formats you need to decode. As a general rule every barcode symbology you add increases the decode time. Get rid of the ones you don’t need. In our app we either enable all 1D barcodes and QR codes, or just all 1D barcodes. Which set is based on the context we are scanning in.
  • If you need to rotate the preview frames, do so efficiently. The Microsoft sample referenced above shows the most efficient way. There is a more discoverable, simpler mechanism but it is significantly slower. Use the one from the sample.

Using these settings we get a decode rate of 80-110ms per frame on a Lumia 636. We average about 10 decoded frames a second. While this might not sound great, we get a pretty good experience.

Auto Focus is Overrated

My last tip concerns focus. There’s a lot written about the need for ‘auto focus’ to get fast accurate reads. There’s also ‘continuous focus’ supported on some phones which sounds like a great idea. We have abandoned both of these. Continuous focus was never good enough to get a barcode read, in my experience. The frames looked ok to my eyes but Zxing.Net never found a barcode.

Auto focus is a bit more complex. It does work. We do use it in our app. We will perform an auto-focus if the user taps on the frame preview control on screen. This helps if the user is having difficulty or is scanning small or large barcodes. We don't perform auto-focus by default or automatically though.

Our default focus when scanning starts is a manual focus. We ask the camera to focus, once, at it’s smallest focal length. Depending on your hardware you may want to use some other (manual) value, but this works well for us.

The manual focus eliminates the camera seeking back and forth through the focal range. This improves the read speed. A user familiar with the system will get an instant read by holding the barcode the right distance away. For those not so familiar the read is still quick. If the barcode isn’t in focus immediately, the user can move the barcode or the phone slightly closer or further away. Scan times under a second are common.

Give manual focus try and see how it works for you.

Obviously none of this matches the performance of dedicated barcode scanning hardware. Yet it’s the best we’ve been able to achieve with what we have. It’s also better than where we started, and so far it’s been free.

One Last Thing

We haven’t tried this. It's just supposition.

It’s possible recompiling Zxing as a UWP library using .Net Native may improve things further. This should provide better performance for the decoding process.

8 comments:

  1. Hi,

    Great article and struggling just as you are. Do you mind me asking, what exactly did you do with the 'CameraGetPreviewFrame'? Do you get your users to tap the screen or did you get it to work and constantly scan the screen until a barcode is found? I've found out that it is possible in Windows 8 as Shop Savy QR Code Reader are doing it, but can't find any relevant tools they have used or to do the same. This is really accurate and fast and would like the same solution in my UWP app. Do you have any suggestions as to how this can be achieved?

    ReplyDelete
    Replies
    1. Hi,

      Glad your found the article helpful. The two ways I've found of getting the preview frame for processing that work best are;

      1. Yes, just 'constantly' get the preview frame - but only one at a time. By this I mean, get the frame, process it for barcodes and while doing so ignore any more preview frames (don't queue them up). If you don't find a barcode, *then* grab the next frame and process that. Queuing up the frames that occur while you're trying to find/process the barcode seems to be a bad idea.

      2. If you use auto focus (on a device that supports it), there is an event that fires when focus is corrected. Capturing the frame at that point (when the event is raised) is likely to give you a good image you can detect the barcode in. However, you have to have a device that supports auto-focus, and you have to wait for th auto-focus to complete.

      There are a number of 3rd party SDK's for detecting/decoding the barcodes in frames. Many of them seem to be much better (or at least quicker) than Zxing, however every one I looked at had licensing that was over the top (for our projects), so we couldn't use them. If you can find one where the licensing works for and that has good performance, then great :)

      Best of luck.

      Delete
  2. Hi Yort, I thought using zxing.net.mobile was great as while buggy here and there, I managed to sort out all issues and I though it would save me having to re-invent the wheel but I just discovered a major problem where it works perfectly in debug mode but it crashes really badly when in release mode. It looks like I've got a long weekend ahead of me!! I'm going to start looking at your solution once again. :). Hopefully I'll figure out something quickly and something stable!

    ReplyDelete
  3. Sorry to hear that, hope you get it solved. I haven't seen any release only issues myself so it should be solvable.

    ReplyDelete
  4. Hey Yort,
    This is a great article you have shared here on Mobile Barcode Scanning With Zxing.Net. I read your post in detail and got very required information. You have done a very good job. Thanks a lot.

    ReplyDelete
  5. Hi Yort!

    I've been trying to find information on how to change the resolution for the scanner interface using ZXing in UWP, but I haven't been able to find anything yet. Could you share the way you choose the resolution? Thanks!

    ReplyDelete
    Replies
    1. Hi,

      Given the video capture is done with the native UWP API's, there is nothing to do with Zxing (at least not in my implementation). I use LINQ over Capture.VideoDeviceController.GetAvailableMediaStreamProperties to find a 'best for purpose' resolution. Here's some sample code;

      var res = _Capture.VideoDeviceController.GetAvailableMediaStreamProperties(MediaStreamType.VideoPreview);

      var selectedResolution = (
      from p in res
      let vp = p as VideoEncodingProperties
      where vp.Height * vp.Width >= 480000 && vp.Height * vp.Width <= 786432
      orderby vp.Height * vp.Width ascending
      select vp
      ).FirstOrDefault();

      if (selectedResolution == null)
      {
      selectedResolution = (
      from p in res
      let vp = p as VideoEncodingProperties
      where vp.Height * vp.Width >= 307200
      orderby vp.Height * vp.Width ascending
      select vp
      ).FirstOrDefault();
      }

      if (selectedResolution != null)
      await _Capture.VideoDeviceController.SetMediaStreamPropertiesAsync(MediaStreamType.VideoPreview, selectedResolution);

      Good luck!

      Delete