Monday, July 27, 2015

Compose Tasks, UWP & Crash Workarounds


If you’ve worked on WP Silverlight applications you’re probably familiar with the ‘compose tasks’. These present consistent UI for composing email, sms messages or making a phone call.

Recently I needed similar behaviour in a UWP application where the API is different. The good news is this functionality still exists. It's just not always be obvious where to look.

The replacement for PhoneCallTask is Windows.ApplicationModel.Calls.PhoneCallManager.ShowPhoneCallUI.

The replacement for EmailComposeTask is the Windows.ApplicationModel.Email.EmailManager.ShowComposeNewEmailAsync method and the Windows.ApplicationModel.Email.EmailMessage class.

The ComposeSmsTask also has a replacement API. There is a significant  issue using this on the desktop versions of Windows 10.

The replacement for ComposeSmsTask is the Windows.ApplicationModel.Chat.ChatMessageManager. ShowComposeSmsMessageAsync method and Windows.ApplicationModel.Chat.ChatMessage class.

This is a universal API and thus is available on desktop and mobile platforms. In the pre-release bits for UWP, this works fine on mobile but will crash the app on desktop. Whether this will be fixed before the official release is unknown. Note I really mean crash. There is no exception to catch and the unhandled exception event doesn't get raised. The process just dies in a nasty way. ApiInformation.IsTypePresent and such don't help because the API is intentionally available on both platforms.  

This leaves you with no (good) way to support the feature and avoid the crash. Thankfully @aruntalkstech has the answer. You can use ChatMessageManager.GetTransportsAsync to determine if the current device supports any messaging protocols. If it does, it’s safe to call the compose API, if not then you should avoid it. For more information sample code, see Arun’s blog post.

Sunday, July 26, 2015

Mobile Barcode Scanning With Zxing.Net


A LOB mobile application I’m working on requires the ability to scan barcodes. A proper hardware solution would be best but sourcing something suitable has proven challenging. As a result we’re trying to scan barcodes using the device’s camera.

When we started the app was a Xamarin project so we tried Zxing.Net.Mobile. Unfortunately performance was abysmal. Many barcodes wouldn’t scan at all and the ones that did required many seconds to get a read. To be fair, we may have configured the settings badly. We found little guidance on the right settings. Even on iOS, enabling Apple's native API’s for decoding barcodes only made marginal difference.

We then looked at commercial solutions. We're not against paying for quality. Most of these SDK's provided a great experience. Yet we found the licenses over the top and difficult to understand. One company wanted tens of thousands of dollars because we will distribute the app via the store. This is despite our app requiring a specialised back end, limiting the audience to a few thousand people. Several others wanted per user per symbology, or per user per platform licenses.  These quickly rack up into the tens of thousands as well. The licenses were all yearly or monthly renewals too. Tracking the exact number of users is problematic. Our devices may not connect to the internet, or connect only periodically.  These devices aren’t personal the same way most mobile phones are. We will have several different users using the same device at different times.  There isn’t an accurate way to track users per device, or total users. No one talked about these issues, or how to report active users for licensing purposes. Another problem is most of these SDK’s only support one or two platforms not all three.

When we changed to a UWP application the situation was even worse. There are few SDK’s for the Windows platform and none that currently target UWP. There are some for WinRT that work in a UWP project, but they have separate binaries for x86/x64 and arm. None have packaged these binaries in a way a UWP project can reference and remain 'universal'. I believe this can be solved by repackaging the binaries ourselves but we never tried.

So we returned to Zxing.Net-, but this time only for decoding the barcodes. We wrote our own camera integration, based on the CameraGetPreviewFrame sample from Microsoft. We also configured Zxing differently. Now we have something with ‘acceptable’ performance for camera based barcode scanning. Unscientific tests show it out performs some implementations our competitors have used on iOS. What follows are tips based on our experience for getting the best out of Zxing. Your mileage may vary. Hardware, environmental conditions and use cases can have an impact. These tips should serve as a good starting point though.

Speed is King

The key take away is this: ensure decoding the barcode is fast. Slow decoding of the barcode is death to the user experience. This might seem counter-intuitive because accuracy would seem just as important. The trick is, with camera based scanning you have a lot of ‘dead’ frames. Frames where the barcode isn't present, isn't in focus or isn't lit correctly. The longer it takes to process each dead frame, the longer it will take to get a read.

  • Use a low to medium resolution. In my testing resolutions between 640x480 and 1024x768 work best. A higher resolution might seem better, as it should improve the barcode definition. In reality it slows down the decoding process and doesn’t help. Most mobile devices pick the highest resolution by default. They assume you’re trying to take video for the sake of video. You need to choose a sensible resolution for barcode scanning.
  • Don’t queue frames. I never actually tried this, but I did think about it. Later I found someone who had tried it, and abandoned it as a bad idea. Most queued frames will be ‘dead’ frames. Processing dead frames means getting a valid read will take longer. Just get a frame, attempt a decode, then get another frame. Don’t worry about the frames that get dropped.
  • Configure Zxing for best performance. Your use cases determine the best settings for you. I recommend these settings as a starting point. 
    • AutoRotate = false. Turning this on means decodes take about three times as long. Note you may need to enable this if you're scanning 2D barcodes. For 1D barcodes it is generally not helpful as orientation doesn't matter anyway. 
    • TryHarder = false. This setting isn’t intended for mobile scanning. It’s for large documents (say A4 document scans) that may contain barcodes anywhere. It makes a big impact to speed. Turn this off. You don’t need it. Trust me.
    • TryInverted = false. I haven't found this ever helped. 
    • Only add barcode formats you need to decode. As a general rule every barcode symbology you add increases the decode time. Get rid of the ones you don’t need. In our app we either enable all 1D barcodes and QR codes, or just all 1D barcodes. Which set is based on the context we are scanning in.
  • If you need to rotate the preview frames, do so efficiently. The Microsoft sample referenced above shows the most efficient way. There is a more discoverable, simpler mechanism but it is significantly slower. Use the one from the sample.

Using these settings we get a decode rate of 80-110ms per frame on a Lumia 636. We average about 10 decoded frames a second. While this might not sound great, we get a pretty good experience.

Auto Focus is Overrated

My last tip concerns focus. There’s a lot written about the need for ‘auto focus’ to get fast accurate reads. There’s also ‘continuous focus’ supported on some phones which sounds like a great idea. We have abandoned both of these. Continuous focus was never good enough to get a barcode read, in my experience. The frames looked ok to my eyes but Zxing.Net never found a barcode.

Auto focus is a bit more complex. It does work. We do use it in our app. We will perform an auto-focus if the user taps on the frame preview control on screen. This helps if the user is having difficulty or is scanning small or large barcodes. We don't perform auto-focus by default or automatically though.

Our default focus when scanning starts is a manual focus. We ask the camera to focus, once, at it’s smallest focal length. Depending on your hardware you may want to use some other (manual) value, but this works well for us.

The manual focus eliminates the camera seeking back and forth through the focal range. This improves the read speed. A user familiar with the system will get an instant read by holding the barcode the right distance away. For those not so familiar the read is still quick. If the barcode isn’t in focus immediately, the user can move the barcode or the phone slightly closer or further away. Scan times under a second are common.

Give manual focus try and see how it works for you.

Obviously none of this matches the performance of dedicated barcode scanning hardware. Yet it’s the best we’ve been able to achieve with what we have. It’s also better than where we started, and so far it’s been free.

One Last Thing

We haven’t tried this. It's just supposition.

It’s possible recompiling Zxing as a UWP library using .Net Native may improve things further. This should provide better performance for the decoding process.

Saturday, July 25, 2015

PointerDownThemeAnimation, PointerUpThemeAnimation & Null Reference Exceptions


WinRT and UWP projects have access to PointerDownThemeAnimation and PointerUpThemeAnimation. These animations provide a consistent visual feedback experience with the OS and other apps. Unfortunately these animations have some specific implementation details and don’t always behave as expected. This is most often true when using them from code behind instead of declaring them in XAML. This usually happens when you’re building a templated control, behaviour or similar reusable component.

The most common issue that occurs is an unexpected null reference exception. To avoid this;

  • Ensure the target element (the one to be animated) has a name. The easiest way is to assign one in XAML at design time. If no name is present at runtime you need to assign one.
  • Ensure you set the TargetName property on the animation. Setting the Target property won’t work. You must set the TargetName property, and that’s why you need a name set on the target.
  • You must add the animation into the logical tree for the target. My suggestion is to add it to the resources collection for the target. If you don’t do this the system has no context to search in so it still won’t find the target. This will also result in a null reference exception.

It's a shame there isn’t a better error message. The inability to assign a target object without a name is also disappointing.

Another issue is the duration property for these animations. You can set it without receiving an error, but the duration won’t change. This is fine as you should be seeking consistency with other apps. It's just sad that this is a silent failure. It is noted on MSDN though.

Below is a draft XAML Behaviour to apply these animations to any visual element. Feel free to use it (Apache 2.0 license). I have Windows 10 UWP code based on this and it should work in WinRT for Windows 8.1 too. In both cases it requires the Behaviours SDK for WinRT

This custom behaviour adds both the animation and a command binding. If specified the bound command will execute after the animation completes. If the command reports as not executable, neither the animation nor the command executes.

   1:      public class BehaviourBase<T> : DependencyObject, IBehavior where T : DependencyObject
   2:      {
   3:   
   4:          private T _AssociatedObject;
   5:   
   6:          public DependencyObject AssociatedObject
   7:          {
   8:              get
   9:              {
  10:                  return _AssociatedObject;
  11:          }
  12:          }
  13:   
  14:          public T TypedAssociatedObject
  15:          {
  16:              get { return _AssociatedObject; }
  17:          }
  18:   
  19:          public void Attach(DependencyObject associatedObject)
  20:          {
  21:              if (associatedObject == null) throw new ArgumentNullException("associatedObject");
  22:   
  23:              _AssociatedObject = (T)associatedObject;
  24:              Attached(_AssociatedObject);
  25:          }
  26:   
  27:          public virtual void Attached(T associatedObject)
  28:          {
  29:          }
  30:   
  31:          public void Detach()
  32:          {
  33:              if (_AssociatedObject != null)
  34:              {
  35:                  Detatching();
  36:                  _AssociatedObject = null;
  37:              }
  38:          }
  39:   
  40:          public virtual void Detatching()
  41:          {
  42:          }
  43:   
  44:      }
  45:   
  46:      public class VisualTapBehavior : BehaviourBase<UIElement>
  47:      {
  48:   
  49:          public static readonly DependencyProperty CommandProperty = DependencyProperty.Register("Command", typeof(ICommand), typeof(VisualTapBehavior), new PropertyMetadata(null));
  50:          public static readonly DependencyProperty CommandParameterProperty = DependencyProperty.Register("CommandParameter", typeof(object), typeof(VisualTapBehavior), new PropertyMetadata(null));
  51:   
  52:          private Storyboard _PointerDownStoryboard;
  53:          private Storyboard _PointerUpStoryboard;
  54:   
  55:          public Storyboard Storyboard { get; internal set; }
  56:   
  57:          public ICommand Command
  58:          {
  59:              get { return (ICommand)this.GetValue(CommandProperty); }
  60:              set
  61:              {
  62:                  this.SetValue(CommandProperty, value);
  63:              }
  64:          }
  65:   
  66:          public object CommandParameter
  67:          {
  68:              get { return this.GetValue(CommandParameterProperty); }
  69:              set
  70:              {
  71:                  this.SetValue(CommandParameterProperty, value);
  72:              }
  73:          }
  74:   
  75:          public override void Attached(UIElement associatedObject)
  76:          {
  77:              base.Attached(associatedObject);
  78:   
  79:              //Stupidly, these theme animations require the object to have a name :(
  80:              var fe = ((FrameworkElement)associatedObject);
  81:              var name = fe.Name;
  82:              if (String.IsNullOrEmpty(name))
  83:              {
  84:                  name = Guid.NewGuid().ToString();
  85:                  fe.Name = name;
  86:              }
  87:   
  88:              if (fe.Resources.ContainsKey("VisualTapDownAnimation"))
  89:                  _PointerDownStoryboard = (Storyboard)fe.Resources["VisualTapDownAnimation"];
  90:              else
  91:              {
  92:                  var pointerDownStoryboard = new Storyboard();
  93:                  var downAnimation = new PointerDownThemeAnimation();
  94:                  Storyboard.SetTargetName(downAnimation, name);
  95:                  pointerDownStoryboard.Children.Add(downAnimation);
  96:                  _PointerDownStoryboard = pointerDownStoryboard;
  97:                  fe.Resources.Add(new KeyValuePair<object, object>("VisualTapDownAnimation", pointerDownStoryboard));
  98:              }
  99:   
 100:              if (fe.Resources.ContainsKey("VisualTapUpAnimation"))
 101:                  _PointerUpStoryboard = (Storyboard)fe.Resources["VisualTapUpAnimation"];
 102:              else
 103:              {
 104:                  var pointerUpStoryboard = new Storyboard();
 105:                  var upAnimation = new PointerUpThemeAnimation();
 106:                  Storyboard.SetTargetName(upAnimation, name);
 107:                  pointerUpStoryboard.Children.Add(upAnimation);
 108:                  pointerUpStoryboard.Completed += PointerUpStoryboard_Completed;
 109:                  _PointerUpStoryboard = pointerUpStoryboard;
 110:                  ((FrameworkElement)associatedObject).Resources.Add(new KeyValuePair<object, object>("VisualTapUpAnimation", pointerUpStoryboard));
 111:              }
 112:   
 113:              associatedObject.PointerPressed += AssociatedObject_PointerPressed;
 114:              associatedObject.PointerReleased += AssociatedObject_PointerReleased;
 115:          }
 116:   
 117:          private async void PointerUpStoryboard_Completed(object sender, object e)
 118:          {
 119:              await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal,
 120:                  () =>
 121:                  {
 122:                      var command = this.Command;
 123:                      var commandParameter = this.CommandParameter;
 124:                      if (command != null && command.CanExecute(commandParameter))
 125:                          command.Execute(commandParameter);
 126:                  });
 127:          }
 128:   
 129:          public override void Detatching()
 130:          {
 131:              base.Detatching();
 132:   
 133:              TypedAssociatedObject.PointerPressed -= this.AssociatedObject_PointerPressed;
 134:              TypedAssociatedObject.PointerReleased -= this.AssociatedObject_PointerReleased;
 135:   
 136:              _PointerDownStoryboard = null;
 137:          }
 138:   
 139:          private void AssociatedObject_PointerPressed(object sender, Windows.UI.Xaml.Input.PointerRoutedEventArgs e)
 140:          {
 141:              ((FrameworkElement)sender).CapturePointer(e.Pointer);
 142:   
 143:              var command = this.Command;
 144:              var commandParameter = this.CommandParameter;
 145:              if (command == null || command.CanExecute(CommandParameter))
 146:                  RunStoryboardIfNotNull(_PointerDownStoryboard);
 147:          }
 148:   
 149:          private void AssociatedObject_PointerReleased(object sender, Windows.UI.Xaml.Input.PointerRoutedEventArgs e)
 150:          {
 151:              ((FrameworkElement)sender).ReleasePointerCapture(e.Pointer);
 152:              RunStoryboardIfNotNull(_PointerUpStoryboard);
 153:          }
 154:   
 155:          private void RunStoryboardIfNotNull(Storyboard storyboard)
 156:          {
 157:              if (storyboard != null)
 158:              {
 159:                  storyboard.Stop();
 160:                  storyboard.Begin();
 161:              }
 162:          }
 163:      }