Wednesday, October 22, 2014

Smartphone image processing development made easy!

A very popular open source C# project for image processing, neural networks, machine learning etc. is the AForge.NET Framework, developed by Andrew Kirillow. AForge.NET has been used in numerous applications. It also forms the basis for the Accord.NET Framework, developed by César de Souza, which substantially extends AForge.NET with additional image processing filters, machine learning algorithms, statistical distributions and much much more. Numerous examples of what Accord.NET Framework is capable of are available here.

AForge.NET was developed to run on the .NET Framework and uses the System.Drawing assembly as basis for image manipulation and processing. In particular the dependency on System.Drawing heavily limits the applicability of AForge.NET (and subsequently, Accord.NET) to more modern platforms, like WPF, Windows (for tablet and desktop, as opposed to desktop-only applications), and Windows Phone.

It seemed like a terrible waste that these frameworks would only be available to the outdated range of Windows Forms applications. So, to enable the invaluable functionality of AForge.NET and Accord.NET for modern Microsoft platforms, I started the project of porting these frameworks into Portable Class Libraries, PCL. These efforts are also open sourced and are available on Github: Portable AForge and Portable Accord.

It should be noted that Portable AForge and Portable Accord cover a subset of the assemblies in the pure .NET based frameworks. Video and robotics support is not included, and audio support is limited. Windows Forms UI controls have not been ported. Nevertheless, practically all image processing and scientific computing functionality is included in the portable frameworks.

To facilitate continuous inclusion of updated code for the two frameworks, I have striven to keep the original AForge.NET and Accord.NET code bases as untouched as realistically possible. To cover up for those parts of the framework codes that are not applicable in the portable class libraries, I have therefore developed two support assemblies called Shim and Shim.Drawing.

Shim, which contains simple implementations of non-UI types and methods unavailable in the PCL profiles, is maintained in a separate Github repository, since this assembly is also immediately applicable as a legacy code bridge in other modern target adaptations, such as the port of the fo-dicom library to the (modern) Windows and Windows Phone platforms.

Shim.Drawing is internal to the Portable AForge and Accord assemblies, and is contained in the Portable AForge Github repository. It provides a down-scaled replacement for the System.Drawing assembly. Along with the platforms of primary interest, Shim.Drawing can also be contained in a .NET Framework WPF project as long as System.Drawing is not simultaneously referenced. Portable AForge and Accord is thus a also shortcut to using AForge.NET and Accord.NET in WPF applications.

To enable the portable framework assemblies on a specific target platform, the Shim and Shim.Drawing assemblies come in various flavors. There are for example specific Windows Phone (Silverlight) 8 versions of Shim and Shim.Drawing that should be referenced in a WP8 application using the Portable AForge and Accord assemblies.

Currently, Portable AForge and Portable Accord are available for the following target platforms:

  • .NET Framework 4.5 and higher (for WPF applications)
  • Windows 8 and higher (modern tablet/desktop applications)
  • Windows Phone Silverlight 8 and higher
  • Universal applications, i.e. Windows Phone 8.1 + Windows 8.1
  • Xamarin.iOS
  • Xamarin.Android

Yes, you read it right! My most recent developments also incorporate the ability to target the Xamarin.iOS and Xamarin.Android platforms. The Xamarin assemblies are however not available as open source. If you are interested in developing AForge and Accord based applications for Xamarin.iOS and/or Xamarin.Android, please contact me directly at licenses@cureos.com.

And now, with AForge and Accord available on the three mobile platforms, cross-platform development using Xamarin.Forms has also become a reality! This means that it is practically possible to contain an entire C# code base in a PCL core library that uses AForge and Accord, and consume this core library from end applications for all three mobile platforms, Windows Phone, Xamarin.iOS and Xamarin.Android.

Just to show what can now be very quickly done, I copied the functionality of the Face Detection (Haar Object Selector) sample application from the Accord.NET samples and contained it in a Xamarin.Forms project. All application code, including setting up the graphical layout and the face detection operation, is contained in the PCL core project. The platform specific end application projects only wrap the core assembly.

And this is what the face detection sample application looks like on the three mobile platforms:


A first small step, perhaps, but the possibilities that have now been opened up are practically infinite :-)


37 comments:

  1. This is awsome, thank you Anders!

    ReplyDelete
  2. hai,,, can u tell me,,how to used that library in windows phone 8.1.. i quite difficult to understand... thank you for the reply

    ReplyDelete
    Replies
    1. Hi Al Kisah,

      You can download AForge and Accord libraries from Github, https://github.com/cureos/aforge/releases and https://github.com/cureos/accord/releases .

      These libraries (together with some libraries from NuGet) are all you need to create a WPF desktop application or a Windows Phone Silverlight (8.0 or 8.1) application.

      If you are developing for Windows Phone 8.1 non-Silverlight you will need to obtain the necessary Shim.Drawing DLL for universal applications from me. If you plan to use it in any application distributed on Windows Store or elsewhere, the license cost is currently (Dec 2014) $100.

      If you are interested in this arrangement, please e-mail me at licenses@cureos.com

      Best regards,
      Anders @ Cureos

      Delete
    2. Hi Anders,


      CssShim is under LGPL so binaries should be free also for commercial use
      http://answers.google.com/answers/threadview/id/439136.html

      Delete
    3. Hi Mateusz,

      yes, CSShim is LGPL, and the source code is available on Github. You are free to use it as long as you adhere to LGPL.

      However, the Shim.Drawing source code (which is required for image related functionality), for PCL, Windows Phone 8 Silverlight and .NET/WPF is licensed under GPL. You are free to use it as long as you adhere to GPL, which in practice means that you would need to publish the source code of your end-user application if you want to use Shim.Drawing. If you do not want to publish your own source code, you can purchase a closed-source license from Cureos for Windows Phone 8 Silverlight (and WPF).

      I also have developed Shim.Drawing versions for Android, iOS and Windows Universal, but these versions are only available as closed-source. Licenses for these platforms can be purchased directly from Cureos.

      Regards,
      Anders @ Cureos

      Delete
  3. Very nice and helpful information has been given in this article. I like the way you explain the things. Keep posting. Thanks..

    mobile app development company usa

    ReplyDelete
    Replies
    1. Thanks, Peter, nice to hear the article has been helpful.

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. This is great!!!

    @Anders I have a problem to reference and use Shim.Drawing and Aforge (portable) on Xamarin projects.

    I'm basically including Shim via nuget and the release DLLs (https://github.com/cureos/aforge) on my project.

    But that is not working!

    Do I need to obtain the especific Shim.Drawing DLL for Xamarin?

    I'm trying the cast from Android.Graphics.Bitmap to System.Drawing.Bitmap (WriteableBitmap) without success.

    :-(

    Please, can you help me?

    --
    Râmede Bento

    ReplyDelete
    Replies
    1. Hi Ramede,

      yes, for Portable AForge image functionality to work on Xamarin (Android or iOS), you would need to have a Shim.Drawing DLL specifically for Android and iOS, respectively.

      The license cost to use Shim.Drawing DLL on one platform is currently $100, and $150 for both Android and iOS..

      Best regards,
      Anders @ Cureos

      Delete
    2. Thanks Anders, that's great !!!
      How do I get a version of the DLL Shim.Drawing for Android and IOS?
      I want an evaluation copy for testing

      Delete
  6. I would like to use this in a PCL project that is targeting android and iOS. However I'm getting this error: Could not install package 'shim.drawing 2.2.5.50504'. You are trying to install this package into a project that targets 'portable-net45+win+wp80+MonoTouch10+MonoAndroid10+xamarinmac20+xamarinios10', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.

    ReplyDelete
    Replies
    1. Okay, learned I need to change my profile to 111. Are there any projects that use aforge in the PCL with android and iOS projects?

      Delete
    2. Hi Aaron,
      great that you found out about the PCL profile yourself. You can find some samples here: https://github.com/cureos/accord-samples , including iOS and Android.
      Best regards,
      Anders @ Cureos

      Delete
  7. Thanks for giving knowledge Excellent overview :)
    android developer USA

    ReplyDelete
  8. Do you have any documentation for this? or it is the same of AForge.NET?

    ReplyDelete
    Replies
    1. Yes. However, I have noticed on iOS and Android that RGB processes seem to be returned as BGR instead.

      Delete
  9. Nice collection of moiles and tab but this day its really very difficult to keep update any price list information site due to frequently changing product price, i appreciate your effort.
    latest mobile price
    repair
    Mobile Phone Accessories

    ReplyDelete
  10. great!. I would like to record video using xamarin android. is it possible to capture video and editing it before playing it?
    thanks

    ReplyDelete
    Replies
    1. Hi,

      And sorry for the late response. Unfortunately, video capturing is not supported in the portable version. Only still images are supported.

      Best regards,
      Anders @ Cureos

      Delete
  11. Hi Anders

    I plan to use this port in 1 project. Before buy I should check how it works on Android (in wp8 all is fine). However I can't convert byte array from camera to Bitmap - Bitmap s = (Bitmap)Bitmap.FromStream(new MemoryStream(image));
    I have Not Implemented exception. How I can get Bitmap from byte array and then convert it to ImageSource? I hope you can help me with that (I spent a lot of time trying to make that simple sample)(((

    ReplyDelete
    Replies
    1. Hi Stanislav,

      it sounds like you have not referenced Shim.Drawing properly, the exception indicates that your application is using the PCL assembly instead of the Android one.

      You need to explicitly include the Shim.Drawing package in your Android project, either directly or via the AForge or Accord package that has a dependency on Shim.Drawing. NuGet will then automatically select the Shim.Drawing instance that fits the Android platform.

      Best regards,
      Anders @ Cureos

      Delete
  12. Hi Andres,

    We are considering using the portable.aforge library with shim in our Xamarin.forms application. To prove the concept I tried to use the evaluation shim library for Windows 8.1, what I'm trying to do is to get the file, convert it to Bitmap and to show it.

    Bitmap bmp = (Bitmap)Bitmap.FromStream(stream);
    Stream ms = new MemoryStream();
    bmp.Save(ms, ImageFormat.Bmp); <--------------------
    ms.Seek(0, SeekOrigin.Begin);

    image.Source = ImageSource.FromStream(() => { return ms; });

    When I try to save the Bitmap to stream, I see that nothing is actually written into the stream. Can you let me know, the evaluation versions of shim library, do they implement Save methods or not? Or may be I'm doing something wrong.

    Thank you very much,

    Andrew Kaplanovsky

    ReplyDelete
    Replies
    1. Hi Andrew,

      Do I understand correctly that you are trying this together with Windows Forms? The Portable AForge libraries do not support the "real" Windows Forms, since it would interfere with the Shim.Drawing library that is used as a "bridge" for the other platforms. This means that if you want to try out Portable.AForge under .NET, you should target WPF instead.

      Best regards,
      Anders

      Delete
    2. Hi Andres,

      Thank you so much for your help, sorry, I think I expressed myself incorrectly. I'm not trying to use the Portable.AForge libraries together with Windows forms. We're developing a mobile application using Xamarin.Forms and PCL. I tried to use the examples you'd provided here

      https://github.com/cureos/accord-samples

      but with no luck. When I try to save a Bitmap object into a stream, nothing actually happens. I get exactly the same error when trying to run the FaceDetectionExample:

      https://github.com/cureos/accord-samples/issues/1

      Thank you,
      Andrew

      Delete
    3. Hi again Andrew,

      OK, I see. I will look into the Github issue as soon as possible.

      Best regards,
      Anders

      Delete
  13. Hi, Is a portable version of aforge.video coming anytime soon ? IE: for cross platform Camera/Video capture ? Thanks Nick

    ReplyDelete
  14. Environment: Xamarin.Forms App. using Aforge.net portable version:2.2.5.0 and Shim.Drawing eval Version 3.0.1.0

    Passing an Image with a Pixel Format of Format32bppPArgb to a ResizeBilinear Filter throws an exception that the Pixel Format is not recognized. So I attempted to clone with _BitmapOrignal.Clone(PixelFormat.Format32bppArgb)but it also fails. could you recommend on how to convert a Format32bppPArgb to a format that would be recognized by a filter.

    ReplyDelete
  15. Hi Nick!

    Regarding a portable version of aforge.video, there are no plans for the moment. It is quite an effort to adopt the video library on all platforms of interest, and I don't have the necessary (financial) incentive to attempt this, I am afraid.

    Regarding the pixel format problem you have: I have not checked particularly for the ResizeBilinear filter, but generally the filters only work for a sub-set of pixel formats, and as you correctly assume you need to change the pixel format through cloning before applying the filter. Depending on platform, not all pixel formats are supported in practice though. If you specify which platform your test app is running on, I could give more detailed information about the supported pixel formats.

    Regards,
    Anders @ Cureos

    ReplyDelete
  16. Hi, I am working in a project using Xamarin.Forms. I need to scan a document using the smartphone camera and using the image to convert it to a PDF later.

    Is it possible to use your library to handle the document scan, skewing correction, etc with your product?

    Do you have any samples for a similar scenario?

    Thanks.

    ReplyDelete
  17. Hi, I am working on Android with Xamarin. We are having some issues on images taken from camera.

    To apply filters on this image we clone the bitmap, converting the pixel format to system.Drawing.Imaging.PixelFormat.Format24bppRgb
    The operation is done using AForge.Imaging.Image.Clone

    After this conversion, all other operations (filters) on the image are very slow.

    How can I avoid this problem ?

    Thanks,
    Gianluca

    ReplyDelete
  18. Hi,
    We are currently using accord.net in windows desktop, we are more fun in developing with face detection.

    we are planning to build an application for Android and ios.

    can we take a look all the sample programs that is possible to build a portable application for android and ios?

    we can't find .apk or .ipa

    i hope to catch better executable samples for a test,
    thank you

    ReplyDelete
  19. Nice Information provided in the blog
    Best mobile app development agency firm in Paris which provides custom Android, iphone, ios application development services in France Paris, Lyon, Toulouse, Nantes, Strasbourg, Bordeaux, Lille
    Best Mobile App Development Agency Paris
    

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. Hello,

    I wrote you an email, but didn't get an answer so far. So another try on your blog. As I wrote, we're looking after a technology to write an Xamarin.Forms based app providing a similiar functionality like the Office Lens app to scan documents (shape recognizing through the camera of the mobile). What do you think: Can wie use portable aforge to realize such an app?

    I would really appreciate your feedback on this topic. Thank you.

    Best regards,
    Matthias

    ReplyDelete