Q:

How to return Item's XML properly in an xsl extension function? How to return node sets in C# in order to use it in XSLT?

A:

You should use property-like methods with get_ or set_ prefixs to return Item's XML, for example if the property is named ItemID you should use the following statements

get_ItemID()

and

set_ItemID()

This is the way .Net Framework converts properties to methods.

If you wish to return an Item set that will be possible to loop through, you need to return XpathNodeIterator:

using System;
using System.Xml;
using System.Xml.XPath;

using Sitecore;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Xml;
using Sitecore.Xml.XPath;
...
      public XPathNodeIterator GetItemsIterator()
      {
          Item itm = Sitecore.Context.Database.Items["/sitecore/content/Home"];
          XPathNodeIterator nav = new ItemNavigator(itm).Select(".");
          return nav;
      }