There is no really any way to increase perfomance other then use cache feature...
Jan once told me that fetching information about all categories will slow down script with lots of categories and that he's trying eliminate this feature...but unfortunetly it will not be included in next version...