The other day at work I spent several hours tracking down this precious little gem of confusing behavior with NSMenuItem.
The framework is supposed to give you some menu-related niceties when a window or dialog is running modal in your app. Non-applicable Main menu items are supposed to get disabled for you. We were seeing that several, though not all, of our menu items were not playing nice.
Obviously, if you've bound the NSMenuItem's "enabled" binding to a custom function, you certainly shouldn't expect to get the default OS behavior for free. This wasn't the case for us. However, we had bound the title to a custom function. Apparently, if you bind *any* property of an NSMenuItem you will no longer get the default OS auto-enabling/disabling behavior.
The work around I implemented was to implement a generic "isAppModal" function on our App controller. I then bound all the offending menu items' enabled property to this method and used a NSNegateBoolean transformer. A bit hacky, if functional.
Matt Neuberg posted a great summary on Menu Item enabling. Though it doesn't directly mention the behavior introduced by bindings, it is very insightful and condenses information from several developer docs.